AI Engineer World's Fair 2026

Shipping AI to a Million Patients Without an A/B Test

Loading slides…

Read the talk

Shipping AI to a Million Patients Without an A/B Test

Dora’s safety stack combines simulated patients, clinician-grounded hazard detection and prompt optimization, then uses staged clinical evidence to decide what can reach patients.

From a talk by Jared Joselowitz

Before you start: Familiarity with LLM prompts, evaluation datasets and false positives versus false negatives will help; no clinical background is required.

What must be true before the first patient hears it?

How do you establish that a clinical voice agent is safe before a patient hears its advice? That is the job Jared Joselowitz describes at Ufonia: building the safety and evaluation stack behind Dora. After studying electrical engineering in South Africa and moving into AI, he joined the company’s science team as a research engineer. The release decision starts before the conversation, because its consequences cannot always be repaired afterward.

Joselowitz identifies three constraints. Deliberately assigning patients to a worse variant is ethically unacceptable and, he says, often illegal; this is a constraint on exposing patients to harmful experiments, not a blanket prohibition on clinical randomization. Once the agent has spoken, a software rollback cannot retract its advice. And a vendor’s benchmark score cannot explain away a failure in a particular clinical workflow. The safety case must belong to the deployed system and its intended use.

Slide states: You can't A/B test on patients; you can't unsend a phone call; the model card won't save you at the post-incident review.Open full source frame
Three constraints on shipping clinical AI safely.

Dora phones patients for clinical conversations such as postoperative follow-ups and preoperative checks. These are tasks clinicians would otherwise perform themselves; the aim is to return clinician time, rather than replace doctors. Joselowitz reports approximately 200,000 UK clinical calls across 20 hospitals, with contracts to reach one million patients over the following two years. At the time of the talk, he reports two live US clinics and six more signed across four states. The million-patient figure is contracted expansion, not an achieved deployment total.

0:230:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:23 · section reference included

A cataract follow-up becomes a safety problem

The demonstration makes the responsibility concrete. Dora asks whether a patient’s difficulty seeing after cataract surgery affects distance vision, near vision or both. The patient says distance vision is fine but seeing up close is difficult. Dora then asks when the blur began, and the patient places its onset after surgery. Rather than simply collecting a yes-or-no symptom flag, the agent is eliciting the symptom’s scope and timing.

The patient begins asking how long the problem will last while Dora briefly acknowledges the previous answer. Dora recognizes the duration question and, in this demonstration, says the blur should improve rapidly over the first few days. The patient then asks about swimming, and Dora advises avoiding it for a month after surgery. These are examples of the recorded agent’s advice, not postoperative instructions for the reader: the conversation has moved from gathering information to giving clinically consequential answers.

Joselowitz identifies those activities—asking about symptoms, giving advice and answering medical questions—as the basis for Dora’s regulation as a medical device. He reduces the engineering problem behind regulation to three questions: what does the software do, what could go wrong, and how do you prevent it? For a conversational system, the second question opens a large space of possible failures.

The starting point is patient harm, expressed as specific hazards:

  • Missed red flags: the agent fails to recognize sudden vision loss or severe pain.
  • Invented medical answers: a patient asks a question and the agent hallucinates advice.
  • Ignored distress: the patient is distressed, but the agent continues without acknowledging it.

These are examples from a larger inventory of documented hazards, not a complete checklist. Each names something the evaluation must actively try to provoke and detect.

2:152:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:15 · section reference included

Replace first exposure with simulation

The familiar release loop is to expose a small group, watch the dashboard, roll back if something breaks and iterate. It works when being wrong briefly is affordable. In clinical conversations, even a small rollout can expose many patients to unproven advice, and a red dashboard may describe harm that has already occurred. Monitoring still matters, but it cannot serve as the first test of a known dangerous scenario.

Joselowitz draws an analogy with autonomous driving: extensive simulated mileage lets teams explore failures before exposing passengers to them. Ufonia applies that principle to clinical history-taking through MATRIX, a framework for generating and evaluating conversations without a real patient in the loop. Simulation provides somewhere to test the hazards that would be unacceptable to introduce deliberately into a patient’s first encounter.

Inside MATRIX, PatBot is an LLM playing the patient opposite Dora. Simulated patients make it practical to run concurrent scenarios while the target system changes; hiring actors for every iteration would be too slow. A scenario specifies what the patient should do—for example, ask whether Dora is a human or an AI. Clinical configuration keeps that behavior inside a particular care workflow. PatBot and Dora then exchange messages to produce the dialogue that will be evaluated.

Diagram connects Patient Input Type and Clinical Configuration to Patbot, a simulated patient, alongside Agent (Dora) and a central simulated clinical conversation containing four message bubbles.Open full source frame
Inside MATRIX: a simulated patient exchanges messages with Dora.
4:254:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:25 · section reference included

Following a scenario is not the same as sounding like a patient

A simulated patient must first perform the requested behavior. Ufonia starts with a binary script-adherence check: did PatBot do what the scenario instructed? That filters out models that fail to follow instructions, but passing it says little about whether the conversation resembles a real patient encounter. A compliant simulator can still produce unnatural tests.

The team therefore conducted a patient and public involvement study. Participants saw real doctor–patient conversations alongside Dora–PatBot conversations and tried to identify the real patient. Joselowitz repeats the exercise with the audience, showing two conversations and asking for a show of hands. Across four conversation sets, he reports that a majority judged the simulated patient more realistic in three.

The useful conclusion is not that there is one ideal patient persona. Some people speak at length, hesitate and use fillers; others answer with terse yeses and nos. The test suite needs that diversity. The study gave the team confidence that PatBot was realistic enough for its simulation work, while also showing why realism cannot be reduced to a single conversational style.

7:257:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:25 · section reference included

Judge hazards with clinical criteria

Generating thousands of dialogues creates another scaling problem: someone must determine whether they contain hazards. Engineers cannot review every conversation manually, and engineering judgment is not a substitute for clinical expertise. MATRIX uses BehvJudge, an LLM judge that receives the dialogue, expected behaviors and hazardous scenarios developed with clinicians. It produces a structured pass or fail; failures identify the triggered hazard and explain what went wrong.

To validate the judge, the team used a corpus with hazard ground truth and compared its labels with those from ten clinicians representing ten specialties. The original MATRIX paper reports Gemini 2.5 Pro hazard-detection F1 of 0.96 and sensitivity of 0.999 on 240 synthetic transcripts: 80 safe and 160 deliberately hazardous. This is the paper-era model comparison Joselowitz describes, rather than a measurement of Dora’s patient outcomes or necessarily the model used at the time of deployment.

Joselowitz characterizes the judge as performing at least on par with the expert clinicians in that comparison. Sensitivity matters because the costs are asymmetric: a false alarm sends a conversation for unnecessary scrutiny, whereas a missed hazard can allow a dangerous failure to escape. The team would rather over-call hazards than overlook them. A validated automated judge makes broad simulation practical without removing clinicians from the definition of safe behavior.

9:239:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:23 · section reference included

Turn evaluation failures into prompt improvements

A pile of pass/fail results locates failures but does not fix them. The next step used to be manual prompt engineering: inspect the failing component, edit its instructions and try again. That process becomes difficult to reproduce when seemingly incidental prompt changes alter behavior.

Joselowitz cites formatting effects as large as 76 percentage points and few-shot ordering effects ranging from near-random to near-state-of-the-art performance. Supporting research on format sensitivity reports the 76-point maximum accuracy spread for LLaMA-2-13B under meaning-preserving few-shot formatting changes; research on example ordering documents the ordering effect in few-shot text classification. These are particular experimental findings, not expected swings for every model. They explain why hand-tuning individual strings can be a fragile optimization strategy.

Ufonia instead uses GEPA, short for Genetic-Pareto, which Joselowitz associates with the creators of DSPy. Its optimization loop makes the feedback explicit:

  1. Define the metric that determines a good result.
  2. Run examples through the candidate prompt and collect failures.
  3. Have a strong LLM reflect on the failures and propose prompt revisions.
  4. Evaluate the revisions and retain a Pareto frontier of promising candidates.
  5. Repeat until the optimization budget is exhausted, then select a prompt.

Keeping a frontier preserves candidates with different strengths while the search continues, rather than committing to every new revision.

Slide titled “We don't write prompts. We compile them.” compares an orange manual trial-and-error diagram with a blue automated workflow using metrics, datasets, an optimisation algorithm and feedback loops.Open full source frame
Manual prompt engineering versus automated prompt optimisation.

Joselowitz reports that their optimized prompts normally take approximately 30–60 minutes to produce, compared with hours to days of manual tuning. He emphasizes the repeatable process, audit trail and explicit feedback loop. The engineering work shifts toward the data, feature engineering and the metric defined with clinicians: those inputs determine what the optimizer learns to favor.

11:0511:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:05 · section reference included

Optimize the cost of an error, not just accuracy

The optimizer needs a definition of good that reflects clinical consequences. Flat accuracy treats mistakes as interchangeable, but missing a red flag and asking an unnecessary follow-up question are not equivalent outcomes.

Actual stateAgent detects a red flagAgent detects no red flag
Red flag presentCorrect detectionPotentially catastrophic miss
No red flagExtra questions in this exampleCorrect negative

In Joselowitz’s example, over-calling a red flag inconveniences the patient with additional questions; missing one may cause serious harm. That asymmetry is the reason to prioritize sensitivity.

A cost function can express this preference directly. The following Python example leaves the clinical weights explicit: when miss_cost is greater than false_alarm_cost, a missed red flag incurs the larger penalty. It returns feedback alongside the score so a prompt optimizer can distinguish the failure it needs to address.

python

def red_flag_feedback(
    present: bool,
    detected: bool,
    *,
    miss_cost: float,
    false_alarm_cost: float,
) -> dict[str, float | str]:
    if miss_cost < 0 or false_alarm_cost < 0:
        raise ValueError("Costs must be nonnegative")

    if present and not detected:
        return {"score": -miss_cost, "feedback": "Missed red flag"}
    if not present and detected:
        return {
            "score": -false_alarm_cost,
            "feedback": "Flagged a red flag that was absent",
        }
    return {"score": 0.0, "feedback": "Correct red-flag decision"}

The important choice is the relative cost, which must come from the clinical use case. Joselowitz describes rewarding red-flag detection more strongly and missed flags less strongly; clinicians may instead prioritize accuracy or another objective. Changing that objective means recompiling the prompt against the revised metric.

13:1813:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:18 · section reference included

A candidate prompt still has to earn deployment

The replacement release loop starts with real call data, supplemented by synthetic edge cases such as rare symptoms and mistranscriptions. GEPA or another optimizer produces a candidate prompt. MATRIX then acts as a simulation safety gate. If it reveals a failure, the team revisits the data, corrects labels or adds examples before running the process again. A candidate that clears the gate can proceed toward gated deployment, whose new calls supply data for later iterations. Optimization proposes a change; the safety gate evaluates whether it should advance.

Passing every simulated test does not establish that Dora helps real patients. Simulation is the fast inner loop, allowing many runs before human exposure; real encounters provide the outer loop of clinical evidence. Simulation is necessary but insufficient. It earns the right to test carefully, rather than permission to deploy without further scrutiny.

The transition proceeds through stages:

  1. Simulation: establish performance against the defined scenarios and hazards.
  2. User testing: test the interaction with people after the simulation results are satisfactory.
  3. Supervised clinical evaluation: evaluate with real patients while clinicians remain involved throughout. Actors can help with testing, but real patients provide the most realistic encounters.
  4. Monitored deployment: allow independence in proportion to the evidence collected.

Each stage must justify the next. More autonomy follows stronger evidence, not merely a successful prompt-optimization run.

Underneath these gates is traceability. Every call, dataset, pinned prompt and judge verdict must connect to the hazard it addresses. That connection makes the evaluation record useful for regulation: it explains which failure was considered, which system version was tested and what evidence supports its release. As Joselowitz puts it, “you ship the evidence.”

Building this into another product begins with defining its particular harms and deliberately manufacturing rare dangerous cases. The evaluation metric should encode the real cost function, while prompt versions and traces preserve the record of how a candidate was produced and assessed. Waiting for rare failures to appear naturally would leave the most consequential cases outside the development loop.

14:2814:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:28 · section reference included

Voice changes what can fail

The hazard inventory keeps growing as the product enters new languages and modalities. Voice introduces timing and turn-taking failures that text evaluations can miss. An agent can be halfway through safety advice when a patient interrupts with an unrelated question. Joselowitz gives the unfinished phrase “You must avoid bright lights and...”: a weaker model may answer the new question and never return to the remaining advice.

Even a backchannel—a brief acknowledgment while the agent is speaking—can cause trouble. The system may treat it as a reason to stop, abandon the safety advice and wait for the patient to say more. The visible voice-hazards slide also lists accepting half-answers as complete answers. These failures concern whether the conversation completes its clinical obligations, not just whether an isolated generated sentence is correct. The evaluation must therefore exercise the interaction around an answer as well as the answer itself.

Slide titled “New modality, new hazards. Same framework” lists interruptions, half-answers accepted as full answers, and interrupted safety advice that never resumes. A closing line says to black-box the system and evaluate the new hazards as before.Open full source frame
New voice hazards, the same evaluation framework.

Joselowitz says Ufonia already uses MATRIX for voice. The target remains a black box: document the new hazards, simulate the conditions that trigger them and judge the resulting behavior. Voice adds another module to the same safety case. The framework persists, but the scenarios must evolve so that unfinished advice, interruptions and other new failure modes are discovered before a real patient encounters them.

17:1517:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:15 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…