AI Engineer World's Fair 2026
Shipping AI to a Million Patients Without an A/B Test
Loading slides…
Read the talk
Shipping AI to a Million Patients Without an A/B Test
Dora’s safety stack combines simulated patients, clinician-grounded hazard detection and prompt optimization, then uses staged clinical evidence to decide what can reach patients.
From a talk by Jared Joselowitz
Before you start: Familiarity with LLM prompts, evaluation datasets and false positives versus false negatives will help; no clinical background is required.
What must be true before the first patient hears it?
How do you establish that a clinical voice agent is safe before a patient hears its advice? That is the job Jared Joselowitz describes at Ufonia: building the safety and evaluation stack behind Dora. After studying electrical engineering in South Africa and moving into AI, he joined the company’s science team as a research engineer. The release decision starts before the conversation, because its consequences cannot always be repaired afterward.
Joselowitz identifies three constraints. Deliberately assigning patients to a worse variant is ethically unacceptable and, he says, often illegal; this is a constraint on exposing patients to harmful experiments, not a blanket prohibition on clinical randomization. Once the agent has spoken, a software rollback cannot retract its advice. And a vendor’s benchmark score cannot explain away a failure in a particular clinical workflow. The safety case must belong to the deployed system and its intended use.
Dora phones patients for clinical conversations such as postoperative follow-ups and preoperative checks. These are tasks clinicians would otherwise perform themselves; the aim is to return clinician time, rather than replace doctors. Joselowitz reports approximately 200,000 UK clinical calls across 20 hospitals, with contracts to reach one million patients over the following two years. At the time of the talk, he reports two live US clinics and six more signed across four states. The million-patient figure is contracted expansion, not an achieved deployment total.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A cataract follow-up becomes a safety problem
The demonstration makes the responsibility concrete. Dora asks whether a patient’s difficulty seeing after cataract surgery affects distance vision, near vision or both. The patient says distance vision is fine but seeing up close is difficult. Dora then asks when the blur began, and the patient places its onset after surgery. Rather than simply collecting a yes-or-no symptom flag, the agent is eliciting the symptom’s scope and timing.
The patient begins asking how long the problem will last while Dora briefly acknowledges the previous answer. Dora recognizes the duration question and, in this demonstration, says the blur should improve rapidly over the first few days. The patient then asks about swimming, and Dora advises avoiding it for a month after surgery. These are examples of the recorded agent’s advice, not postoperative instructions for the reader: the conversation has moved from gathering information to giving clinically consequential answers.
Joselowitz identifies those activities—asking about symptoms, giving advice and answering medical questions—as the basis for Dora’s regulation as a medical device. He reduces the engineering problem behind regulation to three questions: what does the software do, what could go wrong, and how do you prevent it? For a conversational system, the second question opens a large space of possible failures.
The starting point is patient harm, expressed as specific hazards:
- Missed red flags: the agent fails to recognize sudden vision loss or severe pain.
- Invented medical answers: a patient asks a question and the agent hallucinates advice.
- Ignored distress: the patient is distressed, but the agent continues without acknowledging it.
These are examples from a larger inventory of documented hazards, not a complete checklist. Each names something the evaluation must actively try to provoke and detect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Replace first exposure with simulation
The familiar release loop is to expose a small group, watch the dashboard, roll back if something breaks and iterate. It works when being wrong briefly is affordable. In clinical conversations, even a small rollout can expose many patients to unproven advice, and a red dashboard may describe harm that has already occurred. Monitoring still matters, but it cannot serve as the first test of a known dangerous scenario.
Joselowitz draws an analogy with autonomous driving: extensive simulated mileage lets teams explore failures before exposing passengers to them. Ufonia applies that principle to clinical history-taking through MATRIX, a framework for generating and evaluating conversations without a real patient in the loop. Simulation provides somewhere to test the hazards that would be unacceptable to introduce deliberately into a patient’s first encounter.
Inside MATRIX, PatBot is an LLM playing the patient opposite Dora. Simulated patients make it practical to run concurrent scenarios while the target system changes; hiring actors for every iteration would be too slow. A scenario specifies what the patient should do—for example, ask whether Dora is a human or an AI. Clinical configuration keeps that behavior inside a particular care workflow. PatBot and Dora then exchange messages to produce the dialogue that will be evaluated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Following a scenario is not the same as sounding like a patient
A simulated patient must first perform the requested behavior. Ufonia starts with a binary script-adherence check: did PatBot do what the scenario instructed? That filters out models that fail to follow instructions, but passing it says little about whether the conversation resembles a real patient encounter. A compliant simulator can still produce unnatural tests.
The team therefore conducted a patient and public involvement study. Participants saw real doctor–patient conversations alongside Dora–PatBot conversations and tried to identify the real patient. Joselowitz repeats the exercise with the audience, showing two conversations and asking for a show of hands. Across four conversation sets, he reports that a majority judged the simulated patient more realistic in three.
The useful conclusion is not that there is one ideal patient persona. Some people speak at length, hesitate and use fillers; others answer with terse yeses and nos. The test suite needs that diversity. The study gave the team confidence that PatBot was realistic enough for its simulation work, while also showing why realism cannot be reduced to a single conversational style.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Judge hazards with clinical criteria
Generating thousands of dialogues creates another scaling problem: someone must determine whether they contain hazards. Engineers cannot review every conversation manually, and engineering judgment is not a substitute for clinical expertise. MATRIX uses BehvJudge, an LLM judge that receives the dialogue, expected behaviors and hazardous scenarios developed with clinicians. It produces a structured pass or fail; failures identify the triggered hazard and explain what went wrong.
To validate the judge, the team used a corpus with hazard ground truth and compared its labels with those from ten clinicians representing ten specialties. The original MATRIX paper reports Gemini 2.5 Pro hazard-detection F1 of 0.96 and sensitivity of 0.999 on 240 synthetic transcripts: 80 safe and 160 deliberately hazardous. This is the paper-era model comparison Joselowitz describes, rather than a measurement of Dora’s patient outcomes or necessarily the model used at the time of deployment.
Joselowitz characterizes the judge as performing at least on par with the expert clinicians in that comparison. Sensitivity matters because the costs are asymmetric: a false alarm sends a conversation for unnecessary scrutiny, whereas a missed hazard can allow a dangerous failure to escape. The team would rather over-call hazards than overlook them. A validated automated judge makes broad simulation practical without removing clinicians from the definition of safe behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn evaluation failures into prompt improvements
A pile of pass/fail results locates failures but does not fix them. The next step used to be manual prompt engineering: inspect the failing component, edit its instructions and try again. That process becomes difficult to reproduce when seemingly incidental prompt changes alter behavior.
Joselowitz cites formatting effects as large as 76 percentage points and few-shot ordering effects ranging from near-random to near-state-of-the-art performance. Supporting research on format sensitivity reports the 76-point maximum accuracy spread for LLaMA-2-13B under meaning-preserving few-shot formatting changes; research on example ordering documents the ordering effect in few-shot text classification. These are particular experimental findings, not expected swings for every model. They explain why hand-tuning individual strings can be a fragile optimization strategy.
Ufonia instead uses GEPA, short for Genetic-Pareto, which Joselowitz associates with the creators of DSPy. Its optimization loop makes the feedback explicit:
- Define the metric that determines a good result.
- Run examples through the candidate prompt and collect failures.
- Have a strong LLM reflect on the failures and propose prompt revisions.
- Evaluate the revisions and retain a Pareto frontier of promising candidates.
- Repeat until the optimization budget is exhausted, then select a prompt.
Keeping a frontier preserves candidates with different strengths while the search continues, rather than committing to every new revision.
Joselowitz reports that their optimized prompts normally take approximately 30–60 minutes to produce, compared with hours to days of manual tuning. He emphasizes the repeatable process, audit trail and explicit feedback loop. The engineering work shifts toward the data, feature engineering and the metric defined with clinicians: those inputs determine what the optimizer learns to favor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Optimize the cost of an error, not just accuracy
The optimizer needs a definition of good that reflects clinical consequences. Flat accuracy treats mistakes as interchangeable, but missing a red flag and asking an unnecessary follow-up question are not equivalent outcomes.
| Actual state | Agent detects a red flag | Agent detects no red flag |
|---|---|---|
| Red flag present | Correct detection | Potentially catastrophic miss |
| No red flag | Extra questions in this example | Correct negative |
In Joselowitz’s example, over-calling a red flag inconveniences the patient with additional questions; missing one may cause serious harm. That asymmetry is the reason to prioritize sensitivity.
A cost function can express this preference directly. The following Python example leaves the clinical weights explicit: when miss_cost is greater than false_alarm_cost, a missed red flag incurs the larger penalty. It returns feedback alongside the score so a prompt optimizer can distinguish the failure it needs to address.
python
def red_flag_feedback(
present: bool,
detected: bool,
*,
miss_cost: float,
false_alarm_cost: float,
) -> dict[str, float | str]:
if miss_cost < 0 or false_alarm_cost < 0:
raise ValueError("Costs must be nonnegative")
if present and not detected:
return {"score": -miss_cost, "feedback": "Missed red flag"}
if not present and detected:
return {
"score": -false_alarm_cost,
"feedback": "Flagged a red flag that was absent",
}
return {"score": 0.0, "feedback": "Correct red-flag decision"}
The important choice is the relative cost, which must come from the clinical use case. Joselowitz describes rewarding red-flag detection more strongly and missed flags less strongly; clinicians may instead prioritize accuracy or another objective. Changing that objective means recompiling the prompt against the revised metric.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A candidate prompt still has to earn deployment
The replacement release loop starts with real call data, supplemented by synthetic edge cases such as rare symptoms and mistranscriptions. GEPA or another optimizer produces a candidate prompt. MATRIX then acts as a simulation safety gate. If it reveals a failure, the team revisits the data, corrects labels or adds examples before running the process again. A candidate that clears the gate can proceed toward gated deployment, whose new calls supply data for later iterations. Optimization proposes a change; the safety gate evaluates whether it should advance.
Passing every simulated test does not establish that Dora helps real patients. Simulation is the fast inner loop, allowing many runs before human exposure; real encounters provide the outer loop of clinical evidence. Simulation is necessary but insufficient. It earns the right to test carefully, rather than permission to deploy without further scrutiny.
The transition proceeds through stages:
- Simulation: establish performance against the defined scenarios and hazards.
- User testing: test the interaction with people after the simulation results are satisfactory.
- Supervised clinical evaluation: evaluate with real patients while clinicians remain involved throughout. Actors can help with testing, but real patients provide the most realistic encounters.
- Monitored deployment: allow independence in proportion to the evidence collected.
Each stage must justify the next. More autonomy follows stronger evidence, not merely a successful prompt-optimization run.
Underneath these gates is traceability. Every call, dataset, pinned prompt and judge verdict must connect to the hazard it addresses. That connection makes the evaluation record useful for regulation: it explains which failure was considered, which system version was tested and what evidence supports its release. As Joselowitz puts it, “you ship the evidence.”
Building this into another product begins with defining its particular harms and deliberately manufacturing rare dangerous cases. The evaluation metric should encode the real cost function, while prompt versions and traces preserve the record of how a candidate was produced and assessed. Waiting for rare failures to appear naturally would leave the most consequential cases outside the development loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Voice changes what can fail
The hazard inventory keeps growing as the product enters new languages and modalities. Voice introduces timing and turn-taking failures that text evaluations can miss. An agent can be halfway through safety advice when a patient interrupts with an unrelated question. Joselowitz gives the unfinished phrase “You must avoid bright lights and...”: a weaker model may answer the new question and never return to the remaining advice.
Even a backchannel—a brief acknowledgment while the agent is speaking—can cause trouble. The system may treat it as a reason to stop, abandon the safety advice and wait for the patient to say more. The visible voice-hazards slide also lists accepting half-answers as complete answers. These failures concern whether the conversation completes its clinical obligations, not just whether an isolated generated sentence is correct. The evaluation must therefore exercise the interaction around an answer as well as the answer itself.
Joselowitz says Ufonia already uses MATRIX for voice. The target remains a black box: document the new hazards, simulate the conditions that trigger them and judge the resulting behavior. Voice adds another module to the same safety case. The framework persists, but the scenarios must evolve so that unfinished advice, interruptions and other new failure modes are discovered before a real patient encounters them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Methods and experiments for scenario-driven patient simulation and automated clinical hazard detection.
Introduces prompt optimization using execution feedback, language-model reflection and Pareto-based candidate selection.
Official implementation with installation instructions, examples and integration resources.
Further reading
Studies how meaning-preserving formatting changes affect few-shot language-model accuracy.
ACL 2022 study of performance variation caused by rearranging few-shot examples.
Full transcript
Loading transcript…