AI Engineer World's Fair 2026
Inside 847 Production Clinical AI Notes — Sebastian Fox, Composo
Loading slides…
Read the talk
Inside 847 Production Clinical AI Notes
The title points to a corpus of 847 production notes, while the recording also discusses a separate real-world study and a newly generated multi-scribe dataset without establishing that they share the same denominator. Sebastian Fox uses those sources to show why plausible clinical notes conceal dangerous omissions—and why evaluation must learn continuously from expert judgment.
From a talk by Sebastian Fox
At a glance
Ideas worth remembering
The title’s 847 notes, the cited real-world study, and the generated notes used for judge comparisons are not established as one dataset. Keep their denominators separate.
A fluent output can be dangerous because of what it omits; checking whether every written sentence is plausible does not establish completeness.
Ambient-scribe failures occur in transcription and generation, but additions, changes, and omissions become difficult chiefly when an evaluator must decide which difference matters.
A second language model is not automatically a safety net. Static-rubric judges can compare words accurately while missing intent, temporal decisions, and case-specific significance.
Store moving, explainable judgment as retrievable cases, expert corrections, and references rather than relying only on a fixed rubric or repeatedly retrained weights.
Begin by collecting expert comments on real outputs, then continuously discover failure modes, capture reasoning, and calibrate each evaluation against relevant prior cases.
The most dangerous note may look completely fine
A clinical note describes an ordinary new headache: likely tension-type, take paracetamol, return if it does not settle. Nothing on the page looks obviously false. But the consultation also included jaw pain while chewing in a patient over fifty. Fox, a medical doctor by background who now runs Composo, identifies that combination as a red flag for giant cell arteritis—a same-day emergency because untreated disease can threaten sight within days. The symptom that changes the case never reached the note.
This is a harder failure than visible nonsense. A spectacularly bad note might invent chest pain, angina, diabetes medication, and a nonexistent hospital address for a young patient with tonsillitis. Such additions attract attention. A fluent note that quietly removes the decisive fact can remain in the record because every sentence it contains is plausible.
The title identifies 847 production clinical AI notes, but the recording does not define that corpus or say that every result comes from it. Fox separately cites what he calls the largest real-world study, then later describes notes generated across three production scribes and reused for a judge comparison. Those datasets should not be assumed to be identical.
For the real-world study, Fox reports that about one in twenty notes carried a potentially seriously harmful error, nearly one in five had an important omission, and more than one in ten had a hallucination. He also says ambient scribes operate in about a third of US practices and that most systems lack adverse-event reporting. The recording does not provide the study design needed to assess how broadly those rates generalize.
That reporting gap changes the meaning of apparent safety. An unreported incident count does not show that deployment is harmless when errors can simply remain embedded in records. Healthcare makes the stakes visceral, but the same structure appears anywhere a fluent output can omit the fact, clause, or decision that controls what happens next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Correct words can still produce the wrong meaning
Notes generated across three production ambient scribes produced a revealing distribution of failures. Fox plots each error by clinical importance on one axis and whether a strong automated check catches it on the other. A few errors are caught; the troubling cluster is both consequential and missed. Better prose quality has moved the problem away from easy-to-spot gibberish and toward subtle distortions of intent.
One patient cannot say whether her headache began suddenly or gradually; she only says that it “just happened.” The generated note converts that uncertainty into “abrupt sudden onset.” The observable change is one phrase, but its consequence is large: sudden onset is a red flag for bleeding in the brain. The model did not copy a stated fact. It resolved ambiguity, inserted the stronger interpretation, and made that interpretation drive the workup.
Another consultation discusses tests before the patient asks to try antibiotics instead. The doctor and patient agree to defer testing, treat, and observe. The note nevertheless says the tests were arranged that day. The proposed plan really did appear in the conversation, so simple transcript matching finds supporting words. What the note loses is the conversation’s temporal structure: proposal, negotiation, rejection, final decision.
Failures can enter at two layers. Transcription can confuse soundalike drug names, reverse hyperthyroidism into hypothyroidism, or drop the “no” from “no evidence of cancer.” Even with a perfect transcript, generation can still fail in three parallel ways:
- Addition: Insert something that was never said, ranging from a mild inference to a dangerous fabrication.
- Change: Rephrase supplied information in a way that may preserve or reverse its meaning.
- Omission: Remove material ranging from small talk to a clinically decisive allergy or red flag.
Detecting a difference is therefore only the mechanical half of evaluation. The evaluator must decide which difference matters in this case. Fox calls that “taste,” meaning domain judgment rather than aesthetics. It is tacit because experts cannot fully enumerate it, contextual because the same detail changes importance across cases, and moving because models, guidelines, institutions, and expert opinions change. A model may know the medical facts while still lacking a case-specific sense of priority.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A checker can become a second silent failure
The natural response is to separate generation from verification. Let one model write the note, then give a second model more time, tokens, context, and a narrow instruction to find what is wrong. This asymmetry works especially well in mathematics and code, where a compiler, unit test, or executable answer can cheaply reject many bad outputs.
A serious clinical judge can receive the transcript, generated note, surrounding context, a detailed faithfulness rubric, and worked pass/fail examples. Teams can optimize the rubric and add deterministic NLP checks that count medical concepts appearing in one document but not the other. Expert review remains the offline gold standard, but the automated judge is supposed to extend coverage to every production note.
Fox built that version and ran the same multi-scribe notes through it. The judge flagged a handful and approved most; one in five of its clean passes still contained a serious error, often an omission. This is a result from the presented judge comparison, not necessarily from all 847 notes or from the separate real-world study. It exposes the central failure: a capable model can compare two documents without knowing which missing or changed fact deserves priority.
Where does the safety net break? The flow below makes the two silent decisions visible. The generator can preserve words while misrepresenting intent, and the checker can approve that note because its static standard does not rank the case’s decisive difference highly enough.
How a plausible note and its checker can fail together
Read the diagram as text
- Consultation. source — Contains symptoms, proposals, uncertainty, negotiation, and final decisions.
- Generated note. generation — May add, change, or omit information while remaining fluent.
- Automated judge. verification — Compares transcript, note, context, and a predefined rubric.
- Approved record. outcome — A confident pass may still contain a serious omission or intent error.
- Consultation → Generated note: summarized into.
- Generated note → Automated judge: checked against source.
- Automated judge → Approved record: passes if rubric is satisfied.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The same omission can be noise or the answer
Verification is cheap when correctness has an external test. Clinical completeness has no equivalent of a compiler. A judge can spot differences between transcript and note, but it must also construct the standard that tells it which differences matter. A predefined rubric contains only the judgment its authors managed to express in advance; the hardest part is the judgment they could not fully write down.
Two patients have blood in their urine, and both notes omit where they recently travelled. One went to France; the other went to Lake Malawi. The omissions have the same textual shape. In the first case, the holiday detail is irrelevant. In the second, freshwater exposure in sub-Saharan Africa points toward schistosomiasis and changes the management plan. A concept-counting check sees two missing travel locations. Clinical judgment sees noise in one and a diagnostic clue in the other.
That leaves three places to store the standard. A prompt or rubric is inspectable and easy to change, but it freezes only what people can specify. Model weights can absorb examples through fine-tuning or continual learning, but Fox considers them poorly suited to a moving, explainable standard because updates require retraining and the resulting score cannot point cleanly to the judgment that changed it. Retrieved examples preserve past decisions, expert corrections, and references outside the weights, then bring the relevant subset into the judge’s context for each output.
The third option treats expert judgment as cases rather than a complete specification. A new correction can affect the next evaluation call immediately, and reviewers can inspect which examples or references moved the score. This is a retrieval and context-assembly problem: find prior cases with the same meaningful shape, not merely the same words.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover, capture, and calibrate the standard continuously
Fox organizes the alternative as a repeating three-step loop: discover failures in real outputs, capture how experts judge them, and calibrate each new output against the cases and references that bear on it. The standard remains editable because it lives in data and context rather than only in a fixed prompt or retrained model.
Discover from production. Cluster observed failures and name the recurring modes to build a failure-mode ontology. Synthetic cases remain useful, but they cover failures someone already imagined. Real outputs reveal unexpected behavior. The ontology is not a checklist the judge blindly applies; it organizes what experts should review and how prior cases should be indexed for retrieval.
Capture reasoning, not just scores. Put real outputs before clinicians for focused review and collect comments, corrections, and explanations. A useful seed can come from a few concentrated hours rather than a months-long labeling project. The resulting record shows how experts actually draw distinctions, including distinctions they would struggle to turn into universal rules.
Calibrate per output. Keep generic instructions such as “be faithful” or “do not omit important information,” then assemble the case-specific standard at evaluation time. Retrieve similar judged outputs, applicable expert corrections, reference documents, and guidelines. Each new judgment and correction becomes material for later evaluations; newly discovered failure modes feed back into the same system.
What relationship does this loop make visible? Evaluation produces both a decision and new memory. The next judge call is therefore informed by the failures and corrections accumulated before it, while discovery keeps the retrieval scheme from being limited to an ontology guessed at launch.
The discover–capture–calibrate evaluation loop
Read the diagram as text
- Real production outputs. production — Supply failures that synthetic tests may not anticipate.
- Discover. learning — Cluster failures and maintain an ontology grounded in observed behavior.
- Capture. learning — Store expert scores, reasoning, comments, and corrections.
- Calibrate. evaluation — Retrieve similar cases, corrections, references, and guidelines for this output.
- Case-specific judgment. evaluation — Scores the output against a standard assembled for that case.
- Real production outputs → Discover: reveal unexpected failures.
- Discover → Capture: organizes expert review.
- Capture → Calibrate: supplies judged examples.
- Calibrate → Case-specific judgment: assembles the relevant standard.
- Case-specific judgment → Real production outputs: adds judgments and corrections.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The missing red flag becomes visible when the judge knows what matters
Return to the opening headache. The evaluator need not have memorized that patient. It can retrieve earlier cases with the same shape—a red flag summarized as routine—plus expert corrections about new headaches in patients over fifty and applicable criteria or guidelines. Once that context sets the priority, the omitted jaw pain stands out against an otherwise plausible note.
Fox compares three judges on the same generated-note dataset: an off-the-shelf model with a rubric, a more engineered judge with rubric optimization and deterministic checks, and a judge supplied with discovered failure modes and per-output expert context. The third performs better on this specific dataset. The recording supplies no result showing that this comparison covered the full 847-note corpus; its useful lesson is narrower. The model and notes stay the same while the evidence and judgment standard placed in context change.
The pattern extends beyond medicine. A contract reviewer can omit the clause that changes the deal, while a support agent can promise a refund the company does not offer. Both may be watched by a generic judge that detects obvious inconsistency but lacks the organization’s sense of consequence.
The smallest useful starting point is also the most human: ask experts to leave free-form comments on real outputs. Those comments are raw material for discovering failure modes, preserving corrections, creating retrieval indexes, and eventually automating more of the review. Evaluation cannot be built once and frozen when the standard itself changes with cases, policy, and experience. It is an operating loop, not a possession.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Composo’s site links its failure-mode taxonomy, clinical omission work, benchmarks, and papers on improving language-model judges—the most direct supplied resource for investigating the methods behind the presentation.
Related talks
- The Future of Evals: From LLM as a Judge to Agent as a Judge — Aparna Dhinakaran, Arize AI
Extends the discussion from fixed-rubric judges to evaluators that inspect traces and discover unexpected production failures.
- The maturity phases of running evals
Explains how expert rationales can become automated scoring functions and how production clustering helps teams discover new failure modes.
- AI-powered entomology: Lessons from millions of AI code reviews
Shows the same evaluation problem in code review: models can identify many possible issues, but undocumented team judgment determines which feedback developers actually want.
Full transcript
Loading transcript…