Sebastian Fox

Sebastian Fox develops AI evaluation systems that help distinguish plausible outputs from work that is safe and useful in context. A physician by background, he is identified as Composo’s CEO and co-founder in its January 2026 architecture article and presented at AI Engineer World’s Fair 2026. His work addresses silent AI failures: an answer can read convincingly while dropping a warning sign, changing an agreed plan, or overlooking information that determines what someone should do next.

From medicine to Composo

Fox’s career moved from medicine into consulting and applied AI. He studied medicine at Oxford and qualified as a doctor, then became an engagement manager at McKinsey, where he helped build the UK COVID testing programme. He subsequently worked as a product manager at QuantumBlack, McKinsey’s AI business, developing life-sciences AI products. His career across medicine, consulting, and AI brought him from individual clinical decisions into systems intended to support professional work at scale.

He co-founded Composo in 2023. Its early no-code product allowed subject-matter experts to test and improve generative AI applications alongside developers. That division of responsibility addressed a practical problem: clinicians, lawyers, and other practitioners can recognize inadequate work even when they cannot express every objection as evaluation code.

Separating quality reasoning from scoring

In his January 2026 explanation of Composo’s architecture, Fox describes separating reasoning about quality from numerical scoring. Frontier models analyze an output against requested criteria, and a purpose-built reward model scores those analyses using learned preference comparisons. Customer-specific evaluation memory retrieves earlier judgments, expert annotations, and relevant guidance to calibrate later assessments. This gives an evaluator concrete examples of the expected standard while retaining explanations that help people understand what needs attention.

How clinical notes lose meaning

• Misrepresented decisions: His World’s Fair presentation on clinical-note evaluation explains why that context matters. An AI note can record tests that a doctor and patient discussed but ultimately decided to defer. It can turn a patient’s uncertainty about how a headache began into a definite claim of sudden onset. Both outputs may contain words grounded in the consultation, yet misrepresent the decision or introduce a finding that changes the clinical response. Checking whether a note resembles its transcript does not resolve whether it preserves the encounter’s meaning.
• Missing context: Omissions are especially difficult because the remaining text can be entirely plausible. Fox opens with a note that describes a headache but leaves out jaw pain on chewing in a patient over 50—a detail he uses to illustrate a potentially urgent clinical warning sign. His travel-history example makes the contextual problem explicit: two consultations about blood in the urine both lose a line about a recent trip, but a visit to France and freshwater exposure around Lake Malawi have different diagnostic significance. The evaluator must judge which missing information matters in this encounter, rather than treat all omissions alike.

Fox calls this professional judgment “taste”: knowledge that is partly tacit, depends on context, and changes as models, guidelines, and local expectations change. A detailed rubric captures the distinctions its authors can specify in advance. It cannot exhaust the judgments practitioners make when reviewing unfamiliar cases. Adding an automated judge can therefore introduce another quiet failure if it confidently approves the same omissions the generator missed.

Discovering failures and learning from expert judgments

His proposed response is a continuing discover, capture, and calibrate loop. Teams first inspect real outputs and cluster their failures into an evolving set of failure modes. Synthetic tests remain limited to situations their authors imagined; production outputs reveal additional ways the application can go wrong. Those discovered modes organize the cases experts review and the examples an evaluator later retrieves.

The capture step preserves more than a score. Experts leave free-form comments, explain why an output fails, and supply corrections. A focused review session can begin this record without a lengthy labeling programme. The useful material is the reasoning: which detail changes the decision, why a rephrasing is acceptable, or how an apparently faithful summary misstates intent.

For each subsequent output, the evaluator retrieves comparable judgments, applicable corrections, and reference documents or guidelines. A general instruction such as “be faithful” remains useful, but the standard for this particular case is assembled from relevant evidence. In the headache example, prior cases involving missed warning signs and applicable clinical guidance help the judge recognize why the absent detail deserves attention. New expert corrections can inform the next evaluation without waiting for retraining, and reviewers can inspect the examples that influenced a judgment.

Fox favors retaining these judgments as accessible examples when standards are moving and assessments need explanations. In his presentation, he argues that storing the standard only in model weights makes it harder to update and inspect. He reports that the retrieval-based loop performed better than the rubric-based judges he compared on the dataset presented. That is a result from his comparison, rather than proof that the approach resolves clinical-note safety generally.

Testing the evaluators themselves

• Deployed-scribe audit: Fox also co-authored research examining how evaluators miss errors. With Luke Markham, Ryan Lail, and Michael Karotsieris, he published two August 2026 preprints on commercial AI scribes and automated note review. Their audit of three deployed scribes examined 565 notes generated from 142 consultations, combining recorded encounters with authored scenarios. Under the study’s verification procedure, 31.3% of notes contained a verified failure. Review instructions and reviewer models substantially changed the measured failure rate, making the evaluation procedure an important part of interpreting the result.
• Omission detection: Their companion study of omission detection used 500 paired notes with controlled errors and compared eight judge designs. Conventional judges detected added or altered content more successfully than missing information. Establishing transcript facts first, then checking each fact against the note, improved omission detection by giving the reviewer an explicit account of what should have survived the summary. The improved methods still missed omissions, and benchmark thresholds did not transfer directly to real vendor notes.

Composo released OmissionBench, the benchmark and judging code behind the two papers, as an archival research release. This was collaborative work, not Fox’s sole-authored project. The studies give concrete form to his central concern: the evaluator itself needs scrutiny, especially when fluent outputs conceal missing information. His practical starting point is to collect expert reasoning on real outputs and keep using it as the application and its standards evolve.