▶ Watch ↗AI Engineer World's Fair 202619:48
Conference affiliation: Composo · 2026
Sebastian Fox develops AI evaluation systems that help distinguish plausible outputs from work that is safe and useful in context. A physician by background, he is identified as Composo’s CEO and co-founder in its January 2026 architecture article and presented at AI Engineer World’s Fair 2026. His work addresses silent AI failures: an answer can read convincingly while dropping a warning sign, changing an agreed plan, or overlooking information that determines what someone should do next.
Fox’s career moved from medicine into consulting and applied AI. He studied medicine at Oxford and qualified as a doctor, then became an engagement manager at McKinsey, where he helped build the UK COVID testing programme. He subsequently worked as a product manager at QuantumBlack, McKinsey’s AI business, developing life-sciences AI products. His career across medicine, consulting, and AI brought him from individual clinical decisions into systems intended to support professional work at scale.
He co-founded Composo in 2023. Its early no-code product allowed subject-matter experts to test and improve generative AI applications alongside developers. That division of responsibility addressed a practical problem: clinicians, lawyers, and other practitioners can recognize inadequate work even when they cannot express every objection as evaluation code.
In his January 2026 explanation of Composo’s architecture, Fox describes separating reasoning about quality from numerical scoring. Frontier models analyze an output against requested criteria, and a purpose-built reward model scores those analyses using learned preference comparisons. Customer-specific evaluation memory retrieves earlier judgments, expert annotations, and relevant guidance to calibrate later assessments. This gives an evaluator concrete examples of the expected standard while retaining explanations that help people understand what needs attention.
Fox calls this professional judgment “taste”: knowledge that is partly tacit, depends on context, and changes as models, guidelines, and local expectations change. A detailed rubric captures the distinctions its authors can specify in advance. It cannot exhaust the judgments practitioners make when reviewing unfamiliar cases. Adding an automated judge can therefore introduce another quiet failure if it confidently approves the same omissions the generator missed.
His proposed response is a continuing discover, capture, and calibrate loop. Teams first inspect real outputs and cluster their failures into an evolving set of failure modes. Synthetic tests remain limited to situations their authors imagined; production outputs reveal additional ways the application can go wrong. Those discovered modes organize the cases experts review and the examples an evaluator later retrieves.
The capture step preserves more than a score. Experts leave free-form comments, explain why an output fails, and supply corrections. A focused review session can begin this record without a lengthy labeling programme. The useful material is the reasoning: which detail changes the decision, why a rephrasing is acceptable, or how an apparently faithful summary misstates intent.
For each subsequent output, the evaluator retrieves comparable judgments, applicable corrections, and reference documents or guidelines. A general instruction such as “be faithful” remains useful, but the standard for this particular case is assembled from relevant evidence. In the headache example, prior cases involving missed warning signs and applicable clinical guidance help the judge recognize why the absent detail deserves attention. New expert corrections can inform the next evaluation without waiting for retraining, and reviewers can inspect the examples that influenced a judgment.
Fox favors retaining these judgments as accessible examples when standards are moving and assessments need explanations. In his presentation, he argues that storing the standard only in model weights makes it harder to update and inspect. He reports that the retrieval-based loop performed better than the rubric-based judges he compared on the dataset presented. That is a result from his comparison, rather than proof that the approach resolves clinical-note safety generally.
Composo released OmissionBench, the benchmark and judging code behind the two papers, as an archival research release. This was collaborative work, not Fox’s sole-authored project. The studies give concrete form to his central concern: the evaluator itself needs scrutiny, especially when fluent outputs conceal missing information. His practical starting point is to collect expert reasoning on real outputs and keep using it as the application and its standards evolve.
▶ Watch ↗AI Engineer World's Fair 202619:48
Loading more talks and highlights…