AI Engineer World's Fair 2026
Don't be data poor
Loading slides…
Read the talk
Don't be data poor: generate the records your evaluations need
When patient records cannot be retained, synthetic evaluation data starts with policy outcomes, grows through patient encounters, and improves under clinician control.
From a talk by Anuj Iravane
Decisions over records you cannot keep
A scanned fax bundle is the input to a consequential decision: does the patient's documented history satisfy a health plan's policy? At Anterior, the clinician-led company where Anuj Iravane leads AI, this pattern appears across prior authorization, payment integrity and HEDIS measures. The shared task is policy-guided decision-making over highly unstructured data. Iravane estimates that around 70% of medical communication still happens via fax.
Those bundles contain rich clinical histories, but neither the histories nor their presentation are uniform. Each patient's trajectory contributes to a long tail of rare, nuanced cases. Handwriting, tables, checkboxes, key-value pairs and images all carry information that the workflow may need. A record is an imperfect observation of a person's life, not a neatly populated form.
The accuracy expectations make representative evaluation data essential. Iravane's standard for these healthcare workflows is that 95% accuracy is not good enough; this is a requirement framing, not a reported benchmark. Anterior therefore invests heavily in datasets and evaluations built around the kinds of medical records its workflows encounter.
But the team cannot simply accumulate those records. Under the contracts Iravane describes, protected health information cannot be retained, reused or preserved through derived copies—even redacted or anonymized versions. These are Anterior's stated contractual restrictions, not a universal rule about de-identification. The engineering problem is immediate: how do you evaluate a high-stakes workflow when the data it most needs is the data you are least allowed to keep?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why not generate the whole record at once?
Synthetic generation offers a way to create persistent evaluation data from scratch. LLMs are a natural tool for the job: synthetic data already serves pretraining, reinforcement learning, computer use and agent development, and Iravane points to successful applications both inside and outside healthcare. The difficulty is getting realistic, diverse records when generation must scale.
Iravane says medical records often exceed 300 pages. Asking an LLM to produce an entire record in one response resembles asking it to write a novel in one shot: the request combines long-range coherence with an enormous amount of detailed generation. Repeating the request introduces another problem—mode collapse, where supposedly different records converge on similar scenarios.
He proposes two explanations for that lack of variety. First, models have little exposure to this particular data source during pretraining. Second, pretraining and post-training objectives favor helpful assistance rather than the diversity and creativity needed for synthetic record generation. In this account, a larger prompt alone does not solve the problem; the pipeline needs a deliberate source of variation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reverse the task and sample policy paths
Start with the workflow the evaluation is meant to test. In the forward direction, a policy is applied to an unstructured record, producing a reasoning trace and an outcome label. Anterior reverses that construction: choose the outcome, select a compatible reasoning trace, then generate a record conditioned on both.
| Direction | Starting point | Intermediate step | Result |
|---|---|---|---|
| Forward task | Record and policy | Apply policy through a reasoning trace | Outcome label |
| Synthetic generation | Sampled outcome | Select a compatible policy trace | Conditioned record |
The crucial change is where diversity enters. Instead of asking the LLM to invent varied cases unaided, the system supplies varied outcomes and paths as generation inputs.
A medical necessity policy for a CPAP device makes this concrete. The policy describes conditions under which the device should be approved or rejected. Those conditions can be represented as branches in a decision tree. Anterior invests in explicit symbolic policy representations, which Iravane says improve the accuracy and consistency of policy execution in LLM workflows. The same representation also makes it possible to sample reasoning paths deterministically for a chosen outcome.
Those paths become the conditioning input for synthetic records. Iravane describes their distribution as a more uniform, useful prior than the one obtained from unconstrained LLM generation. This is a way to control scenario coverage; it does not mean that uniformly sampled policy paths reproduce the frequency of cases in production.
That distinction is useful precisely because a production sample can miss important cases. In Iravane's hypothetical example, a 95% score on 200 customer cases says little about performance on rare cases absent from that sample. Sampling policy paths can deliberately reach scenarios that ordinary data collection may never include. He compares this use of symbolic structure with Synthea, which samples scenarios from symbolic causal state representations; the comparison concerns structured scenario generation, not an assertion that Synthea implements this reverse-label pipeline.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build a patient journey before its documents
With a diverse set of conditioning inputs, the LLM pipeline builds the record from coarse to fine. It first establishes the patient and their history, then produces the documents that would arise from that history:
- Generate patient invariants. Establish attributes such as biological sex, birthdate and blood group.
- Create the patient journey. Combine those invariants with the sampled reasoning trace to generate an ordered list of events and provider encounters. This is a high-level history expressed in natural language.
- Plan documents for each encounter. In Anterior's source data, most documentation originates during provider encounters, so the pipeline models those encounters explicitly.
- Generate the documents. Use each encounter's document plan and the patient's preceding history, then fan out generation to fill the documents with synthetic information.
The intermediate journey is more than an outline for a long response: it models the process through which the underlying evidence would have been created.
This layering also controls context size. Each generation stage works on a bounded task rather than receiving and emitting the entire medical record. Iravane describes the resulting input and output payloads as token efficient. Longer patient journeys can produce more documents through fan-out without requiring one model call to hold the complete record in its context window.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check consistency and close the label round trip
Independent document generation creates a new responsibility: the documents must agree. A refinement loop uses evaluation feedback to improve particular parts of the generated record. One evaluation is an LLM-based consistency check that looks for contradictions, inaccuracies and conflicting information across documents. This check is especially relevant because parallel fan-out produces the documents independently.
The initial outcome label supplies a second check. Run the original task on the generated record, then compare its result with the outcome the generation was supposed to support. This round-trip check asks whether the synthetic evidence is concordant with the intended task inputs and outputs. The theoretical benefit is correctly labeled data by construction, potentially avoiding the expensive ground-truthing process required for real records. That benefit depends on successful generation and checking; choosing a label at the start does not itself prove that the finished record supports it.
All of this generation happens in plain text and Markdown. Iravane considers rendering those records into PDFs unnecessary for this pipeline because available PDF parsers can convert source documents into useful Markdown representations. The evaluation therefore operates in the text domain. Its scope does not establish robustness to scan quality, page layout or OCR errors, which are separate from the consistency and policy reasoning being tested here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let clinicians steer cases and change the pipeline
A technically coherent pipeline still needs someone qualified to judge its output. An AI engineer cannot independently determine whether a generated medical record is credible and useful for the intended workflow. Anterior therefore gives clinicians ownership, beginning with intervention points throughout generation. A clinician can steer an individual record at any stage rather than accepting whatever the pipeline produces.
Clinicians use ideas from cases they encounter in production to guide new synthetic scenarios. They can shape records around situations they have seen, or around failures they want to anticipate. This turns generation into a way to reproduce a kind of challenge for evaluation, within the contractual restrictions already described, rather than a way to retain the original patient record.
The second form of ownership reaches beyond individual records to the pipeline's logic. Anterior runs a skills-based workflow on an internally built generic agent harness. Patient journey creation, document generation, document enrichment and evaluations are all skills. Clinicians can therefore change the instructions governing those stages, not merely submit feedback about their outputs.
For example, a new customer may need an intake form with a particular structure. A clinician can create a skill file for that document type and attach it to the pipeline without an engineering change. Iravane presents skills as an effective interface between AI engineers and domain experts, and says Anterior uses this pattern in other internal and production workflows as well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Create evaluation data when a deployment needs it
At the time of the talk, Anterior uses synthetic data only for evaluation. Iravane reports that roughly 90% of Anterior's datasets consist of synthetic data. He credits that investment with supporting high production accuracy across customer deployments, but supplies no numerical production accuracy result or controlled comparison establishing its effect.
In a reported blind review, clinicians distinguished synthetic from real records about 60% of the time. Iravane sees this as promising, with room to improve. The talk does not supply the sample size or detailed review protocol, so the result is evidence about reported distinguishability under that review, not a measure of clinical correctness or downstream decision accuracy.
The operational benefit is just-in-time evaluation data. Iravane says most of Anterior's datasets are now created for upcoming customer deployments. Instead of waiting for a customer to supply records, the team can model edge cases, generate them and test the workflow before it goes live. Rapid generation changes when evaluation can begin, as well as which scenarios it can include.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Apply the construction beyond healthcare
For another domain, the design starts with the same question: can the inference workflow be reversed? Outcomes and compatible reasoning paths can supply a controllable starting point, while the sampling distribution should reflect the scenarios the evaluation needs to cover. Generation then follows the process that produces the real data—as provider encounters produce medical documents here—rather than attempting to imitate only the final artifact.
Domain experts need control of that process so they can improve both the cases and the machinery that creates them. Their knowledge is what turns repeated generation into a useful improvement cycle. The opportunity extends beyond protected health information: whenever needed data is ephemeral, sensitive or expensive to label, generating it may make evaluation possible before a reusable real-world dataset exists.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source patient simulator with modular clinical rules and instructions for generating synthetic health records.
Further reading
Explains how JSON states and transitions model patient histories and clinical encounters.
Anterior's account of policy decision trees, document processing and clinician oversight in healthcare AI.
Full transcript
Loading transcript…