AI Engineer World's Fair 2026

Don't be data poor

Loading slides…

Read the talk

Don't be data poor: generate the records your evaluations need

When patient records cannot be retained, synthetic evaluation data starts with policy outcomes, grows through patient encounters, and improves under clinician control.

From a talk by Anuj Iravane

Decisions over records you cannot keep

A scanned fax bundle is the input to a consequential decision: does the patient's documented history satisfy a health plan's policy? At Anterior, the clinician-led company where Anuj Iravane leads AI, this pattern appears across prior authorization, payment integrity and HEDIS measures. The shared task is policy-guided decision-making over highly unstructured data. Iravane estimates that around 70% of medical communication still happens via fax.

Green slide with a fax statistic above overlapping documents showing handwriting, a fax cover sheet, and a medical report with an image and table.Open full source frame
Scanned medical records illustrate the slide’s claim that about 70% of medical communication still goes by fax.

Those bundles contain rich clinical histories, but neither the histories nor their presentation are uniform. Each patient's trajectory contributes to a long tail of rare, nuanced cases. Handwriting, tables, checkboxes, key-value pairs and images all carry information that the workflow may need. A record is an imperfect observation of a person's life, not a neatly populated form.

The accuracy expectations make representative evaluation data essential. Iravane's standard for these healthcare workflows is that 95% accuracy is not good enough; this is a requirement framing, not a reported benchmark. Anterior therefore invests heavily in datasets and evaluations built around the kinds of medical records its workflows encounter.

But the team cannot simply accumulate those records. Under the contracts Iravane describes, protected health information cannot be retained, reused or preserved through derived copies—even redacted or anonymized versions. These are Anterior's stated contractual restrictions, not a universal rule about de-identification. The engineering problem is immediate: how do you evaluate a high-stakes workflow when the data it most needs is the data you are least allowed to keep?

0:160:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Why not generate the whole record at once?

Synthetic generation offers a way to create persistent evaluation data from scratch. LLMs are a natural tool for the job: synthetic data already serves pretraining, reinforcement learning, computer use and agent development, and Iravane points to successful applications both inside and outside healthcare. The difficulty is getting realistic, diverse records when generation must scale.

Iravane says medical records often exceed 300 pages. Asking an LLM to produce an entire record in one response resembles asking it to write a novel in one shot: the request combines long-range coherence with an enormous amount of detailed generation. Repeating the request introduces another problem—mode collapse, where supposedly different records converge on similar scenarios.

He proposes two explanations for that lack of variety. First, models have little exposure to this particular data source during pretraining. Second, pretraining and post-training objectives favor helpful assistance rather than the diversity and creativity needed for synthetic record generation. In this account, a larger prompt alone does not solve the problem; the pipeline needs a deliberate source of variation.

3:063:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:06 · section reference included

Reverse the task and sample policy paths

Start with the workflow the evaluation is meant to test. In the forward direction, a policy is applied to an unstructured record, producing a reasoning trace and an outcome label. Anterior reverses that construction: choose the outcome, select a compatible reasoning trace, then generate a record conditioned on both.

DirectionStarting pointIntermediate stepResult
Forward taskRecord and policyApply policy through a reasoning traceOutcome label
Synthetic generationSampled outcomeSelect a compatible policy traceConditioned record

The crucial change is where diversity enters. Instead of asking the LLM to invent varied cases unaided, the system supplies varied outcomes and paths as generation inputs.

A medical necessity policy for a CPAP device makes this concrete. The policy describes conditions under which the device should be approved or rejected. Those conditions can be represented as branches in a decision tree. Anterior invests in explicit symbolic policy representations, which Iravane says improve the accuracy and consistency of policy execution in LLM workflows. The same representation also makes it possible to sample reasoning paths deterministically for a chosen outcome.

Clinical Policies slide showing a CPAP policy document on the left and a branching graph on the right, with arrows along one path and a yellow outcome node.Open full source frame
A CPAP clinical policy is represented as a symbolic reasoning graph with a marked path to an outcome.

Those paths become the conditioning input for synthetic records. Iravane describes their distribution as a more uniform, useful prior than the one obtained from unconstrained LLM generation. This is a way to control scenario coverage; it does not mean that uniformly sampled policy paths reproduce the frequency of cases in production.

That distinction is useful precisely because a production sample can miss important cases. In Iravane's hypothetical example, a 95% score on 200 customer cases says little about performance on rare cases absent from that sample. Sampling policy paths can deliberately reach scenarios that ordinary data collection may never include. He compares this use of symbolic structure with Synthea, which samples scenarios from symbolic causal state representations; the comparison concerns structured scenario generation, not an assertion that Synthea implements this reverse-label pipeline.

4:535:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:53 · section reference included

Build a patient journey before its documents

With a diverse set of conditioning inputs, the LLM pipeline builds the record from coarse to fine. It first establishes the patient and their history, then produces the documents that would arise from that history:

  1. Generate patient invariants. Establish attributes such as biological sex, birthdate and blood group.
  2. Create the patient journey. Combine those invariants with the sampled reasoning trace to generate an ordered list of events and provider encounters. This is a high-level history expressed in natural language.
  3. Plan documents for each encounter. In Anterior's source data, most documentation originates during provider encounters, so the pipeline models those encounters explicitly.
  4. Generate the documents. Use each encounter's document plan and the patient's preceding history, then fan out generation to fill the documents with synthetic information.

The intermediate journey is more than an outline for a long response: it models the process through which the underlying evidence would have been created.

This layering also controls context size. Each generation stage works on a bounded task rather than receiving and emitting the entire medical record. Iravane describes the resulting input and output payloads as token efficient. Longer patient journeys can produce more documents through fan-out without requiring one model call to hold the complete record in its context window.

8:098:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:09 · section reference included

Check consistency and close the label round trip

Independent document generation creates a new responsibility: the documents must agree. A refinement loop uses evaluation feedback to improve particular parts of the generated record. One evaluation is an LLM-based consistency check that looks for contradictions, inaccuracies and conflicting information across documents. This check is especially relevant because parallel fan-out produces the documents independently.

The initial outcome label supplies a second check. Run the original task on the generated record, then compare its result with the outcome the generation was supposed to support. This round-trip check asks whether the synthetic evidence is concordant with the intended task inputs and outputs. The theoretical benefit is correctly labeled data by construction, potentially avoiding the expensive ground-truthing process required for real records. That benefit depends on successful generation and checking; choosing a label at the start does not itself prove that the finished record supports it.

Flow diagram from sampled seed through patient invariants, a patient journey, and document plans to multiple document types, followed by a refinement loop and evaluation checks.Open full source frame
The full pipeline connects sampled labels and reasoning traces to documents and consistency, realism, and round-trip checks.

All of this generation happens in plain text and Markdown. Iravane considers rendering those records into PDFs unnecessary for this pipeline because available PDF parsers can convert source documents into useful Markdown representations. The evaluation therefore operates in the text domain. Its scope does not establish robustness to scan quality, page layout or OCR errors, which are separate from the consistency and policy reasoning being tested here.

9:5310:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:53 · section reference included

Let clinicians steer cases and change the pipeline

A technically coherent pipeline still needs someone qualified to judge its output. An AI engineer cannot independently determine whether a generated medical record is credible and useful for the intended workflow. Anterior therefore gives clinicians ownership, beginning with intervention points throughout generation. A clinician can steer an individual record at any stage rather than accepting whatever the pipeline produces.

Clinicians use ideas from cases they encounter in production to guide new synthetic scenarios. They can shape records around situations they have seen, or around failures they want to anticipate. This turns generation into a way to reproduce a kind of challenge for evaluation, within the contractual restrictions already described, rather than a way to retain the original patient record.

Dimmed pipeline diagram with a handwritten human in the loop label and three bright curved arrows pointing toward the seed, patient journey, and evaluation stages.Open full source frame
Human-in-the-loop arrows identify intervention points across the generation pipeline.

The second form of ownership reaches beyond individual records to the pipeline's logic. Anterior runs a skills-based workflow on an internally built generic agent harness. Patient journey creation, document generation, document enrichment and evaluations are all skills. Clinicians can therefore change the instructions governing those stages, not merely submit feedback about their outputs.

For example, a new customer may need an intake form with a particular structure. A clinician can create a skill file for that document type and attach it to the pipeline without an engineering change. Iravane presents skills as an effective interface between AI engineers and domain experts, and says Anterior uses this pattern in other internal and production workflows as well.

11:3411:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:34 · section reference included

Create evaluation data when a deployment needs it

At the time of the talk, Anterior uses synthetic data only for evaluation. Iravane reports that roughly 90% of Anterior's datasets consist of synthetic data. He credits that investment with supporting high production accuracy across customer deployments, but supplies no numerical production accuracy result or controlled comparison establishing its effect.

In a reported blind review, clinicians distinguished synthetic from real records about 60% of the time. Iravane sees this as promising, with room to improve. The talk does not supply the sample size or detailed review protocol, so the result is evidence about reported distinguishability under that review, not a measure of clinical correctness or downstream decision accuracy.

The operational benefit is just-in-time evaluation data. Iravane says most of Anterior's datasets are now created for upcoming customer deployments. Instead of waiting for a customer to supply records, the team can model edge cases, generate them and test the workflow before it goes live. Rapid generation changes when evaluation can begin, as well as which scenarios it can include.

14:0714:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:07 · section reference included

Apply the construction beyond healthcare

For another domain, the design starts with the same question: can the inference workflow be reversed? Outcomes and compatible reasoning paths can supply a controllable starting point, while the sampling distribution should reflect the scenarios the evaluation needs to cover. Generation then follows the process that produces the real data—as provider encounters produce medical documents here—rather than attempting to imitate only the final artifact.

Domain experts need control of that process so they can improve both the cases and the machinery that creates them. Their knowledge is what turns repeated generation into a useful improvement cycle. The opportunity extends beyond protected health information: whenever needed data is ephemeral, sensitive or expensive to label, generating it may make evaluation possible before a reusable real-world dataset exists.

15:2215:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:22 · section reference included

Resources

From the talk

  • Open-source patient simulator with modular clinical rules and instructions for generating synthetic health records.

Transcript download
Full transcript

Loading transcript…