AI Engineer World's Fair 2026
Verifiable Environments for AI in Biology — Kenny Workman, LatchBio
Loading slides…
Read the talk
Verifiable Environments for AI in Biology
Biological agents need more than coding skill and textbook knowledge: they need experimental data, executable tests, and grading that respects valid scientific alternatives.
From a talk by Kenny Workman
Before you start: Basic familiarity with AI agents, Python functions, and data-analysis workflows is helpful; the biological measurement techniques are introduced as needed.
When one experiment outgrows a laptop
How does a scientist analyze an experiment whose output can exceed the storage capacity of a consumer laptop? This is the practical starting point for Kenny Workman’s work at Latch: biological measurement is producing large datasets, and turning those measurements into scientific conclusions requires increasingly capable infrastructure.
A few experimental classes drive much of this growth. Single-cell biology separates cells and measures their RNA. Spatial biology adds location information, showing how RNA is distributed across tissue. Proteomics measures proteins through a broader collection of techniques. Workman gives the following examples of experimental output; these are illustrative volumes, not universal sizes for every assay.
| Experimental class | Reported output |
|---|---|
| Single-cell | 2–6 TB per run |
| Spatial | 7 TB per run |
| Proteomics | A few hundred GB |
Even before interpretation begins, moving and storing the data can become a substantial part of the work.
Improvements in molecular capture point toward further throughput gains. The infrastructure problem therefore grows with the measurement technology: a useful agent needs access to the data and the compute that can process it, not just a chat window.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Data analysis supplies something executable
A useful way to read a biological paper is to reconstruct its sequence of work:
- Choose a biological model—an experimental biological system, not an AI model.
- Generate measurements from that system.
- Process the resulting data.
- Interpret the results against prior literature.
- Make a scientific claim.
The details vary, but the recurring task is to find a signal in noisy measurements.
Data analysis could provide an executable foundation for biological agents, much as code provides one for software agents. A broad scientific goal may be difficult to verify directly, while individual computations along the way can have checkable outcomes. Those components offer places to benchmark capability and potentially train improvements. The slide makes this concrete with analysis steps including cell-line quality control, differential expression, and Perturb-seq.
The analogy is a proposed foundation, not a claim that executing correct code establishes a biological discovery. Workman explicitly qualifies it: data analysis might scaffold harder scientific work in the way executable software tasks scaffold broader engineering work, but the biological case needs its own justification.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From experimental files to an agent loop
Latch began as a data-tool vendor for biotech and pharmaceutical companies, storing and transforming large experimental files. It later moved toward manufacturers of experimental kits, packaging analysis software that those manufacturers could provide to scientists under their own brands. The same infrastructure components then became tools an agent could call. The operational difference from many chat-based tools is substantial: some biological operations take days or weeks.
The first promising prototypes used coding models that, to the team’s knowledge, had not received serious biology-specific post-training. Their interfaces were familiar: ask a question in chat, build a dashboard, and dispatch computation to external infrastructure.
Consider a cancer biopsy measured with a spatial assay. A scientist wants to know which genes are overexpressed in malignant tissue compared with non-malignant tissue. The agent receives large experimental files, and the scientist iterates with it toward that question. Answering requires working with the measured tissue data, not merely recalling facts about cancer. These early agents were weak, but they showed enough promise to suggest a progression like coding agents: increasingly capable individuals, followed by harnesses that coordinate teams. Workman presents those teams of scientific agents as a likely future, not an already demonstrated capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Knowing biology and writing code are not enough
The missing capability lies between biological knowledge and programming: extracting scientific insight from real data. Workman argues that frontier models at the time of the talk could not be trusted with that work without focused post-training. The task combines software execution, data analysis, and scientific reasoning. Spatial biology offered a useful starting point because Latch already had customers, the technical field was open, and the measurements made biological patterns directly visible—even the organization of a developing mouse embryo.
Building agents required understanding how those measurements were made. Spatial capture technologies draw on chemistry, optics, semiconductors, and physics, with different branches reflecting decades of work on measuring particular molecules. In one sequencing-based approach, a slide carries small beads bearing DNA that captures RNA from a tissue section. Placing tissue over the slide allows the experiment to recover RNA measurements together with information about where that RNA was located.
The resulting inputs include a large numerical matrix and a high-content image. Reaching a biological conclusion requires a sequence of analysis steps, but that sequence varies with the capture technology, tissue, and disease. There is also limited consensus about how to perform many steps. An agent can therefore execute a plausible workflow without establishing that it has done good science. Measuring that distinction becomes the next engineering problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Break the workflow into verifiable tasks
The available benchmarks largely tested academic questions about what a scientist should do, or covered experiments too broadly to measure this particular work. Latch built SpatialBench to evaluate agents working with spatial data. The original SpatialBench contained 146 problems, attempting coverage across experimental kits and analysis tasks.
Grading only the final scientific outcome produced too little information when models usually failed. SpatialBench instead decomposed the analysis into components of a directed acyclic graph, or DAG. Each task starts with data prepared at an intermediate state where a scientist could take the next meaningful step. Benchmark authors then establish what can reasonably count as success from that state. The construction diagram connects this decomposition to data snapshots, graders, manual inspection, and checks for agent cheating.
Each evaluation packages four pieces:
- Data nodes: One or more experimental inputs, such as a matrix or image.
- Task prompt: A carefully described scientific goal.
- Grader configuration: The information needed to evaluate the submitted result.
- Deterministic grader: A function that checks the success condition.
The grader can be a Python function, and the evaluation specification is represented as JSON. SWE-bench supplied an early design reference: give an agent a concrete environment and task, then evaluate the outcome through executable checks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A deterministic grader can still be wrong
A useful biological test needs more than a function that returns True or False.
- Verifiability: A function can check the success condition.
- Durability: The grader accepts valid scientific alternatives rather than enforcing one author’s analysis path.
- Data dependence: Reaching the answer requires interaction with the supplied measurements, not a memorized response in one turn.
Durability is especially difficult because science often lacks a single canonical answer. A correct analysis may follow a different path and produce a different result from the benchmark author’s. The target should therefore be a property that remains invariant across acceptable paths.
For example, if a task’s contract is to identify a set of genes, ordering should not determine whether the answer passes. This small Python grader illustrates that distinction:
python
from collections.abc import Iterable
def grade_gene_set(
submitted: Iterable[str],
expected: Iterable[str],
) -> bool:
return set(submitted) == set(expected)
expected = ["GENE_A", "GENE_B"]
submitted = ["GENE_B", "GENE_A"]
assert grade_gene_set(submitted, expected)
The check preserves an order-independent success condition. It does not establish that expected is the right biological target; constructing that target remains the harder scientific responsibility. Benchmarking helped the team choose models for customer tasks, but also exposed the need for human verification and longer evaluations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Human attempts expose hidden choices
Watching model trajectories across successive releases revealed that many initial task assumptions were poor. Without a canonical scientific answer, having scientists attempt tasks and grade one another’s work became a practical way to find those problems.
One task asked an agent to split a gene list into microglial-activation and oligodendrocyte-inflammation groups, score cells, find neighboring oligodendrocytes using an appropriate radius, and compute a Spearman correlation at two time points. Each instruction concealed a decision:
| Instruction | Unspecified choice |
|---|---|
| Split the genes | Membership of each activity group |
| Identify inflammation | Which genes qualify |
| Score cells | How to normalize measurements |
| Find neighbors | What radius is appropriate |
| Pool local counts | How to aggregate within the radius |
These choices can change the final correlation. A grader tied to one unspoken interpretation can reject a defensible answer.
Human attempts also challenged conventional numerical thresholds used for quality control. A threshold that a bioinformatician routinely applies is not automatically a durable success criterion. Requiring a machine to satisfy an explicit test forces the task author to explain what scientific structure the threshold is supposed to capture. After two rounds of human attempts, the team published a verified subset. Here, verification means an independent expert could reconstruct a passing answer from the prompt and data; it does not imply a uniquely correct scientific answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From individual steps to a scientific result
The next extension was SpatialBench-Long. Real biological work combines experiment types and complete workflows. At every step, the analyst must interpret the data against experimental design, prior literature, and the original scientific goal. The longer tasks aim to resemble the results section of a paper or the analysis behind an industry decision to continue or stop a drug program. Workman reports that constructing each task took roughly a week for a group of three people.
One example asks whether an agent can reconstruct a tumor’s metastatic niche. Given a primary-tumor biopsy and biopsies from metastatic lesions elsewhere in the body, the agent must combine genetic and mRNA evidence to identify the region of the original tumor that seeded the spread. The scientific motivation is to identify the populations that cause problems and might warrant targeted intervention.
Workman reports that none of the evaluated models solved this particular metastatic-origin task at the time of the talk. That result is specific to the example, not a claim of zero success across the benchmark. It also exposes a training problem: when a long workflow ends in failure, a terminal reward says little about which parts of the analysis were useful or where it went wrong.
The team began experimenting with intermediate rubrics. Imagine the possible analysis paths as a tree: different legitimate branches may still pass through common scientific checkpoints. Domain experts can identify those invariant points and use them to assess progress without prescribing every step. But the rubric scores were only loosely correlated numerically with verifiable final outcomes. Workman therefore did not yet regard them as sufficiently established for reinforcement learning or benchmarking. Intermediate assessment remained experimental; verifiable outcomes remained the preferred foundation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extending evaluation across biological work
The spatial work began reaching frontier-model developers. Workman reports that Anthropic included the benchmarks in model cards and presented them at a Claude Science launch, without advance coordination with Latch. He connects that adoption to the team’s early experience shipping products and experimenting with coding agents on customer workflows.
The research then expanded beyond spatial assays into single-cell and epigenomics tasks, including longer workflows. Drug discovery adds another organizational problem: a benchmark must identify which part of a program it measures. Latch’s first small-molecule preclinical pharmacology benchmark begins a broader effort to organize tasks across discovery, development, and translation, further separated by therapeutic type and experimental technique. The slide places TxBench-PP beside a grid of these benchmark categories.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Refusal is also an evaluation problem
Latch also formed a biosecurity team through an acquisition. Workman introduces collaborations with American Wetware and Aclid; the refusal study is associated with American Wetware, while Aclid’s collaboration concerns surveillance. The practical concern is that safety behavior can fail in both directions: blocking basic scientific assistance and overlooking risky requests.
Benchmarking Refusals in Agentic Biology contrasts routine requests a legitimate scientist might make with red-team requests designed to appear innocuous. The conceptual deception example is a request that labels biological material as GFP, a fluorescent protein, while the underlying material is hazardous. Evaluating only the benign description would miss the risk.
In the reported evaluation, routine tasks were refused more often than red-team tasks. The comparison concerns the tested configurations under direct task framing; it is not a universal refusal rate for biological assistants. Interpreting these results also requires distinguishing provider API blocking from a model’s own refusal. The evaluation target is whether the system recognizes the relevant risk while still supporting ordinary scientific work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Customer work determines what to measure next
Latch is aggregating results, preprints, evaluations, and agent trajectories into a central research resource. But the benchmark program remains connected to deployment: the company continues serving customers and working with kit manufacturers, using those relationships to identify the scientific tasks that matter in practice.
The intended cycle is concrete. Customer workflows reveal missing capabilities; those gaps become benchmarks; competition among model developers improves performance on the measured tasks; and better models return to the products scientists use. Workman closes with an invitation to engineers and scientists to join that research and deployment effort.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original benchmark of 146 spatial-analysis problems with experimental data and deterministic graders.
Two rounds of independent expert attempts produce 115 verified tasks and illustrate ambiguous prompts and overly narrow grading tolerances.
A 24-task benchmark testing recovery of scientific conclusions through complete spatial-biology workflows.
A paired evaluation of legitimate biological research and concealed hazards, separating API blocking from model refusal behavior.
Code, datasets, and evaluation instructions for resolving real software issues with language models.
Further reading
Workman's argument for data analysis as the executable foundation of biological agents, including the limits of the software analogy.
Full transcript
Loading transcript…