Don't Trust Your AI Agents. Test Them Like a Scientist — Stephanie Jarmak, Omni Analytics
Loading slides…
Read the talk
Don't Trust Your AI Agents. Test Them Like a Scientist
Stephanie Jarmak follows agent evaluation from a misleading cost improvement to production measurement, showing why realistic workloads, full traces, enforceable gates, and trustworthy records answer different questions.
From a talk by Stephanie Jarmak
At a glance
Ideas worth remembering
Define the decision before interpreting the metrics. A run that became 28% cheaper still failed the experiment's objective when the agent continued to ignore the intended tool.
Evaluate the model, harness, and workload together. Public benchmarks help shortlist candidates; representative internal tasks and repeated runs support decisions about the intended configuration.
Trace availability, invocation, returned information, and final outcome separately. A better tool can be called successfully without improving the task result.
Measure the retrieval behavior you need. Finding all six reference files among 295 returned files illustrates why recall alone can miss the value of precision.
Separate detecting a problem from enforcing a stop, and budget human review capacity. A low false-alarm percentage can still create an unmanageable review queue.
Check whether the retained record deserves trust, then continue measuring in production. Risk and operating scope should determine when an agent may act autonomously and when it must escalate.
Twenty-eight percent cheaper—and still ignoring the tool
After a prompt change, a coding-agent run cost 28% less. The agent still ignored the code-search tool the experiment was meant to test. That discrepancy opens Stephanie Jarmak's practical case for treating agent evaluations as experiments. An AI engineer at Omni Analytics at recording time, she previously worked on Amp and evaluation infrastructure at Sourcegraph, including CodeScaleBench. Before that, she was an astronomer. The scientific habit she brings to agents is to question what a result actually establishes.
The Sourcegraph experiment concerned a code-search MCP tool. The intervention was a change to the agent's initial prompt, intended to encourage tool use; the run retained traces and produced several metrics. The question to inspect in the comparison is simple: did the intended behavior change alongside the cost? It did not. The 28% reduction was an observation about spending, while the agent continued to skip the tool.
The cheaper run could justify a separate investigation into cost variation. It could not establish that the code-search integration was working. Keeping those questions separate prevents an attractive secondary metric from deciding an experiment whose primary objective remains unmet.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The harness can reverse the model ranking
A second comparison asks what happens when the same two models run under different harnesses. In a third-party Terminal-Bench test on 63 tasks requiring no GPU, Claude Opus 5 solved 36 tasks under OpenHands and GPT-6 Astra solved 31. Under PI, Opus solved 19 and Astra solved 38, giving Astra a lead of about 30 percentage points. Each configuration had one scored run per task after infrastructure retries.
The harness is the surrounding setup through which the model receives instructions, accesses tools, and carries out work. A benchmark result therefore describes a configuration, not a model in isolation. If a harness change can reverse the ordering, choosing a model from a leaderboard without testing the intended setup can select the wrong combination for the actual job.
This gives public benchmarks and product evaluations different roles:
- Public benchmarks narrow the search. They provide a standardized reference set and a view of the wider landscape, helping a team shortlist configurations worth investigating.
- Internal evaluations support a decision. They measure changes to a product or process using tasks and environments that reflect its own work. A standardized internal benchmark can then make successive configurations comparable.
The law-firm example makes the relevance test concrete. Terminal-Bench results might help the firm's developers choose a coding agent; they do not automatically tell the firm how an agent will perform its other work. Match the benchmark to the task, then test the configuration as close as possible to the environment where it will run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose workloads, repetitions, and interacting variables
Once a shortlist exists, experimental design starts with the work the system must cover. Grouping tasks into families makes it harder for one convenient task type to stand in for the whole product. Jarmak built a broader coding benchmark because, at the time, the bug-fixing emphasis of SWE-bench did not cover the range of developer work she wanted to study. Understanding a codebase and fixing a bug are different tasks; a model can be effective at one without being equally effective at the other.
Three design choices deserve separate attention:
- Workload coverage: Select representative tasks and organize them into meaningful families. Tooling may need to work differently across those families.
- Repeated runs: Decide how many runs to perform because the same setup can produce different results from one attempt to the next. A single run cannot reveal that variation.
- Interaction effects: Test relevant combinations of changes. The model-and-harness comparison shows why the effect of one component can depend on which other component accompanies it.
These choices determine what a result can mean. A configuration that improves bug fixing may leave code understanding unchanged; a strong run may not repeat; a model that wins in one harness may lose in another. The experiment needs enough coverage to distinguish those possibilities before the result becomes a product decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trace the path from tool availability to task outcome
Returning to the unused search tool, the next question is where the information path broke. Retain the traces, then begin with the context actually available to the agent. Was the tool available at all? If not, stop the investigation there and repair the setup. If it was available, inspect whether the agent called it and what values came back. Tool availability, tool invocation, and useful returned information are separate facts.
These checks answer a narrower question than a final score can. Which prerequisite failed before the information could help? Even after a successful call, one question remains. Did the agent use the returned information in a way that changed the task outcome?
Even a successful tool correction can leave the final result unchanged. Jarmak describes the scenario of an agent finally using a much better tool, yet failing to benefit from its information. That moves the investigation downstream: the integration now works far enough to return useful material, but the agent or configuration still does not turn it into a better result. Keep non-use and failed attempts in view rather than examining only successful calls.
Where did useful information stop helping?
Read the diagram as text
- Was the tool available?. Inspect the context and tools the agent actually received.
- Repair the setup. setup-failure — A missing tool must be fixed before investigating its use.
- Was the tool called?. Availability does not establish invocation.
- Investigate non-use. invocation-failure — Retain the attempt where the available tool was not used.
- What came back?. Inspect the values returned by the tool.
- Was the information used?. A better tool can return information the agent does not use.
- Did the task outcome change?. Check the end result separately from tool-level progress.
- Was the tool available? → Repair the setup: No: stop and fix availability.
- Was the tool available? → Was the tool called?: Yes: inspect invocation.
- Was the tool called? → Investigate non-use: No: inspect the skipped tool.
- Was the tool called? → What came back?: Yes: inspect returned values.
- What came back? → Was the information used?: Trace whether returned information was used.
- Was the information used? → Did the task outcome change?: Check for an end-task benefit.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Returning 295 files is not the same as finding the right six
The search-tool comparison also exposes a measurement choice. In Jarmak's example, Claude used grep and returned 295 files—the “entire kitchen sink” approach. All six files in the answer key were present. The other 289 files did not affect recall.
Two retrieval measures answer different questions:
- Recall: What fraction of the reference files did the search retrieve? Here it found six out of six, for a recall of one.
- Precision: What fraction of the returned files matched the reference set? Here six out of 295 matched, or about 2%.
Jarmak describes wanting a balance of precision and recall. The scorer used an F-beta score with beta 0.5, which gives precision more weight than recall.
The business objective decides which tradeoff matters. A search that returns every possibly relevant item can be appropriate under a recall-focused objective and poor under a precision-sensitive one. Before adopting a single benchmark score, ask what its validator measures and why that criterion was chosen. A higher score answers the validator's question; it only answers yours if those questions align.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A warning must be able to stop work
Evaluation overlaps with operational safety when a detected failure is supposed to block further action. In Jarmak's experimental software factory, work requests passed through approval queues and required checks, with regular audits. The agent could notice that something was wrong. The missing mechanism was a way to enforce a stop.
Separate the responsibilities:
- Detection: An agent or another checking system flags a problem.
- Enforcement: Code implements the gate that prevents work from continuing when required conditions are not met.
- Escalation: A defined path brings the blocked decision to a human when necessary.
A checker that produces the right warning has only performed the first job. Without enforcement, the workflow can continue through a failure it has already recognized.
Sending every alarm to a person creates a second engineering problem: review capacity. In a separate LLM-as-a-judge system Jarmak cites, a 1% false-alarm rate would produce 10,000 human reviews at the scale under discussion. The percentage sounds small; the queue does not. The design must account for whether domain experts can handle that load and how consequential the flagged cases are.
Alarm quality and downstream operations therefore belong in the same evaluation. A threshold does not become useful merely because it catches problems: its false alarms consume scarce attention, while the consequences of missed problems affect how much risk the organization can accept. The bottleneck may move from automated execution to expert review.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The retained trace can also be wrong
Retaining traces is necessary for diagnosis, but the record itself can be unreliable. Jarmak describes a security experiment in which researchers seeded issues into an agent setup and used a hash check intended to detect tampering. That check passed without noticing the injected issues. An independent comparison of records—including traffic history and model changes—detected all the seeded issues in the described experiment.
The comparison asks what each checking method can establish. Here, a passing hash check did not establish that the security-relevant history was sound, while the independent record comparison exposed discrepancies. The talk does not detail the hash coverage or the injection mechanisms, so this is a warning about relying on that particular check, not a general claim that hashes cannot protect records.
Observability needs a trust model of its own. Decide whether the retained record is reliable enough for the system's security needs before using it to explain behavior or certify a result. More traces help only if the relevant events are recorded in a form the investigation can trust.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production tests whether the workload proxy was good
At Omni, the workload design begins with internal production data: how users actually interact with agents. Those interactions are classified into task types, including Q&A ranging from easy to hard questions, dashboard building, and app building. These categories give the evaluation harness a way to compare changes against recognizable kinds of user work rather than an undifferentiated average.
The purpose is to identify where a system change adds value and where failures appear downstream. Even a carefully constructed internal suite remains a proxy for actual workloads. “Production is truth” is Jarmak's shorthand for the next obligation: after deployment, keep measuring the system in its real environment.
Shadow operation offers a way to gather information when directly changing live behavior would carry a larger blast radius. The choice is risk-dependent: obtain observations from realistic conditions while limiting the effects of an insufficiently trusted change. Offline results guide the deployment decision; measurements in the deployed environment test whether those results carried over.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make the next action and the escalation decision explicit
The final decision is how much risk to accept before a human must intervene. In her software factory, Jarmak allows some autonomy for lower-risk decisions while making the proposed next action and its scope explicit. When human input is needed, the system identifies why a person should weigh in. That makes escalation a bounded decision about an action, rather than an unspecified request to inspect everything the agent has done.
The distinction to encode is when autonomous deployment is acceptable and when work must stop and escalate. It connects back to the earlier gates: risk criteria decide when human involvement is required, and enforcement must make that requirement real. More autonomy is a choice about which actions the system may take under which conditions.
Jarmak closes by offering reusable skills for evaluation design, harness traceability, and reliability procedures, alongside a roughly 300-page archive that agents can consult. These materials supply procedures to bring into a system. The decisions they support remain concrete: what work is representative, what result matters, what record can be trusted, and when the system must hand the next action to a human.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Jarmak built this public benchmark to study agents working across complex codebases and to cover developer tasks beyond bug fixing.
Further reading
Jarmak describes a fix that passed automated review and was marked complete while its commits remained on an unmerged branch.
Companion repository to Jarmak’s essay, with executable reliability patterns and fault drills for AI software factories.
Related talks
- Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust
Pairs Jarmak’s coding-agent retrieval example with a comparison of agentic search and vector search.
- How to Know When an AI Agent Can Work Unsupervised — Evan Kormos, Coinbase
Shares Jarmak’s closing question about when an agent can safely work unsupervised.
- Fixing the PR Bottleneck — Matt Pocock, AIHero
Connects Jarmak’s discussion of human review capacity to the pull-request bottleneck.
Full transcript
Loading transcript…