AI Engineer World's Fair 2026
Your Agent Evolved. Your Evals Didn't.
Loading slides…
Read the talk
Your Agent Evolved. Your Evals Didn’t.
As agents move from single model calls to tool loops, memory, and code execution, evaluations must expand from answer quality to the behavior of the whole system.
From a talk by Ameya Bhatawdekar
Before you start: Familiarity with model prompts, tool calls, and basic retrieval-augmented generation will help; no evaluation framework experience is required.
Why a better model can require a different system
Why is an AI demo easy to build, while maintaining its quality in production is so difficult? A chatbot, agent, or batch-processing application can change substantially after its first working version: prompts become pipelines, orchestration grows, and eventually the application needs a rewrite. Ameya Bhatawdekar, Braintrust’s Field CTO, frames this as a moving-target problem. Models change, users discover different ways to use the application, and the underlying data changes too.
Better tool calling, longer usable context, code generation for sandboxed execution, and practical memory systems can each change what an application is capable of doing. These are sometimes large enough advances to require re-platforming, rather than another round of prompt tuning.
Simply replacing the model may leave those capabilities inaccessible. If the old model struggled to choose tools, the surrounding software probably made those choices for it. A stronger model placed inside the same constraints still cannot exercise its new judgment. Restructuring the application can unlock that capability, but it also introduces places where behavior can go wrong. Architecture follows model capabilities; evaluations must follow architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with an SRE agent—and the simplest evaluation
Consider an SRE agent that can read system information and take operational actions. Its tools include deployment rollback, escalation to a human, and paging. The distinction between read tools and write tools matters: the system can change the environment, not merely describe it. Bhatawdekar uses this agent to ground a series of notional evaluations as the architecture evolves.
The starting architecture is one input, one model call, and one output. Here, the final answer is a reasonable unit of evaluation. Does it answer accurately? Are its claims factual? Does it invent information or rely on stale training knowledge rather than the information relevant to the task?
A golden dataset supplies representative inputs and expected behavior. Scorers encode what a good answer looks like and evaluate the model’s responses against that definition. This is a useful baseline precisely because the architecture is narrow: there is no retrieval stage, tool execution, or orchestration to inspect independently.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A chain creates intermediate failure points
Retrieval-augmented generation adds work before the answer. A typical chain follows a fixed procedure:
- Parse the user’s input to extract information needed for retrieval.
- Retrieve relevant information.
- Assemble that information into context.
- Ask the model to reason over the context and synthesize an answer.
The answer is now the product of several stages, each of which can affect correctness.
| Stage | Failure to evaluate |
|---|---|
| Parsing | Extracting the wrong information |
| Retrieval | Returning the wrong context |
| Context construction | Overloading the model with material it cannot use effectively |
| Generation | Failing to reason correctly over the supplied context |
A larger advertised context window does not itself establish that a model can reason well over everything placed inside it. Bhatawdekar recalls early systems in which context stuffing impaired performance even as window sizes increased. Intermediate evaluations make a bad final answer actionable: they help distinguish a retrieval problem from a reasoning problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Letting the model choose the next step
A fixed chain performs the same kind of sequence for every request. ReAct offers a different structure: interleave reasoning and action, allowing the model to decide what to do next. The paper appeared in 2022; Bhatawdekar places its growing popularity among application builders in 2023 and early 2024.
The loop is straightforward: choose a tool call, inspect its return, reason about the new information, and select the next step. Continue until the user’s intent is satisfied or the iteration budget is exhausted. Instead of requiring a developer to predefine the entire route, the model can organize the work around the request and the observations it receives.
The early models made that flexibility difficult to use reliably. They supplied incorrect tool arguments, selected the wrong tools, made reasoning mistakes, and struggled to preserve useful context over longer executions. An appealing orchestration pattern still depended on capabilities the model did not consistently possess.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Workflow graphs buy predictability with explicit control
When the model could not reliably control execution, teams moved control into software. A graph or state machine encoded planning, orchestration, and execution; models performed narrower tasks inside individual nodes. This made behavior more predictable because the surrounding program constrained how the system could proceed.
Those constraints also defined the system’s supported distribution of requests. An intent that fit the graph could work well. An intent requiring a different sequence could expose a gap in the orchestration. Teams responded by adding branches and special cases, increasing the graph’s complexity as they expanded its coverage.
Evaluation consequently expanded beyond the answer and the individual model call:
- Branch behavior: Does the system choose the appropriate path and behave consistently along it?
- Inter-node contracts: Does one node produce information the next node can correctly consume?
- Classifier accuracy: Does an early classification send the request down the right workflow?
- Node behavior: Does each bounded model task work on its intended inputs?
- Retries and orchestration: Do retry loops and the overall execution behave correctly?
These checks supplement earlier evaluations. Moving decisions into deterministic software does not remove the need to evaluate those decisions or their interaction with model outputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The loop returns, and evaluation becomes a distribution
Bhatawdekar places widespread production use of graph frameworks in late 2024 and early 2025. In his account, Anthropic and OpenAI releases in mid-to-late 2025 then substantially improved tool calling, orchestration, long-horizon planning, and the ability to recognize mistakes and course-correct. This is an architectural assessment rather than a model-specific benchmark: he identifies no model versions or measured success threshold.
Graphs built to compensate for earlier weaknesses could still constrain those stronger models and retain their own brittleness. The model-driven loop became practical again. Yet successful execution did not imply a fixed path: repeated runs on the same input could take substantially different trajectories while arriving at the correct answer.
A single successful trace is insufficient evidence of repeatable behavior. Run the same evaluation multiple times and inspect the distribution of results. Bhatawdekar proposes repeated trials without specifying a fixed value of k or an uncertainty calculation; the required number depends on the reliability question being investigated.
| Metric | Success condition across k attempts | Question |
|---|---|---|
pass@k | At least one attempt succeeds | Can the agent solve the task? |
pass^k | Every attempt succeeds | Can it solve the task consistently? |
The talk’s verbal description of pass^k as how many trials succeed needs a precise distinction. In the original τ-bench definition, it is the probability that all k independent, identically distributed trials succeed, averaged across tasks—not the count or fraction of successful trials. That definition dates to 2024; these metrics become useful here because the architecture makes repeated execution especially consequential.
For a single task with independent trials and success probability p, the distinction is:
More attempts increase the chance of seeing at least one success, while requiring every attempt to succeed becomes more demanding. A high pass@k therefore does not establish dependable repeated performance.
A small TypeScript helper can keep those questions separate when inspecting one batch of graded SRE-agent runs. Each boolean represents whether a run met the evaluation’s success criteria; the helper summarizes the batch, rather than estimating a population probability from it.
typescript
type BatchResult = {
attempts: number;
successes: number;
successFraction: number;
anySucceeded: boolean;
allSucceeded: boolean;
};
function summarizeTrials(passed: readonly boolean[]): BatchResult {
if (passed.length === 0) {
throw new Error("At least one graded trial is required");
}
const successes = passed.filter(Boolean).length;
return {
attempts: passed.length,
successes,
successFraction: successes / passed.length,
anySucceeded: successes > 0,
allSucceeded: successes === passed.length,
};
}
Keeping the success fraction, any-success condition, and all-success condition distinct prevents a capability result from being mistaken for a reliability result. The traces behind those outcomes then help locate the sources of failure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The agent becomes a product system
The next expansion surrounds the loop with persistent capabilities. Memory systems store and retrieve information within a session and across sessions, allowing later runs to benefit from earlier ones. Sandboxes provide an execution environment for model-generated code. The behavior being evaluated now depends on more than the current prompt and the model’s immediate tool choices.
Model Context Protocol connections and skill directories add extensibility. Skills augment the agent through symbolic instructions, expanding what it can do without changing model weights. Memory, execution environments, and these extensions all add surfaces where the product can fail. An evaluation suite inherited unchanged from the previous generation covers only part of the resulting system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production must keep changing the evaluation suite
Across these architectural generations, evaluations remain a durable description of intended behavior. Their implementation and coverage must change, but they preserve a useful contract: new capabilities should not break the experiences that already work.
That contract needs a continuous connection to actual usage. Production data supplies cases for evaluations; those evaluations guide improvements; the improved system generates new production evidence. Bhatawdekar reports that many teams endorse this flywheel but leave their suites largely static. Even without an architectural change, a suite can become less representative as users and their requests evolve.
The harder requirement is discovering failures the existing suite does not know how to look for. Collecting another example of a known error improves coverage within an established category. Discovering a new category changes the definition of what must be evaluated. Architectural changes make this second kind of discovery especially important: a new component or execution pattern can fail in ways the previous definitions of good behavior never anticipated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Finding missing categories with Topics
Braintrust combines evaluations, observability, and production insights to feed new cases back to the development team. Its Topics feature adds clustering as a way to find patterns in production data, including failure categories a team has not already encoded in a scorer or guardrail.
Bhatawdekar describes this as cluster analysis over all production data. The operational scope is more specific: Braintrust’s later engineering account describes clustering sampled, extracted facets and then classifying incoming data. The mechanism supports discovering patterns; it does not guarantee discovery of every failure. The useful result is a surfaced category that prompts the team to investigate behavior its existing checks missed.
Those observations can become evaluation cases, expanding the suite beyond new examples of already-known failures. After an architectural change, this gives teams a route from unexpected production behavior to a dataset that explicitly exercises the new failure mode.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make evaluation part of how the system changes
Bhatawdekar expects further model capability unlocks and, with them, more substantial changes to agent architectures. The operational requirement is to make those changes without losing existing capabilities: the system should improve at new tasks while continuing to perform the work users already depend on.
That requires evaluation discipline supported by tools, automation, and an everyday production-feedback workflow. The same process must accommodate both small improvements and changes large enough to reshape the system. Evaluations earn their durability by continuing to describe—and test—the behavior that matters as the implementation underneath them changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original research on interleaving model reasoning with actions and observations.
Instructions for finding patterns in production traces and turning selected examples into evaluation datasets.
Official introduction to connecting AI applications with external tools and data through MCP.
Further reading
A tool-and-user interaction benchmark introducing pass^k to measure success across every repeated trial.
Practical guidance on graders, outcomes, regression coverage, and repeated-trial metrics.
- Inside Braintrust TopicsArticle
An engineering account of trace preprocessing, facet extraction, embeddings, clustering, and continuous classification.
An explanation of packaging instructions and supporting resources into reusable agent skills.
Full transcript
Loading transcript…