AI Engineer World's Fair 2025

Five Hard-Earned Lessons About Evals

Loading slides…

Read the talk

Five Hard-Earned Lessons About Evals

Useful evals turn model releases and user feedback into product decisions. Building them requires engineering the data, scoring rules, tools, and optimization process together.

From a talk by Ankur Goyal

Before you start: Familiarity with LLM prompts, tool calls, and evaluation datasets will help you follow the examples.

Can your evals support the next model release?

A new model comes out. Can your team determine whether to use it and ship the change while the opportunity is still fresh? Ankur Goyal proposes a concrete test: your evals should prepare you to incorporate a new model into your product within 24 hours. He reports that Notion had incorporated new models within 24 hours across several releases. The point is operational: the evaluation process gives the team enough confidence to make a product decision.

The next test starts with a complaint. Can a user-reported failure become an eval through a clear, straightforward process? Without that path, valuable information disappears into support conversations and scattered feedback. With it, the team can preserve the failure and use it to guide subsequent improvements.

Slide stating that user feedback can be saved as an eval, with a downward arrow from Human Feedback to Eval.Open full source frame
User feedback can become an eval.

The third test looks ahead: can your evals tell you which use cases are feasible before you ship them? Regression tests help protect behavior that already works. Evals can also explore behavior that does not work yet, giving you evidence about how well a proposed product might perform before you commit to launching it.

0:190:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:19 · section reference included

Datasets and scorers are engineering work

A synthetic dataset and an off-the-shelf LLM judge do not automatically produce a useful evaluation system. Most datasets assembled before deployment fail to represent what users will actually experience. Goyal offers competition math as a narrow counterexample: some tasks have a relatively clear relationship between benchmark problems and the intended use. Most products need an ongoing process for reconciling their dataset with reality. That process—not just the initial collection of examples—is the engineering problem.

Scoring requires the same investment. Teams often ask which built-in scorers will let them avoid thinking about scoring. Braintrust provides AutoEvals, but Goyal emphasizes that its openness and flexibility are deliberate: the advanced customers he works with write their own scoring functions and continually modify them.

Scorers are the specification for the application. They express what counts as acceptable behavior, much as a product requirements document does. This makes custom scoring a product-design responsibility, not incidental test infrastructure. A generic scorer may be a useful starting point, but its definition of success can amount to the specification for somebody else’s project.

Slide titled “Scorers are the spec for your project,” with labels for hallucination, correctness, brand alignment, and conciseness.Open full source frame
Scorers are the specification for your project.
2:102:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:10 · section reference included

Tools construct the context the model sees

In an agent, the system prompt is only the beginning of the model’s context. The application repeatedly:

  1. Calls the model with the current context.
  2. Executes the tools the model requests.
  3. Incorporates the tool results into the next prompt.
  4. Repeats the process.

Each iteration adds material that the original system prompt did not contain. In the agent trajectories Goyal sampled, most prompt tokens came from outside the system prompt. Improving that prompt still matters, but it leaves much of the model’s input untouched.

Tool definitions and outputs therefore deserve the same care as instructions written directly to the model. An existing API reflects the way a product was built; it does not necessarily expose the operations or information an LLM needs. Good tool design starts with what the model should see and do. That can require substantial changes to the product, rather than simply placing an agent-facing wrapper over its existing APIs.

Output format is part of that design. In one internal analysis project, Goyal reports a significant improvement after changing a tool’s output from JSON to YAML. He attributes the difference to more efficient token use and easier reading by the model. This is a task-specific observation, not a general ranking of serialization formats.

The distinction is between text the model interprets and structured data that code consumes. Once equivalent JSON or YAML has been parsed into the data a JavaScript charting library expects, the original representation need not matter to the chart. Before parsing, the model encounters different textual structures. The tool’s interface and its returned text are both opportunities to make the next model call more effective.

4:054:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:05 · section reference included

Keep an eval for the feature that does not work yet

A new model can change which product architectures are worth building. Goyal credits people at Replit with helping pioneer the practice of preparing for that possibility: the product, team, and development habits should let a capability improvement become something users can actually use.

Braintrust had maintained an eval for a prospective feature and rerun it every few months. GPT-4o had once been the strongest available option; later runs improved with GPT-4.1, Claude 3.7 Sonnet, and then Claude Sonnet 4, called Claude 4 Sonnet in the talk. A result around 10% was not viable for this feature, Goyal says. The later improvement crossed the team’s threshold for shipping. These are results from Braintrust’s internal feature benchmark, not a general model ranking; the talk does not define its scoring rule, dataset size, or run settings.

Goyal reports shipping the feature’s first version two weeks after the Claude Sonnet 4 release. The standing eval made that response possible: the team already had a way to recognize that a previously impractical capability had become usable.

This is a reason to maintain ambitious evals even when current models perform poorly. Make the model replaceable in the evaluation setup, so trying a release is a small operation. Goyal describes Braintrust Proxy and similar tools as a way to switch providers without rewriting integration code.

A new release is an opportunity to measure, not a guarantee of progress. Goyal reports that Gemini 2.5 Pro 0520 scored 1% on this particular benchmark. He had not yet tested the newer Gemini version mentioned during the talk; the point was that the existing eval would make it easy to find out whether it helped.

6:386:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:38 · section reference included

Give the optimizer the data and scores

Optimizing an AI application means working on three connected components: the evaluation data, the task implementation—including prompts, the agent loop, and tools—and the scoring functions. A prompt can only be judged in relation to the examples it must handle and the criteria used to assess its output.

Braintrust compared two ways of asking an LLM to optimize prompts on the same benchmark. One supplied only the prompt. The other supplied the prompt, dataset, and scores, with a request to optimize the whole system. The slide reports:

Optimizer contextDisplayed benchmark result
Prompt Only8.9%
Dataset+Prompt+Scorers39.14%

The difference is the evidence available to the optimizer: it can inspect both the task and how performance is being assessed, rather than editing instructions in isolation.

Two bars beneath “Not only the prompt, but the data and scorers, too”: a gray Prompt Only bar labeled 8.9% and a taller green Dataset+Prompt+Scorers bar labeled 39.14%.Open full source frame
Prompt Only: 8.9%; Dataset+Prompt+Scorers: 39.14%.

Goyal describes the improvement as enough to move the feature from unviable to viable. The practical lesson is to inspect the entire evaluation system when improving an application. Better wording alone cannot resolve every mismatch among the task, the examples, and the definition of success.

9:099:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:09 · section reference included

Loop turns evaluation artifacts into working context

The feature behind these experiments is Loop, which Goyal introduces as launching during the talk. The launch-era activation path was to open Braintrust’s feature flag section and enable Loop. In the playground, users could provide a prompt, dataset, and scores and work with Loop to improve them. Loop could also create prompts, datasets, and scoring functions.

The requests extend beyond rewriting a prompt:

  • Prompt improvement: optimize the current instructions.
  • Dataset coverage: identify missing cases worth testing for the use case.
  • Score diagnosis: explain why results are unexpectedly low or high.
  • Stricter evaluation: help write a harsher scorer when the current one is too forgiving.

These requests make the evaluation artifacts themselves part of the model’s working context.

Goyal reports strong results with Claude Sonnet 4 and says Claude Opus 4 performed a couple of percentage points better in this evaluation context. He also encourages experimentation with o3, o4-mini, Gemini, and custom or fine-tuned models.

The proposed workflow change is that a model can inspect prompts and data and make constructive improvements automatically. Goyal expects this to reduce the manual labor of iterating through evals; the launch is an opportunity to test that workflow with users and learn from their feedback.

10:2110:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:21 · section reference included

Build the organization to act on the evidence

An organization may not yet have evals that support these decisions. Goyal’s closing emphasis is to assess that honestly and invest in the missing engineering: supplement synthetic or downloaded datasets with application evidence, write scorers that encode the intended behavior, and treat tool interfaces and outputs as deliberate communication with the model.

Readiness also has an architectural cost. Recognizing a meaningful improvement on release day is useful only if the team can act on it—even when that means replacing a substantial part of the system with an architecture that takes advantage of the new model. Whole-system optimization includes how the team obtains its data, not just the dataset currently on disk, alongside the task and scoring functions.

12:2112:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:21 · section reference included

Does adding user feedback cause overfitting?

The first audience question challenges the feedback loop: if complaints become evals, could the dataset overfit to those complaints? Goyal is more concerned about overfitting to a static dataset that excludes user feedback. What matters is not the dataset’s state at one moment, but the team’s ability to keep reconciling it with the reality the product should serve.

That does not mean automatically accepting every complaint as a test case. At the time of the talk, Braintrust deliberately did not add user feedback to datasets automatically. A person with domain intuition selected interesting examples and applied judgment: is this something the product should clearly support, and is its failure worth preserving in the evaluation set? Human curation connects feedback to intended behavior.

14:3614:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:36 · section reference included

What the token percentages measure

Another question returns to the earlier context chart. Its percentages describe the relative number of tokens by message type in traces from a few agents:

CategoryWhat it contains
System promptSystem instructions
Tool definitionsSpecifications of callable tools
User and assistantText exchanged in the conversation
Tool responsesContent generated by tools

These are shares of token composition, not accuracy figures or the frequency of tool calls.

In Goyal’s account, tool-related content dominates the token budget of these agentic systems. The sample supports paying close attention to tool definitions and returned content; it is not a universal estimate for all agents. Simply turning an existing GraphQL API into a collection of tool calls misses the opportunity to design the information the model will actually consume.

16:1016:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:10 · section reference included

An accurate answer can still receive a thumbs down

An audience member working with government services gives a concrete reason to curate feedback. Users sometimes give an accurate answer about taxes a thumbs down because they dislike what the answer means for them. The team added an explicit feedback option: “The answer is right, but I just don't like it.” Correctness and satisfaction are different signals; a negative reaction alone does not establish that the model made an error.

17:3317:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:33 · section reference included

Model breakthroughs depend on the task

The same audience member questions how often a new model really changes everything. Their team had seen limited differences across Claude and OpenAI upgrades, apart from an anecdotal instruction-following failure with GPT-4.1 mini. What kinds of tasks actually experience a large change?

Goyal returns to Loop’s demanding job: inspect prompts, datasets, and scores, then automatically optimize prompts using that evidence. Braintrust had run its standing benchmark across successive model launches, with results staying near the earlier GPT-4o level for a long time.

The chart revisited during the answer shows the following results for the internal Loop benchmark:

Model, as labeled on the slideDisplayed result
GPT 4o10%
GPT 4.146%
Claude 3.7 Sonnet51%
Claude 4 Sonnet58%

But Loop’s evaluation set consists of evals it attempts to optimize, and not every constituent task needs a breakthrough. Identifying the movie that a quote comes from, for example, had already worked well since GPT-3.5, according to Goyal.

Bar chart showing GPT 4o at 10%, GPT 4.1 at 46%, Claude 3.7 Sonnet at 51%, and Claude 4 Sonnet at 58%, with the final bar highlighted purple.Open full source frame
Loop performance across four models.

For mature tasks, a model upgrade may make little practical difference. For ambitious tasks, it can determine whether the product works at all. Keep those currently unsolved use cases expressed as runnable evals, so that when a new model arrives, discovering whether it changes the answer is a routine test rather than a new research project.

17:5718:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:57 · section reference included

Resources

From the talk

  • Python and TypeScript evaluation library with built-in metrics, customizable model judges, and examples of custom scoring functions.

Updates since the talk

Transcript download
Full transcript

Loading transcript…