AI Engineer World's Fair 2025

Why should anyone care about Evals?

Manu Goyal· Founding engineer, Braintrust5:41

Loading slides…

Read the talk

Why should anyone care about Evals?

A higher model score does not establish that an application works. Evals connect model changes to real tasks, support experiments before deployment, and turn production feedback into the next iteration.

From a talk by Manu Goyal

From Nintendo to adaptive software

A boy absorbed in a Nintendo 64 game looks like an unlikely starting point for a discussion of AI evaluation. Manu Goyal, an engineer at Braintrust, offers a comic reinterpretation of his childhood photograph: here is someone already disappointed by technology’s limitations. Rule-based systems repeat the same behavior; he wants technology that grows, adapts, and becomes a thought partner. The childhood career revelation is a joke, but the ambition sets up a practical question: how do we know whether adaptive software actually works?

0:240:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:24 · section reference included

A better classifier is not a road-ready car

In Goyal’s accelerated origin story, he drops the Nintendo, starts grinding LeetCode, and lands in the self-driving car industry. That experience supplies the central engineering problem. You can spend all day tuning a model, changing its architecture, or adjusting its loss function without acquiring enough evidence to ship. In his hypothetical example, improving image classification from 98% to 99% does not establish that a car is ready for the road.

A two-panel cartoon shows buttons labeled “blame the model” and “blame the prompt,” followed by a sweating man struggling to choose.Open full source frame
Blame the model or blame the prompt?

The missing evidence concerns the application, not just the model’s component task:

QuestionWhat it tests
Does it avoid pedestrians?Behavior around people
Does it negotiate traffic appropriately?Behavior in driving scenarios
Does it obey the law?Compliance with driving rules

Evals put model performance in the context of the real-world application. An aggregate classification score cannot answer all of these questions. Calling evals unit tests for AI captures only part of their purpose.

1:291:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:29 · section reference included

Build a laboratory before going to production

Regression detection is useful, but evals also provide a place to experiment. Without them, Goyal argues, getting feedback on a change means shipping it to production and observing what happens. That loop is expensive, slow, and risky: the real world becomes the testing environment.

Investing in good evals creates a laboratory where developers can repeatedly test changes before exposing users to them. Goyal estimates that good evals can move 90% of the product iteration loop before production. This is an advocacy estimate, with no measurement protocol supplied; the practical claim is that earlier feedback makes iteration faster and deployment more confident.

The laboratory should remain connected to production. Apply the same metrics used offline to online production data, and those measurements can identify which real examples will be most useful for the next iteration. Production feedback helps decide what to evaluate next. The loop therefore does more than check a release: it helps improve the evidence available for subsequent changes.

2:362:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:36 · section reference included

The enthusiasm around evals

Goyal closes his personal journey with a before-and-after transformation gag: understanding evals completes the transformation. He then turns to industry enthusiasm, naming Kevin Weil, Gary Tan, Mike Krieger, and Greg Brockman as advocates. His joking inference is that, with everyone endorsing evals, there must be something worth investigating—it cannot all be a scam. The appeal to recognizable names follows the concrete reason to care: a safer, faster way to learn whether changes help an application.

3:323:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:32 · section reference included

Connect experimentation, evaluation, and observation

At Braintrust, Goyal describes the goal as building a development platform around this loop. Evals sit alongside prompt tweaking and playground experiments, while logging supplies data and observability makes application behavior visible. Connecting those activities creates the data flywheel: experiments produce changes, observation supplies feedback, and that feedback informs further evaluation and experimentation. The accompanying slide groups the workflow into Play, Eval, Log, Review, and Monitor.

A slide lists Play, Eval, Log, Review, and Monitor beside a mint-green panel with the Braintrust symbol.Open full source frame
Braintrust’s workflow: Play, Eval, Log, Review, Monitor.

After joking that this has been a dense, content-heavy presentation, Goyal distills his pitch into a broad assertion: evals are the key to industry transformation and success. His concrete invitation is to continue the discussion in the Evals track, in Golden Gate Ballroom B.

4:184:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:18 · section reference included

Resources

  • Braintrust's original introduction connects evaluations, production examples, datasets and prompt experimentation.

Updates since the talk

Transcript download
Full transcript

Loading transcript…