AI Engineer World's Fair 2025

Evals Are Not Unit Tests

Ido Pesok· Vercel15:22

Loading slides…

Read the talk

Evals Are Not Unit Tests

A fruit-counting app shows why successful demos are not enough—and how representative data, shared production logic, simple scoring, and regular evaluation make improvements measurable.

From a talk by Ido Pesok

Before you start: Familiarity with prompts, API calls, and basic software testing is helpful; the code example uses TypeScript.

What would make a fruit-counting app reliable?

How do you know an AI application will work for its users, rather than just during your demo? At the time of this talk, Ido Pesok was an engineer at Vercel working on v0, a platform for generating full-stack web applications. Its recently launched GitHub Sync could push generated code, pull repository changes into a chat, switch branches, and open pull requests. Pesok reported that v0 had crossed 100 million messages. That product context motivates application-level evaluation: model-release benchmarks describe capabilities, but your application still needs evidence about its own users, inputs, and behavior.

Consider Pesok’s illustrative app, Fruit Letter Counter. The premise plays on the familiar question about how many Rs appear in strawberry. ChatGPT supplies a logo, v0 builds the UI and backend, and the generated backend uses the AI SDK’s streamText call. In the story, GPT-4.1 answers three correctly in two consecutive trials. The slide makes the temptation clear: it worked, then worked again, so perhaps it is ready to ship.

Slide titled “Worked first try” showing code that asks GPT-4.1 how many r’s are in strawberry, with an output comment of “3 letters.”Open full source frame
“Worked first try”: the strawberry prompt returns three letters.

The app launches on Vercel with Fluid Compute enabled. Then John reports that the same strawberry question returned two. The earlier successes did not establish dependable behavior. A polished demo can conceal failures that become visible only after deployment, and the problem applies well beyond letter counting: users need the application’s central function to work.

0:160:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

A better prompt still needs representative tests

The next move is prompt engineering: perhaps additional instructions or chain of thought will make counting more reliable. Pesok’s story escalates to an elaborate fruit-loving persona prompt. The revised prompt succeeds in ten consecutive ChatGPT trials. After another deployment, however, John asks for the Rs across strawberry, banana, pineapple, mango, kiwi, dragon fruit, apple, and raspberry. The app answers five and fails again. These trial counts are illustrative outcomes, not a controlled benchmark of the model.

The second failure exposes a different weakness in the testing approach. Repeating one familiar question does not cover the range of questions users will invent. A production user can expand the input in an entirely reasonable way and reach behavior the developer never tested.

Tweet-style slide from John Smith listing strawberry, banana, pineapple, mango, kiwi, dragonfruit, apple and raspberry, and reporting that the app counted five r’s.Open full source frame
A multi-fruit query exposes another counting failure.

Conventional tests remain useful. Individual functions can have unit tests; authentication, login, and sign-out can have end-to-end tests. Pesok illustrates the gap as 95% of the application working all the time while the crucial remaining 5% can fail. Those percentages are a rhetorical split, not measured reliability. Correct surrounding software does not establish the quality of the model behavior at its center.

3:253:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:25 · section reference included

Map the application onto a basketball court

A basketball court gives this uncertainty a useful shape. The basket is a glowing golden circle. Blue dots represent made shots; red dots represent misses. Distance from the basket stands for difficulty. The court also has boundaries: even a successful shot from outside the court does not count toward the game you are playing. For an application, those boundaries define the work it is supposed to do.

Place the strawberry question near the basket and color it blue after the prompt improvement. Place the longer fruit list farther away and color it red. These points become the evaluation dataset: collect the questions users ask, retain them over time, and build a picture of where the application succeeds and fails.

Now consider counting Rs in strawberry, pineapple, dragon fruit, and mango after replacing every vowel with R. That is difficult, but still within this app’s domain. Counting syllables in carrot is outside the boundary Pesok chooses for Fruit Letter Counter. Difficulty and relevance are separate dimensions: a hard case can deserve attention, while an easy success can be irrelevant.

The metaphor maps directly onto the components of an eval. Braintrust calls the system execution a task.

Eval componentBasketball equivalentApplication meaning
DataShot locationThe input being tested
TaskTaking the shotRunning the system on that input
ScoreMake or missChecking the resulting output

Before optimizing the shot, understand the court: what belongs in the application’s domain, which cases are difficult, and what success means.

4:595:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:59 · section reference included

Build coverage from actual usage

Two mistakes can make a growing eval suite misleading. One is collecting out-of-bounds cases that users do not care about. The other is concentrating points in a small part of the court. A large collection of similar, easy questions may reveal little about the application’s broader behavior. Cover the relevant domain, including its difficult regions.

Production evidence helps locate those regions:

  • Thumbs-up and thumbs-down feedback: Noisy, but useful for finding where users struggle.
  • Random log review: Observability lets you inspect interactions even when users leave no explicit feedback. Pesok suggests reviewing about 100 random samples once a week as a practical routine, not an established optimal sample size.
  • Community reports: Forums and X can surface failures users care enough to describe, though social feedback also contains noise.

These sources complement one another: explicit complaints identify pain, while random review helps reveal usage that never becomes a complaint.

There is no shortcut around understanding the data. A useful court has recognizable boundaries, coverage within them, and visible clusters of success and failure. If many users struggle in the bottom-right corner, that region can become next week’s engineering priority. The objective is to turn relevant failures from red to blue, rather than merely increase the number of eval cases.

7:207:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:20 · section reference included

Keep inputs stable and share the production logic

Put constants in the data and variables in the task. The user’s strawberry question stays fixed while the implementation changes. A system-prompt experiment should change the task, not require rewriting every dataset entry. The same separation applies when testing preprocessing or retrieval-augmented generation, or RAG.

Keep in the dataVary in the task
User’s questionSystem prompt
Stable test casePreprocessing
Same input across experimentsRetrieval behavior

This makes the cases reusable and keeps the experiment’s moving parts visible.

Text slide stating, “Just like in math or programming, factoring constants improves clarity, reuse, and generalizations.”Open full source frame
Factoring constants improves clarity, reuse and generalization.

AI SDK middleware provides a place to share prompting and preprocessing logic, including retrieval, between the production completion route and the evaluation task. The linked documentation is for AI SDK 4; it supports this middleware pattern without establishing the exact SDK version used in the talk.

Reuse that logic in both paths so an eval exercises substantially the same behavior that users receive. Basketball practice is useful because it resembles the game. Likewise, a model comparison becomes less informative if the evaluation task omits the retrieval or prompt preparation that production uses. Shared code reduces that mismatch.

9:159:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:15 · section reference included

Score the failure you can recognize

Scoring depends on the domain. A fruit counter has a concrete correct count; judging writing can be much harder. Where possible, prefer deterministic pass/fail scoring. Evaluation produces many outputs and logs, so an understandable score makes debugging easier. An elaborate scoring system can also make it harder for colleagues and other teams to understand or reuse the evals.

Start with the evidence you would inspect manually to recognize failure, then write a check for that evidence. For v0, one concern is whether the generated code works. For writing, the relevant signal might be a linguistic property. If code cannot capture the judgment reliably, human review is acceptable: the purpose is to collect trustworthy signal about the court, even when obtaining it takes work.

Pesok also suggests requesting the final answer inside answer tags to make extraction and string matching easier. This is a deliberate exception to keeping evaluation and production identical: the extra formatting may be unwanted in the user-facing response. For the same strawberry case, the following TypeScript example makes that extraction rule concrete. It uses a constructed tagged response, not a recorded model output.

typescript

const testCase = {
  input: "How many Rs are in strawberry?",
  expected: 3,
};

const formattingInstruction =
  "Output your final answer inside <answer>...</answer> tags.";

function scoreAnswer(output: string, expected: number): 0 | 1 {
  const match = /^\s*([0-9]+)\s*$/.exec(
    output.match(/<answer>(.*?)<\/answer>/s)?.[1] ?? "",
  );
  return match !== null && Number(match[1]) === expected ? 1 : 0;
}

const evaluationPrompt = `${testCase.input}\n${formattingInstruction}`;
const exampleOutput = "<answer>3</answer>";

console.log({
  evaluationPrompt,
  score: scoreAnswer(exampleOutput, testCase.expected),
});

The check isolates the answer field instead of searching all response text for a digit that might appear in an explanation. It scores the formatted answer; it does not establish how reliably the model will produce it.

10:4010:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:40 · section reference included

Review changes across the whole court

Adding evals to CI makes them part of reviewing a change. Braintrust can run the task across the dataset and produce a report of improvements and regressions. For a pull request that changes the prompt, the review becomes concrete:

  1. Run the proposed task against the evaluation data.
  2. Inspect cases that changed from failure to success.
  3. Inspect cases that previously passed but now fail.

A prompt may fix one region while breaking another. The report exposes both movements; Pesok does not prescribe a universal score threshold that must block deployment.

The same court supports other comparisons. Switching models resembles trying a different player in the same practice environment. Changing retrieval or the system prompt becomes an experiment whose effects can be shown to a colleague. Measurement gives a claim of improvement something firmer than a handful of convincing outputs.

Pesok connects this discipline to better reliability and quality, higher conversion and retention, and less support and operations work. These are anticipated product benefits, not measured business results presented in the talk. The court itself was built with a v0-generated app that let him place made and missed shots—a small tool for making the evaluation landscape visible.

12:3612:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:36 · section reference included

Scheduled runs and repeated trials answer different questions

The closing Q&A asks whether to run the same eval repeatedly. Pesok initially answers in terms of practice over time: even a generally accurate basketball player misses some shots. Pesok says his team runs evals at least daily to locate failures and detect regressions. A regular schedule helps reveal where the system is going wrong.

The questioner then clarifies a different idea: run the same question five times and distinguish four successes from five. That concerns variability for one input, rather than the schedule for checking the system. Pesok begins relating harder questions to shots farther from the basket, but the recording ends during his answer. The exchange leaves the repetition count and success-rate estimation method unresolved.

14:3614:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:36 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…