AI Engineer World's Fair 2025

How to Run Evals at Scale: Thinking Beyond Accuracy or Similarity

Muktesh Mishra· Lead Engineer, Adobe9:25

Loading slides…

Read the talk

How to Run Evals at Scale: Beyond Accuracy and Similarity

Reliable evaluation starts with application-specific data and extends to code behavior, agent trajectories, tool calls, and a repeatable process for balancing speed with human judgment.

From a talk by Muktesh Mishra

Before you start: Familiarity with LLM applications, retrieval-augmented generation (RAG), and basic software testing will help you follow the examples.

Testing an application whose answers change

How do you test an AI application when the same input can produce different answers—and deciding whether an answer is good requires judgment? Muktesh Mishra approaches this problem from his work leading applied AI engineering for developer platforms at Adobe, alongside coauthoring CI/CD Design Patterns and contributing to open source. After invoking a Twitter snapshot as a sign of growing interest in evaluations, he turns to the practical difficulty shared by RAG applications, chatbots, and agents: ordinary expectations of repeatable output no longer suffice.

A prompt edit makes the problem immediate: “If I am changing a prompt, what is gonna break?” You need a way to detect regressions, not just inspect whether the new answer looks plausible. As model capabilities evolve, that same evaluation process must help you choose tools, metrics, and models suited to the application.

Slide titled “Have you seen these questions before?” with three colored boxes describing testing challenges, prompt changes, and evaluation choices.Open full source frame
Three evaluation questions: testing nondeterministic outputs, checking prompt changes, and choosing metrics, tools, and models.

Evals are test cases for measuring application behavior. Their purpose extends beyond assigning an accuracy score: they connect system outputs to business consequences and application goals, establish whether successive changes improve the product, and support the trust and accountability customers need. Without that measurement, an application can change without anyone knowing whether it has become better.

0:390:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:39 · section reference included

Start small, then refine the data

The first practical step is to assemble evaluation data. Start with a small synthetic dataset: artificial inputs against which you can validate application outputs. That gives you an initial testing surface before you have accumulated a mature collection of examples.

The dataset should then change as you observe the system. Generated outputs reveal where the existing examples are insufficient; those observations feed back into refinement. Label cases by the aspects of behavior and application flows they exercise, so the collection expresses what it covers rather than becoming an undifferentiated pile of prompts.

For example, a small support application could separate factual lookup from a clarification flow using a TypeScript data structure like this:

typescript

type EvalCase = {
  id: string;
  flow: "policy-lookup" | "clarification";
  input: string;
  expectedBehavior: string;
};

const cases: EvalCase[] = [
  {
    id: "return-policy",
    flow: "policy-lookup",
    input: "Where can I find the return policy?",
    expectedBehavior: "Point to the supplied return-policy source.",
  },
  {
    id: "unspecified-change",
    flow: "clarification",
    input: "Can I change it?",
    expectedBehavior: "Ask what the customer wants to change.",
  },
];

function casesForFlow(flow: EvalCase["flow"]): EvalCase[] {
  return cases.filter((testCase) => testCase.flow === flow);
}

The labels make it possible to select cases by flow; expectedBehavior records what a later evaluator must assess. Mishra’s experience is that one dataset is never sufficient: different applications, flows, and intended outcomes require different collections of cases.

3:183:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:18 · section reference included

Turn “evaluate everything” into explicit coverage

Wanting to evaluate everything is an ambition, not yet a testing strategy. Make that ambition concrete in three steps:

  1. Define goals and objectives. Specify which system behaviors you want to evaluate.
  2. Separate components into modules. Organize evaluation around those components and handle the data so different flows receive appropriate datasets.
  3. Test flows, outputs, and paths. When the application can follow several routes, cover those routes rather than checking only a single final response.

This connects the dataset design to the application’s structure: coverage means exercising the behaviors the system can actually perform.

4:505:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:50 · section reference included

Match the evaluation to the task

There is no universal evaluation. A typical RAG question-answering application and a code generator produce different kinds of artifacts, so they need different evidence of success.

ApplicationEvaluation focus
RAG question answeringAccuracy, similarity, usefulness
Code generationFunctional correctness, robustness

For question answering, the listed criteria examine qualities of the response. For code generation, Mishra calls for testing the generated code against the actual codebase. A plausible-looking implementation is insufficient: evaluation needs to establish whether it functions correctly and behaves robustly in that setting.

Agents add another dimension: the path taken through the task. Trajectory evaluation examines which route an agent follows while executing a flow, because agents can take different paths. Multi-turn simulation then extends the evaluation to conversations, where behavior unfolds across exchanges rather than within one answer. Tool use adds checks for call correctness, associated test suites, and the data being generated. These are separate evaluation targets that a final-answer score alone does not describe.

Adaptive Evals slide with four colored rows: RAG accuracy, similarity, usefulness and conciseness; code correctness, robustness, efficiency, quality and HITL; agent trajectories and multiturn simulation; tool-call correctness, test suites and Pass@K.Open full source frame
Adaptive Evals lists different evaluation criteria for RAG, code generation, agents, and tool calls.
5:335:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:33 · section reference included

Scale the execution and the feedback loop

Once the evaluation targets are clear, scaling becomes an execution problem. Mishra points to caching intermediate results, regression evaluation, and deliberate orchestration and parallelism. These are the operational concerns behind running a collection of evaluations repeatedly: deciding how work is coordinated, what can run concurrently, and which intermediate results can be reused.

Execution must lead to analysis. Aggregate the results, run evaluations frequently, and use the findings to improve the application. Mishra describes the cycle as “measure, monitor, analyze and repeat.” The metrics and methodologies still depend on the use case; running more evaluations does not remove the need to decide what is worth measuring.

6:577:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:57 · section reference included

Make evaluation a development process

There is also no fixed division between human judgment and automation that suits every application.

  • Human-in-the-loop evaluation: Give human judgment precedence where the task requires it.
  • Automated tests and evals: Use automation where repeatable, frequent assessment serves the goal.

Mishra frames this as a balance between speed and fidelity. The right balance depends on what the application is trying to achieve, rather than on a preference for automating every check.

Establish the process before relying on the tools. Not everything can be automated, so the evaluation strategy must define how assessments happen, including where human judgment belongs. A tool can participate in that process; it cannot substitute for deciding how the application should be evaluated.

This leads to eval-driven development, analogous to test-driven development in conventional software. Define evaluations around the use case, and include negative cases as well as positive ones: the dataset must cover situations where the application should not succeed in the ordinary way, not just examples of desired output. Data remains central as development continues. Measuring, monitoring, analyzing, and iterating become a recurring engineering practice, with the balance between fidelity and speed revisited as the application changes.

7:447:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:44 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…