AI Engineer World's Fair 2026
Production Evals For Agentic AI Systems
Loading slides…
Read the talk
Production Evals for Agentic AI Systems
Evaluating an agent means checking whether its workflow delivers dependable outcomes, from tool execution and failure recovery to the cost of keeping it running.
From a talk by Nishant Gupta
Before you start: Familiarity with LLM tool calls and basic production monitoring will help; no evaluation framework is required.
From benchmark scores to system behavior
A model scores 90% on a benchmark; its replacement scores 92%, and the team celebrates. Nishant Gupta opens with this hypothetical improvement to pose a harder question: what does that gain tell us about an agent operating in production? An agent does more than generate an answer. It plans, calls tools, retrieves information, executes workflows and interacts with production infrastructure. The evaluation question expands from whether the answer was right to whether the system behaved correctly.
The gap appears when improving offline scores coexist with unpredictable production reliability. Tool failures, API outages, changing context, different users and long-running workflows all affect execution. A capability benchmark does not capture that whole operating environment. As autonomy increases, more of the outcome depends on behavior outside the final response.
The workflow becomes the unit of evaluation. Planning quality, tool usage, execution, recovery and decision-making all matter. A correct-looking answer alone cannot establish that those steps worked correctly, so the evaluation architecture needs evidence about the path to the outcome as well as the outcome itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Failure layers and the SRE lens
Hallucination is only one category of production failure. Gupta organizes the broader problem into layers:
| Layer | Failure modes |
|---|---|
| Foundation | Memory, retrieval, safety |
| Reasoning and execution | Reasoning mistakes, poor planning, incorrect tool execution |
| Coordination | Multi-agent coordination failures |
Evaluating only the final output leaves these different sources of failure difficult to distinguish. A workflow can fail because information was unavailable, because the plan was poor or because execution did not carry out the plan. Those are different problems to diagnose and repair.
The next shift is operational: think like a site reliability engineer. Evaluate reliability, availability, latency, cost and recovery alongside correctness. The point is not to exclude accuracy, but to stop treating it as a sufficient definition of success. Dependable outcomes are the objective; accuracy is one input.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Benchmarks, scenarios and production evidence
Gupta's evaluation pyramid retains benchmarks at its base. They are useful and scalable, but their operational value is limited. Scenario evaluations occupy the middle because they simulate realistic workflows. Production telemetry sits at the top: in his framing, real users interacting with real systems provide the highest-value evaluation signals. These layers complement one another rather than making offline evaluation obsolete.
Offline evaluation therefore moves from isolated prompts to scenarios. A customer-support agent, code-generation agent or research agent operates inside a simulated environment. The evaluator measures task completion, tool correctness, planning quality and resource usage. This preserves the sequence of work and its dependencies instead of grading only the response at the end. Gupta describes resource usage at scale as exponential, but supplies neither measurements nor a scaling model. The actionable requirement is to measure resource consumption alongside success.
A minimal TypeScript record can make those separate judgments explicit for a customer-support scenario:
typescript
type ScenarioEvaluation = {
workflow: "customer-support";
taskCompleted: boolean;
toolChecks: Array<{
tool: string;
correct: boolean;
}>;
planning: {
acceptable: boolean;
rationale: string;
};
resources: {
toolCalls: number;
elapsedMs: number;
};
};
const exampleEvaluation: ScenarioEvaluation = {
workflow: "customer-support",
taskCompleted: false,
toolChecks: [
{ tool: "lookupOrder", correct: true },
{ tool: "issueRefund", correct: false },
],
planning: {
acceptable: true,
rationale: "Look up the order before attempting the refund.",
},
resources: {
toolCalls: 2,
elapsedMs: 1800,
},
};
In this illustrative record, an acceptable plan and a correct lookup coexist with an unsuccessful task and an incorrect refund operation. Keeping those judgments separate makes the evaluation more informative than a single answer score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Production signals need human judgment
After deployment, interactions become evaluation evidence. Execution traces show what happened; user outcomes show whether the work helped; escalations, failures and feedback reveal where the system struggled. Gupta characterizes production as an organization's largest and most representative evaluation dataset. The architectural implication is to collect these signals as part of operating the system, rather than treating traffic only as work to process.
Humans contribute more than a fallback when automation fails. They evaluate correctness, trust, usefulness and safety—judgments that automated metrics may miss. Targeted human review then calibrates the evaluation pipeline and exposes its blind spots. Automated evaluation supplies breadth; human review supplies additional judgment where the automated assessment is incomplete.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Detecting drift requires workflow traces
Agent behavior changes as model versions, prompts, tools and users change. Each individual update may appear modest while the combined system gradually becomes less reliable. Declining success rates, increasing escalations and rising tool failures are warning signals. Without continuous evaluation, user complaints may become the first indication that the system has drifted.
Diagnosing that decline requires visibility into recorded planning steps and available outputs, tool calls, memory access, execution timelines and state transitions. Gupta calls for visibility into reasoning paths; operationally, traces can capture the steps the system exposes, not guarantee access to hidden model reasoning.
Traditional logs alone are insufficient for reconstructing these workflows. Agent traces play the role of distributed tracing in a deeply nested microservice architecture: they connect operations into an execution path. The slide pairs a user prompt, planner iteration, vector database lookup and parallel API calls with operational measurements such as latency, retries, step costs and memory usage. That connection between workflow structure and operational evidence makes evaluation diagnosable rather than guesswork.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The evaluation loop continues after deployment
Evaluation becomes an always-running service rather than a phase that ends at deployment. The loop connects live evidence to the next round of offline validation:
- Telemetry identifies issues in production behavior.
- Humans review edge cases that need closer judgment.
- Feedback improves datasets, incorporating what the system encountered.
- Offline scenarios validate updates against those accumulated cases.
Production reveals problems that predeployment tests missed; improved scenarios help evaluate subsequent changes. Repeating that cycle makes evaluation an operational capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose metrics by the outcomes they represent
The operational metrics matter because they connect system behavior to business consequences:
| Evaluation measure | Business consequence |
|---|---|
| Task completion | Value delivered |
| Tool success | Operational reliability |
| Escalation rate | Human burden |
| Safety evaluations | Risk exposure |
| Latency | User experience |
| Cost | Scalability |
| Recovery rate | Resilience |
Accuracy is absent from this particular list because the list broadens the definition of success, not because correctness has stopped mattering. An agent must deliver useful work within acceptable operational constraints, and it must recover when execution goes wrong.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation becomes infrastructure
The architectural destination is a separation between doing the work and evaluating its behavior:
- Execution plane: Performs the agent's work.
- Control plane: Observes the system, collects telemetry, runs simulations, coordinates human review, and measures and governs behavior.
Gupta presents this separation as an emerging direction for production AI systems. Evaluation becomes part of the control plane rather than a separate tool used only offline. Its responsibilities persist while the execution plane serves users.
That architecture brings the earlier layers together. Benchmarks remain necessary, but scenarios evaluate whole workflows and production telemetry reveals how those workflows behave in use. Reliability, rather than raw model accuracy alone, determines whether the resulting system is dependable. The substantive change is permanence: evaluation is no longer a bounded testing or QA phase. It is infrastructure that must keep operating as the agent, its dependencies and its users change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Practical guidance on evaluating agent trajectories and outcomes, combining graders, maintaining regression suites and calibrating evaluation with human review.
Google's SRE chapter explains how to select user-centered reliability indicators, define objectives and interpret latency distributions.
Development-stage conventions for instrumenting agent invocations, workflows, planning and tool execution with OpenTelemetry spans.
Full transcript
Loading transcript…