# From Agent Traces to Agent Simulations — Rustem Feyzkhanov, Snorkel AI

Rustem Feyzkhanov, Snorkel AI

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/Ib5t2RLtxvM-from-agent-traces-agent-simulations) · [Video](https://www.youtube.com/watch?v=Ib5t2RLtxvM)

## From Agent Traces to Repeatable Agent Simulations

Turn production failures into executable tasks that hold the environment steady while you test the agent’s model, tools, harness, cost and reliability.

Familiarity with tool-using agents, containers and integration tests will help you follow the implementation discussion.

### What would happen if you changed the agent?

[0:19](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=19s)

How do you know whether a changed agent will work with your company’s real tools, API services, policies and workflows? A useful benchmark must reproduce those conditions and keep growing as production exposes new cases. Rustem Feyzkhanov, who leads AI platform work at Snorkel AI, treats benchmark construction as an engineering discipline. Snorkel produces benchmarks using agents and subject matter experts; Feyzkhanov reports that it runs millions of agent simulations per month.

Start with a production trace: an input prompt, the actions the agent took and its final output. An evaluator can inspect that record to determine whether the agent succeeded or encountered an edge case. The opening slide makes this concrete with a billing-dispute trace and an observed failure. That record gives you evidence about what happened.

![Slide showing production users flowing into a live agent run and production traces, beside a billing-dispute trace and an observed failure.](https://ai.engineer/api/talk-essay-media/e8fdc4a63bc637a90e17df543115f07bde2368de115f9089d1226e1546bf18c5)

A trace is evidence, not an experiment.

Testing what would happen with a different configuration is harder. Production A/B tests encounter changing database contents and tool versions, so two agents may not face the same problem. **Offline simulation turns a trace into a repeatable experiment:** construct a task from the production interaction, then run different agent configurations against controlled conditions. Those experiments can run in parallel and compare success rate, cost, latency and retries.

### Benchmark the system you need to ship

[3:05](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=185s)

Public benchmarks help establish expectations about capabilities. [SWE-bench](https://www.swebench.com/) tests repository issue resolution; [Terminal-Bench](https://arxiv.org/abs/2601.11868) tests work in a terminal. Computer-use benchmarks cover another interaction surface. Your company’s benchmark must go further into its own use cases, tooling, policies and production environment. Public results help you form a prior; private evaluations help you decide whether to ship.

![Two-column table contrasting public benchmarks with company benchmarks covering codebase and CI, tools and workflows, policies, and a production-like sandbox.](https://ai.engineer/api/talk-essay-media/ea69ef16beb2f772d52105bb19a8434961b98094b0425cab1257a96ff4ca4d61)

Public benchmarks are priors, not verdicts.

Pass rate describes only part of the production tradeoff. A successful agent may still cost too much, respond too slowly or require too many retries. Simulation lets you compare those dimensions under the same conditions. It also lets you evaluate the entire agent stack, rather than treating model selection as the only variable.

| Change between experiments | Hold constant for comparison |
| --- | --- |
| Model and thinking level | Task environment |
| Prompt | Task environment |
| Harness and skills | Evaluators |
| Available tools | Evaluators |

The unit being evaluated is **the full system**. A model change, a different reasoning budget or a revised skill can each affect behavior; keeping the environment and evaluators fixed makes those changes easier to assess.

### Use the benchmark beyond the first release

[5:12](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=312s)

The same benchmark supports successive stages of agent development:

1. **Make it work.** Check edge cases, select a model and inspect traces to understand failures before the first release.
2. **Make it reliable to change.** Put the benchmark in the release path as an integration test, checking that modifications to the stack or harness do not introduce regressions.
3. **Optimize it.** Tune cost and latency, and use simulation traces for reinforcement learning.

Feyzkhanov cites a task-specific example in which fine-tuning a smaller **Qwen** model brought it to the performance of a larger Qwen model; he does not give model sizes or measured scores in the talk. The broader architectural point is that a simulation benchmark can serve three roles: evaluation suite, release integration test and source of training data.

### Package an executable task and a known solution

[6:24](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=384s)

A benchmark task has an execution sequence:

1. Give the agent an input prompt.
2. Let it interact with APIs, MCP tools, databases and files in the environment.
3. Collect its trace, the final environment state and any output artifacts.
4. Run verifiers over those results to produce success and quality metrics.

The output is therefore more than the agent’s last message. It includes the world the agent leaves behind.

A task also needs an **oracle solution**: a constructed solution that runs through the same sequence in place of the agent. Its purpose is to establish that the task can actually be solved. Without that check, an agent failure may be evidence of a broken task rather than a capability gap.

[Harbor](https://www.harborframework.com/docs/tasks), created by the Terminal-Bench team, gives this contract a file-based structure. The talk groups the files by what they expose:

| Part | Contents | Agent access |
| --- | --- | --- |
| Instructions | `instruction.md` | Visible |
| Environment | `Dockerfile`; Compose for multiple containers | Interacts with it |
| Reference solution | Oracle | Hidden |
| Evaluation | Verifiers | Hidden |
| Metadata | Task configuration | Separate task description |

The packaging is deliberately straightforward. Instructions, an environment and hidden evaluation code are enough to turn an interaction into an experiment that can be repeated.

### Build a small environment with production behavior

[8:23](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=503s)

The environment should behave like a miniature production system without requiring a full production deployment for every experiment. Its databases, API services, tools and files need to match the interactions the agent will encounter. An agent should not get an easier or recognizably artificial task simply because it is being evaluated.

A real user cannot be packaged into that environment. A **simulated user** can instead be an LLM with its own prompt and additional context describing the behavior and interaction it should reproduce. The user’s replies become part of the environment that the verifiers inspect after the run.

The implementation patterns resemble integration tests and development environments:

- **Database snapshots:** provide the relevant state instead of copying an entire production database into every run.
- **Sidecar containers:** place the agent in a main environment and expose API services, databases and MCP tools through other containers.
- **Mocked services:** reproduce the required service interactions without deploying every production service.
- **Intermediate task steps:** for work that spans hours, give each step its own prompt and verifiers. Stop the simulation early when a step fails instead of spending resources completing a trajectory that has already failed.

These choices bound the experiment while preserving the interactions that matter to the task.

### Verify the changed world, not just the answer

[10:21](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=621s)

Output checking is familiar from coding tasks: run tests against the code the agent produced. An agent simulation exposes more evidence. The database may have changed, APIs may have returned particular responses and a user may have supplied additional information. Verifiers should inspect the **final state, trace and artifacts**, choosing evidence appropriate to the task.

- **Deterministic checks** work well when correctness is directly testable, such as an exact output requirement or a tool call.
- **LLM, harness or agent judges** can assess trace quality and whether the agent planned appropriately.
- **Combined evaluation** uses both when the task contains mechanically checkable outcomes and behavior that requires judgment.
- **Subject matter experts** review selected traces and outputs where discrepancies warrant human attention, rather than reviewing every run.

### Test the benchmark before trusting its scores

[11:55](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=715s)

An executable task is not automatically a trustworthy release gate. An agent may recognize the simulation and exploit it to obtain a reward. A task may be too easy, or its verifiers so permissive that incorrect behavior passes. A defective verifier may reject every attempt. Success may also vary substantially across runs. These are defects to investigate during benchmark development, not signals to accept uncritically as model performance.

![Six cards list benchmark symptoms, causes, and fixes, including incorrect failures, universal passes, suspicious results, score variation, crashes, and agent disagreement.](https://ai.engineer/api/talk-essay-media/c813cc4cf608d4194f8f539c4e595a7c5dcb163c74cc40d987025b05bcd57d11)

Debug the benchmark before you debug the model.

**The benchmark is software and needs its own CI pipeline.** That starts with ordinary engineering checks: pinned dependencies, the correct base image and complete fixtures. For a single-container task, even a small shell check can catch missing inputs before an expensive simulation starts:

```bash
#!/usr/bin/env bash
set -euo pipefail

cd "${1:?Usage: check-task TASK_DIRECTORY}"

required_files=(
  instruction.md
  environment/Dockerfile
)

for path in "${required_files[@]}"; do
  if [[ ! -s "$path" ]]; then
    printf 'Missing or empty task file: %s\n' "$path" >&2
    exit 1
  fi
done
```

This illustrates a file-presence check using Harbor’s current documented directory layout. It is only the first layer: dependency, image and task-specific fixture checks still belong in the pipeline.

Behavioral validation comes next:

1. Run the oracle solution and require the verifiers to pass.
2. Run the verifiers without executing the solution and require them to fail, checking that the task does not pass in its initial state.
3. Run the agent several times to assess solvability and difficulty.
4. Tag the task as simple, medium or hard based on observed success frequency, then approve it for inclusion in the benchmark.

The positive and negative checks matter together: a verifier that accepts the oracle but also accepts untouched state is not testing the intended work.

### Change one thing, then fix the right layer

[13:57](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=837s)

With a validated benchmark, improvement becomes a controlled loop. Establish a baseline, inspect the failed cases, change one thing and rerun the experiment. Record the results with an experimentation tool such as Arize. After the targeted fix works, rerun the full evaluation before releasing to production.

This process gives you an alternative to continually adding prohibitions and urgent instructions to the prompt. Because simulation exercises the whole system, it can test a fix in the component responsible for the failure:

| Failure or requirement | Place for the fix |
| --- | --- |
| Context overload | Harness |
| Missing procedure | Skill |
| Required output schema | Structured output |

A longer prompt may change behavior, but it should not become the default storage location for every missing system capability.

### Connect production observation to release experiments

[15:09](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=909s)

Once the agent is deployed, two loops operate together. The **benchmark expansion loop** takes failures from production observability and turns them into additional tasks. The **release loop** runs simulations against the expanded benchmark or a new agent configuration, records the experiments and uses the results as a gate before deployment.

Observability and experimentation need to be connected because each supplies what the other lacks. Traces reveal edge cases that actually occurred in production. Simulations let you test what would happen under a changed configuration. Keeping that connection alive makes the benchmark part of agent operations rather than a dataset used once before launch.

### Keep unseen cases and calibrate the reviewers

[16:48](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1008s)

The audience discussion adds an important constraint to repeated experimentation: maintain a holdout that the agent has not seen during development. Asked about example counts and dataset splits, Feyzkhanov suggests an 80/20 training–validation split, depending on the use case. He does not prescribe a benchmark size. The central requirement is that the holdout remain unseen throughout experimentation, so it can provide a separate check on the resulting agent configuration.

Coverage requires more than collecting memorable failures. An audience question contrasts handpicked production runs with tasks created before an agent reaches production. Feyzkhanov’s guidance is to include both everyday workflows and edge cases: the happy paths users depend on, tool failures and database problems. The integration-test analogy applies again—successful routine behavior and failure handling both need coverage.

Task construction need not be entirely manual. In response to a follow-up about simulation versus handcrafting, Feyzkhanov points out that agents can already write much of the necessary code. The valuable foundation is to build the environment once and supply context that reproduces production interactions; automation can then help construct tasks around that foundation.

The final question returns to human involvement in verification. Snorkel uses subject matter experts at scale, but the goal is not to have experts review everything. Direct their attention to disagreements among agents and verifiers: a task appears solved but receives a failure judgment, or a trace is judged insufficiently good despite an apparently acceptable outcome. Experts can then tune the agent performing the review. **The evaluator itself becomes something to improve**, with human expertise concentrated where automated judgments diverge.

## Resources

- [Harbor task structure](https://www.harborframework.com/docs/tasks) — Task directory layout and instructions for constructing executable agent evaluations.
- [Terminal-Bench and Harbor task formats](https://www.harborframework.com/docs/migration) — Explains how Harbor's task format evolved from Terminal-Bench and maps their file structures.
- [SWE-bench](https://www.swebench.com/) — Official project site for evaluating software agents on repository issue-resolution tasks.
- [Terminal-Bench research paper](https://arxiv.org/abs/2601.11868) — Describes realistic terminal tasks with isolated environments, reference solutions and executable verification.
- [Train and run a FinQA financial agent](https://rllm-project.readthedocs.io/en/latest/projects/finqa/) — Practical setup for a financial reasoning agent trained with reinforcement learning, including tools, datasets, inference and training.

## Slides

### From agent traces to agent simulations

Rustem Feyzkhanov introduces how agent traces can become simulations for repeatable, production-like benchmarks.

[0:19](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=19s) [Title slide introducing repeatable, production-like benchmarks for AI agents, presented by Rustem Feyzkhanov.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1334b10b89a6a773fbab59177f4294b3b4e414968f6e427ade783df44a20276.webp)

Snorkel | The Frontier AI Data Lab

THE NEXT ERA OF AGENT EVALUATION

From agent traces to agent simulations

Building repeatable, production-like benchmarks for AI agents.

Rustem Feyzkhanov
AI Platform Engineering, Snorkel AI

AI Engineer
World’s Fair

### Three things to take away

Each company needs its own benchmark that mimics production tools and workflows. Production traces keep it current, while a CI pipeline gates every change before release.

[0:41](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=41s) [Three takeaways: build a company benchmark, make it production-like, and integrate it into the agent lifecycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/632f07bceef6fbcac4dde6157a29d1754cf78c173f5252ae001467857d69eb5f.webp)

THE TAKEAWAYS
Three things to take away

Every company needs its own benchmark
The only reliable way to evaluate, improve, and release your agents. Public benchmarks are priors, not verdicts on your workload.

It has to be as close to production as possible
Mimic production surfaces — tools, schemas, APIs, policies, and workflows — without standing up full production for every test.

It has to live in the agent lifecycle
Populated by production traces and gated by a CI pipeline — so every change has to pass it before it ships.

Snorkel | The Frontier AI Data Lab
© 2026 Snorkel AI
2

### Why Snorkel AI is giving this talk

Snorkel AI runs millions of agent simulations per month and applies its experience in environments, secure execution, and evaluation to building private benchmarks.

[1:19](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=79s) [Snorkel AI's benchmark capabilities and lessons from building simulations and evaluations at scale.](https://assets.ai.engineer/sites/aiecode2025/sha256/0879c475335c37c02639f167e360f60f6c69b3a1ec0c98558cd82214a239a9bb.webp)

WHY OFFLINE EVALS MATTER
Why Snorkel AI is giving this talk

Snorkel AI builds expert-authored datasets, simulation environments, and evaluation pipelines for frontier AI models and production agents. We treat benchmark construction as an engineering discipline.

What we build
Expert-authored datasets
Simulation environments
Evaluation pipelines
Multi-layer quality pipelines
Secure sandbox execution
Frontier-lab collaboration

Millions of agent simulations per month

What we've learned building benchmarks at scale
High-volume simulation runs
Secure sandbox execution
Expert-authored environments
Dense rewards + verifier design
Production-like task construction
Release-relevant evaluation

This talk shares the practical patterns we've learned building private benchmarks at scale.

Snorkel | The Frontier AI Data Lab
© 2026 Snorkel AI
| 3

### A trace is evidence, not an experiment

A billing trace exposes an agent attempting a refund when it should have escalated a dispute. Comparing variants is harder because user intent, database state, tools, context and time can drift between runs.

[1:50](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=110s) [A production trace illustrates a billing dispute failure and variables that drift between runs.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3df15e6906cda7346a27dc1a4676a7e781281cfc39a8043d5bdcda250dacbbb.webp)

WHY OFFLINE EVALS MATTER
A trace is evidence, not an experiment

Production users
↓
Live agent run
↓
Production traces

What drifts between runs
user intent
DB / API state
tool versions
context / history
wall-clock time

Example production trace
user  “I was charged twice for April. Fix it now.”

agent
  → billing_api.lookup_invoice(user_id)
  ← invoice status: disputed

agent
  → attempts refund workflow

final  “I’ve adjusted your bill.”

observed failure  should have escalated the dispute to a specialist

Great for finding real failures. Weak for comparing variants — it happened once, under one state of the world.

Snorkel  The Frontier AI Data Lab
© 2026 Snorkel AI
| 4

### Offline simulation turns traces into repeatable experiments

Production traces become representative tasks for an offline benchmark. Agent configurations run under the same conditions, allowing comparisons of success rate, cost per success, latency, retries and trace quality.

[2:37](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=157s) [Diagram showing production traces converted into representative tasks for comparing agent configurations in an offline simulation benchmark.](https://assets.ai.engineer/sites/aiecode2025/sha256/f181c24b1e21f21831c5fe6939ba2d1e4c7436aad4c099a6017b8b5ee084808c.webp)

WHY OFFLINE EVALS MATTER

Offline simulation turns traces into repeatable experiments

Production traces
observability · Arize AX

extract / cluster

Representative tasks
the failures worth fixing

Agent config A
model · prompt · tools · harness

Agent config B
model · prompt · tools · harness

Agent config C
model · prompt · tools · harness

Offline simulation benchmark
same task · same environment
same seed · same state · same tools

score

Comparable metrics
success rate
cost per success
latency
retries · trace quality

Offline and bounded — so the same task can run thousands of times in parallel, and every configuration is judged on the same footing.

Snorkel  The Frontier AI Data Lab
© 2026 Snorkel AI
5

### Public benchmarks are priors, not verdicts

Public benchmarks provide an initial orientation; company benchmarks test agents against the codebase, workflows, policies, and production-like sandbox that matter for a release.

[3:03](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=183s) [Table contrasts public benchmarks with company-specific codebases, workflows, policies, and production-like sandboxes.](https://assets.ai.engineer/sites/aiecode2025/sha256/c89b978bfcddcbc77fdc1f650486937930071032e1503f2d11e3ce6f435671a5.webp)

WHY OFFLINE EVALS MATTER

Public benchmarks are priors, not verdicts

Public benchmark
orientation · a prior

Company benchmark
release-relevant measurement

SWE-bench
Can it fix GitHub issues?
Your codebase + CI
Can it change your service safely?

WebArena / BrowseComp
Can it browse and research?
Your tools + workflows
Can it drive your CRM, DB, and APIs?

GPQA / MMLU
Can it answer hard questions?
Your policies
Can it follow your operating constraints?

Terminal-Bench
Can it solve terminal tasks?
Your production-like sandbox
Can it complete your end-to-end task?

Use public benchmarks to orient. Use your benchmark to ship — your release-relevant distribution, frozen into repeatable tasks.

Snorkel The Frontier AI Data Lab
© 2026 Snorkel AI
| 6

### What you actually need to compare

In this illustrative comparison, Agent C has the highest success rate at 88%, while Agent A costs less per success and has lower latency and fewer retries. Production comparisons need all four measures.

[3:51](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=231s) [Three agents compared by success rate, cost per success, latency, and retries.](https://assets.ai.engineer/sites/aiecode2025/sha256/737fb22c728f4e424f46d0879b63a4052db27c5161e150e2cd4b9e1361dcb6a2.webp)

WHY OFFLINE EVALS MATTER
What you actually need to compare

Illustrative example

Agent A — best $ / success
Success rate: 82%
Cost / success: $0.18
Latency: fast
Retries: low

Agent B — middle
Success rate: 79%
Cost / success: $0.26
Latency: medium
Retries: medium

Agent C — highest success
Success rate: 88%
Cost / success: $0.21
Latency: slow
Retries: high

A bigger model can be cheaper per success. Optimize successful outcome per dollar and per second.

Snorkel — The Frontier AI Data Lab
© 2026 Snorkel AI
| 7

### Benchmark the whole agent stack

Compare agent configurations by tuning the model, prompt, harness and tools while holding the environment and evaluators constant. A separate observer scores each run.

[4:32](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=272s) [Configurable agent components and runtime environment alongside an observer and evaluation layer.](https://assets.ai.engineer/sites/aiecode2025/sha256/22ed6bf77431d63dbeddb5ccbd2092d4c16d565b18d8aa97dce62e0ad9386b5a.webp)

WHY OFFLINE EVALS MATTER

Benchmark the whole agent stack

Agent behavior

Agent configuration — what you tune
Model + thinking level — configurable
Prompt / policy — configurable
Harness / context / memory — configurable
Skills / tools — configurable

Runtime environment
MCP servers / APIs / DBs
Environment data

Observer / evaluation layer
Verifiers
Scoring
Traces
Metrics

Observes and scores the run — it is not part of the stack.

We benchmark configurations, not models.

Snorkel — The Frontier AI Data Lab
© 2026 Snorkel AI
8

### The simulation flywheel

Simulation, measurement and diagnosis generate data to improve the agent, while harder benchmark tasks feed the next cycle. Uses progress from model selection and debugging to release gates, cost and latency tuning.

[5:10](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=310s) [A cycle from simulation through measurement, diagnosis, data generation, and agent improvement, with short-, mid-, and long-term uses.](https://assets.ai.engineer/sites/aiecode2025/sha256/2860c05b02e14e63d9c3e8bc1d656f9af6fe431a67d725602f764ca8c8c26a45.webp)

WHY OFFLINE EVALS MATTER
The simulation flywheel

Simulate → Measure → Diagnose → Generate data → Improve agent
harder benchmark tasks

Make it work
Short term
Model selection
Baseline pass rate
Trace debugging

Make it right
Mid term
Regression testing
Release gates
Prompt / tool / harness iteration

Make it fast
Long term
Cost tuning
Latency tuning
RL / post-training

Evaluation becomes a data engine, not just a scorecard.

Snorkel  The Frontier AI Data Lab
© 2026 Snorkel AI
9

### Anatomy of a benchmark task

A benchmark task needs to be runnable, solvable, and scorable. The section also introduces two pieces teams often forget: oracle solutions and metadata.

[6:23](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=383s) [Section introduction to runnable, solvable, and scorable benchmark tasks, including oracle solutions and metadata.](https://assets.ai.engineer/sites/aiecode2025/sha256/fc3e2205d85d520bdd0957ec1302affa96be280d516a387a16e2cef63e942c67.webp)

1
Anatomy of a benchmark task

The pieces that make a task runnable, solvable, and scorable — including the two pieces teams often forget: oracle solutions and metadata.

Snorkel  The Frontier AI Data Lab
© 2026 Snorkel AI
| 10

### What a simulation benchmark contains

An agent and an oracle run against separate copies of the same environment. The same verifiers score the agent’s trace and check that the oracle solution passes reliably.

[6:32](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=392s) [Parallel diagrams show an agent run producing verifier results and an oracle run validating the task and verifiers.](https://assets.ai.engineer/sites/aiecode2025/sha256/6af63265388f103c45b14a07cc2b1507fab4ba280e733de12ba3a468ac1386c7.webp)

ANATOMY OF A BENCHMARK TASK
What a simulation benchmark contains

Agent run evaluates candidate agent
Task spec
Agent
Environment copy A
files · DB · APIs · tools
Trace + final state + artifacts
read-only
Verifiers read env + trace
Reward · metrics · labels

Oracle validation validates task + env + verifiers
Task spec
Oracle solution
Environment copy B
same setup
Oracle trace + final state + artifacts
read-only
Same verifiers read env + trace
Must pass reliably

The agent is evaluated by verifiers. The oracle proves the task is solvable and the verifiers are correct.

Snorkel  The Frontier AI Data Lab
© 2026 Snorkel AI
| 11

### The environment is mini-production

The agent works with a simulated user and production-like services in a sandbox. Mocks preserve production contracts; a separate verifier-only zone hides the oracle and checks from the agent.

[8:31](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=511s) [Diagram showing an agent interacting with a simulated user, database, API service, files and tools, alongside a separate verifier-only zone.](https://assets.ai.engineer/sites/aiecode2025/sha256/a6463494e813f9fa112868b7c10e9fb25e6d09e00086a354e9baeaf38934f1c4.webp)

BENCHMARK TASK ENVIRONMENT
The environment is mini-production

AGENT-VISIBLE ZONE
SIMULATION SANDBOX · network disabled / controlled

Simulated user
Agent
Database
API service
Files / fixtures
MCP / tools

Mocks preserve production contracts, not production data

VERIFIER-ONLY ZONE
Oracle solution
Hidden checks
Tests

Walled off from the agent — this is what prevents leakage and cheating.

Small enough to run in parallel. Realistic enough to preserve production contracts.

Snorkel The Frontier AI Data Lab
© 2026 Snorkel AI
14

### Simulation environment patterns

Sidecar containers provide databases and API services, while local mocks preserve API contracts. For long-horizon tasks, multi-step simulations use separate prompts and verifiers and can stop early on failure.

[9:20](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=560s) [Six simulation patterns: multi-container, MCP servers, simulated user, skills, multi-step and production-like APIs.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b9977d11a6daa8418b0b154d9f492663a78e95f22daafcd01bea708d989026a.webp)

BENCHMARK TASK ENVIRONMENT
Simulation environment patterns

Multi-container
App + DB + API as separate services.

MCP servers
Tools over the same protocol as prod.

Simulated user
An interactive counterpart that adapts.

Skills
Reusable, packaged capabilities.

Multi-step
Shared state evolving over time.

Production-like APIs
Same contract, mocked locally.

Choose patterns based on what your production agent actually touches.

Snorkel — The Frontier AI Data Lab
© 2026 Snorkel AI
| 15

### Benchmark task verifiers

This section introduces verifiers that define success and score the agent’s path as well as its answer.

[10:20](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=620s) [Section 3 introduces benchmark task verifiers and scoring the path as well as the answer.](https://assets.ai.engineer/sites/aiecode2025/sha256/d2846a3ce84fec25d2f862c73aac98a0acd23098d42b3bf9e37d3ff2722d8c62.webp)

3
Benchmark task verifiers
Defining what success means — and scoring the path, not just the answer.

Snorkel The Frontier AI Data Lab
© 2026 Snorkel AI
| 16

### Don’t just grade outputs — simulate worlds

Instead of scoring one answer, a simulated world responds to the agent’s tool calls and records their consequences. A verifier can then inspect the final state, trace, and artifacts.

[10:24](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=624s) [Static grading is compared with a simulated world where a verifier reads final state, trace and artifacts.](https://assets.ai.engineer/sites/aiecode2025/sha256/e4ae2a05f2d4822a594d683e84442adf40780c9b1ec9d6f6411911b05f369018.webp)

BENCHMARK TASK VERIFIERS
Don’t just grade outputs — simulate worlds

Static grading
Agent → answer → Grader → score
One shot. A single score against a key.

World simulation
Agent → action / tool call → Simulated world
Agent ← observation / response ← Simulated world
state changes — consequences
world = DB state · API responses · user replies · files / artifacts · tool outputs
Verifier reads final state + trace + artifacts

Don't just grade the output. Simulate the world the agent operates in.

Snorkel — The Frontier AI Data Lab
© 2026 Snorkel AI
17

### Verifiers define what success means

Deterministic checks primarily assess outputs, tool calls and constraints; LLM judges assess traces, planning and recovery. Human experts can review selected cases, including discrepancies.

[11:00](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=660s) [Matrix comparing programmatic checks, LLM judging and human review across output, trace, tool calls, planning, recovery and constraints.](https://assets.ai.engineer/sites/aiecode2025/sha256/bbcd6731d4ae7be4bd64147e22fe3af2cb9650f4cccad0d86116b7cf1ac5cbc0.webp)

BENCHMARK TASK VERIFIERS

Verifiers define what success means

Signal | Programmatic / deterministic | LLM judge | Human / SME
Final output | Primary | Optional | Optional
Trace quality | — | Primary | Optional
Tool calls | Primary | Optional | —
Process / planning | — | Primary | Optional
Error recovery | — | Primary | Optional
Constraint following | Primary | Optional | —

Example · billing dispute: route to specialist · no refund tool call · policy passes · clear response

Snorkel The Frontier AI Data Lab
© 2026 Snorkel AI
| 18

### Benchmark creation pipeline and lifecycle

This section connects benchmark tasks to agent release and improvement through quality gates, CI and an eval-driven loop.

[11:53](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=713s) [Section 4 introduces benchmark creation, quality gates and the lifecycle connecting benchmarks to production.](https://assets.ai.engineer/sites/aiecode2025/sha256/1fafe6fa5bcf13e5b199f901300513c679a60176ec90b0323012fa9d3d58a41c.webp)

4

Benchmark creation pipeline and lifecycle

Treating tasks like software — quality gates, CI, an eval-driven loop — and the lifecycle that connects the benchmark to production.

Snorkel | The Frontier AI Data Lab
© 2026 Snorkel AI
| 19

### Debug the benchmark before you debug the model

Benchmark failures can come from brittle graders, leakage, noisy runs or infrastructure. Match each symptom to its cause before judging the model: fix the verifier, strengthen tasks or separate infrastructure errors from task failures.

[12:04](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=724s) [Six benchmark symptoms with their causes and suggested fixes.](https://assets.ai.engineer/sites/aiecode2025/sha256/d60573d5b1f16ce886ed4e4e800d15b36f94d2523829df4423b40f079568d762.webp)

BENCHMARK CREATION PIPELINE AND LIFECYCLE

Debug the benchmark before you debug the model

Symptom
Correct answer fails
Cause · brittle grader
Fix · normalize output or use a semantic check

Symptom
Everyone passes
Cause · silent saturation
Fix · add harder, adversarial tasks

Symptom
Result looks too good
Cause · leakage or a shortcut
Fix · trace review + isolate hidden verifier

Symptom
Score swings between runs
Cause · noise, not signal
Fix · pin seeds, average over runs

Symptom
Model blamed for a crash
Cause · infra failure miscounted
Fix · separate infra errors from task failures

Symptom
Agents disagree wildly
Cause · underspecified task
Fix · tighten the spec and acceptance criteria

Snorkel | The Frontier AI Data Lab
© 2026 Snorkel AI
| 20

### Treat benchmark tasks like software

A benchmark CI pipeline checks the environment, oracle, no-op, determinism and leakage before sampling agents and approving tasks. The oracle must pass, while a no-op must fail.

[13:04](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=784s) [Table of benchmark CI gates and the problems each catches.](https://assets.ai.engineer/sites/aiecode2025/sha256/8cc0b986e6550b03dbcc58ec8e5d93e7f8c18e6eca4b08ea4fdc8164fa4ec37d.webp)

BENCHMARK CREATION PIPELINE AND LIFECYCLE
Treat benchmark tasks like software

CI gate | What it catches
Build environment — reject | Missing deps · bad image · missing fixtures
Oracle passes — reject | Unsolvable task · wrong spec · broken env
No-op fails — reject | Leakage · trivial task · broken reward
Deterministic across runs — quarantine | Flaky tests · random data · time deps
Leakage / anti-cheat checks | Shortcut paths · exposed tests · cheating
Sample agent baselines | Too easy · too hard · no signal
Approve · store pass rates | Ready for release gating

Two gates do most of the work: the oracle must pass; a no-op must fail.

Snorkel The Frontier AI Data Lab
© 2026 Snorkel AI
21

### Build-time loop: every change becomes an experiment

After establishing a baseline, the team runs an eval, inspects failures, changes one component and records the rerun. Failures or unclear signals loop back; passes proceed to production release.

[13:55](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=835s) [Workflow from baseline and evaluation through experiment recording, with failures looping back and passes proceeding to release.](https://assets.ai.engineer/sites/aiecode2025/sha256/c7778dcb9c6c8aa784f5f8487b426d707ddf21f730529d8f3abb0d37da340d62.webp)

Build-time loop: every change becomes an experiment

BENCHMARK CREATION PIPELINE AND LIFECYCLE

Use simulation during development to compare agent-stack changes before they ship.

Establish baseline
current agent config

run →

Run eval set
repeatable benchmark

inspect →

Read failures
read traces · identify layer

change →

Change one thing
model · prompt · tool
skill · harness

rerun →

Arize AX
Record experiment
compare to baseline

fail · regression · unclear signal — back to Run eval set

pass → Production release

Failing experiments loop back into development. Passing experiments become release candidates.

Snorkel — The Frontier AI Data Lab
© 2026 Snorkel AI
22

### Fix failures with structure, not vibes

Match each failure to a fix in the agent stack: use the harness for context overload, a skill for missing procedures, and a structured output schema for shape issues.

[14:24](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=864s) [A prompt patch pile contrasted with a table assigning failure types to appropriate fixes.](https://assets.ai.engineer/sites/aiecode2025/sha256/1be896caa8780beb8df067ecad9201503bff63ac99b19ca04dc3fdf4a96ef066.webp)

BENCHMARK CREATION PIPELINE AND LIFECYCLE

Fix failures with structure, not vibes

Before — the prompt patch pile

Prompt
  + IMPORTANT: never...
  + NEVER output...
  + CRITICAL: always...
  + edge-case patch
  + another edge-case patch

refactor →

After — give every fix a home

Failure type | Fix home
Bad calculation | Tool / code path
Policy ambiguity | Policy check / decision rule
Output shape issue | Structured output schema
Context overload | Harness / context strategy
Brittle decision | Decision schema
Missing procedure | Skill

Don’t turn every eval failure into another prompt patch.

Snorkel The Frontier AI Data Lab
© 2026 Snorkel AI
| 23

### Closing the loop with observability + experiments

Production traces surface failures that expand the benchmark. A simulation runner tests cases and agent configurations; experiments compare the results to inform release decisions and promotion to production.

[15:08](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=908s) [Diagram connecting production traces, benchmark cases, simulation, experiments, and release decisions.](https://assets.ai.engineer/sites/aiecode2025/sha256/3d67940d7a857d9709d766fd2b8087a71a02939f8fe69c5f07fa801e2cec0e7b.webp)

BENCHMARK CREATION PIPELINE AND LIFECYCLE

Closing the loop with observability + experiments

Observability finds failures. Simulation makes them repeatable. Experiments make changes comparable.

Benchmark expansion loop

Agent in production
current config

traces

Observability
traces · sessions · spans
failures · cost · latency
Arize AX

failures

Benchmark cases
selected failures
task specs · labels · metadata

new cases

Simulation runner
sandboxed envs · verifiers
runs config variants

record

Experiments
runs · evaluator scores
metadata · artifacts
Arize AX

compare

Release decision
ship · hold · iterate

ship

Approved config
promoted to production

promote

Agent config loop

Agent config variants
model · prompt · tools · harness

Snorkel | The Frontier AI Data Lab
© 2026 Snorkel AI
| 24

### AgentOps requires simulation

Feyzkhanov recommends using production traces to find edge cases, then testing agent-stack changes against a benchmark built from the company’s own workload before release.

[16:07](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=967s) [Three takeaways on traces, workload-specific benchmarks, and testing agent-stack changes before release.](https://assets.ai.engineer/sites/aiecode2025/sha256/5a54d6f2eea55f120ee609a303fc53dd86b4e039c5fc88936827f8e0993180d0.webp)

Snorkel  The Frontier AI Data Lab

AgentOps requires simulation

1  Traces explain what happened; simulations test what would happen.
2  Every company needs a benchmark built from its own workload.
3  Every agent-stack change should pass the benchmark before it ships.

© 2026 Snorkel AI
25

### Thank you

Rustem Feyzkhanov invites questions about benchmarks at booth L-G12. The slide also offers a QR code to explore Snorkel’s research.

[16:28](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=988s) [Closing slide inviting attendees to visit the research team’s booth, with a QR code to explore research.](https://assets.ai.engineer/sites/aiecode2025/sha256/91e40f86ebc66f694134e658044948a53c6594f7f92e9fa689219b1ba7303074.webp)

Snorkel
The Frontier AI Data Lab

AI Engineer
World’s Fair

Thank you!

Continue the conversation with our research team. ↳ Booth L-G12

Rustem Feyzkhanov
AI Platform Engineering, Snorkel AI
@ryfeus

↳ Explore all research

© 2026 Snorkel AI
| 26

## Full transcript

[0:00](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=0s) [upbeat music] Okay. Yeah, thanks everyone for coming, and I know this is the last session before lunch, so thanks for staying here.

[0:19](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=19s) Let's make it smooth and with good vibes, just as Ned said, and thanks Ned for introduction and for inviting me. So yeah, my name is Rustem. I'm leading AI pod platform team, uh, at Snorkel.

[0:31](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=31s) And, uh, today, I want to tell you how to turn agent traces into agent simulations and why this, uh, un- becomes the next stage for agent evaluations.

[0:42](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=42s) So three main things that I want you to take away from my talk is, uh, every company needs a benchmark. It's the only way to reliably evaluate, release, and improve your agents.

[0:54](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=54s) It has to be as close to production as possible. Uh, it has to mimic your real tools, real API services, policies, and workflows.

[1:04](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=64s) And finally, it has to be part of your agentic life cycle. It's not a static benchmark. It's a constantly populated data set from your production traces.

[1:17](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=77s) So why is Snorkel AI giving this talk? We are a data as a service company, and we, uh, basically selling, uh, benchmarks and we're producing benchmarks at scale, and for us, benchmark construction is an engineering discipline.

[1:33](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=93s) We run millions of agent simulations per month, and, uh, we learned how to do, like, uh, en- environment build and scale, working using both agents, uh, and subject matter experts to build reliable benchmarks that are close to production and, uh, specific domains.

[1:52](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=112s) So a lot of the time when people say about agent evaluation, they, they're focused on traces, and traces are very useful. The usual traces, like you can see an example on the screen, it basically shows, okay, here is the input prompt, here are the actions that agent took, and here is the agent o- output.

[2:09](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=129s) And then evaluation can analyze it, eval, and say, "Okay, was agent successful or not? Was there any edge case?"

[2:16](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=136s) Uh, so it is useful to find failures in production, but it's hard to test different variants. You can run A/B testing, and that's one way of checking different agent configurations.

[2:28](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=148s) But it's hard to make sure that everything is repeatable because you will get different da- database state, different tool versions, and so on, so you never fully compare apples to apples.

[2:38](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=158s) Offline simulation turns traces into repeatable experiments. Now you take tr- production traces, you construct tasks, and then you can run simulation, uh, benchmark, um, with different agent configuration offline, and you can compare agents using different metrics, not just success rate, but cost, latency, and retries, and you can run those in parallel.

[3:05](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=185s) But you can ask, "Okay, but why do we need it? Like, we already have public benchmarks." The challenge with public benchmarks is that usually they are focused on a very specific domains.

[3:14](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=194s) For example, SWE-bench is focused on, like, uh, fi- fixing GitHub issues, Terminal-Bench will focus on agent running in Terminal, and COAW-bench will focus on computer use agent. In your case, you want your benchmark to be focused on your company's domain, both from perspective of use cases and in terms of tooling that your agent has, whe- whether it

[3:35](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=215s) follows the policies that your company uses, and whether you get full production environment.

[3:41](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=221s) Basically, public benchmark is useful to o- orient and build your prior, but your private benchmark is useful to ship.

[3:52](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=232s) And a lot of the time, public benchmarks, they're specifically focused on pass rate. Every time you see a new model release, you see performance, like pass rate on different benchmarks, which makes sense because it tells us, like, w- uh, like about the frontier, how good is the, like, new model.

[4:10](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=250s) But when you release agent to production, you actually care about more metrics. You care about cost of you're solving the task. You care about latency. You care about number of retries.

[4:20](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=260s) And by running, uh, evaluation offline within simulations, you can effectively compare apples to apples and, and iterate on agent.

[4:34](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=274s) In this case, you can test the full stack of your agent. You're not just checking, okay, is model, uh, one pe- uh, is one model performs, uh, better than the other model.

[4:45](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=285s) You check even like thinking to- like thinking level, you can change the prompt, you can tune the full harness and skills, tools available to agent. Because in your production, you don't care about the model, you care about the full system.

[5:00](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=300s) And here you can configure and test the full system while keeping environment and, uh, evaluators the same between runs.

[5:12](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=312s) And that raises a point about like what benchmark is in this case. Because first of all, you can use it to first release the agent. You can make sure that it works, you can handle edge cases well, you can select the optimal model, and you can debug the traces.

[5:27](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=327s) That's how you make sure that it works. Next, you can make it right. You can put it as a release gate for your agent and verify that a- any change to agent stack in- uh, didn't introduce regression suddenly, and iterate on harness.

[5:42](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=342s) And finally, you can optimize it. You can tune it for better cost or latency, or you can use the traces to even do RL training.

[5:52](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=352s) Uh, later I will say, uh, I will share a link to our website where we have example how we used simulation environments to fine-tune Qwen, small Qwen model to match performance of large Qwen model for specific tasks.

[6:05](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=365s) Here effectively that becomes like trifecta of use cases. Like for you, benchmark becomes por- part of, uh, agent evaluation, becomes part of, uh, integration test for agent for release, and it becomes also training set for agent to improve it.

[6:24](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=384s) So I hope I explained why you need benchmark. So now let's take a look, like how can you actually construct it at scale for your company? Like, what is the anatomy of the benchmark task?

[6:33](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=393s) If you take a step back, like what, what is the sequence of running the benchmark? It's straightforward. Agent gets input prompt, it interacts with environment trying to solve the task with APIs, MCP tools, database, files.

[6:48](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=408s) Then it produces the output. Trace, final state of the environment, and artifacts, basically output files. Then we run verifiers, and we can produce the metrics. How-- Was a- agent able to solve the task?

[7:03](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=423s) How well did it do, and so on. The important second part of the task is oracle. When oracle solution runs, it runs through the whole sequence, but just instead of running the real agent, it runs oracle and it-- and when we construct oracle ourselves to make sure that task is solvable in the first place.

[7:23](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=443s) Because if it's not solvable, agent won't be able to solve it. So oracle is important part of the task.

[7:30](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=450s) If you look at the anatomy of the, like, uh, of the benchmark task, how it looks in terms of files. We can look at o- uh, one of the most popular formats nowadays, Harbor format, which was done by the same team who, uh, uh, maintains Terminal-Bench.

[7:45](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=465s) In this case, basically, it's just three set of files. So basically fi- what agent sees and interacts with, instruction.md, classic markdown file. Environment, which you can see Dockerfile, could be Docker Compose in case you have multiple Docker containers.

[8:00](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=480s) You have something that agent doesn't see, which are-- which is oracle solution and verifiers. And finally, you have some metadata. The-- It may s- look very straightforward, yeah. I'm saying like, okay, simulation environment is just Dockerfile and bi- bunch of stuff, but it is useful because now you have repeatable way of running experiments in agent simulation.

[8:23](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=503s) So now let's dive deeper into two main parts of benchmark. First is environment. The main challenge with environment is that effectively it has to be mini production, but you don't want to run full production for every experiment.

[8:37](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=517s) So you want to make sure that, uh, y- your database, API service, tools, and files match production. Just as like previous speaker mentioned, uh, like pre-- uh, that you don't, you don't want your agent to know that it's running within simulation, so it has to be real.

[8:52](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=532s) One thing though, you cannot put real user in your simulation task. So you can simulate the user. In this case, that becomes effectively LLM with its own prompt, which with additional context that can mimic human behavior and interaction with your system.

[9:11](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=551s) So in the port system, all these things exist in the environment and verifiers just interact with the environment afterwards.

[9:18](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=558s) There are certain patterns to make sure that you can, uh, organize environment this way. And think about it as like how you construct integration tests. Basically, how you give effectively dev environment to your agent.

[9:31](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=571s) You don't run the full production database. You have a certain snapshot. You can run side, uh, containers sidecars in this case. So your, uh, agent runs in one main, uh, environment, but there are other containers which contain API services, databases, MCP tools, and so on available to your agent.

[9:49](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=589s) You don't need to have like full production API services, you can mock them. I already mentioned simulated users. Uh, and uh, one important piece here is multi-step. To handle long-horizon tasks, if your agent needs to handle tasks that span hours, you want to ensure that you have intermediate v- like steps, and for each step you have separate

[10:08](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=608s) prompt, separate verifiers, and, uh, you can, uh, finish simulation early if you see agent failing. And basically, that enables you to simulate long-running horizon tasks.

[10:21](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=621s) So next part is verifiers. Like in traditional sense, usually when people speak about verifiers, you just verify the output. You get agent output, you verify it, that's it. That's why, let's say, how coding works, like we just verify the output code with tests and so on.

[10:39](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=639s) Here it's more complex. The way agent interacts with the simulation, we get a lot of different data. We get the world, basically how environment changed and the final environment state.

[10:49](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=649s) What is your database state? What are the API responses? What are user replies? And so on. And your verifier analyzes final state, trace, and artifacts. So how can you analyze it?

[11:02](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=662s) Effectively, there are multiple ways to do it. You can have deterministic checks, basically. Uh, and that can work really well for things like final output or tool calls where like it's very easy to check whether it was correct or not.

[11:13](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=673s) It-- Sometimes you can use LLM as a judge or even harness as a judge or agent as a judge to evaluate basically whether the, uh, trace quality was successful, whether, um, planning of the agent was correct, and so on.

[11:29](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=689s) In this case, it really depends on the use case. So you can use one or another or both, depending on what's, what works better. And finally, it's important to keep in mind that you can use sub- subject matter expert to review some of the traces, some of the outputs.

[11:44](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=704s) Not for everything, but for cases where you see discrepancy in agent behavior and where you want human involvement.

[11:55](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=715s) So final piece, how can we kind of organize everything together as part of agent release and agent improvement process?

[12:06](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=726s) So can we just start? Not just yet. Can something go wrong with benchmark task? Hundred percent. Agent can try to reward hack simulation environment because it can understand that it's in simulation and it can hack it.

[12:19](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=739s) Task could be too simple and, like, our verifiers could be too broad, and in this case, agent will always pass, even if it does something incorrectly. It could be that agent always fail because verifiers are incorrect.

[12:30](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=750s) Or it could be that agents not perform in a stable way and, like, you have high variation of agent success.

[12:38](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=758s) So all of these are effectively edge cases that you need to catch during your benchmark development. Because benchmark development is an art on its own. We saw already, like, hundreds of benchmarks appear over the course of last years.

[12:52](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=772s) But this is something that the car- culture and engineering discipline that needs to be built in each, uh, engineering team that needs to ship AI agents to production.

[13:06](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=786s) Because as you saw, effectively benchmark is software. It's code, it's files. You need to treat it as such. You need to have a separate CI pipeline for it. It-- And you can check pretty obvious things like, for example, making sure that all dependencies are pinned or, like, your base image is correct, or you don't have any missing

[13:27](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=807s) fixtures. Then you can run oracle, uh, uh, solution and making sure that it passes. Or, like, if you don't run oracle, that verifiers fail. You can run several agent runs, uh, on the task and verify that, okay, it is solvable and it is hard for agent.

[13:43](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=823s) You can tag the task whether it's simple, medium, or hard, depending on how much time-- uh, how many times the agent is successful. And finally, you can approve it to make it part of your benchmark.

[13:57](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=837s) In this case, the process for improving agent becomes straightforward. You establish baseline in, uh, like, on your benchmark, you run evaluation dataset, you see the failures where it doesn't perform well, you change one thing, you rerun experiment and where you can use, uh, something like Arize to record your experiments, and then, uh, once you fix it, you

[14:18](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=858s) rerun the full experimentation again. And then you can finally release to production.

[14:24](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=864s) What it unlocks is to make sure that you fix issues correctly. There is a bit of anti-pattern in the industry where, like, folks try to fix things in the prompt, and they populate the prompt with things like, "Never do this," or, "Only do that," "Never output this," "Critical that," which is one way of handling it for sure.

[14:45](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=885s) But with simulation, you control the full stack, you can evaluate the full stack, and you can make sure that fix lives in the correct, uh, place. You don't push everything to the prompt.

[14:54](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=894s) You fix harness if you want-- uh, if, if you have context overload. Or you can-- If there is a missing procedure, you put it into skill. Or, like, if you need to have specific output schema, you put it as part of structured output.

[15:09](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=909s) So once agent is in production, effectively you will have two loops. One loop for, uh, benchmark expansion.

[15:19](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=919s) You take, uh, observability traces, uh, traces from observability, basically using something like Arize. You record failures and use failures to build your benchmark further.

[15:30](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=930s) Then you have simulation runner that will run experiments on new be-- on extended benchmark on a ne- or on new agent config, uh, will record these experiments. Then you can use it f- as a release gate, whether, uh, agent performs significantly well, and finally release it to production.

[15:48](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=948s) Uh, in this case, it's important that you have system, like it's important to underst- uh, to make sure that your observability piece and experimentation piece are connected

[16:00](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=960s) because they're part-- they're two sides of the same coin.

[16:07](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=967s) Um, so yeah, to summarize my talk, everyone needs-- every company needs a benchmark. Traces are useful for, like, the finding edge case in production, but simulation helps to, uh, test what would happen.

[16:21](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=981s) And finally, uh, you want, uh, your, um, benchmark to be part of your, uh, agent ops, uh, loop.

[16:29](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=989s) Thank you very much for, uh, for your time and, uh, I think maybe I will have answer for one question, but will be happy to answer any questions outside and please check our booth, uh, if you have any questions about benchmarks. [audience applauding]

[16:48](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1008s) All right. A couple questions. All right. Hi, great talk. I have a question about structuring the benchmark, how many examples you should include ideally, and do you split them into, uh, like a train test split?

[17:04](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1024s) If so, like how do you recommend structuring examples, um, across those two splits?

[17:11](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1031s) Great question. So, uh, the question was basically how can we structure benchmark for training and validation? And just as previous speakers shared, basically, they also had a pa-pattern of like having train val split.

[17:22](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1042s) This is very close to traditional machine learning, where we want to have standalone dataset that agent didn't see, wh- basically, where we can verify agent config. I think the classic approach applies when, like, you want to have eighty, twenty percent.

[17:35](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1055s) Always depends on the use case, but, uh, you do want to have a holdout dataset that agent didn't see through the experimentation process.

[17:45](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1065s) Awesome. Next question.

[17:47](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1067s) Um, so when you create the benchmarks, um, what data do you include in the benchmark? Do you include production runs that you've handpicked into the benchmark? Or if your agent is not yet in production, do you create, uh, datasets or problems to solve for the agent in the benchmark?

[18:08](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1088s) And if you're, if you're like handpicking problems for the agent to solve, um, how do you make sure you have like- Enough coverage, uh, similar to what you'd see in production

[18:19](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1099s) Gr- great question. So the question is how to ensure c- coverage, and distribution of benchmark is a very important piece. So effectively, you want to keep, keep-- make sure, like, you have both.

[18:28](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1108s) You have bread and butter use cases, basically covering all main use cases that work, but also making sure you have edge cases. Basically, how you can make sure, like, your agent can handle edge case when tools fail, if, like, there is problem with database and so on.

[18:41](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1121s) So you want to have both, um, e- basically think about this integration tests. You, you have happy path, but you also have edge cases.

[18:49](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1129s) Would you simulate it with LLMs or would you create handcrafted LLMs?

[18:54](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1134s) Yeah, uh, great question. Basically, simulate versus handcrafting. Uh, I think basically... I mean, people use agents to write code, so a lot of things here can be automated. So the-- what is handcrafted kind of changes.

[19:05](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1145s) Like, the, the most important piece that you can provide is, like, the build environment once and then build the context, uh, for basically that mimics your production.

[19:15](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1155s) Um, you mentioned, um, sometimes you use, um, LLM or sometimes you use a human expert to build verifiers. Um, can you elaborate on that? What's the best practice over there?

[19:28](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1168s) Great question. So in our, uh, case, basically we have a lot of subject matter experts, so we do things at scale. So, uh, the important piece here is to spec- you don't need subject matter experts to review everything, but you specifically want to find cases where there is disagreement between, uh, a agent, uh, and different verifiers.

[19:49](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1189s) If you, you think that basically task was supposed to be solved, but agent somehow marks it as un- like, not solved correctly, or, like, it marks that trace wasn't optimal enough.

[20:00](https://www.youtube.com/watch?v=Ib5t2RLtxvM&t=1200s) And this is where you want to have subject matter expert that can basically tune the agent that does the review. [outro jingle]
