AI Engineer World's Fair 2025
From Self-driving to Autonomous Voice Agents — Brooke Hopkins, Coval
Loading slides…
Read the talk
From Self-driving to Autonomous Voice Agents
Voice agents need more than a convincing demo: they need simulations, reusable metrics, and continuous evaluation that preserve autonomy while exposing failures.
From a talk by Brooke Hopkins
Before you start: Familiarity with LLM agents, tool calls, and basic software testing will help; no self-driving background is required.
Production trust without sacrificing autonomy
Why aren't voice agents everywhere? They promise to automate difficult workflows, yet trusting them with customer interactions remains a barrier. Brooke Hopkins, founder of Coval, approaches that problem through her experience leading Waymo's evaluation job infrastructure team. The mismatch starts with scope: teams overestimate agents when they attempt to automate every call or workflow at once, while underestimating the useful experiences they could already build around a smaller problem.
An agent can handle 10 conversations convincingly and still struggle across 10,000 or 100,000; Hopkins uses these counts to illustrate scale, not to report benchmark results. That gap leaves enterprises reluctant to move beyond proofs of concept, internal tools, or tightly controlled workflows.
Two familiar approaches appear to offer a tradeoff:
- Constrain the conversation. Force the agent down predetermined paths to make its behavior predictable. Taken far enough, this becomes an expensive interactive voice response tree with an LLM inside it.
- Allow autonomy. Let the agent adapt to unfamiliar situations, accepting that its interactions and actions become harder to predict.
Reliability and autonomy need not be mutually exclusive. Waymo provides Hopkins's motivating comparison: a system can respond to unfamiliar interactions while delivering a dependable experience. The engineering question is how to build confidence in that flexibility.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From road observations to aggregate simulation
Hopkins traces the self-driving evaluation process from manual observation toward large-scale simulation. Initially, teams drove cars through streets, noted failures, and brought those observations back to engineers. That produced useful evidence but was difficult to scale. Explicit scenario tests came next: construct a particular situation and specify what the vehicle should do. Those tests were more repeatable, but complicated scenarios and exact expectations were expensive to maintain and could quickly become obsolete.
The next step broadened the question from what must happen in one scripted instance to how often a type of event happens across many simulations. Aggregate evaluation characterizes behavior across varied runs instead of making every test depend on an exact sequence. This is the emphasis Hopkins brings from self-driving to voice; focused tests still have a role within that larger evaluation strategy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Conversations need a responsive environment
Driving and conversation share a crucial property: each action affects what happens next. A car must respond to its environment, which continues changing as it moves. A conversational agent must respond to a person whose next answer depends on what the agent just said. Asking for a name produces a different continuation from asking for an email address. A useful simulation therefore needs an environment that responds to the agent's actual behavior.
A test that prescribes every conversational turn becomes fragile when the agent changes its wording or asks questions in a different order. Simulated users can instead generate varied responses as the conversation unfolds. Here, LLM nondeterminism becomes useful: it supplies variation across repeated interactions, helping explore possible continuations without writing each one by hand. The objective is broad coverage, not a claim that a finite simulation suite exhausts everything a caller might do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure behavior without prescribing every answer
Repeated simulations let a team estimate how often its agent succeeds. That shifts evaluation beyond the familiar pattern of running fixed inputs through a prompt and comparing outputs against a golden dataset. In a conversation, enumerating every acceptable response to every possible input becomes cumbersome. Reference-free evaluation assesses behavior without requiring a reference answer for each turn.
| Evaluation approach | What it checks |
|---|---|
| Input-output evaluation | Correctness for specified inputs |
| Scripted scenario assertions | Expected events in a particular scenario |
| Reference-free conversational metrics | Behaviors across varied conversations |
Reusable criteria might ask whether the agent resolved the inquiry, repeated itself, or said something prohibited. These criteria still define what good behavior means; they simply avoid prescribing every intermediate response. Metrics that apply across many scenarios provide the basis for a continuous evaluation loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A local fix must survive broader regression tests
Without a repeatable process, maintaining a deployed voice agent can turn into an ongoing professional service: every enterprise needs another adjustment. Hopkins estimates that enterprise-specific tweaks can consume 80% of a team's time after initial setup when processes are inadequate; she supplies no study conditions for that estimate. The alternative is to make each fix pass through increasingly broad checks.
Consider a car that fails to stop at a stop sign. An engineer reproduces the problem, changes the system, and confirms that the car now stops. The focused test passes—but perhaps the change made the car stop every ten seconds. A larger regression suite reveals that the apparent fix damaged ordinary driving. Passing the reproducer establishes that the local symptom changed, not that the whole system improved.
The evaluation layers then extend through delivery and operation:
- Focused feature evaluations reproduce the bug and check the fix.
- Larger regression sets check for damage elsewhere.
- Pre-submit and post-submit CI/CD checks keep verification connected to code changes and deployment.
- Large-scale release evaluations assess readiness using automated tests, manual review, or both.
- Live monitoring and detection find production failures and feed them back into development.
Each layer answers a broader question than the one before it. Monitoring closes the loop by supplying new cases for future fixes and regression tests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build metrics by reading conversations
The goal is not to automate every evaluation. Automated checks provide speed and scale; human reviewers handle judgments that need a human perspective. For an appointment-booking agent, begin with a happy path such as booking an appointment for tomorrow, then simulate that request repeatedly.
Read the resulting conversations before deciding what to measure. Identify how they fail, turn those failure modes into automated metrics, and iterate through simulation and evaluation. Once the agent is ready to deploy, run the evaluations again on production conversations. Flag cases for human review and return the findings to the simulation suite. The test collection grows from observed behavior rather than remaining a fixed checklist written before anyone used the product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match simulation realism to the question
Does a simulated caller need to sound exactly like a real customer? That depends on the component being tested. As in an experiment, the task is to control variables and expose the behavior of interest. A driving simulation that looks like real video is not automatically the most useful test environment. For some decisions, the system only needs inputs identifying a dog, a cat, or a person crossing the street. Controlling those relevant inputs matters more than rendering every detail.
| Simulation level | Suitable questions |
|---|---|
| Text | Workflows, tool calls, instruction following |
| Simple voice | Interruptions, latency, instructed pauses |
| Realistic audio conditions | Accents, background noise, audio quality |
For workflow iteration, text is often the fastest and cheapest option. Basic voices can exercise timing and turn-taking behavior without recreating every caller's accent or acoustic environment. More realistic audio becomes necessary when those properties are the suspected cause of a production issue. Keep end-to-end voice tests, but avoid paying for full realism in every component test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Denoise a failure by rerunning its scenario
One failed run tells you that a failure is possible. It does not tell you how frequently it occurs. Hopkins calls the next step denoising: take the failed scenario and simulate it repeatedly to estimate its failure frequency. Her example uses 100 reruns of the same scenario.
| Hypothetical failures in 100 reruns | Interpretation |
|---|---|
| 50 | Roughly coin-flip behavior |
| 99 | Near-consistent observed failure |
| 1 | A rarer observed failure |
These are illustrative outcomes, not experimental results. A 99-in-100 result does not prove that failure is certain, and a 1-in-100 result is acceptable only if the application's requirements permit it.
The bookkeeping can stay simple. This TypeScript function summarizes already-scored reruns of one scenario:
typescript
type RunResult = {
scenarioId: string;
failed: boolean;
};
function summarizeScenario(runs: readonly RunResult[]) {
const first = runs[0];
if (!first) throw new Error("At least one rerun is required");
if (runs.some(run => run.scenarioId !== first.scenarioId)) {
throw new Error("Summarize one scenario at a time");
}
const failures = runs.filter(run => run.failed).length;
return {
scenarioId: first.scenarioId,
runs: runs.length,
failures,
observedFailureRate: failures / runs.length,
};
}
The output is an observed fraction, not a guaranteed underlying probability. Hopkins invokes cloud infrastructure's six-nines reliability as a way to ask what level of reliability each part of a voice product needs—not as a result established by these reruns or a universal target for voice agents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose metrics from the product's purpose
Evaluation design is product design. Choosing metrics means deciding what the agent must do well. General voice systems, including those available through OpenAI APIs, already cover broad conversational capabilities. A vertical agent needs more specific priorities: which tasks justify its existence, and which capabilities matter less?
- Refund support: A caller who wants a refund may tolerate some delay to obtain the outcome.
- Outbound sales: Latency can be critical because the recipient is ready to hang up.
- Appointment booking: Instruction following and complete information collection are essential; a booking without the required details is useless.
- Interviewing or therapy: Conversational workflows may deserve more emphasis than rigid task completion.
Interruptions deserve evaluation alongside latency. The relative importance of each behavior comes from the interaction the product is supposed to support.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Calibrate automated judges with human feedback
LLM-as-a-judge makes customer-specific evaluation criteria practical, but flexibility brings noise. Asking only “Was this conversation successful?” leaves success underspecified. Hopkins illustrates the problem by suggesting that judging the same conversation ten times may produce inconsistent answers. Before trusting an automated metric, make its judgments align with what human reviewers actually mean by success.
Hopkins presents Coval's Metric Studio as a way to iterate on automated metrics using human feedback. Review conversations, supply judgments, and refine the metric until its evaluations align with those judgments. She illustrates scaling from 10 or 100 human-labeled conversations to automated evaluation of 10,000 conversations; those counts are not a guarantee that a small labeled sample is sufficient.
The calibration effort should reflect the reliability the product needs. A team may begin with a small number of labeled conversations, while another application warrants much more work to establish agreement. The essential step is to set the desired reliability deliberately and use human feedback to improve the metric before relying on it at scale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Benchmark components, then test the full system
Build the evaluation strategy in stages:
- Review public benchmarks. Use them for rough directional guidance about promising approaches.
- Benchmark on your own data. A medical application should test transcription methods on the medical terminology callers will actually use.
- Run task evaluations. Exercise individual modules or workflows, using text where appropriate. As with a car, testing one component does not require enabling every component.
- Run end-to-end evaluations. Exercise the complete system at scale under conditions resembling production.
This progression keeps early tests focused while still requiring the integrated system to demonstrate that its parts work together.
Hopkins describes Coval's work on continuous model benchmarking, then emphasizes custom comparisons. A voice that suits one kind of conversation may not suit another; an LLM that performs well on one task may be a poor fit for a different workflow. Benchmark the components of the voice stack against their intended use, then establish task-level baseline performance. That baseline makes the next development question concrete: where does the application work, where does it fail, and which component or workflow needs attention?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give production failures an owner and a test set
An evaluation process must specify what happens after a customer reports a production bug. Decide who owns the investigation, how the failure becomes a test case, and which regression sets should include it. Organize those sets around meaningful boundaries: important customers, customer types, workflows, or individual features. This gives a discovered failure a route back into routine verification.
Dashboards and recurring review processes make that structure useful over time. A successful customer pilot answers a question about the agent during that pilot. Continuous monitoring and evaluation answer the ongoing question of whether it still works as the product and its operating conditions change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Voice as an expected enterprise interface
Hopkins's closing platform thesis compares voice with the web and mobile. Each shift changes where companies can meet users and which interactions users expect to complete there. Natural voice interfaces could bring services into additional moments of daily life, though that does not make voice the right interface for every task.
She predicts that every enterprise will launch a voice experience within three years of the talk. Her analogy is an airline's mobile app: a good voice experience could become a baseline expectation rather than a novelty. That remains a forecast, but it explains the emphasis on sustained reliability. As users expect more from voice, a persuasive pilot will be insufficient. Hopkins presents integrated evaluation—and Coval's work on it—as the infrastructure needed to keep those experiences dependable as they grow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- Waymo's SimulationCityArticle
Waymo explains how scenario variation, realistic behavior distributions and full-trip simulation complement focused tests.
- Confidence intervals for failure proportionsDocumentation
Statistical guidance for expressing uncertainty around observed failure rates, including small samples and rare failures.
Updates since the talk
Current guides for simulating conversations, monitoring production agents, incorporating human review and integrating evaluations into development.
Full transcript
Loading transcript…