AI Engineer World's Fair 2025

Building Effective Voice Agents

Loading slides…

Read the talk

Building Effective Voice Agents

A voice agent must respond quickly, accept interruptions, and make reliable decisions. Those demands shape its architecture, prompts, tools, evaluations, and guardrails.

From a talk by Anoop Kotha and Toki Sherbakov

Before you start: Familiarity with LLM prompts, tool calling, and basic speech-to-text and text-to-speech pipelines will help.

Can the agent change course while speaking?

What should happen when someone interrupts a voice agent and changes the request? A fluent answer is only part of the problem: the agent also needs to listen, stop, and respond to the new intent. This is the practical promise of speech-to-speech systems. Toki Sherbakov, introducing himself as the lead of OpenAI’s US solution architecture team, and teammate Anoop Kotha place that promise within a broader shift from text applications toward images, video through Sora, and audio. Transcription and text-to-speech already existed; their assessment at the time of the recording is that low-latency speech-to-speech is becoming usable for production applications at scale.

Slide contrasts Before—slower, robotic, brittle—with After—faster, expressive, accurate—and states that audio is now usable at scale.Open full source frame
Audio reaches a “good enough” tipping point: faster, expressive, and accurate.

The first haiku demonstration makes the interaction problem concrete. A user asks for a poem about whispers. After a noticeable pause, the model begins its answer. The user interrupts to ask for stars instead, but the model finishes the whispers poem and subsequently produces a poem about sand. Sherbakov describes this example as less than six months old: slow delivery, a robotic voice, and brittle handling of a changed request.

The second demonstration repeats the sequence. The user requests whispers, the model starts, and the user redirects it to stars. This time the model changes course and delivers a stars haiku. The improvement is audible as well as behavioral: the voice is more expressive, and its delivery can be steered toward excitement or a slower speaking pace. These paired examples illustrate responsiveness and interruption handling; they are not controlled latency or accuracy benchmarks. Voice quality includes how well the system adapts while a conversation is already underway.

0:150:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

From three model stages to speech-to-speech

The established chained architecture takes incoming audio through three stages: transcription, an LLM that decides what to say or do, and text-to-speech that produces the reply. Each stage contributes to the time before the user hears an answer. Converting the conversation into text also creates an information bottleneck: the reasoning model receives the transcript rather than the original vocal delivery.

A speech-to-speech architecture, accessed here through the Realtime API, brings speech understanding, reasoning, and speech generation into a single model. The application no longer has to stitch together three separate model stages.

ArchitectureProcessing pathMain trade-off
ChainedAudio → transcript → LLM → speechSeparate stages add delay and a text bottleneck
Speech-to-speechAudio → one model → audioSimpler path for low-latency conversation

Sherbakov’s rationale is that native audio processing reduces response delay and preserves conversational information that a transcript can discard. That is an architectural advantage, not a guarantee of lossless understanding.

4:084:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:08 · section reference included

Choose for the consequences of an error

Architecture selection depends on five dimensions: latency, cost, accuracy and intelligence, user experience, and integrations and tooling. In the talk’s simple consumer example, quick responses and expressive delivery dominate. The interaction needs few integrations, and neither factual precision nor cost receives the same emphasis as conversational experience. That weighting describes this example, not every consumer product.

Customer service changes the consequences. An agent cannot casually mishear an order number or delete an order that the customer wanted updated. It also needs access to internal systems and, where applicable, SIP and phone providers. A slightly slower or less expressive interaction may be acceptable if it makes the business operation more reliable. The speakers treat cost in this setting relative to the operational expense the service could replace.

DimensionSimple consumer interactionCustomer service
LatencyFast exchange is centralSpeed matters alongside correctness
CostLower emphasis in this exampleWeighed against service savings
Accuracy and intelligenceLess central to the interactionCritical for business actions
User experienceExpressiveness and immediacyBalanced with reliable outcomes
Integrations and toolingFew external connectionsInternal systems and telephony

Realtime can fit customer service when low latency is the leading requirement. A chained system can also make sense when control, determinism, and the intelligence of the reasoning stage matter more. Choose the architecture for the task’s failure costs, not simply for the most natural-sounding demo.

Customer service example with five filled columns: latency, cost, accuracy and intelligence, user experience, and integrations and tooling. Accuracy and intelligence is tallest. A note contrasts Realtime API with a chained architecture.Open full source frame
Customer service places greater importance on accuracy, intelligence, and integrations.
5:255:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:25 · section reference included

Keep the voice responsive while specialists decide

A voice agent still has the familiar agent components: a model, instructions, tools, and a runtime. The runtime includes the execution environment and guardrails. Voice adds distinctive requirements around system design, vocal customization, tool use, and evaluation, but it does not remove those foundations.

A text-based customer-service system might use a small triage model to route difficult work to specialists: o3 for refunds, for example, and o4-mini for cancellations. For voice, Kotha introduces a different arrangement: keep the Realtime API as the frontline conversational agent, let it answer straightforward questions, and expose more capable specialist agents as tools for harder decisions. The voice layer maintains the interaction while another model handles the reasoning it should not attempt alone.

The snowboard-return demonstration follows that division of responsibility:

  1. Establish the request. The voice agent says the customer is authenticated and asks what they want to return. The customer identifies a snowboard; the authentication implementation is not shown.
  2. Acknowledge the work. The agent says it is reviewing the policy and will return with next steps.
  3. Delegate the decision. Kotha identifies o4-mini as the model handling the policy reasoning.
  4. Speak the result. The agent reports that the return falls within a thirty-day policy and says it will process the return and send a confirmation email.

The demonstrated result is a spoken eligibility decision and a promise of follow-up, not evidence that a backend return or email operation completed.

7:277:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:27 · section reference included

Prompt the delivery and the conversation

Voice prompts control both what the agent does and how it sounds. Demeanor, tone, and enthusiasm become explicit parts of the product’s behavior and brand. Kotha points to openai.fm as a place to experiment with voices and sample prompts in the demonstrated workflow. Those experiments help reveal how much delivery changes with prompting, as well as the limits of that control.

For more complex tasks, specify the conversation’s progression too. Text agents often receive examples that show a sequence of steps; voice agents can use the same technique. The illustrated flow begins with a greeting stage, a description of that stage, and its associated instructions, then continues through later stages. Delivery instructions establish how the agent speaks; staged conversation instructions establish what it should accomplish next.

9:5310:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:53 · section reference included

Keep tools focused and carry context forward

Start with a small set of tools for each agent, then add capabilities as requirements become clear. Kotha cautions against beginning with ten or twenty tools attached to one model. Those are illustrative counts, not a hard limit: the design principle is to distribute specialized work instead of giving every agent every capability.

Two ways of distributing work serve different purposes:

  • Handoffs: Transfer the conversation to another agent. Preserve relevant context by summarizing the conversation state and passing it along. The summary should retain what the next agent needs; summarization itself does not guarantee that nothing is lost.
  • Delegation: Let the current agent invoke a specialist through a tool call, as in the snowboard example. Kotha names o3-mini and o4-mini as possible delegates and points to the Agents SDK for these patterns.

The distinction is who continues managing the conversation: a receiving agent after a handoff, or the existing agent after delegated work returns.

Tools slide lists Start simple, Handoffs, and Delegation beside a weather-tool code example.Open full source frame
Tool guidance: start simple, manage context during handoffs, and use delegation.
11:3711:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:37 · section reference included

Evaluate the decisions and the audio

Begin with observability: capture agent traces and audio so the team can inspect what happened. Then have subject-matter experts label examples and use those findings to revise prompts. Kotha describes expert-led iteration as one of the most effective approaches observed in customer work, even though it can look less sophisticated than automating evaluation immediately.

Once examples and labels exist, evaluate different aspects of the interaction with the appropriate evidence:

  • Transcript evaluation: Use an LLM judge, a business rubric, or explicit checks of function-calling behavior to assess whether the agent followed the required process.
  • Audio evaluation: Inspect tone, pacing, and intonation—the qualities a transcript does not adequately capture. Kotha names GPT 4 Audio as an example audio-capable evaluator, without specifying an exact model identifier.

Much of business-task correctness can be assessed from text. Audio evaluation adds information about how the answer was delivered rather than replacing transcript checks.

Synthetic conversations extend the range of scenarios. Connect two realtime clients: one runs the agent being tested, and the other plays customer personas. Let them converse, then evaluate the resulting transcripts and audio. This creates a way to exercise many interactions while reusing the same evaluation criteria developed for inspected conversations.

12:5613:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:56 · section reference included

Use the gap between generation and playback

A fast voice system needs safety checks that can operate alongside its response. In the described Realtime workflow, generated text advances faster than the corresponding spoken audio. That creates a timing window for asynchronous guardrails: inspect the generated text while speech is progressing, rather than treating checking as an entirely separate stage before response generation.

Kotha describes checks every 100 characters, with a configurable cadence. This is a scheduling choice, not a safety score or a universal threshold. The useful mechanism is the lead that text generation has over playback; whether a check can prevent particular words from being heard also depends on timing and playback control. The talk does not demonstrate the cancellation or playback-blocking implementation.

Guardrails slide with Run async and Settings control guidance beside a code example. The text says generated text is faster than spoken speech and describes a flexible check cadence of every 100 characters.Open full source frame
Run guardrails asynchronously and configure how often checks occur.
15:0615:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:06 · section reference included

Build the feedback loop early

The closing customer examples emphasize different investments. According to Kotha, Lemonade focused early on evaluations, guardrails, and feedback mechanisms for its team. Some practices were initially difficult to scale, but the resulting feedback helped the team move faster. For Tinder’s experience, which he calls Rizz Chat, customization and brand realism were the priorities he credits with improving the user experience. These are customer observations from the speakers, not measured comparisons of business outcomes.

The production outlook returns to audio’s place alongside text, images, and video. Sherbakov points to a Realtime snapshot released the day before the talk and describes it as substantially better, without naming a version or presenting a benchmark. That release reference belongs to the recording’s historical context, not a current API recommendation. His closing assessment is that speech-to-speech has become capable enough to justify building scalable applications, with an opportunity for teams that start learning early. The invitation is to build and test against real requirements—not to treat a convincing voice demonstration as a finished deployment.

15:4916:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:49 · section reference included

Resources

Updates since the talk

Transcript download
Full transcript

Loading transcript…