# Building Effective Voice Agents

Anoop Kotha · Toki Sherbakov

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/-OXiljTJxQU-building-effective-voice-agents) · [Video](https://www.youtube.com/watch?v=-OXiljTJxQU)

## Building Effective Voice Agents

A voice agent must respond quickly, accept interruptions, and make reliable decisions. Those demands shape its architecture, prompts, tools, evaluations, and guardrails.

Familiarity with LLM prompts, tool calling, and basic speech-to-text and text-to-speech pipelines will help.

### Can the agent change course while speaking?

[0:15](https://www.youtube.com/watch?v=-OXiljTJxQU&t=15s)

What should happen when someone interrupts a voice agent and changes the request? A fluent answer is only part of the problem: the agent also needs to listen, stop, and respond to the new intent. This is the practical promise of speech-to-speech systems. Toki Sherbakov, introducing himself as the lead of OpenAI’s US solution architecture team, and teammate Anoop Kotha place that promise within a broader shift from text applications toward images, video through **Sora**, and audio. Transcription and text-to-speech already existed; their assessment at the time of the recording is that low-latency speech-to-speech is becoming usable for production applications at scale.

![Slide contrasts Before—slower, robotic, brittle—with After—faster, expressive, accurate—and states that audio is now usable at scale.](https://ai.engineer/api/talk-essay-media/1612b2a6f08c2732f91547d1b4ec727efe71e22b1e9167e1187f65b13edf9d4f)

Audio reaches a “good enough” tipping point: faster, expressive, and accurate.

The first haiku demonstration makes the interaction problem concrete. A user asks for a poem about whispers. After a noticeable pause, the model begins its answer. The user interrupts to ask for stars instead, but the model finishes the whispers poem and subsequently produces a poem about sand. Sherbakov describes this example as less than six months old: slow delivery, a robotic voice, and brittle handling of a changed request.

The second demonstration repeats the sequence. The user requests whispers, the model starts, and the user redirects it to stars. This time the model changes course and delivers a stars haiku. The improvement is audible as well as behavioral: the voice is more expressive, and its delivery can be steered toward excitement or a slower speaking pace. These paired examples illustrate responsiveness and interruption handling; they are not controlled latency or accuracy benchmarks. **Voice quality includes how well the system adapts while a conversation is already underway.**

### From three model stages to speech-to-speech

[4:08](https://www.youtube.com/watch?v=-OXiljTJxQU&t=248s)

The established **chained architecture** takes incoming audio through three stages: transcription, an LLM that decides what to say or do, and text-to-speech that produces the reply. Each stage contributes to the time before the user hears an answer. Converting the conversation into text also creates an information bottleneck: the reasoning model receives the transcript rather than the original vocal delivery.

A **speech-to-speech architecture**, accessed here through the Realtime API, brings speech understanding, reasoning, and speech generation into a single model. The application no longer has to stitch together three separate model stages.

| Architecture | Processing path | Main trade-off |
| --- | --- | --- |
| Chained | Audio → transcript → LLM → speech | Separate stages add delay and a text bottleneck |
| Speech-to-speech | Audio → one model → audio | Simpler path for low-latency conversation |

Sherbakov’s rationale is that native audio processing reduces response delay and preserves conversational information that a transcript can discard. That is an architectural advantage, not a guarantee of lossless understanding.

### Choose for the consequences of an error

[5:25](https://www.youtube.com/watch?v=-OXiljTJxQU&t=325s)

Architecture selection depends on five dimensions: **latency, cost, accuracy and intelligence, user experience, and integrations and tooling**. In the talk’s simple consumer example, quick responses and expressive delivery dominate. The interaction needs few integrations, and neither factual precision nor cost receives the same emphasis as conversational experience. That weighting describes this example, not every consumer product.

Customer service changes the consequences. An agent cannot casually mishear an order number or delete an order that the customer wanted updated. It also needs access to internal systems and, where applicable, SIP and phone providers. A slightly slower or less expressive interaction may be acceptable if it makes the business operation more reliable. The speakers treat cost in this setting relative to the operational expense the service could replace.

| Dimension | Simple consumer interaction | Customer service |
| --- | --- | --- |
| Latency | Fast exchange is central | Speed matters alongside correctness |
| Cost | Lower emphasis in this example | Weighed against service savings |
| Accuracy and intelligence | Less central to the interaction | Critical for business actions |
| User experience | Expressiveness and immediacy | Balanced with reliable outcomes |
| Integrations and tooling | Few external connections | Internal systems and telephony |

Realtime can fit customer service when low latency is the leading requirement. A chained system can also make sense when control, determinism, and the intelligence of the reasoning stage matter more. **Choose the architecture for the task’s failure costs, not simply for the most natural-sounding demo.**

![Customer service example with five filled columns: latency, cost, accuracy and intelligence, user experience, and integrations and tooling. Accuracy and intelligence is tallest. A note contrasts Realtime API with a chained architecture.](https://ai.engineer/api/talk-essay-media/4e101e18a72fbb0880973cf5017f41ca7fee0fd7845230e045629cc10abd24a3)

Customer service places greater importance on accuracy, intelligence, and integrations.

### Keep the voice responsive while specialists decide

[7:27](https://www.youtube.com/watch?v=-OXiljTJxQU&t=447s)

A voice agent still has the familiar agent components: a model, instructions, tools, and a runtime. The runtime includes the execution environment and guardrails. Voice adds distinctive requirements around system design, vocal customization, tool use, and evaluation, but it does not remove those foundations.

A text-based customer-service system might use a small triage model to route difficult work to specialists: **o3** for refunds, for example, and **o4-mini** for cancellations. For voice, Kotha introduces a different arrangement: keep the Realtime API as the frontline conversational agent, let it answer straightforward questions, and expose more capable specialist agents as tools for harder decisions. The voice layer maintains the interaction while another model handles the reasoning it should not attempt alone.

The snowboard-return demonstration follows that division of responsibility:

1. **Establish the request.** The voice agent says the customer is authenticated and asks what they want to return. The customer identifies a snowboard; the authentication implementation is not shown.
2. **Acknowledge the work.** The agent says it is reviewing the policy and will return with next steps.
3. **Delegate the decision.** Kotha identifies o4-mini as the model handling the policy reasoning.
4. **Speak the result.** The agent reports that the return falls within a thirty-day policy and says it will process the return and send a confirmation email.

The demonstrated result is a spoken eligibility decision and a promise of follow-up, not evidence that a backend return or email operation completed.

### Prompt the delivery and the conversation

[9:53](https://www.youtube.com/watch?v=-OXiljTJxQU&t=593s)

Voice prompts control both what the agent does and how it sounds. Demeanor, tone, and enthusiasm become explicit parts of the product’s behavior and brand. Kotha points to `openai.fm` as a place to experiment with voices and sample prompts in the demonstrated workflow. Those experiments help reveal how much delivery changes with prompting, as well as the limits of that control.

For more complex tasks, specify the conversation’s progression too. Text agents often receive examples that show a sequence of steps; voice agents can use the same technique. The illustrated flow begins with a greeting stage, a description of that stage, and its associated instructions, then continues through later stages. Delivery instructions establish how the agent speaks; staged conversation instructions establish what it should accomplish next.

### Keep tools focused and carry context forward

[11:37](https://www.youtube.com/watch?v=-OXiljTJxQU&t=697s)

Start with a small set of tools for each agent, then add capabilities as requirements become clear. Kotha cautions against beginning with ten or twenty tools attached to one model. Those are illustrative counts, not a hard limit: the design principle is to distribute specialized work instead of giving every agent every capability.

Two ways of distributing work serve different purposes:

- **Handoffs:** Transfer the conversation to another agent. Preserve relevant context by summarizing the conversation state and passing it along. The summary should retain what the next agent needs; summarization itself does not guarantee that nothing is lost.
- **Delegation:** Let the current agent invoke a specialist through a tool call, as in the snowboard example. Kotha names **o3-mini** and o4-mini as possible delegates and points to the [Agents SDK](https://openai.github.io/openai-agents-js/guides/voice-agents/build/) for these patterns.

The distinction is who continues managing the conversation: a receiving agent after a handoff, or the existing agent after delegated work returns.

![Tools slide lists Start simple, Handoffs, and Delegation beside a weather-tool code example.](https://ai.engineer/api/talk-essay-media/2721e6086f2d33c24152196c2d6cdfbbb3a696ecf51d286953e385ff4e06eefd)

Tool guidance: start simple, manage context during handoffs, and use delegation.

### Evaluate the decisions and the audio

[12:56](https://www.youtube.com/watch?v=-OXiljTJxQU&t=776s)

Begin with **observability**: capture agent traces and audio so the team can inspect what happened. Then have subject-matter experts label examples and use those findings to revise prompts. Kotha describes expert-led iteration as one of the most effective approaches observed in customer work, even though it can look less sophisticated than automating evaluation immediately.

Once examples and labels exist, evaluate different aspects of the interaction with the appropriate evidence:

- **Transcript evaluation:** Use an LLM judge, a business rubric, or explicit checks of function-calling behavior to assess whether the agent followed the required process.
- **Audio evaluation:** Inspect tone, pacing, and intonation—the qualities a transcript does not adequately capture. Kotha names **GPT 4 Audio** as an example audio-capable evaluator, without specifying an exact model identifier.

Much of business-task correctness can be assessed from text. Audio evaluation adds information about how the answer was delivered rather than replacing transcript checks.

Synthetic conversations extend the range of scenarios. Connect two realtime clients: one runs the agent being tested, and the other plays customer personas. Let them converse, then evaluate the resulting transcripts and audio. This creates a way to exercise many interactions while reusing the same evaluation criteria developed for inspected conversations.

### Use the gap between generation and playback

[15:06](https://www.youtube.com/watch?v=-OXiljTJxQU&t=906s)

A fast voice system needs safety checks that can operate alongside its response. In the described Realtime workflow, generated text advances faster than the corresponding spoken audio. That creates a timing window for **asynchronous guardrails**: inspect the generated text while speech is progressing, rather than treating checking as an entirely separate stage before response generation.

Kotha describes checks every 100 characters, with a configurable cadence. This is a scheduling choice, not a safety score or a universal threshold. The useful mechanism is the lead that text generation has over playback; whether a check can prevent particular words from being heard also depends on timing and playback control. The talk does not demonstrate the cancellation or playback-blocking implementation.

![Guardrails slide with Run async and Settings control guidance beside a code example. The text says generated text is faster than spoken speech and describes a flexible check cadence of every 100 characters.](https://ai.engineer/api/talk-essay-media/22aa1456a685e3a573a2b4eb8af38231aeaed29469116ae37c8755b3ce5a02c0)

Run guardrails asynchronously and configure how often checks occur.

### Build the feedback loop early

[15:49](https://www.youtube.com/watch?v=-OXiljTJxQU&t=949s)

The closing customer examples emphasize different investments. According to Kotha, Lemonade focused early on evaluations, guardrails, and feedback mechanisms for its team. Some practices were initially difficult to scale, but the resulting feedback helped the team move faster. For Tinder’s experience, which he calls Rizz Chat, customization and brand realism were the priorities he credits with improving the user experience. These are customer observations from the speakers, not measured comparisons of business outcomes.

The production outlook returns to audio’s place alongside text, images, and video. Sherbakov points to a Realtime snapshot released the day before the talk and describes it as substantially better, without naming a version or presenting a benchmark. That release reference belongs to the recording’s historical context, not a current API recommendation. His closing assessment is that speech-to-speech has become capable enough to justify building scalable applications, with an opportunity for teams that start learning early. The invitation is to build and test against real requirements—not to treat a convincing voice demonstration as a finished deployment.

## Resources

- [OpenAI Realtime Agents examples](https://github.com/openai/openai-realtime-agents) — Examples of realtime conversation, specialist delegation, handoffs, and a customer-service return flow.
- [Building Realtime Agents](https://openai.github.io/openai-agents-js/guides/voice-agents/build/) — Current SDK guidance for realtime agents, including configurable asynchronous output guardrails.
- [Build Hour: Voice Agents](https://webinar.openai.com/on-demand/ead245e4-c7d6-4e35-b7ca-41a51454a4b4) — A practical voice-agent walkthrough covering delegation, conversation prompts, and evaluation.
- [GPT-4o Realtime model documentation](https://developers.openai.com/api/docs/models/gpt-4o-realtime-preview) — Model reference listing historical GPT-4o Realtime snapshots and their deprecation status.
- [Tinder’s The Game Game](https://ca.tinderpressroom.com/2025-04-16-Tinders-The-Game-GameTM-Is-Now-Available-in-Canada) — Tinder's announcement of an interactive voice experience with AI personas and conversational feedback.

## Slides

### Innovation partner and platinum sponsors

[0:10](https://www.youtube.com/watch?v=-OXiljTJxQU&t=10s) [AWS is listed as innovation partner above six platinum sponsor logos.](https://assets.ai.engineer/sites/aiecode2025/sha256/a1a2985c035f68caf53ab4b635769536c4cbb8a5646b5678607f09a5c5e18f1e.webp)

INNOVATION PARTNER
aws

PLATINUM SPONSORS
Graphite
Windsurf
MongoDB®
daily™
augment code
WorkOS

### Building Practical Audio Agents

Toki Sherbakov and Anoop Kotha introduce a talk on where audio models stand and the patterns and best practices for building audio agents.

[0:27](https://www.youtube.com/watch?v=-OXiljTJxQU&t=27s) [Title slide introducing Toki Sherbakov and Anoop Kotha.](https://assets.ai.engineer/sites/aiecode2025/sha256/926ac07cfb0c2feb5b983df97cf6d57e77f253f3df0f2eef3c27ecf83710bbd7.webp)

Building Practical Audio Agents

Toki Sherbakov
Solutions Architecture Leader

Anoop Kotha
Solutions Architect

### Gen AI applications have primarily been text-based

Generative AI applications have largely centered on understanding and generating text. The speaker says production applications built on text continue to scale.

[0:48](https://www.youtube.com/watch?v=-OXiljTJxQU&t=48s) [A text card describes understanding and generating text with human-like fluency.](https://assets.ai.engineer/sites/aiecode2025/sha256/b46f4019e4b37ecfd9795cc1e13afdfffff8d14796db481045f19730ce9606c8.webp)

Gen AI applications have primarily been text-based

Text

Understand and generate text with human-like fluency.

### Entering the multimodal era

Models now span text, images and video, and audio. The talk focuses on audio, where the speaker sees speech-to-speech improving alongside transcription and text-to-speech.

[1:07](https://www.youtube.com/watch?v=-OXiljTJxQU&t=67s) [Three cards describe text, images and video, and audio capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/3ef214daa46111dba556ce0c000ed6bf1de381ea1ab1b62d7b0ffcca6df977eb.webp)

We’re now entering the multimodal era

Text
Understand and generate text with human-like fluency.

+

Images & Video
Interpret complex visuals and generate high-resolution images and videos.

+

Audio
State-of-the-art transcription, voice generation, and speech-to-speech capabilities.

### Speech-to-speech architecture

A chained architecture passes speech through transcription, an LLM, and text-to-speech. The Realtime API combines those steps in one speech-to-speech model; the speaker says this reduces output latency.

[4:56](https://www.youtube.com/watch?v=-OXiljTJxQU&t=296s) [A chained transcription, LLM, and text-to-speech pipeline is compared with a single speech-to-speech model using the Realtime API.](https://assets.ai.engineer/sites/aiecode2025/sha256/5897851270e9dcaeed412afebade4f5997e93479253416d46bd885351a3520d0.webp)

Speech-to-speech is emerging as a scalable architecture pattern

Chained Architecture
Chaining audio models with a LLM

USER SPEECH → API CALL → SPEECH-TO-TEXT MODEL → API CALL → LLM → API CALL → TEXT-TO-SPEECH MODEL → MODEL SPEECH
GPT-4o-transcribe
GPT-4.1
GPT-4o-mini-tts

Time →

Speech-to-Speech Architecture
Combining audio and LLM steps into a single speech-to-speech model

USER SPEECH → API CALL → SPEECH-TO-SPEECH MODEL → MODEL SPEECH
Realtime API

### Primary considerations and tradeoffs when building audio experiences

Audio architecture choices involve five tradeoffs: latency, cost, accuracy and intelligence, user experience, and integrations and tooling. Their relative importance depends on the application.

[5:23](https://www.youtube.com/watch?v=-OXiljTJxQU&t=323s) [Five considerations: latency, cost, accuracy and intelligence, user experience, and integrations and tooling.](https://assets.ai.engineer/sites/aiecode2025/sha256/2661332e0632ca0a645cb73071db3f234abfdaca42d8ffa2d47317fa32846384.webp)

Primary considerations and tradeoffs when building audio experiences

Latency
Cost
Accuracy & intelligence
User experience
Integrations & tooling

### Consumer application example

Consumer applications prioritize user experience and latency, trading off cost, accuracy and integrations. The slide proposes the Realtime API for this mix.

[5:44](https://www.youtube.com/watch?v=-OXiljTJxQU&t=344s) [Consumer applications prioritize user experience and latency; the proposed architecture is the Realtime API.](https://assets.ai.engineer/sites/aiecode2025/sha256/8cb0b22ccac93b4ebb90eea0166e34bd6e1b73b8eb7d3f8f22c5279f643b29ec.webp)

Consumer application example

Consumer applications are more likely to have a higher importance on user experience and latency and will trade off on cost, accuracy/intelligence, and integrations.

Proposed architecture: Realtime API

Latency
Cost
Accuracy & intelligence
User experience
Integrations & tooling

### Customer service example

Customer service prioritizes accuracy and integrations over latency and user experience. The slide proposes Realtime API when latency matters most, or a chained architecture when accuracy and determinism matter.

[6:22](https://www.youtube.com/watch?v=-OXiljTJxQU&t=382s) [Customer service prioritizes accuracy, intelligence, and integrations; the slide compares Realtime API and chained architectures.](https://assets.ai.engineer/sites/aiecode2025/sha256/620235b29f01e870edf1823f17fa7f07bd74e11ee2df600c2fe9e4f09228b35f.webp)

Customer service example

Customer service is more likely to have a higher importance for accuracy/intelligence + integrations and will trade off on latency and user experience.

Proposed architecture: Realtime API when latency matters most, otherwise chained (STT + intelligence + TTS) when accuracy + determinism matters

Latency
Cost
Accuracy & intelligence
User experience
Integrations & tooling

### What is an agent?

An agent combines a model, instructions, tools, and a runtime to carry out tasks independently. Its tools gather context and take actions within defined guardrails.

[7:28](https://www.youtube.com/watch?v=-OXiljTJxQU&t=448s) [An agent independently accomplishes tasks using a model, instructions, tools, and a runtime.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3ccdd3ce65a86a646928e9aaa41647e1edc2c5722dc8efa78c9da7d9efc137a.webp)

What is an agent?

Agents are systems that independently accomplish tasks on your behalf.

Agents use an LLM to execute instructions and make decisions. They have access to tools to gather context and take actions, always operating within clearly defined guardrails.

Agent =
+ Model
+ Instructions
+ Tools
+ Runtime

[illegible]nfidential and proprietary.

### Building Voice Agents

Voice agents share traditional agent patterns, but system design, prompting and voice customization, tools, evals, and guardrails can differ.

[7:49](https://www.youtube.com/watch?v=-OXiljTJxQU&t=469s) [Voice agent patterns include system design, prompting and customization, tools, evaluations, and guardrails.](https://assets.ai.engineer/sites/aiecode2025/sha256/cdd7366a371045ac90098c3f49ef35fb0b2a02bb05dea0dd9451ecbd8d53cf6c.webp)

Building Voice Agents

Voice agents take many of the same patterns that traditional agentic architectures hold.

Some areas, however, are a bit different.

Key patterns
• System design
• Prompting and customization
• Tools
• Evals
• Guardrails

### Speech-to-speech agents: Delegation through tools

A Realtime API frontline agent exchanges audio with the user, handles easier questions, and calls specialized agents for harder tasks such as refunds and order cancellations.

[8:20](https://www.youtube.com/watch?v=-OXiljTJxQU&t=500s) [A Realtime API frontline agent exchanges audio with a user and delegates through tool calls to refund, order cancellation, and product expert agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/02a3c76567a5626f4639d12af6b4e289e043a5f6491d5d5bb91cc32b024b4779.webp)

Speech-to-speech agents:
Delegation through tools

USER AUDIO

REALTIME API
FRONTLINE AGENT

TOOL CALLS
REFUND AGENT (o3)
ORDER CANCELLATION AGENT (o4-mini)
PRODUCT EXPERT AGENT (GPT-4.1)

15

[8:57](https://www.youtube.com/watch?v=-OXiljTJxQU&t=537s) [Diagram showing a Realtime API frontline agent delegating to refund, order cancellation, and product expert agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/67288096864e5f24ad6968bfc8b799c394c56a14a9dd1f3fba1ed8c5c07e586d.webp)

Speech-to-speech agents:
Delegation through tools

USER AUDIO

REALTIME API
FRONTLINE AGENT

TOOL CALLS
REFUND AGENT (o3)
ORDER CANCELLATION AGENT (o4-mini)
PRODUCT EXPERT AGENT (GPT-4.1)

15

### Customer support return demo

A customer asks to return a snowboard. The presenter says the voice agent delegates the policy review to o4-mini; the demo then says the return is within the 30-day policy and promises an email with next steps.

[9:22](https://www.youtube.com/watch?v=-OXiljTJxQU&t=562s) [A retail support assistant states that a return falls within the return policy and will be processed.](https://assets.ai.engineer/sites/aiecode2025/sha256/caf958e5bc30a53ebb5d4c30703ddb133c192aa757b4e1fd34e09d97ef948c51.webp)

New Season Collection
Conquer The Mountains With Confidence
Premium ski and snowboard gear for enthusiasts and professionals. Ride the powder with the best equipment.
Shop Collection
View Deals

Featured Products
Discover our top-rated gear for your next adventure on the slopes.

PeakRider Support
[illegible] statutory rights.
It looks like the return you are making is within the 30-day return policy. We will process the return and send you a confirmation email with next steps shortly.
11:05 AM
Recording...
Type your message...

Interface text:
PeakRider
Home
Shop
Categories
About
Contact
Sign In
Shop Now
Sale

### Customize and create a personalized brand

A front-desk prompt specifies not only the agent’s task, but how it should sound: composed, friendly yet crisp, measured, and quick. Voice prompts can shape demeanor, tone, and enthusiasm alongside behavior.

[9:52](https://www.youtube.com/watch?v=-OXiljTJxQU&t=592s) [Guidance on using voice prompts to shape an agent's personality, alongside an example prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/183bd9d781aeeff3dd3314e8bdb6f0ce4af744c1a73449a93fbfa2e1a9654fbf.webp)

Customize and create a personalized brand

Be detailed in your personality design

Use prompting of our speech-to-speech and text-to-speech models to influence not just agent behavior but actual tone and personality.

# Personality and Tone

## Identity
You are an efficient, polished, and professional front desk agent, akin to an assistant at a high-end law firm. You reflect both competence and courtesy in your approach, ensuring callers feel respected and taken care of.

## Task
You will field incoming calls, welcome callers, gather necessary details (such as spelling of names), and facilitate any required next steps. Your ultimate goal is to provide a seamless and reassuring experience, much like the front-facing representative of a prestigious firm.

## Demeanor
You maintain a composed and assured demeanor, demonstrating confidence and competence while still being approachable.

## Tone
Your tone is friendly yet crisp, reflecting professionalism without sacrificing warmth. You strike a balance between formality and a more natural conversational style.

## Level of Enthusiasm
Calm and measured, with just enough positivity to sound approachable and accommodating.

## Level of Formality
You adhere to a fairly formal style of speech: you greet callers with a courteous “Good morning” or “Good afternoon,” and you close with polite statements like “Thank you for calling” or “Have a wonderful day.”

## Level of Emotion
Fairly neutral and matter-of-fact. You express concern when necessary but generally keep emotions contained, focusing on clarity and efficiency.

## Filler Words
None — your responses are concise and polished.

## Pacing
Rather quick and efficient. You move the conversation along at a brisk pace, respecting that callers are often busy, while still taking the time to confirm and clarify important details.

## Other details
- You always confirm spellings or important information that the user provides (e.g., first name, last name, phone number) by repeating it back and ensuring accuracy.

Interface text:
17

### Customize and create a personalized brand

OpenAI.fm lets you try different voices and sample prompts to explore how prompting changes voice expressiveness.

[10:35](https://www.youtube.com/watch?v=-OXiljTJxQU&t=635s) [A slide recommends trying OpenAI.fm and shows its voice and style controls.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d31aa9afa3d94a5f0fc13eb5e02b12ca6aaae3a79324115281b48c7eaca830b.webp)

Customize and create a personalized brand

Try out OpenAI.fm to get inspired on how to steer our text-to-speech or speech-to-speech model to create a more personalized voice experience.

Interface text:
OpenAI.fm
An interactive demo for developers to try the new text-to-speech model in the OpenAI API.
START BUILDING ↗

VOICE
Alloy
Ash
Ballad
Coral
Echo
Fable
Onyx
Nova
Sage
Shimmer
Verse

VIBE
Serene
Friendly
Sports Coach
Emo Teenager
NYC Cabbie

Voice Affect: Soft, gentle, soothing; embody tranquility.
Tone: Calm, reassuring, peaceful; convey genuine warmth and serenity.
Pacing: Slow, deliberate, and unhurried; pause gently after instructions to allow the listener time to relax and follow along.
Emotion: Deeply soothing and comforting; express genuine kindness and care.

SCRIPT
Hello, and welcome to your moment of mindfulness. I'm so glad y[illegible]
here. Let's begin by closing your eyes and taking a deep, calmi[illegible]
breath. Breathe in slowly through your nose, and exhale softly,
releasing any tension.

Imagine your thoughts as soft clouds drifting across the sky—[illegible]
them without attachment, letting your mind become clear and pea[illegible]

DOWNLOAD
SHARE
PLAY

18

### Prompting the agent with details

For complex voice conversations, define states in the system prompt with instructions, examples, and transitions. The example moves from a greeting to confirming the caller’s first and last names.

[10:56](https://www.youtube.com/watch?v=-OXiljTJxQU&t=656s) [Slide recommends outlining conversation states and including context in prompts, alongside a partially clipped code example.](https://assets.ai.engineer/sites/aiecode2025/sha256/1ae8e039d8f6a84e39dc473677933965008826799ac4120309f8c123a225af00.webp)

Prompting the agent with details

Outline conversational flows
For more advanced use cases, try defining conversation states inside your system prompt for your agent to follow. These states could be a series of JSON or XML.

Include context
To avoid unnecessary tool calls include any important context directly in the prompt.

# Conversation States (Example)
[
  {
    "id": "1_greeting",
    "description": "Greet the caller and explain the verification process.",
    "instructions": [
      "Greet the caller warmly.",
      "Inform them about the need to collect personal information for their record."
    ],
    "examples": [
      "Good morning, this is the front desk administrator. I will assist you in verifying your details.",
      "Let us proceed with the verification. May I kindly have your first name? Please spell it out letter by letter for clarity."
    ],
    "transitions": [{
      "next_step": "2_get_first_name",
      "condition": "After greeting is complete."
    }]
  },
  {
    "id": "2_get_first_name",
    "description": "Ask for and confirm the caller's first name.",
    "instructions": [
      "Request: 'Could you please provide your first name?'",
      "Spell it out letter-by-letter back to the caller to confirm."
    ],
    "examples": [
      "May I have your first name, please?",
      "You spelled that as J-A-N-E, is that correct?"
    ],
    "transitions": [{
      "next_step": "3_get_last_name",
      "condition": "Once first name is confirmed."
    }]
  },
  {
    "id": "3_get_last_name",
    "description": "Ask for and confirm the caller's last name.",
    "instructions": [
      "Request: 'Thank you. Could you please provide your last name?'",
      "Spell it out letter-by-letter back to the caller to confirm."
    ],

19

### Tools

Start each agent with a small set of tools. Use handoffs to scope tasks and pass a conversation summary between agents; the Agents SDK also supports delegation.

[11:35](https://www.youtube.com/watch?v=-OXiljTJxQU&t=695s) [Slide discusses limiting tool calls, agent handoffs and delegation, with a weather tool example.](https://assets.ai.engineer/sites/aiecode2025/sha256/954a15c3820c7b454372f71e8e97fca144edd94e3dba98993210226a406b1181.webp)

Tools

Start simple
Limit the number of tool calls associated with a single agent.

Handoffs
Utilize handoffs between agents to scope tool calls. Manage context between handoffs by changing system prompts and summarizing conversation state.

Delegation
Built in delegation to the Agents SDK

import { tool } from "@openai/agents/realtime";
import { z } from "zod";

const getWeather = tool({
  name: "get_weather",
  description: "Return the weather for a city.",
  parameters: z.object({ city: z.string() }),
  async execute({ city }) {
    return `The weather in ${city} is sunny.`;
  },
});

const weatherAgent = new RealtimeAgent({
  name: "Weather assistant",
  instructions: "Answer weather questions.",
  tools: [getWeather],
});

20

### Evaluation

Start with traces and SME labels to improve prompts, then evaluate transcripts and audio. Simulate conversations between Realtime clients to scale testing.

[12:54](https://www.youtube.com/watch?v=-OXiljTJxQU&t=774s) [Slide lists observability, transcription evaluations, audio evaluations and synthetic conversations alongside a trace interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/9741df493d45a17fd97fe21fbbe597b5fd067bc3a636e745adbbe375fdd8c09f.webp)

Evaluation

Observability and Annotation
Start with looking at traces and SME’s annotating calls to then iterate on prompts

Transcription Evals
Traditional LLM-as-a-judge evals on function performance and business criteria

Audio Evals
Use gpt-4o-audio to understand tone, pacing, etc.

Synthetic Conversations
Simulate back and forth between Realtime clients to scale testing

21

Interface text:
Traces / Realtime Session
Debug  Refresh
Realtime Session
Audio In
Audio Out
Audio In
Audio Out
transfer_to_Weather_Expert
Audio Out
Audio In
weather
Audio Out
Audio In
Audio Out
Audio In
Audio Out

Audio In
Input Audio
Properties
First content at: Jun 3, 2025, 3:38 PM
Model: gpt-4o-realtime-preview
Input
Transcript configuration
Model: gpt-4o-mini-transcribe
[illegible]

### Guardrails

Generated text arrives faster than speech, giving asynchronous guardrails time to run. The SDK’s 100-character interval is configurable; the example sets it to 200 characters.

[15:05](https://www.youtube.com/watch?v=-OXiljTJxQU&t=905s) [Slide describes asynchronous guardrails and configurable settings, with a code example.](https://assets.ai.engineer/sites/aiecode2025/sha256/23dac1603e3aa3fd37a54482d5c1de40a0e952cb7155bc1bd704e922df4eef9a.webp)

Guardrails

Run async
Define the set of rules and then run them as a response is generated. Generated text is faster than spoken speech

Settings control
In the Agents SDK run every 100 characters, but flexible

const guardrails: RealtimeOutputGuardrail[] = [
  {
    name: 'No mention of Dom',
    execute: async ({ agentOutput }) => {
      const domInOutput = agentOutput.includes('Dom');
      return {
        tripwireTriggered: domInOutput,
        outputInfo: {
          domInOutput,
        },
      };
    },
  },
];

const session = new RealtimeSession(agent, {
  outputGuardrails: guardrails,
  outputGuardrailSettings: {
    debounceTextLength: 200 // run every 200 characters and the end
  }
});

22

### Takeaways

Audio joins text, image and video in the multimodal era. As real-time speech-to-speech technology emerges, the slide urges builders to get started.

[15:51](https://www.youtube.com/watch?v=-OXiljTJxQU&t=951s) [Slide highlights multimodal AI, emerging real-time speech-to-speech technology and the opportunity to build.](https://assets.ai.engineer/sites/aiecode2025/sha256/588ef64f6b857ad68fb1f187059524c80bad59152b9293c20172289b587bc22c.webp)

Takeaways

We’re entering the multimodal era

Real-time speech-to-speech
Audio AI technology is emerging

First movers advantage:
Get building!

## Full transcript

[0:00](https://www.youtube.com/watch?v=-OXiljTJxQU&t=0s) [on-hold music] Hello, everyone.

[0:15](https://www.youtube.com/watch?v=-OXiljTJxQU&t=15s) Thanks for coming to, to our talk here. Um, just to briefly introduce myself, I'm Toki. I lead the solution architecture team, uh, in the US for OpenAI, and I have Anoop here as well on the solution architecture team.

[0:26](https://www.youtube.com/watch?v=-OXiljTJxQU&t=26s) Uh, and we'll talk about building practical audio agents. Um, before I jump in, maybe a quick raise of hands. Who has either, like, built an audio agent or voice agent of some kind, or thinking about building an audio agent?

[0:39](https://www.youtube.com/watch?v=-OXiljTJxQU&t=39s) Okay, so like half-ish of the room. Okay, good to know. It's always good to know what, what people's interests are. So yeah, we'll talk a bit about, like, um, some of the things we, we've learned, where audio models are today, and then some of the kind of patterns and best practices.

[0:50](https://www.youtube.com/watch?v=-OXiljTJxQU&t=50s) So just to take a quick step back, um, gen AI applications broadly to date, as many of you all know, have been very text-based, right? This is, uh, models that really deeply understand, uh, and generate text with very much human-like fluency, and there's a lot of production applications out there built on text, and that's continues to scale.

[1:07](https://www.youtube.com/watch?v=-OXiljTJxQU&t=67s) I think we've really started to enter a bit more of this multimodal era where we now have models that can understand images extremely well. There's now models that can generate images really well.

[1:17](https://www.youtube.com/watch?v=-OXiljTJxQU&t=77s) If you were part of the ImageGen Studio Ghibli craze, uh, or Sora generates videos, for example. Um, and then now also audio. These are models-- We've had models like transcription models or text-to-speech models for some time, but now this kind of emerging speech-to-speech capability is getting a lot better.

[1:33](https://www.youtube.com/watch?v=-OXiljTJxQU&t=93s) And I would say now we, we will see, I think, a much more of an explosion of use cases around these modalities of images and video. And today we're really gonna focus on the audio modality, uh, across this.

[1:42](https://www.youtube.com/watch?v=-OXiljTJxQU&t=102s) So I, I would say, like, one important thing we're starting to see is that audio is really getting to that point where it's becoming much more usable at scale.

[1:51](https://www.youtube.com/watch?v=-OXiljTJxQU&t=111s) And again, we've had the transcription and the TTS side, but especially that speech-to-speech low latency experience. The models are getting, I would say, good enough and are kind of at this tipping point where you can start to m- really meaningfully build high-quality production applications at scale.

[2:04](https://www.youtube.com/watch?v=-OXiljTJxQU&t=124s) So I'd say before, the models were a lot slower, robotic, and brittle, and, and now they're a lot faster, expressive, and accurate. I'll show an example. The first one will show what the models were like not even six months ago.

[2:15](https://www.youtube.com/watch?v=-OXiljTJxQU&t=135s) It's a very painful example, so I hope you feel a bit of the pain, uh, in this.

[2:20](https://www.youtube.com/watch?v=-OXiljTJxQU&t=140s) Hello.

[2:23](https://www.youtube.com/watch?v=-OXiljTJxQU&t=143s) Hello. Please give me a topic for your haiku.

[2:25](https://www.youtube.com/watch?v=-OXiljTJxQU&t=145s) Uh-

[2:28](https://www.youtube.com/watch?v=-OXiljTJxQU&t=148s) Hmm. Well, you can see the, the initial delay. Um, that's actually honestly what the model-

[2:33](https://www.youtube.com/watch?v=-OXiljTJxQU&t=153s) Could you write me a haiku about whispers?

[2:40](https://www.youtube.com/watch?v=-OXiljTJxQU&t=160s) Silent air, faint words.

[2:42](https://www.youtube.com/watch?v=-OXiljTJxQU&t=162s) Actually, could you write me a haiku about stars?

[2:43](https://www.youtube.com/watch?v=-OXiljTJxQU&t=163s) Whispers drift through darkened rooms. Secrets linger near.

[2:52](https://www.youtube.com/watch?v=-OXiljTJxQU&t=172s) Dust slides through my hands. Sifted grains forget the weight. What is left behind?

[2:58](https://www.youtube.com/watch?v=-OXiljTJxQU&t=178s) So I hope you felt the pain a little bit, right? So these models are, uh... This is actually not just-- not even six months ago, right? The models are incredibly slow.

[3:06](https://www.youtube.com/watch?v=-OXiljTJxQU&t=186s) Um, they're very robotic in how they sound, uh, and they're quite brittle. So in that example, we actually tried to interrupt the model and tell it, "No, don't write a, don't write a poem about, uh, whispers.

[3:16](https://www.youtube.com/watch?v=-OXiljTJxQU&t=196s) Write one about stars." It wrote one about sand. I'm not sure where it got that from. So just showing you that's actually where it was. And then now if you, like, fast-forward to today, these are the models that, uh, for a very similar use case, kind of how they-- how they'll behave today.

[3:28](https://www.youtube.com/watch?v=-OXiljTJxQU&t=208s) Hello.

[3:30](https://www.youtube.com/watch?v=-OXiljTJxQU&t=210s) Greetings. Share with me a topic, and I shall weave a haiku for you, rich with emotion and imagery. What shall it be?

[3:42](https://www.youtube.com/watch?v=-OXiljTJxQU&t=222s) Can you make a haiku about whispers?

[3:45](https://www.youtube.com/watch?v=-OXiljTJxQU&t=225s) Whispers-

[3:46](https://www.youtube.com/watch?v=-OXiljTJxQU&t=226s) Actually, could you make a haiku about stars?

[3:49](https://www.youtube.com/watch?v=-OXiljTJxQU&t=229s) Stars, whispers of light, adorning the velvet night, eternal secrets.

[3:58](https://www.youtube.com/watch?v=-OXiljTJxQU&t=238s) So you can see it's much faster. It's incredibly expressive, emotional. You can steer it. You can get it excited. You can slow it down, and it's accurate. We interrupted it, and we had it change course and do something else, and it responded well.

[4:08](https://www.youtube.com/watch?v=-OXiljTJxQU&t=248s) Um, so the models are really, we do believe, at that good enough tipping point where you can start to build much more reliable experiences around it. Um, in terms of actually building on top of the, uh, audio models, there's really two emerging architectures that we see.

[4:22](https://www.youtube.com/watch?v=-OXiljTJxQU&t=262s) One is kinda the one that's been around for a bit, and then I'd say a little bit of the newer one, which I'll talk about. The existing one has been what we call kinda the chained approach.

[4:29](https://www.youtube.com/watch?v=-OXiljTJxQU&t=269s) This is s-stitching together these three approaches. You have audio come in, you transcribe it. You have some sort of intelligence layer with an LLM that does some sort of intelligence, and then finally, text-to-speech on the output.

[4:40](https://www.youtube.com/watch?v=-OXiljTJxQU&t=280s) Um, but again, this is stitching together three models. There's a lot of problems with this. It's slow across ti-- Again, it takes longer time to actually generate your output audio.

[4:48](https://www.youtube.com/watch?v=-OXiljTJxQU&t=288s) Um, it also has some lossiness across this, right? You kind of lose a little bit of the semantic understanding of the conversation. So really, this emerging pattern is more the speech-to-speech architecture, where it really, it takes these three models into a, um, single model, which for us it's the Realtime API.

[5:03](https://www.youtube.com/watch?v=-OXiljTJxQU&t=303s) Uh, and it does all these layers. It does the transcription, it does that intelligence layer, and then finally does the speech as the output. So this simplifies the architecture quite dramatically, really reduces the time it takes to output.

[5:14](https://www.youtube.com/watch?v=-OXiljTJxQU&t=314s) This is really good for low latency experiences. Um, and it doesn't have that lossiness problem. You actually are able to maintain semantic understanding across a conversation. So this is really kind of the emerging pattern we see.

[5:25](https://www.youtube.com/watch?v=-OXiljTJxQU&t=325s) And when building, there's some kind of key trade-offs and considerations to take into account, um, when building on top of these architectures, and we think it's kind of across these five main areas.

[5:35](https://www.youtube.com/watch?v=-OXiljTJxQU&t=335s) You have latency, you have cost, you have accuracy and intelligence, user experience, and integrations and tooling. And depending what you're building, your trade-offs will be different, right? So if, for example, you're talking about a consumer-facing application, um, the trade-offs are, are quite different.

[5:49](https://www.youtube.com/watch?v=-OXiljTJxQU&t=349s) Here... You care a lot about user experience. Um, these are usually a low latency experience that, um, a lot of end users are, are interacting with the model, so user experience matters a lot.

[5:59](https://www.youtube.com/watch?v=-OXiljTJxQU&t=359s) Latency matters a lot. It can't be slow. It needs to be really fast and interactive. But things like cost probably doesn't matter as much. It doesn't need to be accurate, not really, right?

[6:06](https://www.youtube.com/watch?v=-OXiljTJxQU&t=366s) It's more about expressiveness and quick answers. Um, and integrations and tooling, like you don't really need to integrate with, uh, SIP or like Twilio for, for a broader experience.

[6:15](https://www.youtube.com/watch?v=-OXiljTJxQU&t=375s) It's a pretty simple, um, uh, interaction. So things like this would really-- the Realtime API works really well for a consumer-facing application depending on these trade-offs. Now, if you go to customer service as another example, the trade-offs are different, right?

[6:27](https://www.youtube.com/watch?v=-OXiljTJxQU&t=387s) So here, accuracy and intelligence actually probably matters the most. You can't get an order number wrong. If it, uh, goes to delete an order instead of updating an order, that's a big deal.

[6:36](https://www.youtube.com/watch?v=-OXiljTJxQU&t=396s) So accuracy and intelligence matters probably the most. Integrations and tooling start to matter a lot more too, because now you have to integrate your, with your various internal systems.

[6:43](https://www.youtube.com/watch?v=-OXiljTJxQU&t=403s) You have to integrate with, yeah, SIP and phone providers if needed. Um, so those are probably the most important things. But user experience still matters, but you'll probably take a hit on that a little bit to make sure it's more accurate.

[6:53](https://www.youtube.com/watch?v=-OXiljTJxQU&t=413s) Again, latency still matters, but not as much because accuracy and intelligence matters more. And cost, um, you're saving a, a large cost for this anyway. So I think these are just the other considerations to, um, to take into account for customer service.

[7:04](https://www.youtube.com/watch?v=-OXiljTJxQU&t=424s) So for this, Realtime API might make sense for, uh, a situation where latency might matter the most. But if you care about determinism and accuracy and intelligence, then actually the chained architecture would make a lot of sense here too.

[7:16](https://www.youtube.com/watch?v=-OXiljTJxQU&t=436s) Um, but it's just important to consider what your trade-offs are depending on what you're trying to build. So hopefully that gives you-- paints you a picture a bit of like what to consider.

[7:23](https://www.youtube.com/watch?v=-OXiljTJxQU&t=443s) I'll hand it off to Anoop to talk about very specific patterns we've seen when working with customers building on this.

[7:27](https://www.youtube.com/watch?v=-OXiljTJxQU&t=447s) Cool. Awesome, Toki. Um, so maybe just like taking a step back, I think we've probably defined agent a bunch of times, but here's just like a cano- a canonical definition to hopefully keep everyone grounded for this session.

[7:38](https://www.youtube.com/watch?v=-OXiljTJxQU&t=458s) Um, think of an agent as some model, some set of instructions, so your prompts, the tools you give your model, and then the runtime, so like the guardrails, like how you do your execution environment.

[7:47](https://www.youtube.com/watch?v=-OXiljTJxQU&t=467s) Um, and generally, like when you're building voice agents, that's all the same things you should be worrying about, but there's also a few other things that are maybe a bit different that we're gonna focus on today.

[7:57](https://www.youtube.com/watch?v=-OXiljTJxQU&t=477s) Uh, so the main stuff is like your system design is probably a bit different than a traditional LLM agent. Um, it's probably a bit different on the prompting side and customization of the voice.

[8:06](https://www.youtube.com/watch?v=-OXiljTJxQU&t=486s) Um, the tools you're using, probably a bit different. Um, and then also evals and guardrails you might have to think about, um, as well. So these are the things we'll, we'll probably focus on that are like distinct when you're building specifically voice agents compared to like traditional agents.

[8:21](https://www.youtube.com/watch?v=-OXiljTJxQU&t=501s) Cool. So if you've all built, uh, maybe like a text-based agent, let's use like customer service as a canonical example. Um, you have like some user question. You have some sort of triage agent that's like a small model that it goes to other models here that will do like the harder tasks.

[8:36](https://www.youtube.com/watch?v=-OXiljTJxQU&t=516s) So you'll have like an o3 to do like the refund or maybe like an o4-mini to do some of the cancellation. Um, and typically that's like how you build a, a normal agent today.

[8:45](https://www.youtube.com/watch?v=-OXiljTJxQU&t=525s) Uh, for voice, one pattern that we're starting to see more and more of is some sort of like delegation through tools. So you would have like the Realtime API be your like frontline agent, so it does a lot of the normal interaction patterns that you're used to.

[8:57](https://www.youtube.com/watch?v=-OXiljTJxQU&t=537s) So it can respond to like the easy questions that you sort of want your model to answer and you know you're confident in your model answering. Um, but then there's also other questions that might be a bit harder, um, and then you want your model to like do a tool call to these other agents that are run

[9:10](https://www.youtube.com/watch?v=-OXiljTJxQU&t=550s) by smarter models behind the hood. Um, so that's a, a pattern that we'll show a quick video of, of like how this would work.

[9:21](https://www.youtube.com/watch?v=-OXiljTJxQU&t=561s) I've authenticated you now. What item would you like to return?

[9:25](https://www.youtube.com/watch?v=-OXiljTJxQU&t=565s) Yeah. I wanna return the snowboard I bought.

[9:29](https://www.youtube.com/watch?v=-OXiljTJxQU&t=569s) Got it. I'm reviewing the policy and will get back to you with next steps shortly.

[9:34](https://www.youtube.com/watch?v=-OXiljTJxQU&t=574s) So here's a scenario where we're actually using like a much smarter model, o4-mini, uh, and delegating to that because we don't want, uh, a smaller model like the Realtime API to handle that.

[9:41](https://www.youtube.com/watch?v=-OXiljTJxQU&t=581s) It looks like the return you are making is within the thirty-day return policy. We will process the return and send you a confirmation email with next steps shortly.

[9:53](https://www.youtube.com/watch?v=-OXiljTJxQU&t=593s) So that's like one pattern, uh, we've seen on the architecture side. Another point I touched on is like the, the way you basically customize your brand and what you're sort of building.

[10:02](https://www.youtube.com/watch?v=-OXiljTJxQU&t=602s) Uh, so the prompts you're using is a huge part of that. And the way to think about this is when you're prompting models today, uh, in the text-based world, you only can control like the instructions you're giving it and sort of what you want the model to do.

[10:14](https://www.youtube.com/watch?v=-OXiljTJxQU&t=614s) But in voice-based applications, you can also control the expressiveness of the voice and how you want the model to speak or sound. Uh, so this is like an example prompt here of, hey, you can control like the demeanor, the tone, the level of enthusiasm, all of those things that don't really get captured in a, in a text

[10:30](https://www.youtube.com/watch?v=-OXiljTJxQU&t=630s) agent that you do have much more control over in a voice-based agent.

[10:36](https://www.youtube.com/watch?v=-OXiljTJxQU&t=636s) Um, here's just a, a fun website you can use to play around with this, openai.fm, um, where you can play around with different voices and also click through just sample prompts that we've created.

[10:45](https://www.youtube.com/watch?v=-OXiljTJxQU&t=645s) Uh, this is just a great place to get started on like actually playing around with the expressiveness of voices and seeing that there is a ceiling and like a lot of fluctuation you can get in performance just by the prompts that you do use.

[10:58](https://www.youtube.com/watch?v=-OXiljTJxQU&t=658s) Uh, another side of this is when you're, when you're prompting models in the, the text-based approach, you probably have like a lot of few-shot examples of like, "Hey, I want you to do these few steps, uh, and then do these next few steps," um, and go through like, uh, a typical, uh, conversation flow.

[11:15](https://www.youtube.com/watch?v=-OXiljTJxQU&t=675s) Uh, the same thing you can sort of mimic in voice. So if you see in, in this picture, uh, there's like some idea of, "Hey, I want you to do some greeting," the description of it, and then the instructions for the users.

[11:26](https://www.youtube.com/watch?v=-OXiljTJxQU&t=686s) Um, and you can just go through that. So this is like a sample way of you to think through how you should be prompting models, especially for more complex tasks that you want the Realtime API to handle.

[11:37](https://www.youtube.com/watch?v=-OXiljTJxQU&t=697s) Uh, the next big bucket of things that are sort of important or maybe like a bit different for the voice agents you're building is tools. Tool use in general is like very- Uh, important because you want your model to connect to oth-other things that are sort of important for you.

[11:52](https://www.youtube.com/watch?v=-OXiljTJxQU&t=712s) Um, and the first step is honestly just start simple. You don't need to connect 10 tools or, like, 20 tools to a given model. Uh, keep the number of tool calls pretty limited for a given agent.

[12:02](https://www.youtube.com/watch?v=-OXiljTJxQU&t=722s) Uh, and that maybe goes back to the first diagram we, we showed of you wanna delegate to different models or agents to do different tasks. So here, a given agent, don't give it a ton of tools to start with, and then slowly add on more tools as needed.

[12:15](https://www.youtube.com/watch?v=-OXiljTJxQU&t=735s) Um, handoffs, so if you ever u- uh, use the Agents SDK, this is a concept of [chimes] if we have a given model, um, you wanna pass it to another.

[12:23](https://www.youtube.com/watch?v=-OXiljTJxQU&t=743s) Um, and here you can manage context between handoffs as well. So the thing to re-remember here is you might have a given agent, and you're going to another one, but you wanna keep the context the same between them.

[12:36](https://www.youtube.com/watch?v=-OXiljTJxQU&t=756s) Um, so a good pattern to do is actually summarizing the conversation state and then passing it to the other agent so you don't lose any of the, the context, and it's not lossy.

[12:44](https://www.youtube.com/watch?v=-OXiljTJxQU&t=764s) Uh, the final one is delegation, so the concept we showed a bit earlier of, like, using o3-mini or o4-mini to do the tool call. That's, like, natively built into the Agents SDK if you'd like to try it out.

[12:56](https://www.youtube.com/watch?v=-OXiljTJxQU&t=776s) Um, evals. So I think this is a pretty big part of the, like, voice experience, and maybe, like, breaking it down into, like, four key buckets. I think the first and most obvious thing is always start with observability.

[13:12](https://www.youtube.com/watch?v=-OXiljTJxQU&t=792s) Um, you're not gonna get very far if you don't know how your data looks. Um, so just make sure you have traces of what's going on with your agents, uh, the audio and everything, so you can actually look at it and take actions.

[13:25](https://www.youtube.com/watch?v=-OXiljTJxQU&t=805s) Um, and the next thing is when you're actually starting to do evals, like, the most important thing is actually have SMEs to label these data, um, and actually go and iterate on prompts from there.

[13:35](https://www.youtube.com/watch?v=-OXiljTJxQU&t=815s) It sounds maybe a bit too easy or it's, like, not the most appropriate or, uh, maybe the best approach, but it is the most effective that we've seen when working with customers.

[13:46](https://www.youtube.com/watch?v=-OXiljTJxQU&t=826s) Um, it lets you go a lot faster. Uh, and then from there, the next step is transcription-based evals. So this is where you would have your traditional, LLM-as-a-judge evals or, like, testing your function calling performance on certain business criteria or having a rubric in place.

[14:00](https://www.youtube.com/watch?v=-OXiljTJxQU&t=840s) Um, and then there's this next bucket, uh, where a lot of people ask us: How do we actually test the audio that's generated? Um, and for the most part, it's-- There's not a lot of things that audio, uh, evals get you that maybe text evals don't.

[14:17](https://www.youtube.com/watch?v=-OXiljTJxQU&t=857s) But the things that maybe it does get you is, like, understanding the tone, understanding the pacing or other things that are maybe a bit harder to capture just through text.

[14:25](https://www.youtube.com/watch?v=-OXiljTJxQU&t=865s) And this is where you can use, like, a completions model or some other audio model, uh, so for example, GPT 4 Audio, to, like, understand the tone, the pacing and what other intonations that you care about for your business.

[14:36](https://www.youtube.com/watch?v=-OXiljTJxQU&t=876s) Um, and then the final one that we've started to see a bit more is actually synthetic conversations. So for example, maybe you have two realtime clients going back and forth across to each other.

[14:46](https://www.youtube.com/watch?v=-OXiljTJxQU&t=886s) So maybe one is your Realtime API agent or whatever, and then the other is a set of customer personas that you have. And you can s-simulate, like, many, many of those conversations and then extract those evals across maybe, like, transcriptions or audio and then have a much better way to understand what you've sort of built.

[15:06](https://www.youtube.com/watch?v=-OXiljTJxQU&t=906s) Cool. Uh, and then the final thing here is guardrails. Um, because especially for the Realtime API, it's pretty fast. Uh, you want to have guardrails that make sure that you have the safety that's needed f- uh, to actually have confidence in, in the solution you're, you're rolling out to users.

[15:23](https://www.youtube.com/watch?v=-OXiljTJxQU&t=923s) Um, so the main things we have here is, like, run them async, uh, generally because the generated text from the Realtime API is much faster than the spoken audio that you get back from the model.

[15:34](https://www.youtube.com/watch?v=-OXiljTJxQU&t=934s) So you have room to play with the latency here. Uh, the next thing also is you have control to set, like, whatever debounce period you have here. So in the code snippet, uh, to the right-hand side of the screen, you'll see that, like, right now it runs every 100 characters to run the guardrails, but you can control

[15:49](https://www.youtube.com/watch?v=-OXiljTJxQU&t=949s) that if you want to. Awesome. And then maybe just some learnings that we've seen from other folks in the field. Um, Lemonade, so, uh, AI, like, insurance company, they're, like, building e...

[16:02](https://www.youtube.com/watch?v=-OXiljTJxQU&t=962s) One of the things we, like, noticed from them that made them successful is they, like, focused really early on evals, guardrails and, like, feedback mechanisms for their team. And even if it wasn't scalable, they realized that, and it actually made them move a lot faster in the end.

[16:14](https://www.youtube.com/watch?v=-OXiljTJxQU&t=974s) Um, and then finally for Tinder, um, they realized that customization and brand realism was pretty important for them when they were launching their Rizz Chat experience. Um, so that's what they focused on, and it created a much better experience for their users.

[16:28](https://www.youtube.com/watch?v=-OXiljTJxQU&t=988s) And now I'll hand it off to Toki to close us out. Cool. Yeah, just to wrap this all up. So I kind of-- What I said in the beginning is we're really entering this multimodal era, so away from or in addition to text-based models we have now obviously, um, video and image models but now audio, so very

[16:42](https://www.youtube.com/watch?v=-OXiljTJxQU&t=1002s) much in the multimodal era. And this realtime speech-to-speech technology is emerging, and I think we had a kind of low-key update yesterday, our Realtime API. We had a new snapshot that we, that we released that's actually a lot better.

[16:53](https://www.youtube.com/watch?v=-OXiljTJxQU&t=1013s) So I really feel like now is the time to, to build. So you kind of have this first mover's advantage to, to build now. I think the technology is getting to that point where it is good enough to build scalable production applications.

[17:04](https://www.youtube.com/watch?v=-OXiljTJxQU&t=1024s) Um, so really excited to see what you all build. Please test it out. Um, we'll be around after the talk if you guys have any questions, but thanks for the time. [outro music]
