AI Engineer World's Fair 2025

Building voice agents with OpenAI

Loading slides…

Read the talk

Building voice agents with OpenAI

Build a browser voice agent step by step, from typed tools and secure sessions to interruptions, specialist handoffs, reasoning delegation, approval gates, and audio-aware evaluation.

From a talk by Dominik Kundel

Before you start: Familiarity with TypeScript, async functions, React refs, and basic client–server authentication will help you follow the coding examples.

What does a voice agent need to accomplish a task?

What turns a conversation with a model into a system that can accomplish a task for someone? Start with a model, give it instructions and tools, and put those pieces inside a runtime that manages their lifecycle. That is the working definition behind Dominik Kundel’s workshop. Participants begin by downloading the starter repository’s dependencies before coding: even a small application needs preparation when everyone shares the venue’s internet connection.

“What is an agent?” slide with a definition and a diagram connecting user input, a text-based model, tools, and an agent response.Open full source frame
An agent combines a model, instructions, tools, and a runtime.

The workshop coincides with the June 3, 2025 launch of the TypeScript Agents SDK, the counterpart to the Python SDK. Its primitives include handoffs, guardrails, streaming, tools, MCP support, and tracing, plus human approval with resumability. The examples here follow that launch-era workshop; later API distinctions appear where they affect implementation.

Native voice support applies those same primitives to a conversation that is continuously receiving and playing audio. The runtime tracks history, supports audio replay in traces, executes tools, handles handoffs and output guardrails, and manages interruptions. It supports WebSockets for server integrations such as telephone agents and WebRTC for browser applications—the transport used in this workshop.

0:260:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:26 · section reference included

Choose the path from speech to action

Voice can make software accessible to people who would struggle with its conventional interface. It also carries tone and emotion alongside words. A voice agent can even call a business that has no software API, using the conversation itself to get something done. Those benefits lead to an architectural choice: wrap a text agent with speech models, or let a model work directly with audio.

In a chained architecture, speech-to-text converts the user’s audio into text, a text agent processes it, and text-to-speech reads the result. This makes an existing text agent easy to reuse and leaves the application free to choose among text models. It also exposes the exact text entering and leaving the agent. The difficult part is coordinating the stages: every stage adds latency, transcription loses some audio context, and an interruption requires reconciling the generated text with what the user actually heard. Otherwise, the agent may believe it delivered information that never reached the listener.

A speech-to-speech architecture consumes audio directly and can generate speech and tool calls without transcription as an intermediate reasoning step. Removing the separate speech recognition and synthesis stages reduces the latency those stages introduce and preserves access to tone. The tradeoff in the workshop’s models is that complex decision-making and reuse of specialized text agents are harder.

ArchitectureMain advantageMain engineering burden
Speech → text agent → speechReuse and inspect text workflowsCoordinate latency, playback, and history
Native speech-to-speechDirect audio understanding and deliveryScope complex reasoning carefully
Speech-to-speech with delegationCombine conversation with specialist reasoningDesign the tool boundary

Delegation adds a third option: keep a frontline agent talking to the user, and give it tools that invoke reasoning models such as o4-mini or o3. The diagram’s refund, order-cancellation, and product specialists sit behind that tool boundary.

Delegation diagram linking user audio to a realtime frontline agent and branching tool calls to refund, order cancellation, and product expert agents.Open full source frame
A frontline voice agent delegates through tools to specialist agents.
3:263:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:26 · section reference included

A fluent refund answer still needs inspection

The first live conversation starts with a weather request. The agent announces a transfer to a weather expert, then says it can help directly and asks for a location. Kundel supplies San Francisco while demonstrating interruption handling, and the agent responds that it is sunny. Even this simple exchange exercises routing language, a missing argument, and overlapping speech.

The next request needs a backend decision. Kundel describes a skateboard he tried using, discovered he was bad at riding, and now wants to return despite some scratches. A refund tool is intended to invoke o4-mini, but the initial request gets a referral to customer service. He tries again with an explicit refund request; repeated greetings, network trouble, and room echo complicate the exchange. Eventually the agent announces an assessment and approves a full refund, explaining that the skateboard arrived damaged. That explanation changes the facts: the customer described scratches after trying the skateboard, not damage on arrival. The demonstration shows delegation taking place, but the spoken approval does not establish that the refund decision was correct or that a real payment was issued.

Kundel reports that the o4-mini call finished while the agent’s spoken acknowledgement was still playing. That overlap is useful: generated speech takes time to play, and the application can use that interval for backend work. A short announcement of the next task can reduce perceived waiting without requiring the voice model itself to perform the reasoning.

The traces interface provides a way to inspect what happened beneath the answer. Kundel opens the voice trace, checks tool inputs and outputs, replays audio, and follows the execution into the o4-mini agent. That backend agent received the prior conversation transcript plus request-specific information. Because both sides use the Agents SDK, the trace connects the conversational frontend to the reasoning backend—the path needed to investigate a mismatch such as the arrival-damage rationale.

8:168:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:16 · section reference included

Scope the task, then design the conversation

Before writing the application, define a small goal and give the agent only the tools needed to achieve it. Evaluating a text agent is already difficult; voice adds delivery and timing to the problem. Once the first use case works, additional specialists and handoffs can expand coverage, while unsupported requests can go to a human. Build evals and guardrails early enough that they guide this expansion. Kundel points to Lemonade’s custom dashboard and conversation replay as an example of inspecting the whole customer experience.

The selected voice is only part of the agent’s personality. Generative speech models also respond to instructions about emotion, role, and delivery. Kundel uses the openai.fm demonstration to illustrate how prompting changes the reading of the same text, rather than treating the voice preset as the entire design surface.

Lemonade’s interface is customer-built, not an embedded OpenAI dashboard. Realtime audio data and events give an application material for its own interface. In the workshop, Kundel says the Traces dashboard has neither an API nor iframe embedding, and transcripts appear only when transcription is enabled; it was absent in the trace he checks during this exchange. Raw event handling, application history storage, and platform tracing are separate concerns, which become explicit in the later history example.

Conversation instructions can also describe states and transitions in JSON, much like a script for a human service agent. That gives the model an explicit process to follow instead of only a personality. Kundel mentions a custom GPT for drafting those scripts and promises the links and slides in Slack. The coding portion then starts from minimal Next.js boilerplate with dependencies already installed; headphones help participants test simultaneously, and Anoop handles technical questions through Slack. The first exercise opens index.ts and imports Agent from @openai/agents.

12:3012:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:30 · section reference included

Build a text agent and give it a typed tool

The first agent needs a name and instructions. The name, My agent, makes it identifiable in traces. Calling run starts the execution loop, including any tool calls and output validation; finalOutput contains the last agent’s output.

typescript

import { Agent, run } from '@openai/agents';

const agent = new Agent({
  name: 'My agent',
  instructions: "You're a helpful assistant.",
});

const result = await run(agent, 'Hello, how are you?');
console.log(result.finalOutput);

Kundel runs this with the starter’s exercise 01 script, correcting his initial generic start command. He describes GPT-4.1 as the default at the time, then shows how setting model: 'o4-mini' changes the model without changing the surrounding execution pattern.

A tool adds an operation the model can request. Its Zod schema serves two purposes: it describes valid arguments to the model and validates the values supplied at execution time. TypeScript can also infer the argument types from that schema.

typescript

import { Agent, run, tool } from '@openai/agents';
import { z } from 'zod';

const getWeather = tool({
  name: 'get_weather',
  description: 'Get the weather for a city.',
  parameters: z.object({ city: z.string() }),
  async execute({ city }) {
    return `The weather in ${city} is sunny.`;
  },
});

const agent = new Agent({
  name: 'My agent',
  instructions: "You're a helpful assistant.",
  tools: [getWeather],
});

const result = await run(agent, 'What is the weather in Tokyo?');
console.log(result.finalOutput);

This weather function returns a fixed sunny response. The useful behavior is the tool round trip: the model selects the tool, supplies Tokyo, receives the result, and uses it in the final answer. Kundel opens the trace to verify each part of that sequence rather than inferring tool use from the answer alone.

19:4620:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:46 · section reference included

Connect a browser agent without exposing the API key

Moving to voice introduces a distinction between agent configuration and session lifecycle. RealtimeAgent is a specialized agent configuration with a more restricted set of fields. RealtimeSession manages the live interaction: incoming audio, model responses, outgoing audio, tools, handoffs, and guardrails. The session does substantially more than run a text agent’s tool loop.

“Your first Realtime Agent” slide with session responsibilities, browser WebRTC support, and a short agent-and-session code example.Open full source frame
Realtime agents run within sessions that manage connections and agent lifecycle.

Exercise 02 puts the interface in a Next.js page.tsx. Because this agent runs in the browser, it must not receive the application’s long-lived OpenAI API key. The workshop’s server-side token.ts creates a Realtime session and returns its short-lived client secret. The browser uses that ephemeral credential for WebRTC. A server-hosted integration, such as a Twilio application, can instead use the regular API key within its server environment. The realtime classes come from @openai/agents/realtime, a subpath of the same package.

The Connect button follows a short sequence:

  1. Obtain the ephemeral credential through a Next.js server action, or an equivalent request to your backend.
  2. Construct RealtimeSession with the initial agent and the chosen realtime model.
  3. Call connect with the ephemeral credential.
  4. Keep the session in a React ref so re-renders do not recreate it.
  5. On Disconnect, close the session and release the connection.

Kundel selects the launch-day model and describes its improved tool calling. The important lifecycle detail is the ref: session.current holds a live connection, not a fresh object to construct on every render.

The connection logic can be kept separate from the framework-specific token endpoint:

typescript

import { useRef } from 'react';
import {
  RealtimeAgent,
  RealtimeSession,
} from '@openai/agents/realtime';

const agent = new RealtimeAgent({
  name: 'Voice Agent',
  instructions: "You're a helpful assistant.",
});

export function useVoiceSession() {
  const session = useRef<RealtimeSession | null>(null);

  async function connect(ephemeralKey: string, model: string) {
    session.current?.close();
    const next = new RealtimeSession(agent, { model });
    session.current = next;
    try {
      await next.connect({ apiKey: ephemeralKey });
    } catch (error) {
      next.close();
      session.current = null;
      throw error;
    }
  }

  function disconnect() {
    session.current?.close();
    session.current = null;
  }

  return { connect, disconnect };
}

The browser test produces a spoken greeting without explicit microphone or speaker setup. The SDK handles those defaults and permits custom audio sources. On a server, sendAudio accepts incoming buffers and audio events expose generated buffers for the application’s own transport.

24:0624:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:06 · section reference included

Track what the listener actually heard

The same weather tool can now be attached to the realtime agent. Zod validation matters particularly in this launch-era example because the realtime model does not support strict mode: validating arguments protects the execution boundary, but does not make generation itself schema-strict. The San Francisco weather request succeeds through the reused tool.

The next test asks the agent to count slowly to twenty. Kundel stops it around six, then asks how far it counted; it answers six. He describes the timing as approximate. The significant behavior is that the agent does not assume the entire planned response was delivered. For a customer-service agent reading a policy, unheard speech must not become assumed shared knowledge.

RealtimeSession handles the interruption events, while transport_event exposes the underlying event stream for logging and custom logic. Applications retain the ability to read and send lower-level events. Kundel demonstrates this in the browser developer tools and pushes the code so participants can pull the updated version.

The interface then renders conversation history. Speech-to-speech still works directly on audio; the transcription runs alongside it. In the workshop SDK, transcription is enabled by default and can be disabled, whereas direct API usage requires enabling it. History contains both messages and tool calls, so a transcript-only interface filters for messages before displaying them. The browser shows the greeting and the San Francisco weather exchange beside the code that performs this filtering.

Code editor beside a browser showing Voice Agent Demo, a Disconnect button, and user and assistant messages with transcript text.Open full source frame
The Voice Agent Demo displays the conversation history, including a weather question and response.

Interruption creates an important limitation in that displayed history: the launch-era implementation removes an interrupted output’s transcript rather than trimming it to the exact words heard. Removing it avoids showing the full generated text as if it had all been spoken. The corresponding trace still helps inspect the conversation and weather call.

32:0432:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:04 · section reference included

Separate displayed history from session context

There are three distinct places to think about conversation data:

  • Application history: An in-memory array on the client, updated from API events. Listening to history_updated lets the application persist its own copy.
  • Platform traces: Stored on OpenAI when tracing is enabled. Kundel clarifies that SDK tracing is enabled by default, can be disabled, and is unavailable to zero data retention customers.
  • Realtime session context: The API’s authoritative conversation state. The client’s displayed history is a copy, not the authority deciding what enters the next model generation.

This distinction prevents a transcript UI from being mistaken for either durable application storage or complete control over the model’s context.

Usage events provide another view of the session. Kundel locates detailed token usage while examining response.done, allowing the application to track consumption without seeing the exact context used for every subsequent generation. The audio-format question identifies PCM16 in the example and μ-law as an option useful for telephone audio. A follow-up asks whether usage rolls up intermediate tool calls and where tool-output usage appears. Although Kundel describes tool calls as responses, he leaves the precise accounting question for follow-up.

Adapting an existing text agent to voice requires attention to waiting and scope. The refund example uses a tool to invoke a separate o4-mini agent and return its result. Prompting the frontline agent to announce its next task allows playback to overlap that work; it does not mean a response can absorb arbitrary new information halfway through generation. Interruption is a separate path. Long prompts also make the realtime agent’s decision harder, so split distinct steps into focused agents and handoffs.

When the discussion returns to memory, the visible weather conversation is still just a history view. updateHistory changes session context by sending events that add or delete items in the Realtime API. A previous-item ID can position a new item within the conversation. This supports explicit context editing, but the workshop does not provide a separate advanced long-term memory system.

Voice Agent Demo browser window showing a greeting, a San Francisco weather question, and an assistant response as structured transcript entries.Open full source frame
The demo displays conversation history as user and assistant entries.

For language-learning applications, Kundel says the model can switch languages and that customers use it in that domain, but he offers no established recipe for low-fluency speech, broken grammar, or weak pronunciation. The interruption callback also has a narrower contract than an audience member expects: it does not carry the last transcript. The application can inspect the current history when the callback runs, and tool execution context separately includes history for tools that need conversational information.

38:0338:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

38:03 · section reference included

Keep tools responsive and gate side effects

The shared tool syntax also provides a bridge to backend systems over HTTP. For a slow operation, Kundel recommends a short initial call that starts the job and returns a task ID, plus a separate tool that checks status. This avoids leaving the conversation waiting inside one long tool execution. The voice agent can return to the task when useful instead of holding the entire response open.

needsApproval adds a gate before execution. It can be a predicate for conditional approval or an always-required setting. When the SDK emits tool_approval_requested, the application can inspect the request and explicitly approve it. In the Seattle weather example, Kundel wires up the approval event but always approves; the code does not actually branch on the user’s confirmation result. It demonstrates the interception point, not a finished consent interface.

Approval is an application-side execution boundary. The model requests an operation, and the SDK pauses that operation before side effects occur. It does not automatically ask for spoken consent, and it is not merely an instruction telling the model to behave cautiously. Approval is recorded in local context so the pending operation can proceed. Intercepting before execution avoids having to undo work performed while a confirmation was still unresolved.

49:0149:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

49:01 · section reference included

Switch the active agent, preserve the conversation

A handoff is a specialized tool call that changes the active configuration within the existing session. It replaces instructions and tools so a specialist can take over a focused task. The underlying voice remains fixed once the first agent has spoken, even if another agent configuration specifies a different voice.

For the weather specialist, move getWeather off the initial agent and give it to a new RealtimeAgent. A handoffDescription tells the initial agent when the specialist is useful; handoffs makes that destination available.

typescript

import { RealtimeAgent, tool } from '@openai/agents/realtime';
import { z } from 'zod';

const getWeather = tool({
  name: 'get_weather',
  description: 'Get the weather for a city.',
  parameters: z.object({ city: z.string() }),
  async execute({ city }) {
    return `The weather in ${city} is sunny.`;
  },
});

const weatherAgent = new RealtimeAgent({
  name: 'Weather Agent',
  instructions: 'Help with weather questions. Speak with a New York accent.',
  handoffDescription: 'This agent is an expert in weather.',
  tools: [getWeather],
});

export const voiceAgent = new RealtimeAgent({
  name: 'Voice Agent',
  instructions: "You're a helpful assistant.",
  handoffs: [weatherAgent],
});

Kundel asks for New York’s weather. The session transfers to the specialist, which answers with an attempted New York accent. Whether the agent announces a transfer is a prompting choice; forcing an announcement every time can make the interaction awkward.

Code editor showing a Weather Agent with instructions and a weather tool, followed by a Voice Agent whose handoffs include the weather agent.Open full source frame
The voice agent configuration includes a handoff to a weather specialist.

A weather handoff changes configuration within the same session

Constructed example: Instruction wording and object labels match the article's illustrative code; the user request removes spoken hesitations. No particular voice preset is invented.

User request — unchanged
Can you tell me the weather in New York?

Operation: Execute transfer_to_weather_agent within the existing Realtime session.

Current user request

Before: Before the handoff
Can you tell me the weather in New York?
After: After the handoff · Unchanged
Can you tell me the weather in New York?

Active agent

Before: Before the handoff
Voice Agent
After: After the handoff · Changed
Weather Agent

Active instructions

Before: Before the handoff
You're a helpful assistant.
After: After the handoff · Changed
Help with weather questions. Speak with a New York accent.

get_weather

Before: Before the handoff
Not present
After: After the handoff · Added
Available to the active agent; city is a string.

Session voice

Before: Before the handoff
Voice established by the first agent's audio
After: After the handoff · Unchanged
Voice established by the first agent's audio
The weather specialist receives control and its tool without starting a new conversation or changing the established voice.

The follow-up distinguishes a voice preset from prompted delivery. Accent, pronunciation, and tonality can change with instructions while the underlying voice stays the same. The trace shows transfer_to_weather_agent and the new accent instruction; the tool set changes too. Kundel recommends expressing such tool changes through separate agents. He has not tested mid-session speed changes and does not know the complete set of protected settings for scenarios such as automatic noise-reduction adjustment. Unsupported updates produce API errors.

There is also a language-versus-SDK distinction: at the workshop, the Python Agents SDK’s voice support uses the chained approach, while direct Realtime API calls from Python are a separate integration path. The current Python realtime guide now documents a realtime SDK layer as well.

54:0054:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

54:00 · section reference included

Delegate a riddle to a server-side reasoning agent

Delegation keeps the frontline agent in place and calls another agent for a result. Kundel creates a server-side text agent called Riddler, using o4-mini with instructions to generate riddles for a target demographic and topic. It needs no tools of its own. An exported runAgent function accepts those inputs and returns the agent’s output. In Next.js, 'use server' marks the server-action file.

typescript

'use server';

import { Agent, run } from '@openai/agents';

const riddler = new Agent({
  name: 'Riddler',
  model: 'o4-mini',
  instructions:
    'You are excellent at creating riddles based on a target demographic and topic.',
});

export async function runAgent(demographic: string, topic: string) {
  const result = await run(
    riddler,
    `Create a riddle for ${demographic}. Topic: ${topic}.`,
  );
  return result.finalOutput;
}

The frontend tool accepts demographic and topic and invokes this server function. Other frameworks can use HTTP or a WebSocket connection to make the same backend boundary explicit.

The voice agent receives the riddle tool and instructions to announce the task naturally, without saying that it is calling a tool. Kundel asks for a riddle for a Star Wars fan. After a pause, he asks whether the agent is still there; it replies that it is working on the riddle, then begins delivering it. Kundel observes that the tool result arrives before the acknowledgement finishes playing. The design moves the reasoning workload to the backend while keeping conversational delivery with the voice model.

If a task needs several specialists, there are two possible orchestration points: ask the model to call multiple tools, or expose one tool that runs multiple agents internally. Kundel recommends the latter when the application already knows the required work, because it reduces the decisions the voice agent must make.

The reassuring response in the riddle demo is not SDK-generated filler. It comes from the Realtime API responding to Kundel’s follow-up. The transcript UI omits tool-call entries, which makes the sequence harder to see: Kundel describes an interrupted generation, a reset transcript, and a subsequent tool call that yields the answer. That explanation does not establish cancellation or rollback of side effects from an already-running backend operation.

The pricing question likewise cannot be reduced to one per-minute figure. Realtime billing combines audio and text tokens; the spoken content, selected models, transcription, and function calls all affect consumption. Kundel characterizes realtime as generally more expensive than a chained speech-recognition, text-model, and speech-synthesis setup, depending on the use case and model choices. He gives no numerical per-minute estimate.

1:00:101:00:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:00:10 · section reference included

Output guardrails race against playback

Realtime output guardrails are an Agents SDK feature. They check output text asynchronously while the response is being produced. The application can configure how often they run or wait until the complete output is available. In this browser application, the checks run on the client, so they should be efficient enough to react while the audio is still playing. A violation interrupts the response.

An asynchronous guardrail cannot guarantee that every forbidden word stays unheard. If the violation is in the first words, playback may reach it before the check can interrupt. Later in a response, generated text may be far ahead of spoken audio, giving the check time to stop delivery before the offending passage.

Request involving the forbidden name DomTiming consequence
Ask the agent to call the user Dom immediatelyThe name may be spoken before interruption
Ask for a story that introduces Dom in the second actThe check may catch the name before playback reaches it

Kundel uses this word check to explain the timing difference, not to demonstrate a complete content policy. outputInfo supplies policy hints explaining the violation or directing a corrective response.

Transcription is configurable rather than restricted to Whisper. The two March models referenced here are GPT-4o mini Transcribe (gpt-4o-mini-transcribe) and GPT-4o Transcribe (gpt-4o-transcribe), identified in the March 2025 audio-model announcement. With the coding walkthrough complete, Kundel points participants back to the starter repository’s resources and the slides and links to be shared in Slack. The remaining questions extend the implementation into deployment and evaluation concerns.

1:08:221:08:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:08:22 · section reference included

What prompting changes—and what belongs to the application

An audience question asks whether applications can steer prompt-cache routing back to the same system to improve latency. Kundel and a colleague indicate that no such controls are exposed at the time. This is an answer about routing controls, not a claim that caching is absent.

Natural delivery gets a more tangible demonstration. Asked about emotion, cadence, and listening signals, Kundel opens the no-code Realtime playground and selects a bored-teenager system prompt. He asks the agent to share the excitement around the launch. It replies, “I guess it's cool, whatever,” and continues with an unenthusiastic response. Kundel points out that its pauses are delivery choices, not pauses caused by waiting for generation. Prompting can shape how the model speaks, including its timing and tone.

The subsequent questions separate several capabilities:

  • Release changes: Kundel describes improved function calling in the new realtime base model and positive alpha-tester feedback, but no new voice activity detection model.
  • Ambient sound: An application can intercept returned audio and overlay its own office ambience or typing sounds.
  • Multiple speakers: The model has no built-in speaker detection in the workshop account, so a multi-person conversation may be difficult.
  • Custom voices: The available choices are supplied presets. Kundel discusses future custom voices in terms of abuse prevention and safeguards, without promising a launch date or user-uploaded voice support.

These are different implementation surfaces: prompt-controlled delivery, application audio processing, and model capabilities should not be treated as interchangeable.

1:12:411:13:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:12:41 · section reference included

Carry context across sessions and respect runtime permissions

For long conversations, an audience member asks whether the thirty-minute session timeout still applies. Kundel only tentatively says he believes it has not changed. His proposed continuation strategy is to retain the transcript or relevant context outside the session, create a new session, and populate it with conversation items through the API. He does not know whether there is a timeout event and refers that question to the API reference. The current Realtime conversation documentation, inspected in August 2026, specifies a sixty-minute maximum; the workshop’s tentative limit should not be used as a current deployment setting.

Tool capabilities depend on where their implementations run. A server tool can do what its server process is allowed to do; a browser tool remains constrained by browser capabilities. A voice agent running locally over WebSockets can read or write files through local tool code, subject to the program’s actual permissions. The model still runs through the remote Realtime API. Local tool execution does not turn the model into an on-device model or grant the program additional system access.

1:18:211:18:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:18:21 · section reference included

Evaluate the audio, then control when it enters the session

Evaluation must exercise more than the transcript. An audience member asks how to build diverse tests for function selection and arguments, including whether text cases should be converted to speech. Kundel recommends Anoop’s companion talk and emphasizes retaining original audio where possible: a speech-to-speech model acts on speech, including information that a transcript may lose. Chained systems make text replay easier because their central agent actually consumes text.

Before customer conversations exist, begin with predetermined examples and human review. Kundel returns to Lemonade’s approach of reviewing calls and testing a known set of cases during development. A clearly bounded task makes this practical: define both what the agent should do and what it should not do, rather than trying to evaluate an unlimited conversational assistant.

Another participant, already using Promptfoo for text-agent tests, proposes having one voice agent converse with another to exercise prompts and examine transcription accuracy. Kundel initially treats this as a plausible approach, mentions customers using models to prompt humans in training scenarios, and then acknowledges an in-room confirmation that Lemonade uses the illustrated approach. The slide pairs evaluation guidance with Lemonade interface examples, including an audio waveform. No automated evaluation implementation is built in this exchange.

Slide titled “Build evals and guardrails early, collect feedback,” with evaluation and feedback bullets beside two Lemonade screenshots, including an audio waveform.Open full source frame
Evaluation guidance pairs human review and transcript scoring with Lemonade interface examples.

The last technical question moves one step earlier in the audio pipeline: how should an agent activate on a wake word? The workshop supplies no built-in wake-word feature or ready-made cost-avoidance pattern. Kundel instead describes an application-controlled integration:

  1. Disable the API’s automatic voice activity detection.
  2. Use a custom detector or a wake-word-capable model to decide when input should be handled.
  3. Send the selected audio to the Realtime API.
  4. Explicitly commit that audio with a commit event.

This leaves the application responsible for deciding when listening becomes a conversational turn, while the Realtime session handles the interaction that follows.

1:20:411:20:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:20:41 · section reference included

Resources

From the talk

  • OpenAI webinar covering voice-agent development, tracing and Lemonade's evaluation workflow.

Updates since the talk

Transcript download
Full transcript

Loading transcript…