How to Build Voice Agents That Don't Fall Apart When You Interrupt - Jesse Hall, LiveKit
Loading slides…
Read the talk
How to Build Voice Agents That Don’t Fall Apart When You Interrupt
Jesse Hall of LiveKit explains how audible latency, asynchronous tools, audio-aware turn detection and whole-conversation benchmarks work together—and follows a hotel receptionist through the interruption that a brittle agent would mishandle.
From a talk by Jesse Hall
At a glance
Ideas worth remembering
Select a complete voice stack for the workflow. A speech-recognition error such as the 3rd becoming the 30th can propagate into a wrong backend action even when the downstream model follows its input.
Measure time to first audible speech and, for the sentence-buffered pipeline described here, time to first sentence. First-byte and first-token metrics can hide the silence a caller experiences.
Let tool work and conversation progress overlap, with a path to verify affected information or hand off when a late tool result fails.
Turn detection must distinguish silence from completion. A pause after “five hundred” must not truncate an intended payment of $555.23.
Hall frames roughly 1.5 seconds as an outer conversational guide and 500–600 milliseconds as a more humanlike response gap. The objective is to fit useful behavior inside that audible budget.
Benchmark simulated conversations against actual backend outcomes and the steps taken, then turn discovered failures into regression scenarios. Compare models and providers under the same agent and task conditions.
An interruption can change the transaction
A caller starts scheduling a payment, then changes the amount or the date halfway through. If the voice agent keeps working from the earlier request, the problem is more serious than an awkward conversation: the system may act on an instruction the caller has already revised. Jesse Hall of LiveKit opens with this distinction between background agents, whose extra processing time can often go unnoticed, and real-time agents, whose mistakes unfold while a person is still talking.
The familiar failure ends with the caller asking for an operator. Speaking over an agent is ordinary human behavior, yet it can derail a system that expects clean, alternating turns. Financial services make the stakes especially clear: keeping up with a correction is part of executing the right transaction, not merely making the voice sound pleasant.
LiveKit supplies an open-source real-time framework and infrastructure for agents that can hear, see and speak. This discussion focuses on voice, where the goal is humanlike behavior within a finite latency budget. Faster responses help, but speed alone cannot decide when to listen, preserve a changed instruction or keep a caller informed while a backend operation runs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The leaderboard cannot hear the whole conversation
A traditional voice agent uses a cascaded pipeline: speech-to-text turns the caller’s audio into text, an LLM prepares the response or action, and text-to-speech turns the reply into audio. Speech-to-speech models offer an alternative that replaces this three-model pipeline. At the time of the recording, Hall treats that alternative as improving but not yet suitable for every use case. Either architecture leaves builders choosing among many models and providers.
The payment-date example shows why selecting each component from a leaderboard is insufficient. The caller says they want to pay on the 3rd. Speech recognition produces the 30th. The LLM then plans for the 30th, and that date enters the system. The downstream model can behave consistently with its input and still produce the wrong result, because the input has already lost the caller’s meaning.
That is the force of Hall’s distinction: leaderboards score components, but the product ships a conversation. A component score does not tell you how a transcription mistake propagates into a backend action. Nor does clean benchmark audio reproduce a caller on speakerphone in traffic with children in the backseat. The useful selection question becomes: which complete stack works for this agent, with these callers and these tasks?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure when the caller hears an answer
A text-to-speech dashboard can report a response before the caller hears anything. Hall gives an example of 210 milliseconds to the first byte. In tests of popular providers, he also found up to three-quarters of a second of silence after the first byte.
Two measurements bring the pipeline closer to the caller’s experience:
- Time to first audio: Measure when audible speech begins, rather than when the provider first sends bytes. Leading silence still counts as waiting to the person on the call.
- Time to first sentence: In the sentence-buffered TTS pipeline Hall describes, the synthesizer needs a complete sentence before it starts speaking. The relevant LLM milestone is therefore the first usable sentence, rather than the first generated token.
This changes how models should be compared. The model that emits its first token fastest need not finish its first sentence fastest. Likewise, a fast first byte can precede a long audible gap. After a caller asks whether a payment went through, two seconds of silence is two seconds without reassurance, regardless of what the component dashboards report.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the conversation back its turn while tools run
Voice agents also need to do work: look up transactions, check payment status and submit disputes. A backend operation lasting 40 seconds or more cannot fit inside a natural conversational pause. A human handling that delay would explain what is happening or ask another useful question while waiting. The agent should be able to do the same.
What code change lets those activities overlap? Inside find_transactions, the tool first runs await ctx.update("Checking transactions."), then waits for transactions.search(query) to return. Hall describes that progress update as handing the mic back to the agent so the conversation can continue while the lookup runs.
The result is a different use of waiting time. The backend can keep working while the agent gives a progress update or gathers the next piece of information. The operation has not become faster. The caller simply no longer has to spend its entire duration in unexplained silence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The hotel receptionist keeps the booking moving
The Gilded Rose Hotel demonstration makes this concrete. A caller asks to book a room for next Wednesday through Friday. The receptionist agent translates that relative request into July 30 through August 1 and asks for confirmation; the caller agrees. It then establishes that two guests will stay. The useful behavior is the explicit date readback before the conversation moves on.
The interruption arrives during the room-type list. While the agent is listing options, the caller selects a king. The agent’s list trails off, and it moves to the next relevant question: city or ocean view? The caller chooses ocean. Rather than requiring the caller to wait through the full list or repeat the selection afterward, the exchange advances with the preference they have already supplied.
The agent then collects and confirms the caller’s name and contact details. The demonstration stops when it asks for a credit card to hold the room, so this exchange does not demonstrate a completed reservation or charge. It demonstrates the conversational path through gathering booking information, including an interruption that does not derail the next question.
The backend behavior explains why the agent can keep moving. Hall describes tool calls saving the information along the way, without making every next question wait for a save acknowledgment. The conversation does not repeatedly stop for the backend to say that one field is saved before collecting another.
Continuing before a save finishes also requires handling a late failure. Hall describes the intended recovery: if a tool returns an error, the agent can return to the affected information and verify it; persistent trouble can trigger a handoff to a human. Those are described recovery paths, not failures exercised in the demonstration.
Collect the next detail without waiting for every save
Read the diagram as text
- Booking detail received. The caller supplies information during the booking conversation.
- Save through a backend tool. backend — The tool works on saving the supplied information.
- Ask the next question. conversation — The conversation proceeds without waiting for the save acknowledgment.
- Revisit affected information. recovery — If the tool returns an error, the agent asks for verification or correction.
- Hand off to a human. recovery — The fallback for continuing trouble.
- Booking detail received → Save through a backend tool: Information sent for saving.
- Booking detail received → Ask the next question: Conversation continues without waiting for save.
- Save through a backend tool → Revisit affected information: If the tool returns an error.
- Revisit affected information → Hand off to a human: If trouble remains unresolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A pause inside an amount is not the end of a turn
Asynchronous tools address waiting on the backend. Turn detection addresses a different decision: whether the caller has finished speaking at all. LiveKit’s detector uses audio cues—including rhythm, pitch and intonation—alongside the transcript. The purpose is to distinguish a completed turn from a pause inside an unfinished thought.
Consider a caller saying they want to pay $555.23, with a two-second pause after “five hundred.” A detector relying on the partial transcript and silence could take $500 as the completed amount and start responding. In Hall’s example, the caller’s inflection signals that more is coming, so the audio-aware detector keeps listening through the rest of the amount. Waiting longer at that moment preserves the instruction instead of making the agent merely slower.
At the same latency budget, Hall reports premature user-cutoff rates of 9.9% for a leading alternative and 4.5% for LiveKit’s detector in their evaluated use cases, including the hotel agent. These are results for those use cases, not universal interruption rates. He describes the evaluations as open source and recommends running them against the data and conversations the deployed agent will actually encounter.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fit the behavior inside the audible latency budget
The end-to-end latency budget runs from the caller’s last syllable to the agent’s first audible response. Within that interval, turn detection decides whether the caller is done, the language model prepares a response using a provisional transcript, and TTS begins generating speech as soon as it can. Model work, functions and handoffs all consume time in the caller’s experience.
Where does the time go between those two audible events? The latency illustration is useful for locating detection, response preparation and speech generation within the same window, rather than treating each provider’s timing as the whole answer. Hall’s practical guide is roughly 1.5 seconds for a conversation that still holds together, though it is beginning to feel off; a gap of 500–600 milliseconds feels much more humanlike.
Those targets change the engineering question from “How fast can this model go?” to “How humanlike an agent can fit inside this window?” A short response gap is valuable, but so are waiting for the rest of a payment amount, understanding the request and managing tool work without abandoning the caller. The budget constrains the combination of behaviors the stack can deliver.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Benchmark the working agent, then swap the stack
Selecting the stack now becomes an experiment on a complete agent. The hotel receptionist has a real backend, and simulated callers can pursue booking goals while changing their minds, interrupting, mumbling or speaking over background noise. These behaviors exercise the interaction that isolated model tests leave out.
The two simulation modes answer different questions:
- Text-based simulations: Exercise business logic through the conversation, without testing speech recognition or the audio path.
- Audio-based simulations: Exercise speech recognition, turn-taking and the complete voice pipeline, including conditions such as noise and unclear speech.
Each scenario needs a caller goal, the actions the agent should take and an expected outcome. The outcome check reaches beyond the agent’s words: did the correct reservation enter the database? For a financial workflow, was the correct authorized amount charged? A convincing spoken confirmation is not a substitute for checking the state the agent actually changed.
Completion alone is also too weak a score. The benchmark should inspect how the agent reached the result: whether it took ten steps where two would have sufficed, or hallucinated a function call. Those checks distinguish an effective workflow from one that merely satisfies a completion label while wasting turns or taking invalid actions.
With the agent and scenarios held fixed, swap the models and run the comparisons again. Hall describes giving each candidate identical input, byte for byte. Repeated comparisons build a benchmark for this particular agent rather than a universal model ranking. The winner can differ from a public leaderboard because it is being judged on the conversation and backend work the product needs to deliver.
How much can the provider change the wait for the first sentence? In the hotel benchmark, Gemma 4 31B reached that milestone in 381 milliseconds on LiveKit Inference, the fastest of the five configurations shown. The same model took 844 milliseconds on OpenRouter. Hall attributes the difference to LiveKit’s voice-oriented inference serving.
Hall describes the agent, simulations and benchmarking methodology as open source, and LiveKit as model agnostic. The practical invitation is to repeat the experiment on the intended workflow, changing the candidate stack while keeping the task comparable. The hotel result supplies a reason to test provider choices, not a reason to assume its winning configuration will win every conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Launch checks should reach the backend
A voice agent is a real-time system, not simply a chatbot with a speaker attached. Hall closes by translating that distinction into checks that cover both the conversation and its consequences:
- Audible responsiveness: Set a latency budget around what the caller hears, and measure time to first audio and time to first sentence.
- Critical information and access: Confirm important dates and financial amounts. Verify the caller’s identity and authorization before accessing an account.
- Work during delays: Keep listening and keep the conversation moving while slow tools run. Confirm the result when it arrives.
- Whole-conversation correctness: Repeatedly test final backend state, privacy safeguards, unnecessary tool calls and turn-taking.
Start with one workflow or failure that matters. Run it, listen to the exchange and inspect what happened on the backend. When it fails, preserve that failure as a regression scenario so later model or workflow changes have to face it again. This makes a misheard amount, an interruption or an unnecessary tool call a concrete test case rather than a vague request to make the agent better.
The desired experience is ordinary: a caller talks over the agent, changes direction or confuses it, and the conversation keeps going. Building that experience requires checking both sides of the call—the exchange the person hears and the actions the system takes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- Why ChatGPT Keeps Interrupting You
Develops the turn-taking problem in more detail, distinguishing silence-based voice activity detection from conversational completion and examining semantic context and full-duplex approaches.
- Containment Is Lying to You: Measuring AI Agents Like Employees — Druid AI
Extends the evaluation discussion to banking outcomes and human handoffs, showing why a high containment score can reward the wrong behavior.
Full transcript
Loading transcript…