AI Engineer World's Fair 2026
Designing Voice Agents for Real Conversations
Loading slides…
Read the talk
Designing Voice Agents for Real Conversations
A voice agent needs to know when to speak and when to stop. Three Pipecat configurations show how silence timers, provider events and local turn models make those decisions.
From a talk by Chintan Agrawal and Daniel Wirjo
Before you start: Familiarity with Python and the basic roles of speech-to-text, language models and text-to-speech will help with the configuration examples.
The same correction, two different conversations
A user starts planning a flight, then tries to correct themselves. In one interaction, the agent keeps talking for almost two seconds while the user struggles to get a word in. In the other, it catches the interruption in under 200 ms and yields. The model and prompt are identical; the difference is the audio pipeline.
Conversational quality depends on knowing when to stop as much as knowing what to say. Chintan Agrawal and Daniel Wirjo approach this as an audio-engineering problem: detect that someone is speaking, decide whether a pause ends their turn, and stop the agent promptly when the user takes the floor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why a short delay feels long
Agrawal uses roughly 200 ms as the human turn-switching baseline, describes 800 ms as noticeably awkward, and warns that a 1.5-second pause can prompt a hangup. These are conversational design heuristics, rather than measured abandonment thresholds presented in the talk. A chat response can take several seconds without losing its place; silence in a voice conversation can sound like a broken connection.
The cited enterprise realtime voice tutorial illustrates how difficult that budget is. Its reported 755 ms streaming result is nearly four times the stated human baseline, but the measurement boundary matters: the experimental breakdown covers LLM time to first token, sentence detection and TTS synthesis, without separately including endpointing, STT or client playback. It is a particular pipeline result, not an industry-wide voice-to-voice record. Where raw latency cannot reach human timing, turn-taking must at least make the wait intelligible and avoid taking the floor prematurely.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn-taking surrounds the generation pipeline
The familiar pipeline is speech-to-text (STT), an LLM, and text-to-speech (TTS): audio becomes text, the model generates a response, and that response becomes audio. Voice activity detection, or VAD, sits at the front and supplies a more basic signal: is speech present? A turn detector can combine that signal with audio features to decide whether the agent should respond or continue waiting.
An interruption needs a separate path through the system. When the user barges in, the handler propagates a flush downstream, stops TTS and cancels LLM generation. Agrawal describes about 15 ms for this cancellation and cleanup step. That is a pipeline-handling interval, not the complete time from the user's first sound to silence at their speaker. Canceling only generation would leave already-buffered speech playing; stopping only playback would leave obsolete work running.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Level one: let silence end the turn
Silero VAD provides a small, locally controlled speech detector. Agrawal describes an approximately 300,000-parameter, 2 MB model: raw audio becomes spectral features through a short-time Fourier transform; four convolutional layers extract patterns; an LSTM retains information across frames; and a sigmoid produces a speech probability. The memory matters because a speech decision should not depend on one isolated audio chunk.
With silence-only endpointing, the crucial policy is how long speech must be absent before the user is considered finished.
- Short timeout: the agent responds promptly, but can cut into a breath or a thinking pause.
- Long timeout: the agent leaves more room, but the resulting dead air can make users wonder whether it is still connected.
Agrawal suggests 200 ms for a brisk sales interaction and 1,000–1,200 ms for a domain where people need time to formulate an answer. These are example settings to tune for the task.
The limitation is information, not merely calibration. Silence after a completed sentence, an unfinished thought, a thinking pause or a backchannel acknowledgment can look the same to VAD. A pause lasting 200 or 300 ms does not tell the detector whether the speaker is catching their breath or has finished. Nor does detecting overlapping speech decide whether the agent should yield. Speech detection supplies a signal; conversational policy assigns its meaning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Level two: let the STT provider signal completion
The next option moves the decision into the transcription service. Cartesia Ink-2 accepts streaming audio over its STT WebSocket and emits turn events as well as transcription. Instead of inferring completion from a local timer alone, the application waits for the provider's completion signal.
Agrawal reports approximate provider P50 latencies of 300 ms for Cartesia Ink-2 and 250 ms for Deepgram Nova-3, without specifying a common measurement boundary or workload. The two endpointing mechanisms should also be distinguished: current Deepgram endpointing documentation describes VAD-based silence detection and a speech_final signal, not semantic completion detection. The talk's broader description of audio plus linguistic context should therefore not be applied to Nova endpointing as though the features were equivalent.
The trade-off is visibility. A remote service can make useful decisions without exposing its internal evidence. When it ends a turn too early, an event tells the application what happened, but may not explain why. Owning the surrounding logs does not mean owning the decision process inside the provider's server.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Level three: a local model with a silence safety net
A locally owned alternative keeps Silero active and adds Smart Turn. VAD identifies a pause; Smart Turn evaluates the audio to judge whether the turn is complete. Agrawal introduces Smart Turn v3.2, then quotes 58.9% recall and 68.4% precision. Those figures belong to a specific evaluation labeled Smart Turn v3, rather than a universal performance guarantee for v3.2.
The architecture does not require the model to recognize every completion. A confident completion can advance the conversation; otherwise, a silence timer remains available. Agrawal gives 300 ms as an example timeout, referred to in the walkthrough as stopSeconds. His six-in-ten explanation conveys the role of recall: missed completions can fall back to waiting. It does not establish that four in ten turns in every deployment will use the timer.
The matching hierarchical end-of-turn paper makes the precision–recall trade-off explicit:
| Evaluated system | Recall | Precision |
|---|---|---|
| Smart Turn v3 | 58.9% | 68.4% |
| Hierarchical EOT | 87.7% | 57.2% |
The evaluation used 357 conversation samples, with Smart Turn evaluated on eight-second windows at 100 ms hops. Higher recall comes with lower precision here: catching more true endings also means accepting more incorrect completion decisions. Agrawal describes the higher-recall research model as unavailable to deploy because its code had not been released at recording time.
Deployability is part of the choice. Smart Turn is BSD 2-Clause licensed and can be installed for local use. The documented 8 MB size applies to its quantized CPU variant; the unquantized GPU variant is 32 MB. This gives an application a practical way to own the turn decision instead of depending on a remote endpointing service.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast cancellation still needs an intent decision
Starting a response is only half the problem. Once the agent is speaking, Agrawal describes a barge-in path with 32 ms for VAD pickup followed by about 15 ms to flush the pipeline. Pipecat handles the mechanical work of stopping TTS and canceling generation. These are the component timings described in the talk, not a measured end-to-end guarantee for every transport and playback buffer.
The harder question is whether the sound should have stopped the agent at all.
- Correction: a user saying that the answer is wrong needs the agent to stop immediately—the red case in the presentation.
- Acknowledgment or filler: agreement can mean continue; an amber response might keep speaking, perhaps more quietly.
- Cough or background noise: the blue case can be ignored while the agent finishes its sentence.
Local control creates room for this classification, but adding Smart Turn does not automatically implement it. Agrawal describes interruption intent as an improving area where many systems still stop whenever overlapping speech is detected.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three configurations of the same Python pipeline
The three Python examples leave most of the pipeline unchanged. Their central difference is which component answers the question: has the user finished?
| Approach | Completion decision | Control |
|---|---|---|
| Silence only | SileroVADAnalyzer plus a timeout | Local threshold |
| Provider events | Cartesia turn-aware STT service | Provider decision |
| Local model | Silero plus LocalSmartTurnAnalyzerV3 | Local inference and policy |
In the first example, 300 ms of silence is enough to assume completion. The second uses a server event. The third adds a model that examines prosody and intonation during the pause.
The silence configuration can be expressed directly in Python:
python
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.audio.vad.vad_analyzer import VADParams
vad = SileroVADAnalyzer(
params=VADParams(stop_secs=0.3)
)
Here stop_secs is the Python parameter for the silence interval discussed as stopSeconds. The analyzer is one component of the surrounding turn policy, not the complete agent. Keeping that boundary explicit makes the alternatives easier to compare: local silence control, provider-managed completion, or portable local detection with an additional model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where the response budget goes
Agrawal attributes the next latency breakdown to Kwindla, Daily co-founder and Pipecat creator. In the cited production measurements, capture and transport already consume time before the model begins responding:
| Stage | Reported latency |
|---|---|
| Microphone and encoding | About 40 ms |
| Network and jitter buffer | 52 ms |
| Transcription and endpointing | About 300 ms |
| LLM time to first byte | 500–650 ms |
The LLM interval varies with the model, cloud provider and region. Sentence aggregation, TTS, outbound transport and playback add further delay before the user hears the response.
Agrawal puts a standard cloud-API setup at roughly 1,100–1,300 ms and reports a roughly 500 ms voice-to-voice demonstration from Kwindla's team with all models colocated in one GPU cluster. Colocation removes inter-server network hops. These reports have different measurement boundaries from the earlier 755 ms tutorial result, so they do not form a directly comparable leaderboard.
For developers calling APIs, Agrawal gives a broader working range of 800–1,300 ms and estimates that STT plus the LLM consume about two-thirds of the budget. The practical implication is to measure those stages first: endpointing determines when generation may start, and the LLM's initial response determines when downstream speech synthesis can begin. Infrastructure can shorten transport, but choosing when to release a turn remains an application decision.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A fast median cannot hide a slow conversational turn
For their June 2026 voice-model comparison, the speakers set a target below 700 ms time to first token. Agrawal reports the following observations; the workload, regions, percentile methodology and exact Sonnet version are not supplied.
| Model named in the talk | P50 TTFT | P95 TTFT |
|---|---|---|
| Nemotron-3 Ultra | 529 ms | Not given |
| GPT-4.1 | 536 ms | 1.7 s |
| Claude Sonnet | Not given | Over 4 s |
The tail matters because the listener experiences each turn separately. A good median does not repair a long pause that breaks the flow of the current exchange.
Latency is not the only property that can deteriorate during a call. Agrawal reports instruction-following drift after 15–20 turns: a model may become verbose, go off-script or ignore parts of the system prompt. In voice, the user has to sit through the resulting output. Context pruning or session resets can help maintain the intended behavior across a longer conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn errors become operational problems
False interruptions can turn a timing defect into a support escalation. Agrawal reports that users are more likely to ask for a human when an agent repeatedly cuts them off. His production discussion returns to the gap between human timing and cascaded pipelines, and to the need for a timer alongside an imperfect completion model. The earlier 755 ms result and Smart Turn recall figures motivate those concerns; neither establishes a universal best available system.
Deployment also requires coordinating several systems that scale and fail differently. Agrawal frames the operational burden as five constituent systems, then offers two hosting directions: Pipecat Cloud for managed infrastructure that lets a team concentrate on agent logic, and an AWS Guidance reference architecture for enterprise deployments with guardrails. The choice of hosting does not remove the need to understand why a turn ended or why an interruption was accepted.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A travel assistant makes the policy visible
Wirjo's Pipecat turn detection demo puts the three approaches into a shared travel-assistant task. Its examples cover silence detection, STT-integrated turn events and local Smart Turn. Running the turn model under your own control also creates options for compliance requirements and fine-tuning with your own data.
The first example uses Silero defaults, with confidence, silence duration and minimum audio volume available for configuration. The shared services are Cartesia STT, Sonic TTS with a British-sounding female voice, and Anthropic Claude Haiku for generation. Pipecat supplies a local prototyping environment; Wirjo mutes his microphone during connection so his narration does not accidentally interrupt the assistant's opening greeting.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The silence timer mistakes hesitation for completion
The assistant greets the user and asks where they want to go. With real-time terminal logs alongside the conversation, Wirjo begins: “So I'm thinking of going to, um-”. Before he supplies a destination, the assistant replies, “I'm all ears. Where are you thinking of heading?” The response is linguistically reasonable, but its timing exposes the problem: the user's thought was still unfinished.
The logs identify the cause. An end-of-turn event has fired because the silence interval reached the configured stopSeconds threshold, which Wirjo identifies as approximately 300 ms. The agent did not discover that the destination request was complete; the timer authorized it to respond.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The provider waits for Sydney
Wirjo restarts with the STT-integrated turn detector and repeats the unfinished destination request. This time the assistant waits. Cartesia Ink emits turn events in the logs, but the application has not completed the turn or generated a reply. Wirjo remains muted while inspecting the events, then resumes with “So, Sydney.” The assistant responds with questions about travel dates and the length of the trip.
Waiting preserves the unfinished thought, but the extended muted pause reveals the other side of the policy. In a deployed agent, there must also be a way to handle a user who never resumes. Wirjo recommends combining turn detection with silence handling, both to avoid unbounded waiting and to support barge-in. A completion model answers whether an utterance sounds finished; the application still decides how long a conversation may remain idle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The logs distinguish model completion from a timeout
The final example runs Smart Turn alongside silence detection. Wirjo notes that the model can run locally or in a cloud GPU environment, then starts the assistant and repeats the hesitation: “I'm thinking of going to, um-”. The assistant responds before he names a destination, much as it did in the silence-only example.
The debug logs reveal an important distinction: Smart Turn's end-of-turn state is incomplete, but the silence policy nevertheless advances the conversation. The model has not necessarily misunderstood the unfinished thought. Its judgment and the timeout are separate inputs to the application, and the timeout can still authorize a response.
Wirjo then supplies a complete request: “I'm actually thinking of, um, going to Sydney.” The assistant asks about the time of year, trip duration and traveling companions. Rather than infer success from the reply alone, Wirjo returns to the logs to check what caused it.
This time he identifies a complete end-of-turn state and a high completion probability from Smart Turn; no numerical probability is supplied. The final distinction is the one a production debugger needs: did the model recognize completion, or did another policy end the wait? The spoken result can look similar while the underlying decisions are different.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Three runnable Python examples comparing silence detection, Cartesia turn events and local Smart Turn, with setup instructions.
Technical tutorial covering a streaming voice pipeline, function calling and measured time to first audio.
Research combining speaker tracking and conversational-state prediction, including a 357-sample comparison with Smart Turn v3.
Open audio-based turn completion model with local inference examples, model weights and training guidance.
Configuration and response semantics for detecting speech pauses and finalizing streaming transcripts.
Full transcript
Loading transcript…