AI Engineer World's Fair 2025
Why ChatGPT Keeps Interrupting You
Loading slides…
Read the talk
Why ChatGPT Keeps Interrupting You
A pause is not always the end of a turn. Better voice agents need conversational context, acoustic cues, and a deliberate choice about when to start speaking.
From a talk by Tom Shapland, PhD
Before you start: Basic familiarity with LLMs is helpful; speech recognition, synthesis, and turn detection are introduced as they appear.
A pause can cost a patient
A patient calls a dental practice and starts explaining what they need. The voice assistant interrupts; the patient hangs up; the dentist stops paying the developer. That is Tom Shapland’s opening example of why interruptions matter. In ChatGPT Advanced Voice Mode, an interruption is annoying. In a service workflow, it can end the customer relationship.
Turn-taking is the largely unspoken system that determines who controls the conversational floor. It operates quickly, but its timing is not uniform. In the cross-cultural study Shapland describes, Danish listeners take relatively longer to respond, while Japanese listeners respond almost immediately. Individuals differ too: Shapland describes himself as a slow responder, sometimes prompting people to ask whether he intends to answer. Even his own timing changes with the situation—anger can make him respond faster. A single silence threshold has to contend with all of that variation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a silence-based cascade actually detects
The simplified voice-agent cascade starts with incoming audio and ends with generated audio:
- Speech-to-text transcribes incoming audio chunks.
- An endpoint decision determines whether the user has finished speaking.
- The completed transcript goes to an LLM.
- The LLM streams its response into text-to-speech.
- The synthesized audio plays back to the user.
The important gate is the second step. If it opens during a thinking pause, the rest of the pipeline can produce a perfectly coherent response at exactly the wrong time.
Shapland separates the conventional endpointing arrangement into two parts. A voice activity detection, or VAD, neural network classifies audio as speech or nonspeech. A silence algorithm then interprets the absence of speech: in his example, more than half a second of silence means the user is done. This is the distinction that matters: detecting silence is not the same as detecting a completed thought.
The basic decision can be expressed as a small function:
python
def should_end_turn(
speech_detected: bool,
silence_seconds: float,
silence_limit: float = 0.5,
) -> bool:
return not speech_detected and silence_seconds > silence_limit
The function knows nothing about sentence structure, conversational intent, or whether the user stopped after an unfinished clause. Shapland describes this combination of speech detection and silence timing as common in production at the time of the talk, with newer approaches beginning to replace or augment it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Humans prepare an answer before the turn ends
Human conversation presents a timing puzzle: conversational gaps are often around 200 milliseconds, while planning and encoding even a new word can take roughly 600 milliseconds or longer. Shapland uses this mismatch to motivate prediction. The related research account in Timing in turn-taking and its implications for processing models of language explains why a listener cannot simply wait for silence and then begin all the work of responding. Some preparation must happen while the other person is still speaking.
The main predictive input, in Shapland’s account, is semantics: what the speaker is trying to communicate. Syntax supplies sentence structure; prosody supplies tone, expressiveness, and other acoustic information; visual cues provide additional evidence. He presents the following as a useful proposed processing model, not a complete explanation of human cognition:
- Predict the intended message. Infer what the speaker means and use that inference to anticipate an utterance ending. Keep updating both predictions as more speech arrives.
- Refine the endpoint. As the expected ending approaches, combine meaning with sentence structure to narrow the prediction.
- Finalize with acoustic cues. Near the end, use prosody and other acoustic features to decide when to begin speaking.
This account is full duplex: comprehension and response production overlap. In the figure Shapland describes, zero marks the end of the current speaker’s turn. Well before zero, the listener is both interpreting the incoming message and beginning to prepare an answer. The short audible gap between turns therefore does not represent the full time spent generating the response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the endpoint detector conversational context
Compared with that overlapping human process, the conventional cascade is backward-looking: it measures how long speech has been absent, then starts the response stages in sequence. A practical improvement keeps the cascade but augments its VAD with models that consider semantics, syntax, or prosody.
The LiveKit end-of-utterance model is Shapland’s text-based example. In the version he describes, a transformer receives the last four conversational turns in order: agent, user, agent, current user. It predicts an end-of-utterance token using that content and context. If the prediction indicates that the current turn is unfinished, the system extends the VAD’s silence allowance instead of triggering the agent. The semantic model and VAD work together: silence supplies a candidate stopping point, while conversational context can tell the system to wait longer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An interview that keeps interrupting its answer
Shane’s paired demonstration makes the endpoint decision audible. In the first version, the agent uses traditional VAD. Shane tries to explain that he needed to build a demo but did not know what to build. Before he can finish, the agent says, “Please continue. What happened next?” He tries again, explaining that the project is a LiveKit turn-detection demo, and more questions arrive during his pauses. Eventually he identifies the worst part of making it: the agent kept interrupting him. The agent responds by asking how he overcame that challenge.
The second version enables semantic turn detection. Shane asks to be interviewed about building a demo, then delivers another hesitant answer. He pauses after an unfinished negative clause, pauses again while introducing his next thought, and pauses once more before completing his explanation that the comparison should be side by side. This time, the agent leaves those pauses alone. The audience applauds; Shapland describes the difference as dramatic. What the excerpt establishes is the qualitative listening behavior: unfinished thoughts no longer provoke the same stream of premature questions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Listen to the audio as well as the words
A text-based detector gains conversational meaning but does not directly receive the acoustic signal. Audio-aware endpoint models add that evidence: their input is audio tokens, and their output is a probability that the user has finished speaking. Shapland points to Kwin and the Daily team’s open-weight Smart Turn as an example combining transformer modeling with acoustic characteristics.
Endpoint prediction can also be integrated into transcription. Shapland describes AssemblyAI’s streaming speech-to-text release from earlier that conference week as emitting both a transcript and a likelihood that the speaker has finished, using acoustic and semantic features. He also mentions a recently released Kyutai alternative, without giving implementation details. The products here are the versions discussed in the 2025 recording; current repositories and documentation may describe later implementations.
| Approach | Evidence available to the endpoint decision |
|---|---|
| LiveKit’s text model | Recent agent and user turns |
| Audio-aware detector | Speech content and acoustic characteristics |
| User-stream STT endpointing | User speech, without the agent’s turns |
The last distinction is Shapland’s caution about built-in speech-to-text endpointing: a model receiving only the user’s stream lacks the agent’s side of the conversation. Even so, he regards these approaches as a substantial improvement over silence-only endpointing and recommends adopting them through the available voice-agent platforms.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Audio in and audio out still need a turn boundary
Speech-to-speech does not automatically solve turn-taking. A model can accept audio and generate audio while still relying on a separate decision about when to respond. Shapland uses the OpenAI Realtime API as an example: it uses VAD and offers semantic VAD as an option. Despite the name, the useful distinction is the addition of semantic completion judgments to speech activity detection.
Shapland attributes Advanced Voice Mode’s interruptions to premature judgments that the user has finished, based on elapsed silence or preceding content. That is his explanation of the consumer product; the public Realtime API documentation does not establish ChatGPT’s internal configuration. He emphasizes that none of the approaches has perfected interruption handling. He also distinguishes transport from endpointing: LiveKit supplies Advanced Voice Mode’s audio transport, he says, but OpenAI does not use LiveKit’s end-of-utterance model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Continuous listening, continuous output
Full-duplex models move closer to the overlapping process described earlier: they process incoming speech while generating output. At the time of the talk, Shapland says he knows of no commercial applications of these models. Their appeal is that they learn conversational timing from audio rather than depending entirely on handwritten rules.
His analogy is early computer vision. Trying to recognize a stop sign by explicitly checking its color and number of sides proved less effective than letting a neural network learn from images. He links that shift to the field’s emergence from the AI winter. For conversation, the corresponding ambition is to let models learn when to speak from recorded interaction. That describes the timing-learning principle, not a claim that every full-duplex model is trained exclusively on raw recordings. Shapland’s concern is capability: he characterizes the models under discussion as small, trained on limited data, and weak at instruction following.
Moshi makes the continuous-output idea concrete. It is always listening and always emitting output. When it should not speak, its output is natural silence. Silence therefore becomes part of the generated audio stream, rather than merely the absence of a response job.
Shapland next discusses SyncLLM, which he associates with Meta AI’s experimental in-app full-duplex experience; that specific model identity remains his attribution. He describes its anticipation of user speech as about five tokens, or 200 milliseconds. These are audio units, not ordinary text tokens: more precisely, the paper’s mechanism estimates an unavailable user-audio chunk and later replaces that estimate with actual input. Its raw audio units run at 25 Hz before deduplication, so five raw units correspond to 200 milliseconds, rather than a fixed five-text-token forecast. The important mechanism is acting with an estimate of incoming audio while continuing to listen for the real input.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why a better cascade may still win in production
Shapland’s forecast favors improved cascades over full-duplex models for commercial voice agents. The deciding requirement is control, including something as concrete as how an agent pronounces a brand name. Smarter VAD augmentations can improve endpoint decisions, while faster components elsewhere in the cascade leave more time to make those decisions well.
This is a prediction about the engineering path, not an established winner. His reason for resisting mandatory imitation of human cognition is practical: computers already perform mathematics differently from people, and LLMs need not reason through the same mechanisms humans use. Voice AI may likewise achieve useful conversational timing through a different arrangement of components.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the demonstration leaves unanswered
The first Q&A question identifies a missing part of the improved demonstration: the audience never hears the agent respond at the end. Shapland explains that he shortened a two-minute demo to fit his speaking slot. The moderator jokes that turning speech off altogether would also eliminate interruptions. Shapland says the detector eventually identifies the end of the turn from context, but that eventual response is not shown in the excerpt. He directs viewers to LiveKit’s Twitter account for the full demonstration.
The next question asks whether avatars and other real-time systems should use visual cues for turn detection. Shapland again puts semantics first: despite the importance of vision to humans, he ranks visual cues below conversational content for predicting an ending. He expects someone to be exploring multimodal turn detection, but has not seen a concrete implementation he can point to.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cost, integration, and evidence
Asked about call costs and repeated response generation, Shapland says text-to-speech tends to be the most expensive part of a cascade. In that setting, invoking the LLM multiple times within a turn can be comparatively inexpensive. He does not provide an average call price or a provider-specific breakdown: he is not personally building agents and does not have those unit economics at hand. Conversation length matters, and he suggests using cost calculators to evaluate a particular setup.
Another audience member asks why OpenAI does not use LiveKit’s detector, whether developers can use it themselves, and whether the team has measured its improvement. Shapland does not know OpenAI’s reason; he had joined LiveKit only weeks earlier. For developers, he describes following LiveKit’s agent quick start and adding one line to enable the detector. He says the model is open-weight and free to use, while expressing uncertainty about whether the quick start already includes it by default.
On evaluation, he reports that LiveKit has internal test-set benchmarks, but supplies no scores or user-study results. The audience’s suggested percentage improvement is a question, not a result. His substantive concern is the difficulty of constructing a good benchmark dataset: even the team’s recent discussions centered on what suitable evaluation data should look like. In his view, the industry still lacks a good shared benchmark for turn-taking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An acknowledgment is not always an interruption
The final question reverses the problem. What happens when the human speaks while the agent is talking? A brief acknowledgment such as yeah, mm, or yes can mean that the listener is following along. The same words can also begin a substantive response. These backchannels make it unsafe to equate all user speech with a request to take the floor.
Shapland distinguishes this from the talk’s main focus: preventing the AI from interrupting the human. For user interruptions, he describes LiveKit’s then-current approach as a duration rule using Silero VAD or ordinary speech activity detection. Speech lasting more than a configured number of milliseconds is treated as an intentional interruption; he does not specify the threshold. A model that distinguishes backchannels from attempts to interrupt is something the team wants to build, not a capability he claims is already implemented.
Full-duplex models offer an intriguing counterpart on the generation side. Shapland says Meta’s experimental experience can naturally produce acknowledgments while listening, having learned that behavior from conversational audio. Generating a well-timed acknowledgment and recognizing whether a user’s acknowledgment should stop the agent are different tasks. The remaining challenge is not simply to hear speech sooner, but to understand what that speech means for who should hold the floor.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A full-duplex dialogue model that predicts interleaved speech streams and estimates missing user audio to handle latency.
Kyutai's speech-text foundation model and full-duplex spoken dialogue framework.
Further reading
Research on conversational timing and how listeners prepare responses while comprehension continues.
LiveKit's companion explanation of transformer-based end-of-turn detection.
- Kyutai's Unmute announcementArticle
The May 2025 announcement of a voice interface combining streaming transcription, synthesis, and semantic turn detection.
Updates since the talk
Current Smart Turn model, training materials, and examples for running audio-based endpoint detection locally.
Official documentation for voice activity detection in the Realtime API.
Full transcript
Loading transcript…