Contents
  1. Spoken interaction
    1. What a voice system must coordinate
    2. Foundations that still matter
  2. Representing speech
    1. Samples, frames, and speech time
  3. Recognizing and generating speech
    1. From acoustic frames to words
    2. Partial words are revisable evidence
    3. From text to speakable phrases
      1. Waveform generation
      2. Text lookahead
  4. Audio-native models
    1. Learned audio codes
    2. Architecture and duplex behavior
      1. Simultaneous streams
  5. The live audio path
    1. Buffering against playback deadlines
    2. The room enters the microphone
  6. Conversational timing
    1. Speech activity is not turn completion
    2. Acknowledging without taking the floor
  7. Interruption and continuity
    1. Stop obsolete speech
    2. Retain only supported playback history
  8. Audible performance
    1. Measure onset, continuity, and stopping
    2. Prepare early without claiming completion
    3. Reconstruct the audible interaction
  9. Understandable and controllable exchanges
    1. Repair the misunderstanding
    2. Make listening and stopping controllable
  10. Evaluation and design choices
    1. Measure speech properties separately
    2. Test complete conversations
  11. Check understanding
  12. Open questions
  13. Selected talks
  14. References
  15. Talk library
← All topics

Voice AI

A useful voice system must do more than recognize words and produce speech. It must preserve what a person means while their sentence is still unfolding, decide when a response is welcome, and stop when the conversation changes direction. Understanding those responsibilities requires following both information and time: what the system received, what it inferred, what it generated, and what actually played.

Spoken interaction

What a voice system must coordinate

Speech can make an interface usable when a keyboard or display is inconvenient. But spoken output arrives sequentially: a listener cannot scan several options at once or easily revisit an earlier detail. The design therefore needs manageable answers and inexpensive ways to repeat, correct, or switch channels. These concerns predate modern language models; early voice notetakers and telephone information services already had to address them.

A speech model is a learned system that recognizes, represents, or generates speech. Automatic speech recognition (ASR) converts speech into text; text to speech (TTS) generates spoken audio from text. A common conversational architecture places a large language model—a model that generates text responses—between them. This cascade exposes useful intermediate artifacts, but attaching those components does not settle when to answer or how to handle interruption.

The responsibilities remain useful distinctions even when one service performs several of them.
ResponsibilityArtifact or decision
CaptureMicrophone audio with a format and time basis
Recognition or audio understandingProvisional words or audio-derived representations
Response productionAn answer, clarification, or request to external software
SynthesisGenerated audio representing the response
Transport and playbackAudio received, queued, and eventually played
Interaction controlWhen to listen, speak, interrupt, or repair

A conversational turn is a contribution to an exchange; turn-taking coordinates opportunities to make those contributions. Silence, overlap, and acknowledgment are part of that coordination, not merely gaps between API calls. A system needs understandable speech, appropriate timing, recoverability, and useful completion. Here, real-time responsiveness means serving the interaction while it unfolds, not promising a hard deadline for every operation. The broader choice between unified and modular processing is developed in Choose the boundary you can operate.

Foundations that still matter

Recognition and conversation developed around different problems. Recognition must interpret variable acoustic evidence. Conversation must coordinate contributions and resolve misunderstandings. Interactive systems bring these problems together, but improving one does not eliminate the others.

These developments explain several boundaries that remain important in contemporary systems.
DevelopmentContribution and continuing consequence
1974 — Conversational turn allocationHarvey Sacks, Emanuel Schegloff, and Gail Jefferson described possible completion points and rules for selecting the next speaker. A speaking opportunity is not simply an absence of sound. Original paper.
February 1989 — Statistical sequence recognitionLawrence Rabiner's hidden Markov model tutorial synthesized earlier work on hidden evolving states that produce observed acoustic measurements. Scoring a recording and choosing its most likely state alignment are different computations. Tutorial.
May 1997 — Telephone dialogue in public useMIT's JUPITER weather service combined recognition, language understanding, database queries, response generation, and synthesis. Its displayless interface had to narrow requests and keep spoken answers manageable; lack of barge-in also caused clipped input. System account.
2009 — Explicit incremental revisionA general incremental-dialogue framework represented partial results as linked units that could be added, revoked, or committed. Downstream interpretations could begin early while retaining their dependence on revisable evidence. Original paper.

The recurring problem is commitment under incomplete information. A recognizer can prepare a hypothesis before the sound ends; a dialogue system can prepare an answer before the turn ends. Once an answer becomes audible, however, internal revision alone cannot erase it. Later neural methods change how recognition and synthesis work, while these interaction obligations continue.

Representing speech

Samples, frames, and speech time

A waveform describes amplitude over time. Digitization measures it at discrete times and represents each amplitude with finite precision. Sample rate counts measurements per second per channel; channels contain separate signals. Pulse-code modulation (PCM) stores quantized amplitude values, but a PCM contract must also specify width and packing. For example, the documented Windows 16-bit stereo format interleaves signed little-endian left and right samples. The label PCM alone does not establish those details.

tn=nfs,D=Nfs.t_n=\frac{n}{f_s},\qquad D=\frac{N}{f_s}. Here, nn is a sample index relative to the buffer's time origin, fsf_s is samples per second per channel, NN is the number of sample frames, and DD is buffer duration in seconds. At 16 kHz, a 25 ms analysis window contains 400 samples per channel.

The word frame is overloaded. A sample frame contains simultaneous channel values; an analysis frame contains a short interval of samples. A codec frame and a transport packet are other groupings. None is intrinsically a word or a conversational turn. Opus, for example, supports several frame durations and can combine frames into longer packets. Longer packets reduce some overhead but add waiting and put more audio at risk when one packet is lost.

Resampling constructs values on a new time grid while preserving intended duration. Merely interpreting unchanged values at half their original rate doubles duration and halves playback frequencies. A one-cycle-per-second tone sampled at deliberately tiny rates makes the time-grid difference clear; this teaching example is not speech. Preserve stream identity, rate, and time origin through conversion, as explained in Keep the coordinates that make the claim true.

Resampling versus changing the rate label

Resampling changes the grid; reinterpretation changes playback timeEight original samples occupy one second. Four resampled values also occupy one second. Reinterpreting the original eight at four samples per second occupies two seconds. Dots are sample instants and dashed end markers are exclusive buffer ends.Original: 8 samples/s00.511.52-101Exclusive end 1s8 samplesResampled: 4 samples/s00.511.52-101Exclusive end 1s4 samplesUnchanged values: 4 samples/s00.511.52-101Exclusive end 2s8 samplesTime (seconds) · normalized amplitude on each row
Resampling constructs values on a new grid while preserving duration. Reinterpretation keeps the original values and changes their timing. Dots are sample instants, not a reconstruction curve; dashed markers are exclusive buffer ends.

A recognizer often needs changing frequency patterns rather than raw amplitudes. A spectrogram stacks frequency measurements from successive, usually overlapping windows. Log-Mel features pool frequency bands and apply logarithmic compression. These representations describe local acoustics; they do not directly identify words. Prosody—rhythm, stress, intonation, pauses, and related delivery—also carries information that plain transcription only partly retains. At capture time, inspect the track's actual settings: browser constraints request a configuration rather than proving which configuration was selected.

Recognizing and generating speech

From acoustic frames to words

Recognition aligns variable-duration acoustic sequences with fewer linguistic units, usually without frame-level training labels. An encoder maps audio features to numerical representations; see Different signals become different representations. Alignment determines which acoustic steps support each output symbol.

Connectionist Temporal Classification (Alex Graves and colleagues, 2006) learns this mapping by predicting labels plus blanks at each step. Training sums probabilities over all paths consistent with the target. To collapse a path, merge adjacent repeated labels before removing blanks. For example, a a ∅ a a → (merge adjacent repeats) a ∅ a → (remove blanks) aa, where ∅ is blank. A separating blank preserves two occurrences; it does not indicate conversational completion.

Recurrent neural network transduction, described by Graves in 2012, combines acoustic representations with previous output symbols. It also sums over valid alignments during training. At inference, a search procedure instead considers candidate output sequences. A bounded beam retains a limited set of candidates, exchanging computation and memory for an approximation to a broader search. The alignment objective and the decoding search are separate choices.

Neither alignment objective guarantees streaming. A causal encoder uses only audio already received; lookahead, also called right context, requires waiting for additional audio before making a prediction. Attaching a streaming decoder cannot remove an encoder's need for future input. NeMo's cache-aware streaming models train with restricted context and retain activations—the intermediate numerical results computed from earlier audio. Reusing those results avoids some repeated computation from processing overlapping windows. More lookahead supplies additional acoustic evidence at the cost of waiting. Removing lookahead eliminates that wait, not computation or transport delay.

Whisper, presented by Alec Radford and colleagues in 2022, pursued broad speech-recognition robustness through large-scale weak supervision. Its encoder-decoder maps log-Mel features to text and task tokens, supporting several speech tasks. Windowed long-form processing and incremental interactive recognition nevertheless impose different requirements. Wrapping a batch recognizer with chunking, repeated calls, and transcript stitching can expose partial text without making its internal computation streaming-native.

Partial words are revisable evidence

An incremental recognizer produces hypotheses before all relevant audio has arrived. Later sound can change earlier words, not merely append new ones. The incremental-dialogue example of four becoming forty captures the consequence: an interpretation that used the quantity four must be reconsidered. Give partial results stable identities and record support links to dependent interpretations and prepared work. Starting downstream work early is useful only when those links allow the application to invalidate and revise work based on a changed provisional interpretation.

Revised words invalidate dependent work

Support arrows identify dependencies. The separately labeled invalidation row shows revocation propagation; neither is audio transport or a time scale. Unit identities let the application invalidate dependent work.

Before revision

W1: “four”
provisional word
supports →Q1: quantity = 4supports →P1: preparation using 4

After later audio changes the word

W1: revokedinvalidates →Q1: invalidinvalidates →P1: discard or reconsider

P1 is internal preparation, not an executed external action.

W2: “forty”
replacement word
supports →Q2: quantity = 40supports →P2: new preparation

Independent state has no W1/Q1 dependency and stays unchanged in both states.

An application of the published four/forty example: replacing W1 invalidates Q1 and prepared P1; W2 supports quantity40 and new preparation. Independent state remains unchanged. Internal revocation cannot retract audible speech or reverse an external effect.
Recognition progress has several meanings. Read the particular service contract before treating an event as a commitment.
SignalWhat it supportsWhat remains unresolved
Provisional hypothesisThe recognizer's current interpretationEarlier words may change
Estimated stable prefixA prediction that the prefix will persistPersistence is not correctness or an irrevocability guarantee
Amazon Transcribe Stable=trueThe item is fixed under that service's stabilization contractThe fixed word may still be wrong
Final recognition segmentThe recognizer has completed that segmentThe person may be pausing within a larger contribution

Stability and confidence answer different questions. The 2012 word-stability study used evidence such as how long a prefix had survived and how much following audio was available. Waiting for more stable text adds lag. Whether the words are correct requires separate evidence; see Sources of predictive uncertainty.

Event identity matters as much as text content. OpenAI's documented realtime transcription events identify their input items, and completion events from different turns need not arrive in order. Associate each result with its item rather than appending by arrival time. In an audio-native system, a separately generated input transcript may also differ from the model's own interpretation of the audio.

Test recognition where the task is vulnerable: names, numbers, accents, language switching, and brief answers. Context can narrow plausible vocabulary, but it can also favor what the system expects over what was said. A clinical implementation describes using conversation and medication context, with another recognition pass for single-word replies. Separately, research on Whisper shows that non-speech audio can produce text. Fluent output is therefore not proof that corresponding speech was present.

From text to speakable phrases

Producing a response's meaning and producing its sound are different jobs. Text normalization chooses spoken forms for written numbers, dates, and abbreviations. Pronunciation determines how those words sound; grapheme-to-phoneme conversion maps written symbols to speech-sound symbols. A pronunciation lexicon supplies stored pronunciations, but context may still be needed: read sounds like “reed” in “I read every day” and “red” in “I read yesterday.” Phrasing, duration, and prosody determine how the resulting words unfold in speech.

A conventional neural synthesizer predicts an acoustic representation, such as a Mel spectrogram, and a vocoder converts that representation into waveform samples. This is a functional account, not a requirement that every modern model expose separate modules. Explicit pronunciation hints can improve control over difficult names, but supported controls and their interaction with automatic synthesis are implementation-dependent.

Waveform generation

WaveNet, introduced by Aaron van den Oord and colleagues at DeepMind in 2016, changed waveform generation. Earlier approaches included concatenating recorded units and predicting acoustic parameters for a vocoder. WaveNet instead predicted each waveform sample from preceding samples using dilated causal convolutions. Listening tests favored it over the selected English and Mandarin baselines, but sequential sample generation left a substantial serving problem.

Parallel WaveNet addressed that cost by training a parallel student generator against an autoregressive teacher. The November 2017 report described deployment in English and Japanese Google Assistant voices. This is an example of distillation: moving knowledge from a teacher into a student with different execution properties. Faster waveform production did not itself solve recognition or conversational timing.

Text lookahead

Incremental text is not automatically ready for incremental speech. A partial spelling, abbreviation, or unfinished phrase may lack the context needed for pronunciation and emphasis. Waiting for a coherent phrase gives the synthesizer more information; starting earlier reduces onset delay but can make adjacent chunks sound disconnected. A 2020 incremental-TTS study found that added lookahead brought its Tacotron 2 output closer to full-context synthesis, though listeners still distinguished limited-lookahead output. That model-specific result supports testing the tradeoff, not prescribing a universal number of words to buffer.

Audio-native models

Learned audio codes

Predicting every waveform sample creates a long sequence. A neural audio codec offers a more compact interface: an encoder transforms waveform intervals into numerical representations, a quantizer maps them to discrete codebook entries, and a decoder reconstructs sound. A codebook is a learned collection of vectors; its indices identify vectors, not written words. Reconstruction is lossy even when it sounds convincing.

SoundStream, described by Neil Zeghidour and colleagues in July 2021, used residual vector quantization: successive codebooks encode corrections to remaining representation error. Dropping quantization layers during training allowed one model to support different bitrates. EnCodec, first posted on October 24, 2022 by Alexandre Défossez and colleagues, built on this encoder–quantizer–decoder structure with additional training techniques. It distinguished a causal, streamable model from one using future context. In a simplified two-codebook case, the first selected vector q₁ approximates encoded representation z; the second quantizes the residual z−q₁ to select q₂. Their indices identify vectors, and q₁+q₂ approximates z for waveform decoding. Compression fidelity and conversational usefulness remained different evaluation targets.

Indices select vectors; vectors reconstruct the representation

Encode a representation residual, transmit indices, then sum decoded vectorsCodebook one selects q1 approximating z. Codebook two receives z minus q1 and selects q2. Indices i1 and i2 select q1 and q2 again at decoding. Their vector sum approximates z and enters the waveform decoder.Encoding · representation vectorsEncoded zSelect q₁ in C₁Residual z − q₁Select q₂ in C₂zq₁residualzIndex i₁Index i₂Decoding · indices select vectorsLook up q₁ = C₁[i₁]Look up q₂ = C₂[i₂]Vector q₁Vector q₂ẑ = q₁ + q₂Waveform decoder → reconstructed audio
Two codebooks are a simplified example. q₁ approximates z; q₂ approximates z−q₁. The indices select those vectors again, and their sum enters the waveform decoder. This is representation error, not waveform subtraction; reconstructed audio remains lossy.

Once audio has a discrete representation, a sequence model can predict audio codes rather than individual samples. VALL-E demonstrated this approach in its 2023 work: text-derived phonemes condition linguistic content, while a reference recording conditions acoustic characteristics. A codec decoder turns predicted codes into sound. Its evaluations separated intelligibility, speaker similarity, and human judgments; reported omitted or duplicated words show why a plausible voice is not enough.

Keep the role of each encoding explicit. A call codec packages sound for transmission; learned audio codes can serve as a model's prediction vocabulary. These roles can overlap, but an audio code is neither a transport packet nor a waveform sample. In the documented Moshi implementation, Mimi represents 24 kHz audio at 12.5 frames per second: one representation frame covers 80 ms containing many waveform samples.

Architecture and duplex behavior

A text handoff makes the words entering the response model inspectable, but it can discard delivery cues. Direct audio input retains more of those cues while changing which intermediate artifacts the application can inspect or control. These are information-boundary choices, not a guarantee that one architecture is faster or more capable for every task.

The model's input and output contract determines which surrounding components remain necessary.
ArchitectureInformation pathOperational consequence
Text-mediated cascadeAudio → transcript → response text → speechInspectable text boundaries and reusable text agents; recognition errors and lost delivery cues can propagate
Audio input, text outputAudio encoder and connector → language model → textAvoids requiring a transcript-only input handoff, but still needs synthesis for spoken output
Speech-to-speechAudio-derived input → generated audio, sometimes with associated textCan retain acoustic information across response generation; alignment, controls, and diagnosis depend on the implementation

Ultravox illustrates the intermediate architecture: its documented implementation accepts audio and text while emitting streaming text. GPT-4o's May 13, 2024 announcement marked an integration milestone, presenting one model trained across text, vision, and audio instead of the earlier voice cascade. The announcement did not mean immediate general availability of that voice experience: the initial rollout exposed text and image input with text output.

Simultaneous streams

Full duplex means simultaneous input and output; half duplex alternates them. A transport may support both directions while a model still alternates listening and speaking. Conversely, a model that accepts ongoing input still needs a media implementation that handles echo and lag. Moshi models incoming and outgoing audio streams with text associated with its own speech. Kyutai publicly unveiled the experimental prototype on July 3, 2024; that event is distinct from its later paper and downloadable-model availability.

Simultaneous streams do not settle role control or reliable task behavior. PersonaPlex, described by NVIDIA's Rajarshi Roy and colleagues on January 15, 2026, extended Moshi with text conditioning for role and background and audio conditioning for vocal characteristics. Its training mixed real conversations with synthesized role-directed dialogues. The work addresses control without abandoning concurrent speech; developer-reported evaluations do not establish reliable outcomes in every live service.

Retaining tone also does not prove that a model uses it appropriately. If training tasks can be solved from factual content alone, they may provide little incentive to respond to hesitation or delivery. Evaluate the required behavior directly rather than treating audio input as evidence of emotion understanding. Select the architecture around needed information, pronunciation control, diagnosis, and execution reliability—not a speech-to-speech label.

The live audio path

Buffering against playback deadlines

Audio production and audio consumption run on different schedules. Services may generate or deliver chunks in bursts; ordinary playback consumes one second of sound per second. Jitter is variation in arrival timing. A jitter buffer holds received audio before playout so temporary variation is less likely to exhaust available sound. An underflow occurs when playback needs audio that is not ready.

A larger startup buffer can absorb a longer arrival gap, but the listener waits longer before speech begins. Hold chunk arrivals fixed and vary only startup time to see the tradeoff. Its simplified player pauses on underflow; real media systems may instead conceal missing sound or adjust playback speed. WebRTC's NetEq adapts delay, handles late packets, and uses concealment. Concealment substitutes audio; it does not recover the missing original samples.

Startup waiting changes continuity

Only startup delay changes. Four 400 ms chunks arrive at 0, 0.3, 1, 1.2 seconds. Arrivals at an exact depletion boundary become available before playback resumes; no gap is counted there.

Onset 0s · Total gap 0.2s (0.8–1s) · Finish 1.8s

Fixed arrivals and pause-on-underflow playbackChanging startup delay shifts audible onset and can remove the gap. Occupancy is received duration minus played duration and never negative.Cumulative audio duration (seconds of audio)00.40.81.21.6000.50.5111.51.5222.22.2Solid blue: received · Dashed orange: played · Shaded interval: gapBuffer occupancy (seconds of audio)00.81.6Elapsed wall time (seconds)
Fixed comparisonOnsetGapFinish
Immediate0 s0.8–1.0 s1.8 s
Wait 0.4 s0.4 sNone2.0 s

For this fixture, 200 ms is the smallest gap-free startup delay. It is not a production recommendation; later arrivals or an underspeed producer change the requirement.

Four fixed 0.4-second chunks arrive at 0, 0.3, 1.0 and 1.2 seconds. This player pauses on underflow. The received-minus-played difference is buffered audio; NetEq may instead conceal or time-adjust sound, as described in the text.

Backpressure makes a slow consumer constrain upstream production. A bounded queue can block a producer when capacity is reached; cancelling, dropping, or rejecting audio is a separate policy. Measure capacity in the resource that matters. Five fixed-duration chunks bound pending duration, whereas five variable-duration chunks do not. Queue clearing on interruption is cleanup, not continuous backpressure: the described Pipecat FrameQueue implementation inherits an unbounded queue while providing interruption-reset behavior.

Transport changes who owns these responsibilities. WebRTC supplies specialized media handling; WebSockets offer a simpler ordered connection often useful for prototypes, control messages, and server-to-server communication. TCP's ordered delivery can make newer audio wait behind missing older data. Choosing WebSocket audio is therefore a decision to manage the required pacing, buffering, and adaptation—not an automatic error. One application can use both transports for different roles.

Packet order is not source time. The Real-time Transport Protocol (RTP) carries media with sequence numbers for packet ordering and timestamps for sampling instants. The payload format determines the timestamp clock rate. Neither field tells you when a device actually played the sound. Preserve the distinctions among generated, transmitted, received, decoded, scheduled, and played audio; Progress and completion metrics develops the corresponding serving principle. Finally, no finite startup buffer can sustain indefinitely a producer that generates less than one second of playable audio per second.

The room enters the microphone

A microphone receives the acoustic environment, not an isolated user utterance. Reverberation is sound persisting through reflections; distance changes the balance of direct and reflected energy. Background noise adds other signals. Channel bandwidth limits which frequencies survive. Clipping occurs when amplitude exceeds the representable range. Gain control changes level, while noise suppression tries to reduce unwanted sound; neither can restore information already destroyed by clipping.

Loudspeaker output creates another input: the system's own speech travels through the room and returns to the microphone. Acoustic echo cancellation (AEC) uses the playback signal as a reference and estimates how that signal was transformed on its acoustic path. Subtracting the estimated echo aims to preserve local speech. Generated text is not an adequate reference; the canceller needs the rendered signal and its timing relationship to capture.

Double-talk occurs when local speech and returning playback coexist. Local speech can disrupt an adaptive filter's learning, so echo-cancellation methods control adaptation during such interference. Freezing adaptation does not mean muting the microphone or stopping subtraction with the existing filter. Muting capture whenever the assistant speaks prevents ordinary spoken interruption. The engineering objective is to preserve that new contribution while reducing the returning assistant audio, not to remove everything during overlap.

Playback takes two routes: sound returns through the room, while rendered samples provide the estimator’s reference. This conceptual cutaway shows imperfect echo reduction; adaptation control does not close the microphone.

More aggressive preprocessing is not always better. WebRTC's noise-suppression interface explicitly describes stronger suppression as trading less noise for more speech distortion. Test the combined capture path, particularly short acknowledgments and speech onsets, rather than assuming that cleaner-sounding output contains better recognition evidence.

Speaker diarization estimates who spoke when using consistent, usually anonymous labels. It is not echo removal or authentication. Joining diarization with recognition is not a trivial timestamp lookup: systems can disagree about boundaries, detect different speech, or find two speakers where recognition returned only one word. Preserve uncertainty in attribution. Similarly, missing packets are not observed silence; Absence and delay are distinct states explains why unavailable audio cannot establish that nobody spoke.

Conversational timing

Speech activity is not turn completion

Voice activity detection (VAD) estimates whether audio contains speech. Endpointing decides when an utterance or turn is complete enough to advance. These are different decisions: no speech now does not imply no more speech intended. A person can pause to think, retrieve a detail, or continue a difficult word. Recognition finalization also cannot settle this question because its segments need not coincide with conversational intentions.

A detector turns scores into events through stateful rules. Silero's VAD iterator uses one activation threshold and a lower release threshold, plus a minimum silence interval. This hysteresis reduces rapid switching near a boundary. Padding moves reported boundaries to include surrounding audio; the caller must retain the earlier samples. These rules stabilize speech segmentation, not semantic completion.

Semantic endpointing adds an estimate informed by meaning and sometimes acoustic delivery. An equal-length pause after “The destination is…” and after “The destination is Sydney” supplies the same silence duration but different completion evidence. Voice Activity Projection, introduced by Erik Ekstedt and Gabriel Skantze in 2022, predicts both speakers' future activity, providing a way to distinguish holding, shifting, and brief acknowledgments. Such predictions remain fallible and depend on conversational data.

Waiting has a real benefit and a real cost. Colin Lea and colleagues' CHI 2023 study of people who stutter evaluated 5,370 utterances from 41 participants with moderate or severe stuttering. A more patient threshold, tuned on separate data, reduced premature cutoffs while increasing the delay before the recognizer stopped listening. These are endpoint delays in the studied quiet-condition setup, not complete assistant-response times.

PolicyPremature cutoff rateMedian endpoint delay
Baseline23.8%450 ms
Threshold tuned for mild stuttering4.9%1,670 ms

Completion estimates need an application waiting policy. An unfinished sentence can remain unfinished because the microphone is muted or the person has stopped trying. Combine patience with a bounded, contextual check-in rather than an indefinite wait or a forced answer. The duration and wording should fit the task and user controls; a provider's turn event may reveal its decision without explaining the reasoning behind it.

Equal silence, different completion evidence

Illustrative policy fixture: compare both after 600 ms of silence; unresolved input reaches a check-in at 1600 ms. These are not recommended timeouts or calibrated predictions.

Equal silence does not imply equal completionBoth utterances stop at time zero and have the same 600 ms pause. The unfinished destination remains uncertain and keeps listening; if uncertainty persists, a separate check-in occurs at the illustrative waiting limit. Sydney supplies a complete destination, allowing a reply at the comparison point.0600100016002000“The destination is…”Same silenceIncomplete → listenWaiting limitCheck in, do not invent a destination“The destination is Sydney”Same silenceComplete → reply permittedElapsed silence (ms) · both speech stops aligned at zeroActivity: silence. Completion: estimated from the contribution. Action: chosen by policy.

A check-in asks for the missing destination. Response preparation can begin earlier; permission to speak is a separate decision.

The same pause follows “The destination is…” and “The destination is Sydney”. The incomplete contribution can require more listening and then a separate check-in; the complete destination permits a reply. Activity, completion evidence and application action remain distinct.

Acknowledging without taking the floor

The conversational floor is the opportunity to make the current contribution. A backchannel is a brief listener acknowledgment that need not claim it. Consequently, detecting overlap is not enough to choose a response. A short yeah can signal continued attention, agreement, or the beginning of a correction. Duration is useful evidence but not an intent label.

A transition-relevance place is a possible completion at which speaker change can occur. The earlier turn-allocation account explains why a person may continue when nobody else takes the floor. A voice policy therefore needs more than answer versus silence: it may keep listening, acknowledge without expanding, begin a response, or yield to a substantive interruption. In mixed-initiative dialogue, either participant can redirect the exchange rather than following only system-selected questions.

Human timing helps explain expectations, but it is not a universal machine deadline. A 2009 study across ten languages examined responses to questions ordinarily inviting yes or no. The most frequent response timing fell between zero and 200 ms, while averages differed across languages and response types. Those observations do not cover every task, speaking style, or access need.

Choose interruption handling separately from completion detection. LiveKit's documented policies distinguish activity-triggered interruption, adaptive handling intended to recognize backchannels, and false-interruption recovery. One recovery heuristic resumes an answer when detected speech is followed by no transcription. An empty transcript is not proof that nothing meaningful happened, so test this behavior on the intended speech population.

Acknowledgments should claim only what the system knows. A brief indication of listening is different from confirming a correctly interpreted identifier or announcing completed work. Continuous listening does not require constant speech; unnecessary acknowledgments can themselves seize attention or interrupt delivery.

Interruption and continuity

Stop obsolete speech

Barge-in lets user speech interrupt ongoing system output. The distinction between detecting sound and accepting an interruption is longstanding: the March 2004 VoiceXML 2.0 Recommendation distinguished stopping on detected speech from waiting for a complete match to an active grammar. Whichever policy accepts the interruption, stopping the output requires control over more than the language model.

The playback owner must suppress current sound and remove obsolete queued audio. Generation and synthesis should receive cancellation requests, but callbacks may still arrive. Give each response an identity and reject output whose identity is no longer admissible. At the same time, preserve new user input, including the audio preceding a delayed speech-start event. A short pre-roll buffer serves that last obligation.

Pipecat's interruption contract illustrates the coordination: an interruption bypasses ordinary queued processing, cancels current work, and discards interruptible frames. Some work can remain uninterruptible; function-call cancellation is configurable. Its TTS context APIs also check context identity when accepting later audio. These are concrete local controls, not proof of remote-provider cancellation or immediate silence at every device.

Illustrative pseudocode

Python-like pseudocode
# Conceptual guard; callback acceptance and invalidation are serialized.
def accept_audio(chunk, state):
    if chunk.response_id != state.active_response_id:
        return "discarded_stale"
    if not state.output_enabled:
        return "discarded_stopped"
    state.playback_queue.offer(chunk)
    return "queued"

This guard prevents old callbacks from being accepted again; it does not clear audio already held by the device. Likewise, forwarding an interruption event is not an acknowledgment that all downstream buffers have cleared. Distinguish a cancellation request from quiescence, when the covered work has actually stopped, using Define interruption and cancellation honestly.

External effects have another lifecycle. Interrupting a spoken booking confirmation does not undo a booking; a timed-out request may already have committed. Resolve the operation using its authoritative state and supported retry identity rather than replaying it because the audio stopped. Recover when the effect is unknown teaches that boundary.

Retain only supported playback history

Generation commonly gets ahead of playback. If the user interrupts halfway through a response, retaining the complete generated message as delivered history can make the next answer assume the person received an explanation they never heard. The relevant boundary is output progress, not generation completion. Even completed playback does not establish attention or comprehension.

Different playback contracts support different history precision.
ContractAvailable boundaryLimit
Realtime WebSocket truncationClient-accounted played duration identifies an audio endpointThe guide does not provide precise transcript-to-audio alignment or a precise shortened transcript
Pipecat with supported TTS word timingOutput-synchronized text frames can update assistant context incrementallyPrecision depends on provider timing and aggregator placement; it does not prove hearing
Twilio playback marksA normally completed mark identifies preceding playout completionClear also returns outstanding marks, so a returned mark alone does not prove playback

For Twilio Media Streams, maintain the ordered media and mark records together with the clear lifecycle. Treat marks outstanding during clearing conservatively. A partly played chunk may have no exact word boundary available. Do not manufacture one by taking a percentage of the generated text.

Stop response A without inventing delivered history

A conceptual chunk-accounting example across application responsibilities, not a combined provider protocol. External action state is outside this output-control boundary.

LaneBefore interruptionInterruption acceptedLater observations
Incoming captureCapture stays open; retain pre-roll.Preserve the new user contribution and its onset audio.Input survives response A invalidation.
Response productionResponse A generation runs ahead of playback.Invalidate A, then request generation/TTS cancellation.Late A4 → identity guard → discard. A4 cannot enter playback.
Playback ownerA1 complete; A2 playing; A3 queued.Clear queued A3 and separately request stopping current device output.Observe device cessation separately; a cancel request is not proven silence.
History assemblyGenerated text may exceed played content.Retain ordered chunk/mark records and the clear lifecycle.Use the media-status ledger; do not claim the whole generated answer was delivered.
Response A mediaStatus after interruptionSupported next-history claim
A1Normally completedSupported playout boundary; not attention or comprehension.
A2Partly playedKeep uncertainty about exact word coverage without alignment.
A3Pending → clearedExclude from delivered history; a mark returned during clear is not normal completion.
A4Late → discardedFailed the invalidated-response guard; never re-entered playback.

Serialize invalidation with callback acceptance. The guard protects future acceptance; it cannot remove samples already held by a device. Never convert played-audio percentage into a text prefix.

A1 completed normally; A2 was interrupted with uncertain word coverage; A3 was cleared while pending; late A4 was discarded. Generated content is not automatically delivered history, and marks returned during clearing do not establish normal completion.

Keep generated content available for diagnosis if permitted, but distinguish it from content supported as delivered in the next model input. Where delivery is uncertain, a targeted repeat or confirmation is more honest than assuming the whole suffix arrived. Conversation history is a representation assembled for use, not complete conversation truth; State beyond the prompt develops that distinction.

Audible performance

Measure onset, continuity, and stopping

Latency needs a start event, an end event, and an observation location. First generated audio is useful for diagnosing synthesis, but the listener may still wait for transport, buffering, and device output. A system can begin quickly and then stall, or finish generating while a long queue continues playing. Measure those outcomes separately.

MeasurementBoundary to record
Audible response onsetEnd of the relevant user contribution → first audible response
Synthesis startupAccepted synthesis input → first generated audio chunk
Interruption stoppingOnset of the accepted interruption → cessation of obsolete playback
Playback continuityUnintended gaps or concealed intervals during the response
Useful answer completionUser contribution end → delivery of the required answer content
RTF=TcomputeTaudio.\mathrm{RTF}=\frac{T_{\mathrm{compute}}}{T_{\mathrm{audio}}}. The real-time factor compares processing time with generated audio duration. Producing ten seconds of audio in five seconds gives RTF = 0.5. A value below one means generation outpaces playback on average over that accounting interval; it does not guarantee fast startup or regular chunk arrival.

Some tools report RTFX, audio duration divided by computation time. It is the reciprocal only when workload and accounting interval match. Concurrent aggregate throughput must not be mistaken for one stream's sustainable cadence. Riva's performance documentation accordingly separates first-chunk latency, later chunk intervals, and throughput.

Streaming stages overlap. Recognition can run during speech, and synthesis can begin before all response text is complete. The critical path is the dependency chain determining the event of interest; summing every stage duration overcounts concurrent work. In a hypothetical schedule, speech ends at 600 ms, a usable response prefix arrives at 1,100 ms, the first generated audio at 1,200 ms, and playback starts at 1,300 ms. The 700 ms interval to audible onset includes the dependencies and delivery waiting. Latency has boundaries and Attribute elapsed time provide the general measurement framework.

Audible onset is not summed model time

Audible onset follows the dependency path, not summed stage timeUser speech ends at 600 ms, usable response text appears at 1100 ms, first audio is generated at 1200 ms, and playback starts at 1300 ms. The 700 ms interval runs from speech end to audible onset. Playback observation ends at 1800 ms without claiming answer completion.060090013001800User speech captureStreaming recognitionCompletion waitingResponse text generationStreaming synthesisDelivery + bufferingObserved playbackTA600 → 1,300 ms = 700 ms onset delayT: usable text at 1,100 ms · A: first generated audio at 1,200 ms · shared clock in ms
In this hypothetical schedule, speech ends at 600 ms and playback begins at 1,300 ms: 700 ms to audible onset. Recognition and synthesis overlap other work. The 1,800 ms endpoint is an observation cutoff, not answer completion. All spans share the observed exchange window; containment does not encode dependencies.

Record distributions under the intended load, devices, networks, and conversation lengths. A 95th percentile describes a boundary below which 95% of the measured values fall; it exposes slow turns hidden by a median. Device latency remains relevant after generation: AudioContext.outputLatency is an estimate, not an acoustic measurement at the ear. Deployment placement also matters—several distant inference round trips can cost more than one long media path with nearby processing stages.

Prepare early without claiming completion

Response preparation can start before speaking is appropriate. LiveKit's preemptive generation can prepare language-model output before turn completion; synthesis waits by default unless separately enabled. If the completion hook changes context or tools, the prepared response is discarded. The benefit depends on its assumptions surviving. Long dictation and revisable input can turn preparation into wasted work.

A tool call is a structured request to external software, not evidence that the requested effect occurred; see A call is an envelope, not an effect. For a slow lookup, the agent can briefly announce what it is doing while the request runs. Longer operations can return a task identifier and expose a separate status operation. The announcement reduces unexplained silence, not backend execution time.

Tie spoken claims to the state actually reached.
Known stateAppropriate feedback
Possible answer preparedNo claim that the user finished or that the answer is valid
Lookup startedA brief description of the lookup
Result receivedAn answer supported by that result
External action confirmedA completion statement matching the confirmation

An early sound is not the same as an early completed answer. In a 2010 Swedish bargaining-system study, ten participants used incremental and nonincremental versions in counterbalanced order, with a human supplying transcriptions. Incremental generation reduced mean response-onset delay from 2.84 to 0.58 seconds, but completion delay changed from 5.66 to 5.02 seconds. Some ratings improved; overall preference did not differ significantly. This isolates a useful distinction without establishing current end-to-end agent performance.

Output checks also compete for the response budget. A policy violation detected after speech has played cannot retract that speech. Place required checks before the relevant audible commitment, and measure the added waiting. Streaming is a choice about commitment granularity, not permission to postpone every validation until the end.

Reconstruct the audible interaction

A trace correlates operations; an event records an occurrence. Their general structure is covered in Traces, spans, events, and links. For voice, correlate them with media and response identities. A transcript plus successful server calls cannot explain whether the user was cut off, an obsolete chunk played, or a delegated model received incomplete context.

A practical event record preserves enough information to join observations without recording every payload.
Field groupPurpose
Session, stream, turn, response, chunk IDsConnect audio, hypotheses, generated output, and cancellation
Event type and revisionDistinguish proposed text, final text, policy decisions, and later corrections
Observation location and clockIdentify capture, server, transport, or playback timing
Media position and formatMap an event to the corresponding source interval
Queue or playback statusSeparate generated, pending, cleared, and normally completed output
Protected content reference and availabilityPermit authorized inspection without treating omitted data as empty data

Do not compare unrelated clock values as though they share an origin. RTP sampling timestamps, packet arrival times, server clocks, and device presentation times describe different coordinates. The accompanying RTP Control Protocol (RTCP) supplies sender reports that pair media timestamps with a shared reference clock. Those pairs help align streams, but do not measure actual device output. Record the mapping and its uncertainty. Where only an output-latency estimate exists, label the resulting playback time as estimated.

Investigate the first consequential divergence. Compare captured audio with recognition, then recognition with the turn decision, the decision with generated output, and output with playback. Include tool arguments, results, and delegated context when they matter. Permissioned audio replay can reveal timing that text hides, but recordings, transcripts, and derived speaker representations need separate purposes, access rules, and retention. Govern telemetry as sensitive data covers those responsibilities.

Understandable and controllable exchanges

Repair the misunderstanding

Conversational repair resolves a breakdown in mutual understanding. It starts by identifying what is uncertain: the requested operation, a particular detail, or whether the answer was received. Establishing enough mutual understanding to proceed is called grounding. To preserve progress during repair, distinguish the transcript from dialogue state, the interpreted constraints accumulated across exchanges. That state includes the intent, or requested operation, and its slots, arguments such as destination and date. Workflow state separately records what was executed.

Choose the repair from the uncertain claim.
BreakdownRepair
Missing intent or argumentAsk for the specific missing information
Possibly misrecognized detailConfirm the candidate interpretation
Content may not have playedRepeat or summarize the relevant portion
The response was wrongCorrect it explicitly and update dependent state

Explicit confirmation asks the person to verify an interpretation before proceeding. Implicit confirmation incorporates it into the next contribution, leaving room for correction. Use stronger confirmation when a wrong detail would have greater consequences. For example, if only a place name is uncertain, ask “Did you say Newark?” rather than requesting the entire journey again. Checking that interpretation does not itself authorize an external action.

Spoken answers should expose a useful next step before demanding extensive listening. Group choices, pause at meaningful boundaries, and leave room for correction. If clarification repeatedly fails, offer text, another input method, or human help. After connection loss, recover from supported dialogue and workflow state: uncertainty about what played should remain separate from uncertainty about what an external service did.

Make listening and stopping controllable

People need to know whether the system is receiving sound, preparing a reply, or still playing one. Apply Separate receipt, progress, and outcome to those actual states. Mute affects input; stop affects output; cancelling work and deleting a recording are different operations. One indicator cannot truthfully stand for all of them.

StateTruthful cueRelevant control
Microphone input enabledMicrophone enabledMute or an explicit input gate
Response preparation runningPreparing a responseCancel or redirect
Audio queued or playingSpeaking or stoppingStop playback and clear pending speech
Playback cessation observedPlayback stoppedRepeat or continue
Recording retainedRecording and retention statusApplicable retention and deletion controls

Push-to-talk supplies an explicit input boundary: press to begin and release to finish. It can reduce ambiguity when a button is practical, but it neither reverses external work nor removes the need to stop existing playback. The 1993 VoiceNotes prototype combined speech and buttons for different circumstances. Its formative testing also exposed ambiguous acknowledgments: selecting a category and asking one's location initially sounded the same. Distinct feedback made the state change clearer.

Access needs affect timing and recovery. The W3C's 2026 draft guidance for voice interfaces recommends adjustable timing and speech rate, manageable choices, and alternatives when speech fails. Some people need longer to respond; silence must not automatically mean rejection. Provide repeat, captions or a conversation record where appropriate, text input, and human alternatives. Keep audible cues brief enough not to compete with speech. These are design considerations, not a conformance claim; Accessibility is interaction behavior develops the broader obligation.

Live microphone processing does not establish that audio is stored, and a microphone indicator does not establish permission for every later use. Explain collection and retention separately, following Establish permitted uses.

Evaluation and design choices

Measure speech properties separately

Component evaluation should identify what changed without claiming more than it measured. Recognition fidelity, speaker attribution, pronunciation, naturalness, and timing need different references. Follow Choose checks that match the requirement, then connect those checks to the voice-specific failure.

WER=100S+D+IN%.\mathrm{WER}=100\frac{S+D+I}{N}\%. Word error rate counts substitutions SS, deletions DD, and insertions II after aligning a hypothesis with a reference of NN words. Insertions can make WER exceed 100%. This formula is undefined for an empty reference; score non-speech outputs separately. Character error rate applies the analogous calculation to character units.

Specify normalization and tokenization before scoring. Removing punctuation, standardizing written numbers, or changing word segmentation changes the comparison. WER also counts word disagreements rather than consequences. In the invented reference “Please send fifteen boxes,” replacing Please with Kindly and replacing fifteen with fifty each creates one substitution out of four words—25% WER—but only the latter changes quantity. Track critical details separately.

Mean opinion score (MOS) averages human ratings under a stated protocol; it is not a universal objective unit. Speech material, listeners, equipment, noise, instructions, and rating scale affect the result. ITU-T P.800 separates listening quality from effort and loudness preference. For synthesis, test intelligibility and intended words alongside naturalness: pleasant audio can still pronounce the wrong name or omit content.

Disaggregate recognition by relevant speakers and conditions. Allison Koenecke and colleagues' 2020 study evaluated five commercial services on interview speech from 73 Black and 42 white speakers. Average WER across those services was 35% and 19%, respectively. These historical measurements are not current rankings or an inherent difference in recognizability; corpus and geographic differences constrain interpretation. They demonstrate why an aggregate alone is insufficient.

For interaction metrics, define eligible events and record failures alongside latency.
TargetRequired observationWhat it can miss
Premature endpointingWhether listening stopped before the reference contribution endedWords misrecognized after complete capture
Interruption handlingAccepted interruption, obsolete playback cessation, and subsequent responseAn answer-latency score alone misses stopping delay
Backchannel handlingWhether acknowledgment preserves the ongoing contributionShort duration alone does not establish intent
DiarizationSpeaker timelines, including overlapCorrect words assigned to the wrong participant
Response latencyNamed onset boundary plus nonresponse countA conditional percentile can hide turns with no answer

Full-Duplex-Bench separates pauses, backchannels, transitions, and interruptions using timed input and output. Its response latency is conditional on taking a turn, so nonresponses require a separate rate. Post-interruption response latency is also not playback-stop latency. Preserve these distinctions when adapting a benchmark to a product.

Test complete conversations

Component gains matter when they improve the intended exchange. The PARADISE framework, introduced in 1997, separated task success, dialogue costs, and user satisfaction. A corrected misunderstanding adds repair cost; an unresolved one can prevent success. Its illustrative framework does not prescribe universal metric weights or imply that fewer turns are always better.

Use complementary tests rather than expecting one method to establish the whole experience.
MethodWhat it establishesWhat needs another test
Recorded-audio replayBehavior on the same words, pauses, and acoustic conditionsHow a person changes behavior in response to the system
Timed media fault testsResponses to scheduled network changes, echo, and playback disturbancesCorrect interpretation and useful dialogue
Dialogue simulationCoverage of specified dialogue patterns under the simulatorUnmodeled acoustic failures and human adaptation
Representative participant studiesTask success, effort, recovery, and perceived control during interactionBroader populations and conditions not included

Timed integration tests should observe output, not merely successful calls. WebRTC's media testing framework can schedule network changes and export captured or rendered audio with concealment and buffer statistics. Add application assertions: an invalidated response must not re-enter playback, new user audio must survive interruption, and a clear event must not become a false delivery claim. These assertions test the coordination between otherwise functioning components.

Validate simulations against the behavior they are meant to represent. Hua Ai and Fuliang Weng's 2008 restaurant-dialogue study related dialogue properties to satisfaction from 20 participants, then used that relationship with simulated dialogues. Predicted and observed satisfaction did not differ significantly, but the simulator used the human corpus and bypassed recognition and synthesis. The result was not independent end-to-end validation of spoken interaction.

Compare policies on matched tasks with the intended participants and devices. Include completion, critical-detail correctness, repair success, repeated effort, unwanted overlap, abandonment, and control. A controlled human-conversation study found greater sensitivity to delay in more interactive tasks, reinforcing that acceptable timing depends on the activity. Evaluate use, reliance, and recovery develops the complete-workflow perspective.

Finish with a decision record, not a single quality score: which information requires an audio-preserving path; which words need confirmation; what evidence ends a turn; how much audio may queue; who owns stopping; how uncertain delivery is repaired; and which participant and fault tests support those choices. More patient endpointing may be worth extra waiting. A smaller buffer may be worth greater underrun risk. The justified choice is the one whose combined behavior serves the intended conversation.

Open questions

  1. Adaptive turn policies must become more patient without creating persistent unexplained silence. The difficulty is distinguishing hesitation, atypical speech, acknowledgment, and abandoned input across people and tasks. Progress would mean fewer cutoffs and successful bounded recovery, measured together with added waiting in representative conversations.

  2. Full-duplex models must combine fluid overlap with dependable role and task behavior. Timing competence and instruction-following are different targets, and convincing demonstrations do not establish their joint reliability. Progress would require matched interactive evaluations that retain both conversational and task outcomes.

  3. Audio-preserving models need evidence that they use vocal cues appropriately rather than merely receiving them. Delivery depends on context, and factual-answer training can ignore it. Useful progress would demonstrate task-relevant sensitivity to controlled changes in delivery without claiming certainty about a speaker's internal state.

  4. Playback history remains limited by transport and alignment granularity. Systems can know that some audio completed or was cleared without knowing the exact interrupted words. Better contracts would expose compatible output identities and timing uncertainty while preserving the distinction between playout and comprehension.

  5. Replay and simulation need stronger links to human adaptation. A fixed speaker recording cannot slow down, repeat, abandon, or learn a system's timing habits. Progress would establish which replay improvements predict reduced effort and better completion in independent live interactions.

Follow the curated reading path through the speakers and demonstrations behind this entry.

27 min

AI Engineer World's Fair 2025 · 2025

Why ChatGPT Keeps Interrupting You

Tom Shapland, PhD

Cited in this entry

Explains why speech activity, conversational completion, and incoming backchannels require different decisions. Its examples clarify why changing one silence timeout cannot resolve every interruption problem.

Watch talk
86 min

AI Engineer World's Fair 2025 · 2025

Building voice agents with OpenAI

Dominik Kundel

Cited in this entry

Provides concrete examples of delegation, overlapping speech with tool work, and inspecting audio alongside tool context. Product details are historical; the coordination patterns remain the useful contribution.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

142 matching talks

TalkSpeakerEventYear
Anoop Kotha, Toki SherbakovAI Engineer World's Fair 20252025
Chintan Agrawal, Daniel WirjoAI Engineer World's Fair 20262026
Neil ZeghidourAI Engineer Europe 20262026
Ronan McGovernAI Engineer World's Fair 20252025
Kwindla Hultman KramerAI Engineer World's Fair 20242024
Adam TerlsonAI Engineer Summit 20252025
Vinoth GovindarajanAI Engineer World's Fair 20262026
Vivek MuppallaAI Engineer World's Fair 20262026
Kyle KranenAI Engineer World's Fair 20252025
Giving a Voice to AI Agents

Cited in this entry

Scott StephensonAI Engineer World's Fair 20242024
Arjun Desai, Rohit TalluriAI Engineer World's Fair 20252025
Cormac BrickAI Engineer World's Fair 20262026
Joe ReeveAI Engineer Europe 20262026
Travis Bartley, Myungjong Kim, Byungjoong, JaehanAI Engineer World's Fair 20252025
Nishant GuptaAI Engineer World's Fair 20262026
Kwindla Hultman KramerAI Engineer World's Fair 20252025
Lars GrammelAI Engineer Summit 20232023
Allen PikeAI Engineer World's Fair 20262026
Neil Dwyer, Jack DwyerAI Engineer World's Fair 20252025
Mark Backman, AleixAI Engineer World's Fair 20252025
Juan PeredoAI Engineer Summit 20252025
Andres MarafiotiAI Engineer Europe 20262026
Damien MurphyAI Engineer World's Fair 20242024
Todd FisherAI Engineer World's Fair 20262026
Chad Bailey, Brian JohnsonAI Engineer World's Fair 20252025
Rafal Wilinski, Vitor BaloccoAI Engineer World's Fair 20252025
Thor Schaeff, Philipp SchmidAI Engineer Europe 20262026
Thor Schaeff, PaulAI Engineer World's Fair 20252025
Brooke HopkinsAI Engineer World's Fair 20252025
Romain HuetAI Engineer World's Fair 20242024
Thor SchaeffAI Engineer Europe 20262026
Luke HarriesAI Engineer Europe 20262026
Philip KielyAI Engineer World's Fair 20252025
Peter BarAI Engineer World's Fair 20252025
Gregory BrussAI Engineer World's Fair 20252025
Eddie SiegelAI Engineer Summit 20252025
Sidney PrimasAI Engineer World's Fair 20262026
Nik CaryotakisAI Engineer Summit 20252025
Dippu Kumar SinghAI Engineer Europe 20262026
Suman DebnathAI Engineer World's Fair 20252025
Joel Allou, Ornella BahidikaAI Engineer World's Fair 20262026
Logan KilpatrickAI Engineer World's Fair 20252025
Armanas PovilionisAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer World's Fair 20252025
Nathan WanAI Engineer World's Fair 20252025
Patrick LöberAI Engineer Europe 20262026
Richmond AlakeAI Engineer World's Fair 20252025
Filip KozeraAI Engineer World's Fair 20252025
Rita KozlovAI Engineer World's Fair 20252025
Tom RedmanAI Engineer World's Fair 20242024
Cedric VidalAI Engineer World's Fair 20252025
Building Reactive AI Apps

Metadata candidate

Matt WelshAI Engineer Summit 20232023
Jamie Neuwirth, Zack WittenAI Engineer World's Fair 20242024
Ben HolmesAI Engineer World's Fair 20252025
Rachna SrivastavaAI Engineer World's Fair 20252025
Jedrick Kosinski, ComfyAnonymousAI Engineer World's Fair 20252025
Louis-François Bouchard, Omar Solano, Samridhi VaidAI Engineer World's Fair 20262026
Convex Launch

Metadata candidate

Jamie TurnerAI Engineer World's Fair 20242024
Develop at Idea Velocity

Metadata candidate

Jeffrey Lee-ChanAI Engineer World's Fair 20262026
Ornella Bahidika, Joel AllouAI Engineer World's Fair 20262026
Dat Ngo, Aman KhanAI Engineer World's Fair 20252025
Aparna Dhinkaran, Aparna DhinakaranAI Engineer Summit 20252025
Evaling Video Slop

Metadata candidate

Maor BrilAI Engineer World's Fair 20262026
Sumaiya ShrabonyAI Engineer World's Fair 20262026
Craig WattrusAI Engineer World's Fair 20252025
Chaitanya AsawaAI Engineer World's Fair 20262026
Fuzzing in the GenAI Era

Metadata candidate

Leonard TangAI Engineer World's Fair 20252025
Cassidy HardinAI Engineer Europe 20262026
Dave BurnisonAI Engineer World's Fair 20242024
Victoria MelnikovaAI Engineer World's Fair 20262026
Zhou YuAI Engineer Summit 20252025
Jared HansonAI Engineer World's Fair 20252025
Ian WebsterAI Engineer World's Fair 20242024
Xiaofeng WangAI Engineer Summit 20252025
Guillaume VernadeAI Engineer Europe 20262026
Adam BehrensAI Engineer World's Fair 20252025
Cornelia DavisAI Engineer World's Fair 20262026
Kwindla Kramer, Shrestha Basu MallickAI Engineer World's Fair 20252025
Rami AlhamadAI Engineer World's Fair 20252025
Ahmed MenshawyAI Engineer World's Fair 20242024
Simon WillisonAI Engineer Summit 20232023
Phil NashAI Engineer Europe 20262026
Benjamin SteinAI Engineer World's Fair 20242024
Randall HuntAI Engineer World's Fair 20252025
Steve KorshakovAI Engineer World's Fair 20262026
RAG for VPs of AI

Metadata candidate

Jerry LiuAI Engineer World's Fair 20242024
Boris StarkovAI Engineer Europe 20262026
Adrien GrondinAI Engineer Europe 20262026
See, Hear, Speak, Draw

Metadata candidate

Logan Kilpatrick, Simón FishmanAI Engineer Summit 20232023
Mike ChambersAI Engineer World's Fair 20252025
Jared JoselowitzAI Engineer World's Fair 20262026
Sarah GuoAI Engineer World's Fair 20252025
Karan GoelAI Engineer World's Fair 20242024
Isadora Martin-DyeAI Engineer World's Fair 20262026
Rob CheungAI Engineer World's Fair 20242024
Brendan O'DonoghueAI Engineer Europe 20262026
Barr YaronAI Engineer World's Fair 20262026
Zack Reneau-WedeenAI Engineer Summit 20252025
Dani Grant, Chelcie TaylorAI Engineer World's Fair 20252025
Maximillian PirasAI Engineer World's Fair 20252025
Jeremy Silva, Chris HernandezAI Engineer World's Fair 20252025
The End of Apps

Metadata candidate

KitzeAI Engineer Europe 20262026
Stefania DrugaAI Engineer World's Fair 20242024
Kwindla Kramer, Kwindla Hultman KramerAI Engineer World's Fair 20262026
Diego Rodriguez, Eugene, Jonas Bauer, Shijia Liao, David Vorick, Alex AtallahAI Engineer World's Fair 20252025
Ted JohnsonAI Engineer World's Fair 20262026
Amir HaghighatAI Engineer World's Fair 20252025
Thinking Deeper in Gemini

Metadata candidate

Jack RaeAI Engineer World's Fair 20252025
tldraw computer

Metadata candidate

Steve RuizAI Engineer World's Fair 20252025
Veo 3 for developers

Metadata candidate

Paige BaileyAI Engineer World's Fair 20252025
Harald KirschnerAI Engineer World's Fair 20252025
Why MLX

Metadata candidate

AI Engineer Europe 20262026
Zack ProserAI Engineer Europe 20262026
Phlo YoungAI Engineer World's Fair 20242024
Łukasz GandeckiAI Engineer World's Fair 20252025
Paige BaileyAI Engineer Europe 20262026
Paige Bailey, Guillaume Vernade, Ian BallantyneAI Engineer Europe 20262026
Mehedi HassanAI Engineer Europe 20262026
fighting slop with slop

Metadata candidate

Vaibhav GuptaAI Engineer World's Fair 20262026
Cedric VidalAI Engineer World's Fair 20242024
Eugene YanAI Engineer World's Fair 20252025
Angelos PerivolaropoulosAI Engineer Europe 20262026
Sumanyu SharmaEvent unknownUnknown
Jeremy AdamsEvent unknownUnknown
Emile Baizel, Shruti AroraAI Engineer World's Fair 20262026
Zack Proser, Nick NisiEvent unknownUnknown
Charlie Guo, Gabriel ChuaAI Engineer World's Fair 20262026
Bohan LiEvent unknownUnknown
Paula DozsaEvent unknownUnknown
Fuad AliEvent unknownUnknown
Charlie GuoEvent unknownUnknown
Jesse HallAI Engineer World's Fair 20262026
Philipp SchmidAI Engineer World's Fair 20262026
Paige BaileyAI Engineer World's Fair 20262026
Valeria Wu Fon, Tom OuyangEvent unknownUnknown
Matt LawlerAI Engineer World's Fair 20262026
Neil ZeghidourEvent unknownUnknown
Jesse HallEvent unknownUnknown
Midam KimEvent unknownUnknown
Thor SchaeffAI Engineer World's Fair 20262026
Elizabeth Fuentes LeoneAI Engineer World's Fair 20262026
Venky BEvent unknownUnknown

References

Coverage and source review
Processed transcripts
32 processed in full · 6 in the curated path
Automated source review
Passed
Metadata candidates
116 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Web Audio API 1.1: AudioBuffer, decoding and buffer playback

    https://www.w3.org/TR/webaudio/

  2. The Sampling Theorem

    https://dspguide.com/ch3/2.htm

  3. W3C WebRTC: receiver buffering and media transport

    https://www.w3.org/TR/webrtc/

  4. WebRTC NetEq

    https://webrtc.googlesource.com/src/%2B/refs/heads/main/modules/audio_coding/neteq/g3doc/index.md

  5. VoiceNotes: A Speech Interface for a Hand-Held Voice Notetaker

    https://www.media.mit.edu/speech/people/lisa/interchi93.html

  6. Conversational Interfaces: Advances and Challenges

    https://www.cs.cmu.edu/~dgroup/papers/zue97.pdf

  7. Giving a Voice to AI Agents

    https://www.youtube.com/watch?v=heYmh_lsX5s&t=166

  8. How to build the world's fastest voice bot

    https://www.youtube.com/watch?v=dRQHikOrH2A&t=255

  9. Building & Scaling an AI Agent Swarm of low latency real time voice bots!

    https://www.youtube.com/watch?v=7_WhRAuP2Wg&t=1248

  10. A Simplest Systematics for the Organization of Turn-Taking for Conversation

    https://pure.mpg.de/pubman/item/item_2376846_3/component/file_2376845/Sacks_1974_simplest_systematics.pdf

  11. Speech and Language Processing: chatbots and dialogue systems

    https://web.stanford.edu/~jurafsky/slp3/old_aug24/15.pdf

  12. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition

    https://www.fceia.unr.edu.ar/prodivoz/Rabiner_1989.pdf

  13. A General, Abstract Model of Incremental Dialogue Processing

    https://aclanthology.org/anthology-files/anthology-files/pdf/E/E09/E09-1081.pdf

  14. Speech and Language Processing: phonetics and speech feature extraction

    https://web.stanford.edu/~jurafsky/slp3/15.pdf

  15. Devices and Data Types: PCM Waveform-Audio Data Format and PCM Data Packing

    https://learn.microsoft.com/en-us/windows/win32/multimedia/devices-and-data-types

  16. WAVEFORMATEX structure (mmeapi.h)

    https://learn.microsoft.com/en-us/windows/win32/api/mmeapi/ns-mmeapi-waveformatex

  17. RFC 6716: Definition of the Opus Audio Codec

    https://www.rfc-editor.org/rfc/rfc6716.html

  18. Speech Synthesis Markup Language (SSML) Version 1.1

    https://www.w3.org/TR/speech-synthesis11/

  19. Media Capture and Streams

    https://www.w3.org/TR/mediacapture-streams/

  20. Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

    https://www.cs.toronto.edu/~graves/icml_2006.pdf

  21. Robust Speech Recognition via Large-Scale Weak Supervision

    https://arxiv.org/html/2212.04356

  22. Sequence Transduction with Recurrent Neural Networks

    https://arxiv.org/html/1211.3711

  23. NeMo ASR Models: Cache-Aware Streaming Conformer

    https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/models.html

  24. Engineering voice agents: Latency, quality, and scale

    https://www.youtube.com/watch?v=N7b1PJc7SFc&t=457

  25. Amazon Transcribe: streaming partial results and stabilization

    https://docs.aws.amazon.com/transcribe/latest/dg/streaming-partial-results.html

  26. Estimating Word-Stability During Incremental Speech Recognition

    https://www.isca-archive.org/interspeech_2012/mcgraw12_interspeech.pdf

  27. OpenAI Realtime transcription

    https://developers.openai.com/api/docs/guides/realtime-transcription

  28. OpenAI Agents SDK: Building Voice Agents

    https://openai.github.io/openai-agents-js/guides/voice-agents/build/

  29. 200 Million Patient Interactions Later: What the Generic Voice Stack Misses

    https://www.youtube.com/watch?v=AN65uc645mE&t=571

  30. 200 Million Patient Interactions Later: What the Generic Voice Stack Misses

    https://www.youtube.com/watch?v=AN65uc645mE&t=734

  31. Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio

    https://arxiv.org/html/2501.11378

  32. Speech and Language Processing: TTS and other speech tasks

    https://web.stanford.edu/~jurafsky/slp3/old_aug24/16.pdf

  33. Festival Speech Synthesis System: Lexicons

    https://www.cstr.ed.ac.uk/projects/festival/manual/festival_13.html

  34. WaveNet: A Generative Model for Raw Audio

    https://arxiv.org/html/1609.03499

  35. Parallel WaveNet: Fast High-Fidelity Speech Synthesis

    https://arxiv.org/pdf/1711.10433

  36. What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTS

    https://www.isca-archive.org/interspeech_2020/stephenson20_interspeech.pdf

  37. SoundStream: An End-to-End Neural Audio Codec

    https://arxiv.org/html/2107.03312

  38. High Fidelity Neural Audio Compression

    https://arxiv.org/html/2210.13438

  39. Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers

    https://arxiv.org/html/2301.02111

  40. Moshi: Official Implementation

    https://github.com/kyutai-labs/moshi

  41. Building voice agents with OpenAI

    https://www.youtube.com/watch?v=iXhba366fQc&t=316

  42. Pipecat Cloud: Enterprise Voice Agents Built On Open Source

    https://www.youtube.com/watch?v=IA4lZjh9sTs&t=1442

  43. Ultravox: Official Repository

    https://github.com/fixie-ai/ultravox

  44. Hello GPT-4o

    https://openai.com/index/hello-gpt-4o/

  45. Neil Zeghidour - Voice AI: when is the "Her" moment?

    https://www.youtube.com/watch?v=P_RI1kCkRbo&t=560

  46. Moshi: a speech-text foundation model for real-time dialogue

    https://arxiv.org/html/2410.00037

  47. Kyutai Unveils Moshi, the First Real-Time Voice AI

    https://kyutai.org/assets/pdfs/cp_moshi.pdf

  48. NVIDIA PersonaPlex: Natural Conversational AI With Any Role and Voice

    https://research.nvidia.com/labs/adlr/personaplex/

  49. Neil Zeghidour - Voice AI: when is the "Her" moment?

    https://www.youtube.com/watch?v=P_RI1kCkRbo&t=789

  50. Why ChatGPT Keeps Interrupting You

    https://www.youtube.com/watch?v=1v9zBiZKlIY&t=1033

  51. Python asyncio bounded queues

    https://docs.python.org/3/library/asyncio-queue.html

  52. Pipecat FrameQueue capacity and interruption reset

    https://raw.githubusercontent.com/pipecat-ai/pipecat/main/src/pipecat/utils/frame_queue.py

  53. Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Hultman Kramer (Daily)

    https://www.youtube.com/watch?v=E71YtNbCFXY&t=297

  54. Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Hultman Kramer (Daily)

    https://www.youtube.com/watch?v=E71YtNbCFXY&t=259

  55. Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Hultman Kramer (Daily)

    https://www.youtube.com/watch?v=E71YtNbCFXY&t=402

  56. RFC 3550: RTP: A Transport Protocol for Real-Time Applications

    https://www.rfc-editor.org/rfc/rfc3550.html

  57. Engineering voice agents: Latency, quality, and scale

    https://www.youtube.com/watch?v=N7b1PJc7SFc&t=574

  58. A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research

    https://link.springer.com/article/10.1186/s13634-016-0306-6

  59. WebRTC AudioProcessing interface: noise suppression and gain controllers

    https://webrtc.googlesource.com/src/%2B/refs/heads/main/api/audio/audio_processing.h

  60. Speex manual: echo cancellation

    https://www.speex.org/docs/manual/speex-manual/node7.html

  61. WebRTC AudioProcessing interface

    https://webrtc.googlesource.com/src/+/refs/heads/main/api/audio/audio_processing.h

  62. A New Robust Frequency Domain Echo Canceller with Closed-Loop Learning Rate Adaptation

    https://jmvalin.ca/papers/valin_icassp2007.pdf

  63. WebRTC NoiseSuppression interface at revision 159b417c98270f3c134c32d3d5fe763e2221ff8c

    https://webrtc.googlesource.com/src/%2B/159b417c98270f3c134c32d3d5fe763e2221ff8c/modules/audio_processing/noise_suppression.h

  64. Beyond Transcription: Building Voice AI That Actually Understands Conversations

    https://www.youtube.com/watch?v=mFLlVpnGpds&t=605

  65. Beyond Transcription: Building Voice AI That Actually Understands Conversations

    https://www.youtube.com/watch?v=mFLlVpnGpds&t=1168

  66. Why ChatGPT Keeps Interrupting You

    https://www.youtube.com/watch?v=1v9zBiZKlIY&t=224

  67. Silero VAD: VADIterator Implementation

    https://raw.githubusercontent.com/snakers4/silero-vad/master/src/silero_vad/utils_vad.py

  68. OpenAI Realtime: Voice activity detection

    https://developers.openai.com/api/docs/guides/realtime-vad

  69. Voice Activity Projection: Self-supervised Learning of Turn-taking Events

    https://arxiv.org/html/2205.09812

  70. From User Perceptions to Technical Improvement: Enabling People Who Stutter to Better Use Speech Recognition

    https://www.cs.cmu.edu/~jbigham/pubs/pdfs/2023/stutter-speech.pdf

  71. Designing Voice Agents for Real Conversations

    https://www.youtube.com/watch?v=hMlLw1LeIK8&t=1690

  72. Designing Voice Agents for Real Conversations

    https://www.youtube.com/watch?v=hMlLw1LeIK8&t=519

  73. Full-Duplex-Bench: A Benchmark to Evaluate Full-Duplex Spoken Dialogue Models on Turn-taking Capabilities

    https://arxiv.org/html/2503.04721v3

  74. Why ChatGPT Keeps Interrupting You

    https://www.youtube.com/watch?v=1v9zBiZKlIY&t=1552

  75. Building & Scaling an AI Agent Swarm of low latency real time voice bots!

    https://www.youtube.com/watch?v=7_WhRAuP2Wg&t=1629

  76. Universals and Cultural Variation in Turn-Taking in Conversation

    https://cognitionandculture.net/wp-content/uploads/Stivers_2009_universals.pdf

  77. LiveKit Agents: Turns overview

    https://docs.livekit.io/agents/logic/turns/

  78. Grounding in Communication

    https://web.stanford.edu/~clark/1990s/Clark,%20H.H.%20_%20Brennan,%20S.E.%20_Grounding%20in%20communication_%201991.pdf

  79. Voice Extensible Markup Language (VoiceXML) Version 2.0

    https://www.w3.org/TR/voicexml20/

  80. OpenAI Realtime conversations

    https://developers.openai.com/api/docs/guides/realtime-conversations

  81. Pipecat TTSService source: interruption and audio contexts

    https://reference-server.pipecat.ai/en/stable/_modules/pipecat/services/tts_service.html

  82. Pipecat: Interruptions

    https://docs.pipecat.ai/pipecat/fundamentals/interruptions

  83. Pipecat BaseOutputTransport source: interruption ordering

    https://reference-server.pipecat.ai/en/latest/_modules/pipecat/transports/base_output.html

  84. Making Retries Safe with Idempotent APIs

    https://aws.amazon.com/builders-library/making-retries-safe-with-idempotent-APIs/

  85. How to talk to statues — Joe Reeve, ElevenLabs

    https://www.youtube.com/watch?v=u-rJwPPU3QA&t=1698

  86. Pipecat: Context Management

    https://docs.pipecat.ai/pipecat/learn/context-management

  87. Twilio Media Streams: marks and buffer clearing

    https://www.twilio.com/docs/voice/media-streams/websocket-messages

  88. Audio Output Latency

    https://web.dev/articles/audio-output-latency

  89. How to build the world's fastest voice bot

    https://www.youtube.com/watch?v=dRQHikOrH2A&t=808

  90. NVIDIA Riva: TTS Performance

    https://docs.nvidia.com/deeplearning/riva/user-guide/docs/tts/tts-performance.html

  91. Building & Scaling an AI Agent Swarm of low latency real time voice bots!

    https://www.youtube.com/watch?v=7_WhRAuP2Wg&t=1744

  92. Designing Voice Agents for Real Conversations

    https://www.youtube.com/watch?v=hMlLw1LeIK8&t=1135

  93. Pipecat Cloud: Enterprise Voice Agents Built On Open Source

    https://www.youtube.com/watch?v=IA4lZjh9sTs&t=941

  94. LiveKit Agents: Audio

    https://docs.livekit.io/agents/multimodality/audio/

  95. Building voice agents with OpenAI

    https://www.youtube.com/watch?v=iXhba366fQc&t=2675

  96. Towards Incremental Speech Generation in Dialogue Systems

    https://aclanthology.org/W10-4301.pdf

  97. Engineering voice agents: Latency, quality, and scale

    https://www.youtube.com/watch?v=N7b1PJc7SFc&t=1228

  98. Building voice agents with OpenAI

    https://www.youtube.com/watch?v=iXhba366fQc&t=709

  99. NIST Privacy Framework, Version 1.0

    https://www.nist.gov/document/nist-privacy-frameworkv10pdf

  100. Beyond Transcription: Building Voice AI That Actually Understands Conversations

    https://www.youtube.com/watch?v=mFLlVpnGpds&t=256

  101. PARADISE: A Framework for Evaluating Spoken Dialogue Agents

    https://aclanthology.org/P97-1035.pdf

  102. Cognitive Accessibility Research Modules: Voice Systems and Conversational Interfaces

    https://www.w3.org/TR/2026/DNOTE-coga-voice-20260205/

  103. Engineering voice agents: Latency, quality, and scale

    https://www.youtube.com/watch?v=N7b1PJc7SFc&t=260

  104. Speech and Language Processing: ASR evaluation

    https://web.stanford.edu/~jurafsky/slp3/16.pdf

  105. Hugging Face Evaluate: Word Error Rate

    https://huggingface.co/spaces/evaluate-metric/wer/blob/main/README.md

  106. JiWER: Text Transformations and Alignment

    https://jitsi.github.io/jiwer/usage/

  107. ITU-T P.800: subjective determination of transmission quality

    https://www.itu.int/rec/dologin_pub.asp?id=T-REC-P.800-199608-I!!PDF-E&lang=e&type=items

  108. Racial Disparities in Automated Speech Recognition

    https://pmc.ncbi.nlm.nih.gov/articles/PMC7149386/

  109. Beyond Transcription: Building Voice AI That Actually Understands Conversations

    https://www.youtube.com/watch?v=mFLlVpnGpds&t=872

  110. WebRTC PeerConnection Level Framework

    https://webrtc.googlesource.com/src.git/%2B/refs/heads/main/test/pc/e2e/g3doc/index.md

  111. User Simulation as Testing for Spoken Dialog Systems

    https://aclanthology.org/W08-0126.pdf

  112. It Takes Two to Tango: Assessing the Impact of Delay on Conversational Interactivity on Perceived Speech Quality

    https://www.isca-archive.org/interspeech_2010/egger10_interspeech.pdf

  113. Full Workshop: Realtime Voice AI — Mark Backman, Daily

    https://www.youtube.com/watch?v=nxuTVd7v7dg&t=3674

  114. Beyond Transcription: Building Voice AI That Actually Understands Conversations

    https://www.youtube.com/watch?v=mFLlVpnGpds&t=149

  115. Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral

    https://www.youtube.com/watch?v=3jGAU2sbAyY&t=547

  116. Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral

    https://www.youtube.com/watch?v=3jGAU2sbAyY&t=737