AI Engineer World's Fair 2024

How to build the world's fastest voice bot

Loading slides…

Read the talk

How to build the world's fastest voice bot

An interruptible joke bot, a patient-intake call, and a geography game reveal how media transport, conversational state, model placement, and measurement shape voice AI latency.

From a talk by Kwindla Hultman Kramer

Before you start: Familiarity with language-model APIs and basic audio streaming will help; the article explains the voice-specific timing and orchestration concepts.

The user has already heard that joke

A chatbot starts a skeleton joke. The user interrupts: it always starts with that one. The bot switches to fake spaghetti, only to be interrupted again before finishing the punchline. Getting a language model to produce a joke is easy; making it stop, listen, and respond to the interruption is the voice engineering problem.

This roughly year-old demonstration comes from Kwindla Hultman Kramer’s work at Daily, a real-time audio and video infrastructure company. GPT-4 had made text conversation convincing enough to motivate experiments in spoken conversation. The team kept asking for jokes as an informal test throughout development, eventually learning the models’ recurring repertoire. Familiar content made the behavior around the words—especially interruptions—easy to notice.

0:150:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

A simple diagram hides a large runtime

The initial architecture looks straightforward: a person speaks into a phone or laptop, and the device exchanges data with GPUs in the cloud. Two requirements immediately complicate that picture: speed and architectural flexibility. Production systems must account for audio processing and echo cancellation, CPU time spent encoding and decoding media, firewall traversal, and application capabilities such as retrieval augmented generation and tool calling. Deployment and scaling cross all of these boundaries.

Slide titled “Building Voice AI Agents — Architecture,” with a phone connected to an AI cloud by a dashed arrow and surrounding engineering concerns.Open full source frame
The architecture behind a voice AI conversation spans client audio, transport, and AI services.

The runtime responsibilities become easier to reason about when grouped by what they protect:

  • Media delivery: Encode and transport audio reliably without accumulating unnecessary delay.
  • Transcription: Convert speech into usable text quickly. Even as audio-native models arrive, text remains useful for many integrations.
  • Buffers and pipelines: Preserve the audio representation and move streaming data between stages correctly. Kramer recalls answering roughly 20–30 Discord questions about audio that worked on a Mac but failed on an Intel cloud machine, failures he attributed to endianness. That is an anecdote about particular implementations, not a general incompatibility between those platforms.
  • Model composition: Swap models or combine several without rebuilding the surrounding application.
  • External systems and turn detection: Call application services and perform phrase endpointing—deciding when someone has finished speaking and expects the bot to respond.
  • Interruption state: Decide what stops and what conversational state survives when the user speaks over the bot.
  • Echo cancellation and voice output: Prevent speaker audio from returning through the microphone, then generate speech quickly enough to sustain the exchange.

Headphones cannot be an architectural dependency. Daily’s core product is a WebRTC stack, and Kramer’s experience is that universal headphone use is unrealistic—even in professional call-center settings. The application has to handle ordinary speakers and microphones.

1:241:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:24 · section reference included

Replacing a form with a conversation

Repeated medical forms provide a practical reason to assemble these components. Instead of filling out another paper form or following a pre-visit form link, a patient could receive a phone call or open an app link and have a conversation. In the demonstration, Jessica from Tri-County Health Services asks Chad for his complete birthday to confirm his identity.

Chad first supplies a partial birthday. Jessica begins asking for the missing information, but he interrupts to add the missing year. The next exchange follows the same pattern: he names lisinopril and gabapentin, then interrupts the dosage follow-up to add 10 mg and 100 mg respectively. Finally, he supplies latex and penicillin allergies. The system must carry information across turns and accept corrections while it is speaking; treating each transcript as an isolated request would lose the structure of the intake.

The call’s apparent simplicity depends on the underlying integration: conversation access, external data handling, streaming speech, function calling, and interruption support. Kramer describes the recording as unedited and the tool calling as real, but the external-system calls are mocked: no actual electronic health record backend is connected. He says this class of application is beginning to enter production.

A participant beside five callouts describing conversation access, external data storage, fast voice responses, streaming text-to-speech with function calling, and interruptions.Open full source frame
The conversation demo highlights external system integration, fast responses, function calling, and interruption handling.
5:105:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:10 · section reference included

Generating quickly is different from responding quickly

There are two kinds of fast. Throughput describes how quickly content continues to arrive; latency describes how long the user waits for it to begin. Kramer considers contemporary generation throughput generally sufficient to keep up with reading or listening. The difficult part is time to first byte or first token. His broad engineering comparison is that throughput often improves by an order of magnitude every couple of years, while latency improvements tend to be linear and much slower.

People do not wait for a complete utterance before preparing a reply. Kramer likens this anticipation to speculative decoding: predict the next four or five words, prepare a response, and reroute if the prediction is wrong. Kramer cites common human response gaps of 200–300 ms and uses approximately 500 ms as his voice-bot engineering target. Longer waits can work when the listener supplies some cue, but unexplained silence disrupts the conversational rhythm.

Viewed through that lens, the patient-intake demo has useful orchestration but noticeably slow responses. Model choice can exhaust the budget before the rest of the pipeline begins: Kramer’s historical example puts Gemini Pro time to first token at about 900 ms, already above his 500 ms target before other services and network overhead. The talk does not specify the model version or measurement conditions for that comparison.

7:427:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:42 · section reference included

Put the stages close enough to control

Choosing fast models is only part of the solution; where they run and how they connect also matter. Kramer’s strongest optimization is to bring as much processing as possible into one compute container: speech-to-text, phrase endpointing, LLM inference, and voice output. Running those stages locally together gives the developer more control over the combined path.

Kramer informally estimates that 95–99% of contemporary generative AI work calls hosted services. Those services have legitimate benefits, and latency need not be every application’s highest priority. The trade-off changes when the goal is the shortest possible response gap:

Deployment choiceMain advantageLatency consideration
Separate hosted servicesProvider-managed capabilitiesLess control over the combined path
Co-located, self-hosted stagesTune and combine processingRequires operating the models yourself

The reason to self-host here is concrete: it gives the team control over how stages run together, rather than merely control over which API it calls.

10:1710:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:17 · section reference included

Measure the silence between the waveforms

Voice-to-voice latency is the interval from the end of the user’s speech to the first audible bot response. Imagine the user’s green waveform and the bot’s purple waveform in an audio editor: the gap between them is the measurement that matters. It includes encoding, network transit, processing, the return trip, and speaker playout—not just the LLM request.

For an audio recording containing both sides on one timeline, the measurement reduces to sample positions. This Python function makes that boundary explicit; identifying the two speech boundaries is a separate step:

python

def voice_to_voice_ms(
    user_end_sample: int,
    bot_start_sample: int,
    sample_rate_hz: int,
) -> float:
    if sample_rate_hz <= 0:
        raise ValueError("sample_rate_hz must be positive")
    if min(user_end_sample, bot_start_sample) < 0:
        raise ValueError("sample positions must be nonnegative")
    return (
        (bot_start_sample - user_end_sample)
        * 1000.0
        / sample_rate_hz
    )

A negative result represents overlapping speech. For a normal turn transition, the positive result measures the silence the user experiences.

The budget starts before application code sees any bytes. Kramer puts macOS laptop microphone delay at approximately 30–40 ms, including processing such as echo cancellation, noise removal, and gain control. Audio then needs encoding and transmission. At the receiving end, packet handling, jitter-buffer reassembly, and decoding precede transcription.

The remaining stages leave little slack:

StageKramer’s approximate budget or target
Carefully optimized transcription and endpointing200 ms
LLM time to first token100 ms
Text-to-speech and return audio pathAdditional delay still required
Complete voice-to-voice path600–700 ms

Kramer describes the overall budget as optimistic but realistic, based on approximate measured medians with substantial variation and imperfect measurement. The component figures are not a complete additive table: speech generation, transport, and playout still have to fit around the named stages.

11:5312:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:53 · section reference included

A geography game exposes both speed and mistakes

Kramer reports approximately 500–700 ms of latency for the open-source container demonstrated as Fast Bot, which he had posted to Twitter the previous day. After introducing itself, the bot agrees to run a geography quiz. It asks for the world’s largest desert, accepts Sahara, and then qualifies the answer as the largest hot desert—a change in the question’s scope rather than a clean factual success.

The next answer contains a repair within the same utterance: the user starts toward Nile, then corrects himself to Amazon. The bot accepts Amazon and continues through Canberra and the Pyrenees. These short exchanges make the conversational pace easy to hear while the dashboard accumulates timing results.

Browser demo with a circular bot avatar, connected status indicators, four session turns, and charts for LLM, voice, and transcription TTFB.Open full source frame
Fast Bot’s dashboard shows conversation turns and service timing charts.

Then the smooth rhythm breaks. Asked for Scandinavia’s largest city, the user answers Stockholm, but the bot responds about Visits.com, a travel website. He interrupts and repeats Stockholm; the bot then accepts it. On the next question, it corrects his Kraków answer to Warsaw. The interaction demonstrates both recovery from a recognition problem and ordinary quiz feedback.

The user reverses roles and asks for France’s capital. The bot answers Paris, but then misinterprets the elliptical follow-up, Germany? The user has to expand it into an explicit question about Germany’s capital before getting Berlin. Finally, he starts asking for the capital of Stockholm, corrects himself to Sweden, and gets Stockholm. A fast response can still be the wrong conversational move: recognition, contextual interpretation, and repair remain distinct from response speed.

14:2114:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:21 · section reference included

Separate the user’s delay from service timings

The geography demo is intended to test latency, not showcase prompt engineering. Its open-source UI puts the client-measured voice-to-voice delay in the center and server-reported component timings around the edges. Keeping those measurements separate lets a developer correlate individual stages with the delay the user hears and inspect how both vary between turns. A fast server metric alone cannot establish a fast conversation.

The deployment uses Deepgram models for transcription and voice generation, selected as compromises between quality and speed. Deepgram offers hosted services as well as models that can run on premises in Docker containers. Kramer tentatively identifies the language model in this recording as Llama 3 8B: he had not brought Llama 3 70B down to the desired response time, although he thought doing so should be possible.

16:5517:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:55 · section reference included

Reusable orchestration survives better models

Repeatedly solving these integration problems led to Pipecat, an open-source framework for real-time multimodal AI. Kramer positions it as doing for this domain what LangChain and LlamaIndex do for other AI applications. It originated in Daily’s work but is vendor neutral, with community contributions expanding the framework.

The same infrastructure supports several application shapes:

  • Conversation: Spoken bots and speech-to-speech translation.
  • Action: Voice agents that control software interfaces.
  • Perception: Real-time vision services alongside audio.

Service plugins provide an extension point, and Kramer invites builders into the Pipecat Discord to contribute integrations or help maintain the project. The reusable part is the coordination between capabilities, not a commitment to one model provider.

The closing slide connects Pipecat’s sharp GitHub-star growth to the GPT-4o announcement. Kramer’s expectation is that better multimodal models will make fast applications easier to build while leaving orchestration necessary. His observation that the anticipated experience is not yet available belongs to the talk’s historical moment: the announcement had released text and vision capabilities while describing the new voice experience and broader audio/video API access as forthcoming.

Closing slide with two demo images on the left and a Pipecat GitHub stars chart rising sharply on the right.Open full source frame
“Voice is the new, new thing” pairs demo images with Pipecat’s sharply rising GitHub star history.

Kramer reports that the demo posted the previous day had reached 175,000 Twitter views. The invitation behind that interest is to build the surrounding systems together. Better models can simplify the speech pipeline, but a usable voice application still has to connect the user’s device, manage conversational state, call external systems, and deliver a response at the right moment.

18:1118:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:11 · section reference included

Resources

From the talk

  • Open-source Python framework for building voice and multimodal agents with composable pipelines and interchangeable services.

  • Hello GPT-4oArticle19:37

    The original GPT-4o announcement, including its staged rollout of text, vision and audio capabilities.

Updates since the talk

  • Current self-hosted deployment documentation requires Enterprise enrollment and authorization to obtain the deployment assets. This access requirement is separate from the historical demonstration.

Transcript download
Full transcript

Loading transcript…