AI Engineer World's Fair 2024
How to build the world's fastest voice bot
Loading slides…
Read the talk
How to build the world's fastest voice bot
An interruptible joke bot, a patient-intake call, and a geography game reveal how media transport, conversational state, model placement, and measurement shape voice AI latency.
From a talk by Kwindla Hultman Kramer
Before you start: Familiarity with language-model APIs and basic audio streaming will help; the article explains the voice-specific timing and orchestration concepts.
The user has already heard that joke
A chatbot starts a skeleton joke. The user interrupts: it always starts with that one. The bot switches to fake spaghetti, only to be interrupted again before finishing the punchline. Getting a language model to produce a joke is easy; making it stop, listen, and respond to the interruption is the voice engineering problem.
This roughly year-old demonstration comes from Kwindla Hultman Kramer’s work at Daily, a real-time audio and video infrastructure company. GPT-4 had made text conversation convincing enough to motivate experiments in spoken conversation. The team kept asking for jokes as an informal test throughout development, eventually learning the models’ recurring repertoire. Familiar content made the behavior around the words—especially interruptions—easy to notice.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A simple diagram hides a large runtime
The initial architecture looks straightforward: a person speaks into a phone or laptop, and the device exchanges data with GPUs in the cloud. Two requirements immediately complicate that picture: speed and architectural flexibility. Production systems must account for audio processing and echo cancellation, CPU time spent encoding and decoding media, firewall traversal, and application capabilities such as retrieval augmented generation and tool calling. Deployment and scaling cross all of these boundaries.
The runtime responsibilities become easier to reason about when grouped by what they protect:
- Media delivery: Encode and transport audio reliably without accumulating unnecessary delay.
- Transcription: Convert speech into usable text quickly. Even as audio-native models arrive, text remains useful for many integrations.
- Buffers and pipelines: Preserve the audio representation and move streaming data between stages correctly. Kramer recalls answering roughly 20–30 Discord questions about audio that worked on a Mac but failed on an Intel cloud machine, failures he attributed to endianness. That is an anecdote about particular implementations, not a general incompatibility between those platforms.
- Model composition: Swap models or combine several without rebuilding the surrounding application.
- External systems and turn detection: Call application services and perform phrase endpointing—deciding when someone has finished speaking and expects the bot to respond.
- Interruption state: Decide what stops and what conversational state survives when the user speaks over the bot.
- Echo cancellation and voice output: Prevent speaker audio from returning through the microphone, then generate speech quickly enough to sustain the exchange.
Headphones cannot be an architectural dependency. Daily’s core product is a WebRTC stack, and Kramer’s experience is that universal headphone use is unrealistic—even in professional call-center settings. The application has to handle ordinary speakers and microphones.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Replacing a form with a conversation
Repeated medical forms provide a practical reason to assemble these components. Instead of filling out another paper form or following a pre-visit form link, a patient could receive a phone call or open an app link and have a conversation. In the demonstration, Jessica from Tri-County Health Services asks Chad for his complete birthday to confirm his identity.
Chad first supplies a partial birthday. Jessica begins asking for the missing information, but he interrupts to add the missing year. The next exchange follows the same pattern: he names lisinopril and gabapentin, then interrupts the dosage follow-up to add 10 mg and 100 mg respectively. Finally, he supplies latex and penicillin allergies. The system must carry information across turns and accept corrections while it is speaking; treating each transcript as an isolated request would lose the structure of the intake.
The call’s apparent simplicity depends on the underlying integration: conversation access, external data handling, streaming speech, function calling, and interruption support. Kramer describes the recording as unedited and the tool calling as real, but the external-system calls are mocked: no actual electronic health record backend is connected. He says this class of application is beginning to enter production.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generating quickly is different from responding quickly
There are two kinds of fast. Throughput describes how quickly content continues to arrive; latency describes how long the user waits for it to begin. Kramer considers contemporary generation throughput generally sufficient to keep up with reading or listening. The difficult part is time to first byte or first token. His broad engineering comparison is that throughput often improves by an order of magnitude every couple of years, while latency improvements tend to be linear and much slower.
People do not wait for a complete utterance before preparing a reply. Kramer likens this anticipation to speculative decoding: predict the next four or five words, prepare a response, and reroute if the prediction is wrong. Kramer cites common human response gaps of 200–300 ms and uses approximately 500 ms as his voice-bot engineering target. Longer waits can work when the listener supplies some cue, but unexplained silence disrupts the conversational rhythm.
Viewed through that lens, the patient-intake demo has useful orchestration but noticeably slow responses. Model choice can exhaust the budget before the rest of the pipeline begins: Kramer’s historical example puts Gemini Pro time to first token at about 900 ms, already above his 500 ms target before other services and network overhead. The talk does not specify the model version or measurement conditions for that comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Put the stages close enough to control
Choosing fast models is only part of the solution; where they run and how they connect also matter. Kramer’s strongest optimization is to bring as much processing as possible into one compute container: speech-to-text, phrase endpointing, LLM inference, and voice output. Running those stages locally together gives the developer more control over the combined path.
Kramer informally estimates that 95–99% of contemporary generative AI work calls hosted services. Those services have legitimate benefits, and latency need not be every application’s highest priority. The trade-off changes when the goal is the shortest possible response gap:
| Deployment choice | Main advantage | Latency consideration |
|---|---|---|
| Separate hosted services | Provider-managed capabilities | Less control over the combined path |
| Co-located, self-hosted stages | Tune and combine processing | Requires operating the models yourself |
The reason to self-host here is concrete: it gives the team control over how stages run together, rather than merely control over which API it calls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure the silence between the waveforms
Voice-to-voice latency is the interval from the end of the user’s speech to the first audible bot response. Imagine the user’s green waveform and the bot’s purple waveform in an audio editor: the gap between them is the measurement that matters. It includes encoding, network transit, processing, the return trip, and speaker playout—not just the LLM request.
For an audio recording containing both sides on one timeline, the measurement reduces to sample positions. This Python function makes that boundary explicit; identifying the two speech boundaries is a separate step:
python
def voice_to_voice_ms(
user_end_sample: int,
bot_start_sample: int,
sample_rate_hz: int,
) -> float:
if sample_rate_hz <= 0:
raise ValueError("sample_rate_hz must be positive")
if min(user_end_sample, bot_start_sample) < 0:
raise ValueError("sample positions must be nonnegative")
return (
(bot_start_sample - user_end_sample)
* 1000.0
/ sample_rate_hz
)
A negative result represents overlapping speech. For a normal turn transition, the positive result measures the silence the user experiences.
The budget starts before application code sees any bytes. Kramer puts macOS laptop microphone delay at approximately 30–40 ms, including processing such as echo cancellation, noise removal, and gain control. Audio then needs encoding and transmission. At the receiving end, packet handling, jitter-buffer reassembly, and decoding precede transcription.
The remaining stages leave little slack:
| Stage | Kramer’s approximate budget or target |
|---|---|
| Carefully optimized transcription and endpointing | 200 ms |
| LLM time to first token | 100 ms |
| Text-to-speech and return audio path | Additional delay still required |
| Complete voice-to-voice path | 600–700 ms |
Kramer describes the overall budget as optimistic but realistic, based on approximate measured medians with substantial variation and imperfect measurement. The component figures are not a complete additive table: speech generation, transport, and playout still have to fit around the named stages.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A geography game exposes both speed and mistakes
Kramer reports approximately 500–700 ms of latency for the open-source container demonstrated as Fast Bot, which he had posted to Twitter the previous day. After introducing itself, the bot agrees to run a geography quiz. It asks for the world’s largest desert, accepts Sahara, and then qualifies the answer as the largest hot desert—a change in the question’s scope rather than a clean factual success.
The next answer contains a repair within the same utterance: the user starts toward Nile, then corrects himself to Amazon. The bot accepts Amazon and continues through Canberra and the Pyrenees. These short exchanges make the conversational pace easy to hear while the dashboard accumulates timing results.
Then the smooth rhythm breaks. Asked for Scandinavia’s largest city, the user answers Stockholm, but the bot responds about Visits.com, a travel website. He interrupts and repeats Stockholm; the bot then accepts it. On the next question, it corrects his Kraków answer to Warsaw. The interaction demonstrates both recovery from a recognition problem and ordinary quiz feedback.
The user reverses roles and asks for France’s capital. The bot answers Paris, but then misinterprets the elliptical follow-up, Germany? The user has to expand it into an explicit question about Germany’s capital before getting Berlin. Finally, he starts asking for the capital of Stockholm, corrects himself to Sweden, and gets Stockholm. A fast response can still be the wrong conversational move: recognition, contextual interpretation, and repair remain distinct from response speed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the user’s delay from service timings
The geography demo is intended to test latency, not showcase prompt engineering. Its open-source UI puts the client-measured voice-to-voice delay in the center and server-reported component timings around the edges. Keeping those measurements separate lets a developer correlate individual stages with the delay the user hears and inspect how both vary between turns. A fast server metric alone cannot establish a fast conversation.
The deployment uses Deepgram models for transcription and voice generation, selected as compromises between quality and speed. Deepgram offers hosted services as well as models that can run on premises in Docker containers. Kramer tentatively identifies the language model in this recording as Llama 3 8B: he had not brought Llama 3 70B down to the desired response time, although he thought doing so should be possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reusable orchestration survives better models
Repeatedly solving these integration problems led to Pipecat, an open-source framework for real-time multimodal AI. Kramer positions it as doing for this domain what LangChain and LlamaIndex do for other AI applications. It originated in Daily’s work but is vendor neutral, with community contributions expanding the framework.
The same infrastructure supports several application shapes:
- Conversation: Spoken bots and speech-to-speech translation.
- Action: Voice agents that control software interfaces.
- Perception: Real-time vision services alongside audio.
Service plugins provide an extension point, and Kramer invites builders into the Pipecat Discord to contribute integrations or help maintain the project. The reusable part is the coordination between capabilities, not a commitment to one model provider.
The closing slide connects Pipecat’s sharp GitHub-star growth to the GPT-4o announcement. Kramer’s expectation is that better multimodal models will make fast applications easier to build while leaving orchestration necessary. His observation that the anticipated experience is not yet available belongs to the talk’s historical moment: the announcement had released text and vision capabilities while describing the new voice experience and broader audio/video API access as forthcoming.
Kramer reports that the demo posted the previous day had reached 175,000 Twitter views. The invitation behind that interest is to build the surrounding systems together. Better models can simplify the speech pipeline, but a usable voice application still has to connect the user’s device, manage conversational state, call external systems, and deliver a response at the right moment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source Python framework for building voice and multimodal agents with composable pipelines and interchangeable services.
The original GPT-4o announcement, including its staged rollout of text, vision and audio capabilities.
Further reading
Kramer's companion explanation of model colocation, WebRTC networking and voice latency budgets, with an August 2024 demo update.
Research on conversational gaps and how listeners prepare responses before another speaker finishes.
Updates since the talk
Current self-hosted deployment documentation requires Enterprise enrollment and authorization to obtain the deployment assets. This access requirement is separate from the historical demonstration.
Full transcript
Loading transcript…