AI Engineer World's Fair 2025
The Voice-First AI Overlay: Designing Conversational Co-Pilots
Loading slides…
Read the talk
The Voice-First AI Overlay: Designing Conversational Co-Pilots
Live conversational assistance needs more than fast models: it must recognize when help matters, preserve context, and respect the attention of the people speaking.
From a talk by Gregory Bruss
How can AI help while two people are talking?
Conversation is our oldest interface, yet a live conversation between two people can leave AI outside the interaction. How can increasingly capable systems help through voice while keeping the humans involved? Gregory Bruss starts with this gap: the interface people use most naturally is still a difficult place to deliver assistance.
Two developments make that gap worth revisiting. Specialized agents can perform increasingly substantial tasks, while conversational voice agents make tools accessible through a call: ask for information, let the agent search, and hear its response. Ambient agents change the trigger. Instead of waiting for a chat message, they respond to an event. Bruss points to Harrison Chase’s talk, Ambient Agents and the New Agent Interface, as a reference for this emerging interaction model; Chase’s related ambient-agents article provides companion reading.
On the capability side, better retrieval-augmented generation, multi-step tool calling, longer task horizons, and agent orchestration expand what assistance can accomplish. On the voice side, lower time to first token and improved latency bring that assistance closer to conversational speed. Bruss also anticipates full-duplex speech models. The resulting design question is whether these capabilities can provide real-time help within an ongoing human conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Language assistance during a live call
The prototype puts this question into a concrete setting: a live foreign-language call in which a participant is not fluent.
The described pipeline has several distinct responsibilities:
- Caption scraping brings the call’s captions into the assistance pipeline.
- Smart debouncing manages the incoming caption activity.
- Context management keeps language suggestions aligned with what is happening in the conversation.
- Suggestion and translation endpoints provide the two kinds of language support.
- Overlay rendering presents that support alongside the call.
The demonstration includes French and German speech. In the French demonstration, the interface shows a live translation panel on the left and populated response options and useful phrases, with English translations, on the right. The main caller remains visible between them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Assistance without a third speaker
A voice-first overlay assists human-to-human conversation without becoming another speaker. Its voice-first character comes from the interaction it supports, not from a requirement that the AI speak. A conventional voice agent takes turns with a person and may use tools before responding. Here, the humans keep talking while the overlay listens passively and surfaces relevant help.
That help might be a phrase, a language suggestion, or a definition. Otherwise, the interface stays out of the way. Its conversational awareness is scoped to the current interaction: the surrounding dialogue supplies the context in which a suggestion becomes useful.
The overlay occupies an interface layer above speech capabilities and agent machinery. Speech recognition and text-to-speech provide foundational capabilities; intent and orchestration frameworks organize agent behavior. The overlay can decide when and where agent help appears without owning the agent’s internals.
| Interaction | Role of the AI |
|---|---|
| Meeting notes, as contrasted here | Retrospective assistance |
| Voice avatar or AI caller | Direct participation in dialogue |
| Voice-first overlay | In-flow assistance for the humans |
Bruss uses these distinctions to locate the design work: amplifying the people in the conversation requires research into cognitive load, timing, and display behavior at the intersection of human-computer interaction and AI interfaces.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast help can still arrive at the wrong time
Bruss frames conventional voice-agent responsiveness around a 200–400 ms window; this is not a measured requirement for the overlay. In an overlay, speed alone cannot establish usefulness. Help that arrives early interrupts; help that arrives late misses the moment when it would have mattered.
| Failure | Effect on the conversation |
|---|---|
| Too early | Interrupts the speakers |
| Too late | Misses the useful moment |
| Wrong context | Becomes irrelevant noise |
| Disruptive presentation | Derails conversational rhythm |
Even relevant help at an appropriate moment can fail if presenting it pulls the user away from the conversation. Timing, relevance, attention, and latency must be managed together.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control how much help enters the conversation
Three design principles follow from the need to protect conversational flow:
- Transparency and control: The user should understand and control how much the overlay gets involved.
- Minimum cognitive load: Intelligence does not compensate for an interface that overloads the speakers or derails their discussion.
- Progressive autonomy: A beginner may need substantial support, then less assistance as their competence grows. Adjustable help should preserve the opportunity to learn.
Progressive autonomy here includes the human becoming more independent of the assistance. It is not simply permission for an agent to do more.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four practical engineering challenges
Bruss groups the implementation problems into four memorable challenges. The first, Jitterbug Input, starts with ordinary speech: someone pauses to breathe, and speech-to-text activity stops momentarily. Debouncing matters because a pause in input is not necessarily a useful conversational boundary. Context repair accompanies that input handling so assistance remains grounded in the dialogue.
The Sub-Second Speed-Limit puts pressure on the entire live-assistance pipeline, not just model generation. Premature Interrupt, or No Show concerns the decision to surface help: conversational awareness must identify when stepping in would be useful, rather than delivering a hint too early, too late, or not at all.
Finally, Glanceable Ghost treats attention as a limited resource. Each arriving hint consumes some of it. An overlay in a video call should not obstruct the field of view, and its interface needs to be flexible and dismissible. The complete challenge table connects input handling to debouncing and context repair, speed to pipeline optimization, intervention timing to conversational awareness, and presentation to a flexible UI.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast inference, private inference, and connected agents
Bruss reports roughly 500–700 ms round trips to a fast provisioned LLM provider, not an end-to-end overlay benchmark. He separately points to low time to first token as a reason for optimism. These observations make responsive assistance seem attainable, while leaving the rest of the pipeline and the decision about when to display help as engineering work.
Smaller, increasingly capable models open another possibility: running the system entirely on the device. The question is whether an overlay can retain the intelligence it needs while making inference private by default. Bruss presents this as a deployment direction to explore, rather than a demonstrated property of the prototype.
The UX ambition is to respect human conversation as something worth protecting. A more speculative extension treats voice as a linkable surface: if ambient agents operate within calls and live conversations, could those agents be connected, and how would their activity be orchestrated? That expands the problem from one helpful interface to coordination across conversational contexts.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When the transcript changes the meaning
Recognition errors can cascade into advice errors. Bruss uses a hypothetical 5% word error rate to illustrate the risk, not to report prototype accuracy. Transcribing I don't want this as I do want this changes only a small part of the sentence but reverses its intent. Advice based on that transcript may therefore point in the wrong direction. Broader conversational context is a proposed way to help interpret or repair such errors, not a demonstrated solution.
Correctly recognized words do not preserve everything either. Speech carries micro-intonation, prosody, and timing cues that people are sensitive to. Flattening it into text discards those cues. How much useful information disappears, and whether the remaining text is sufficient for relevant assistance, remain open questions.
Introducing agents into live conversation also creates a security surface that needs investigation. The concern extends beyond recognizing the words correctly to what happens when agents interact in that setting. Bruss raises the security question without prescribing a threat model or mitigation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Beyond captions, toward an unfinished interface
The first proposed extension moves beyond the caption-based prototype. Could speech models use raw audio features to produce contextual overlay suggestions without first converting the conversation into text? Bruss places this possibility alongside emerging full-duplex models. The two properties are distinct: full duplex concerns simultaneous listening and speaking; it does not by itself require an architecture with no text representations. The proposed direction is assistance informed directly by audio.
Multimodal understanding could add the live video as another source of conversational context. Speculative execution and caching are further possible directions for making assistance available when it is needed, though the talk does not develop their implementation. These extensions broaden both what an overlay can perceive and how it might prepare its response.
Bruss closes with the assessment that voice AI’s growth points toward a conversational future: the technology appears ready, but the interfaces are not. For overlays, the remaining work is an interface that can bring that capability into a conversation while leaving the humans able to carry it forward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- Introducing ambient agentsArticle
Harrison Chase explains agents triggered by events and patterns for requesting human attention and feedback.
An original study of conversational response timing across ten languages, including contextual and nonverbal influences.
A full-duplex dialogue architecture combining parallel audio streams with time-aligned text generation.
Inference code and setup instructions for experimenting with Moshi, including an MLX implementation for Apple devices.
Full transcript
Loading transcript…