AI Engineer World's Fair 2025

The Voice-First AI Overlay: Designing Conversational Co-Pilots

Loading slides…

Read the talk

The Voice-First AI Overlay: Designing Conversational Co-Pilots

Live conversational assistance needs more than fast models: it must recognize when help matters, preserve context, and respect the attention of the people speaking.

From a talk by Gregory Bruss

How can AI help while two people are talking?

Conversation is our oldest interface, yet a live conversation between two people can leave AI outside the interaction. How can increasingly capable systems help through voice while keeping the humans involved? Gregory Bruss starts with this gap: the interface people use most naturally is still a difficult place to deliver assistance.

Slide titled “Conversation is the oldest interface. Voice is our original API.” Three bullets describe voice, AI being outside live conversation, and keeping humans in the loop.Open full source frame
Voice as our original interface—and AI locked outside the conversation.

Two developments make that gap worth revisiting. Specialized agents can perform increasingly substantial tasks, while conversational voice agents make tools accessible through a call: ask for information, let the agent search, and hear its response. Ambient agents change the trigger. Instead of waiting for a chat message, they respond to an event. Bruss points to Harrison Chase’s talk, Ambient Agents and the New Agent Interface, as a reference for this emerging interaction model; Chase’s related ambient-agents article provides companion reading.

On the capability side, better retrieval-augmented generation, multi-step tool calling, longer task horizons, and agent orchestration expand what assistance can accomplish. On the voice side, lower time to first token and improved latency bring that assistance closer to conversational speed. Bruss also anticipates full-duplex speech models. The resulting design question is whether these capabilities can provide real-time help within an ongoing human conversation.

0:120:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Language assistance during a live call

The prototype puts this question into a concrete setting: a live foreign-language call in which a participant is not fluent.

The described pipeline has several distinct responsibilities:

  1. Caption scraping brings the call’s captions into the assistance pipeline.
  2. Smart debouncing manages the incoming caption activity.
  3. Context management keeps language suggestions aligned with what is happening in the conversation.
  4. Suggestion and translation endpoints provide the two kinds of language support.
  5. Overlay rendering presents that support alongside the call.

The demonstration includes French and German speech. In the French demonstration, the interface shows a live translation panel on the left and populated response options and useful phrases, with English translations, on the right. The main caller remains visible between them.

Video call with a live translation panel on the left and French response options and useful phrases with English translations on the right.Open full source frame
A conversational French demo with live translation, response options, and useful phrases.
2:593:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:59 · section reference included

Assistance without a third speaker

A voice-first overlay assists human-to-human conversation without becoming another speaker. Its voice-first character comes from the interaction it supports, not from a requirement that the AI speak. A conventional voice agent takes turns with a person and may use tools before responding. Here, the humans keep talking while the overlay listens passively and surfaces relevant help.

That help might be a phrase, a language suggestion, or a definition. Otherwise, the interface stays out of the way. Its conversational awareness is scoped to the current interaction: the surrounding dialogue supplies the context in which a suggestion becomes useful.

The overlay occupies an interface layer above speech capabilities and agent machinery. Speech recognition and text-to-speech provide foundational capabilities; intent and orchestration frameworks organize agent behavior. The overlay can decide when and where agent help appears without owning the agent’s internals.

InteractionRole of the AI
Meeting notes, as contrasted hereRetrospective assistance
Voice avatar or AI callerDirect participation in dialogue
Voice-first overlayIn-flow assistance for the humans

Bruss uses these distinctions to locate the design work: amplifying the people in the conversation requires research into cognitive load, timing, and display behavior at the intersection of human-computer interaction and AI interfaces.

5:115:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:11 · section reference included

Fast help can still arrive at the wrong time

Bruss frames conventional voice-agent responsiveness around a 200–400 ms window; this is not a measured requirement for the overlay. In an overlay, speed alone cannot establish usefulness. Help that arrives early interrupts; help that arrives late misses the moment when it would have mattered.

FailureEffect on the conversation
Too earlyInterrupts the speakers
Too lateMisses the useful moment
Wrong contextBecomes irrelevant noise
Disruptive presentationDerails conversational rhythm

Even relevant help at an appropriate moment can fail if presenting it pulls the user away from the conversation. Timing, relevance, attention, and latency must be managed together.

8:108:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:10 · section reference included

Control how much help enters the conversation

Three design principles follow from the need to protect conversational flow:

  • Transparency and control: The user should understand and control how much the overlay gets involved.
  • Minimum cognitive load: Intelligence does not compensate for an interface that overloads the speakers or derails their discussion.
  • Progressive autonomy: A beginner may need substantial support, then less assistance as their competence grows. Adjustable help should preserve the opportunity to learn.

Progressive autonomy here includes the human becoming more independent of the assistance. It is not simply permission for an agent to do more.

9:379:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:37 · section reference included

Four practical engineering challenges

Bruss groups the implementation problems into four memorable challenges. The first, Jitterbug Input, starts with ordinary speech: someone pauses to breathe, and speech-to-text activity stops momentarily. Debouncing matters because a pause in input is not necessarily a useful conversational boundary. Context repair accompanies that input handling so assistance remains grounded in the dialogue.

The Sub-Second Speed-Limit puts pressure on the entire live-assistance pipeline, not just model generation. Premature Interrupt, or No Show concerns the decision to surface help: conversational awareness must identify when stepping in would be useful, rather than delivering a hint too early, too late, or not at all.

Finally, Glanceable Ghost treats attention as a limited resource. Each arriving hint consumes some of it. An overlay in a video call should not obstruct the field of view, and its interface needs to be flexible and dismissible. The complete challenge table connects input handling to debouncing and context repair, speed to pipeline optimization, intervention timing to conversational awareness, and presentation to a flexible UI.

Table pairs Jitterbug Input with smart debouncing and context repair; Sub-Second Speed-Limit with pipeline optimization; Premature Interrupt, or No Show with conversational awareness; and Glanceable Ghost with flexible, dismissible UI.Open full source frame
Four overlay engineering challenges and their corresponding responses.
10:4610:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:46 · section reference included

Fast inference, private inference, and connected agents

Bruss reports roughly 500–700 ms round trips to a fast provisioned LLM provider, not an end-to-end overlay benchmark. He separately points to low time to first token as a reason for optimism. These observations make responsive assistance seem attainable, while leaving the rest of the pipeline and the decision about when to display help as engineering work.

Smaller, increasingly capable models open another possibility: running the system entirely on the device. The question is whether an overlay can retain the intelligence it needs while making inference private by default. Bruss presents this as a deployment direction to explore, rather than a demonstrated property of the prototype.

The UX ambition is to respect human conversation as something worth protecting. A more speculative extension treats voice as a linkable surface: if ambient agents operate within calls and live conversations, could those agents be connected, and how would their activity be orchestrated? That expands the problem from one helpful interface to coordination across conversational contexts.

12:2112:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:21 · section reference included

When the transcript changes the meaning

Recognition errors can cascade into advice errors. Bruss uses a hypothetical 5% word error rate to illustrate the risk, not to report prototype accuracy. Transcribing I don't want this as I do want this changes only a small part of the sentence but reverses its intent. Advice based on that transcript may therefore point in the wrong direction. Broader conversational context is a proposed way to help interpret or repair such errors, not a demonstrated solution.

Correctly recognized words do not preserve everything either. Speech carries micro-intonation, prosody, and timing cues that people are sensitive to. Flattening it into text discards those cues. How much useful information disappears, and whether the remaining text is sufficient for relevant assistance, remain open questions.

Introducing agents into live conversation also creates a security surface that needs investigation. The concern extends beyond recognizing the words correctly to what happens when agents interact in that setting. Bruss raises the security question without prescribing a threat model or mitigation.

“Where I’m Curious” table lists ASR error cascades, prosody and timing complexity, and security surface. The ASR row says missing “don’t” can produce wrong advice.Open full source frame
Missing “don’t” can turn a transcription error into wrong advice.
14:0014:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:00 · section reference included

Beyond captions, toward an unfinished interface

The first proposed extension moves beyond the caption-based prototype. Could speech models use raw audio features to produce contextual overlay suggestions without first converting the conversation into text? Bruss places this possibility alongside emerging full-duplex models. The two properties are distinct: full duplex concerns simultaneous listening and speaking; it does not by itself require an architecture with no text representations. The proposed direction is assistance informed directly by audio.

Multimodal understanding could add the live video as another source of conversational context. Speculative execution and caching are further possible directions for making assistance available when it is needed, though the talk does not develop their implementation. These extensions broaden both what an overlay can perceive and how it might prepare its response.

Bruss closes with the assessment that voice AI’s growth points toward a conversational future: the technology appears ready, but the interfaces are not. For overlays, the remaining work is an interface that can bring that capability into a conversation while leaving the humans able to carry it forward.

15:3115:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:31 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…