AI Engineer World's Fair 2026
Can Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff
Loading slides…
Read the talk
Can Your Agent Hear You Now? Building Live Voice Agents with Gemini
Thor Schaeff connects live conversation to image generation, translation and music, then explains the stateful WebSocket session behind Gemini’s voice and vision agents.
From a talk by Thor Schaeff
At a glance
Ideas worth remembering
The story canvas connects Gemini Live to NanoBanana Lite through tool calls. Passing the previous interaction ID gives later image requests access to earlier generation context.
Live Translate separates the automatically detected speaking language from each listener’s chosen output language, beginning translation as enough context becomes available.
A live agent becomes more useful when conversation can invoke a concrete capability. The DJ gathers a music request, calls Lyria 3 and returns to the conversation after the song.
Gemini 3.1 Flash Live uses the Live API’s stateful WebSocket session for text, audio and video inputs, returning audio chunks plus transcription. Native audio processing does not require the transcript to serve as an intermediate text stage.
Multilingual capability still benefits from clear conversational control: when the attempted Mandarin switch faltered, an explicit request let the exchange continue.
Expressive speech is only the opening act
The opening voice sounds ready to host the conference: “Welcome one and all to the grandest stage on Earth.” But when the audience guesses which model produced it, Gemini Live is the wrong answer. The theatrical welcome comes from Gemini Text to Speech. Thor Schaeff, who works on developer experience at Google DeepMind, starts with a useful distinction: generating expressive speech and conducting a live conversation are different capabilities.
Gemini TTS accepts inline text instructions to steer delivery and can stream its speech output. Schaeff describes its latency as lower than before, without giving a measurement. The rest of the session moves beyond delivering prepared words: an agent will listen to a developing story, invoke image generation, translate a live microphone feed and turn a spoken music request into a generated track.
The first enabling change is faster image generation and editing with NanoBanana Lite. That makes an illustrated conversation more plausible: the user can keep talking while the application adds pictures to the story. Schaeff introduces his recently “vibe-coded” Gemini NanoBanana story canvas as a way to try that combination in front of the room.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A story changes direction while its illustrations retain context
The story starts with a small prompt: Taipei in the 1980s. Gemini Live expands it into a rainy night market, neon reflected on wet pavement, passing scooters and a young woman hiding a glowing object under her jacket. Schaeff then changes the direction: a new kind of drink has emerged. The continuation follows the woman to a stall where a vendor shakes a metal cup and serves a milky drink with tapioca pearls. Bubble tea becomes the next story beat; this is generated fiction, rather than a historical demonstration.
The illustrations come from a separate capability connected through a tool call. Gemini Live handles the conversation, while the application uses the Interactions API to request an image from NanoBanana Lite as the story develops. Schaeff puts the actual image-generation time at roughly two to three seconds in this demo. That is short enough to accompany the exchange, but it remains a noticeable generation step rather than an instantaneous visual response.
The harder problem is continuity. Independently generating each picture would leave the application asking for a new scene without the same explicit connection to the preceding result. The Interactions API provides a stateful continuation: the first image request has no previous interaction ID; the next request supplies the previous interaction ID so generation can use that earlier context.
The useful code comparison is between those first and subsequent requests: where does the previous interaction ID enter? Schaeff points to the first image’s visual feeling and characters, then describes carrying them into subsequent generations through the linked interaction. The ID gives the next request continuity with earlier context; the demo does not establish a guarantee that every character or stylistic detail will remain identical.
typescript
const interaction = await ai.interactions.create({
model: 'gemini-3.1-flash-lite-image',
input: prompt,
previous_interaction_id: previousInteractionId || undefined,
});
There are two kinds of state to keep straight here. The live conversation carries the unfolding narrative, including the user’s intervention about a new drink. The image-generation continuation carries prior generation context. Connecting those capabilities lets the story advance while giving its illustrations a basis for staying recognizable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One microphone feed, different listening languages
The next demo turns the room into a multilingual listening session. Schaeff creates a session for Gemini 3.5 Live Translate, asks the audience to join through a QR code and enables his microphone. Each listener selects a preferred language on their phone. The useful comparison is between the shared speech input and the listeners’ output choices: someone choosing German can listen to a German rendering of the same talk while Schaeff continues speaking to everyone.
The timing depends on recognizing enough context to begin translating. Schaeff describes the output as near real time: the model listens to the ongoing speech and starts producing the desired language once it recognizes what he is talking about. This gives the translation some context before it commits to an utterance. The recording supplies no numerical translation delay or accuracy evaluation.
The language choices operate on two sides of the exchange:
- Speaker input. The model automatically detects the language being spoken, so the presenter can change languages without manually choosing a new source language.
- Listener output. The listener chooses the language they want to hear. That preference can remain different from the presenter’s current language.
Schaeff proposes switching to German and also checks whether anyone is listening in Chinese. The point is to let the presenter and audience use their own preferred languages within the same event.
A listener confirms receiving audio and responds positively when asked how it sounds. Headphones matter in this setting: the translated stream needs to be audible alongside the presenter’s amplified original speech. The demo therefore combines a model capability with a practical delivery arrangement—microphone input, a joinable session and personal audio output. Schaeff presents Live Translate as available through the Gemini API for integration into applications.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A radio DJ turns conversation into a music-generation request
The radio demo borrows a familiar interaction: phone the host and ask for a song. Here, Gemini 3.1 Flash Live plays the DJ, and a callable function gives it access to Lyria 3 through the Gemini API. The conversational model can gather the request and keep the exchange going; the music model supplies the generated song.
The DJ opens by asking for a genre and a lyrical direction. Schaeff requests high-energy reggaeton about the AI Engineer World’s Fair in San Francisco. The agent asks a narrower follow-up: should it include particular technologies or big names, or aim for a festival atmosphere? “Just surprise me, please” leaves those remaining creative choices to the agent.
The DJ announces generation, and the resulting song combines English and Spanish lyrics about San Francisco, neural networks, algorithms and the fair. The request’s genre and subject have become a concrete media result. After playback, the DJ returns to its conversational role and asks what to play next. That return matters: generating the artifact is one action inside the continuing exchange.
Schaeff is candid about the business value of an endlessly enthusiastic jukebox: the application itself may not be what a company needs. Its components are more transferable:
- Conversational input. The agent gathers intent through ordinary speech and asks a follow-up before acting.
- A callable capability. Music generation gives the agent something useful to do beyond answering. Schaeff proposes CRM lookups or backend ticket creation as other possible tools; those actions are not demonstrated here.
- Multilingual interaction. The experience can accommodate multiple languages, with the generated song providing a playful example of mixed-language output.
The implementation decision is to give the live agent access to a capability that serves the application’s purpose.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inside the Live API: a stateful session with native audio
Gemini 3.1 Flash Live is accessed through the Live API. An application creates a stateful WebSocket session, then sends text, audio and video frames over that connection. The model returns audio chunks that the application can stream to the user, together with a transcript of what was said. This is an ongoing session carrying live inputs and outputs, rather than a separate request for each finished recording.
Where does text sit in that exchange? Schaeff distinguishes this native audio model from a cascading voice pipeline: speech understanding and response generation do not require an intervening text representation in the model’s path. The transcript is still available to the application, but it should not be confused with the medium through which all of the model’s audio intelligence operates. The diagram separates the incoming modalities from the two kinds of output returned through the live session.
The session supports several behaviors that make conversation usable:
- Language switching. Schaeff reports 90 supported languages in the preview and connects native audio processing with fluid multilingual responses.
- Barge-in. A user can interrupt the model rather than waiting for a long response to finish. The overly talkative DJ is the obvious example.
- Tool calls. The agent can invoke capabilities such as the music generator instead of limiting itself to spoken answers.
- Transcription. The application receives text alongside the audio response.
These are capabilities of the conversational experience; implementing them still requires the application to handle the live media exchange.
Real-time media can become difficult to scale even when the model interface is straightforward. Schaeff points to LiveKit and Pipecat as partners that help developers get started with Gemini Live. The session does not develop their transport or deployment designs, but it makes a useful separation: choosing the conversational model does not by itself solve the application’s media infrastructure.
What travels through the live session?
Read the diagram as text
- Application inputs. application — Text, live audio and video frames.
- Stateful Live API session. connection — A WebSocket connection carries the ongoing exchange.
- Native audio model. model — Understands speech and produces spoken responses without requiring a text cascade.
- Streamed audio. application — The application plays returned audio chunks.
- Transcript. application — Text of what was said is also available to the application.
- Application inputs → Stateful Live API session: Send inputs over WebSocket.
- Stateful Live API session → Native audio model: Deliver live inputs.
- Native audio model → Stateful Live API session: Return response audio and transcript.
- Stateful Live API session → Streamed audio: Stream audio chunks.
- Stateful Live API session → Transcript: Provide transcription.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Try the voice, add vision, then recover a language switch
Google AI Studio provides the shortest route to trying the interaction before building an application. Schaeff opens its Live interface, selects Gemini 3.1 Flash Live and chooses the Puck voice. The same interface also offers Live Translate. This lets a developer explore the model’s behavior through a conversation rather than starting with a media client.
Another option enables grounding with Google Search. Training alone does not supply live information such as today’s weather; Search gives the model a way to look information up. Schaeff describes this as a built-in function call. He explains the option before the demonstration, but the ensuing exchange tests hearing, vision and languages rather than showing a search result.
The vision test moves from a broad question to an inspectable detail. Asked whether it can see him, the assistant describes a black shirt, black hat and blue background. Asked what the sweatshirt says, it answers that “Gemini” appears in yellow letters. Comparing the camera input with that answer would show what detail the assistant extracts: the task has moved beyond acknowledging video to reading particular lettering.
The next question connects vision to language. Schaeff asks in German what the word means, and the assistant answers “Zwillinge”—twins. The conversation has moved from reading lettering in the camera view to translating that lettering, without starting a separate translation application.
The attempted move to Mandarin is less smooth. The assistant continues with a German response, and Schaeff remarks that his Mandarin “didn’t work out so well.” The precise cause is unclear, so this moment supports a narrow lesson: the intended language switch was not reliably understood in that exchange. He repairs it with an explicit request to speak Mandarin, which the assistant acknowledges.
Schaeff then explains the final Mandarin exchange for the audience: he asked what the model had eaten, and it replied that it did not need to eat because it was a model. The session ends with a small but useful demonstration of recovery. A missed conversational cue does not have to end the interaction; an explicit language request can redirect it, and the same live session can continue. AI Studio is his suggested starting point for trying those capabilities before connecting them to an application’s tools and media infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…