AI Engineer Europe 2026

From Transcription to Live Music: Gemini's Audio Stack — Thor Schaeff, Google DeepMind

Loading slides…

Read the talk

From Transcription to Live Music: Gemini's Audio Stack

Thor Schaeff moves from a multilingual recording to directed speech, live camera conversation and generated music, showing how Gemini’s audio capabilities fit together.

From a talk by Thor Schaeff

Before you start: Basic familiarity with API requests, JSON schemas and client–server connections is helpful.

Start with a multilingual recording

What can you recover from a recording beyond the words? Thor Schaeff begins by making the test material himself: an introduction that includes his name, a French apology for his French, and a Japanese greeting. He then checks that he has preserved the recording; a previous demonstration had gone wrong when he recorded over it. The introduction will become input to an application that extracts more than a transcript.

Schaeff works on developer experience for the Gemini API and Google AI Studio. His overview follows the DeepMind model families behind these audio experiences. He says he joined in November, the day before Gemini 3 launched. He then introduces the open Gemma 4 family, describing its recent release and opportunities for audio understanding on devices and at the edge. His reference to the previous week belongs to the conference’s April 2026 setting.

Slide titled “Shipping at relentless pace,” with blue frontier model releases above a timeline and red open model releases below.Open full source frame
Google DeepMind’s frontier and open model release timeline.

Veo, including the cited Veo 3.1 Lite release, brings audio generation into video generation. Gemini 3.1 Flash Live addresses another interaction: full-duplex, sound-to-sound conversation, with real-time text, voice and vision inputs. The rest of the demonstration moves through audio understanding, speech generation and live conversation before combining conversation with music generation.

0:400:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:40 · section reference included

The information a plain transcript leaves behind

Speech carries context, emotion and pacing as well as lexical content. Audio understanding therefore has a broader target than recovering a sequence of words: it must comprehend, transcribe and reason across languages, dialects, accents and modalities. Schaeff also describes Gemini’s ability to handle overlapping speech and switches between languages. Those are capability claims; the upcoming recording provides a narrower multilingual example.

Quotation describing models that comprehend, transcribe, and reason through audio across languages, dialects, accents, and modalities.Open full source frame
Audio understanding spans transcription, reasoning, languages, accents, and modalities.

EchoScript puts this into an application. Built with Google AI Studio and available in its gallery, it uses Gemini 3 Flash Preview to analyze a recording and extract information for a structured interface. Schaeff now returns to the introduction he recorded at the start.

3:013:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:01 · section reference included

One recording, several kinds of output

EchoScript extracts the analysis in one model request. The summary recognizes that Schaeff introduced himself, and the interface uses that contextual name to label his speech. Alongside the transcript, it presents timestamps, speaker labels, languages and emotions. No audience hecklers join in, so this recording does not exercise the multiple-speaker case Schaeff had hoped to demonstrate.

The results expose both the richness and the fallibility of the extraction. The German passage receives a happy label; Schaeff jokes that earlier attempts had been labeled angry. The prompt also asks for an English translation whenever the speech is not English. His French apology receives a neutral label, rather than the sad label he has seen before. These labels are model interpretations of the performance, not direct measurements of the speaker’s feelings.

The Japanese result misses his intended introduction. The Mandarin result describes him as living in the United States, which Schaeff accepts as correct, though nobody in the audience confirms the Chinese. Keeping these outcomes in view matters: a populated interface can look equally convincing when an individual transcription is right or wrong.

The extraction contract is explicit:

  1. Identify distinct speakers, using names when the recording supplies enough context.
  2. Return timestamps and identify the language.
  3. Translate non-English speech into English.
  4. Assign an emotion from happy, sad, angry or neutral.
  5. Put a brief summary of the whole recording at the beginning.

Schaeff reports that one Gemini 3 Flash Preview API call produced the summary and segment-level analysis.

Structured output turns that request into data the interface can consume. Schaeff supplies a response schema, then uses the result to populate the UI. A JavaScript representation of the same output contract can keep the summary separate from repeated segment records:

javascript

const analysisSchema = {
  type: "object",
  properties: {
    summary: { type: "string" },
    segments: {
      type: "array",
      items: {
        type: "object",
        properties: {
          speaker: { type: "string" },
          start: { type: "string" },
          end: { type: "string" },
          text: { type: "string" },
          language: { type: "string" },
          englishTranslation: { type: "string" },
          emotion: {
            type: "string",
            enum: ["happy", "sad", "angry", "neutral"]
          }
        },
        required: [
          "speaker", "start", "end", "text", "language", "emotion"
        ]
      }
    }
  },
  required: ["summary", "segments"]
};

The field layout expresses the application’s needs; it does not make the Japanese result correct. Schema conformance and accuracy remain separate concerns.

4:404:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:40 · section reference included

Choose a base voice, then direct its performance

The next step reverses the direction: understanding what speech sounds like helps a model generate a particular performance. Schaeff connects the dedicated audio models to Gemini 3 foundational research, emphasizing knowledge of pacing, accents and situations. That connection motivates the speech interface he demonstrates next.

A conventional voice catalog may invite you to filter many recordings or voices by gender, language and accent. Here, the starting point is a smaller inventory followed by performance instructions. Schaeff describes roughly 30 base voices, which can be directed toward different accents and delivery styles. The speech-generation prompting approach treats the chosen voice as a performer whose delivery can change, rather than a fixed accent preset.

The Voice Library application in the AI Studio gallery organizes that direction into several parts:

  • Audio profile: who is speaking and what their voice is like.
  • Scene: the setting in which the performance takes place.
  • Director’s notes: how to deliver the lines.
  • Sample context: supporting material for the performance.
  • Transcript: the words to speak.

Schaeff requests a high-pitched Irish male voice, then uses Gemini 3 Flash to construct the speech-generation system prompt. It names a character, Finian, places him in a cozy, crowded pub on the coast of County Clare, and asks for a strong Irish accent. The scene gives the delivery a social context as well as an accent instruction.

The live synthesis attempt fails, so Schaeff plays a prepared sample. It tells an animated pub-style story about an enormous thing, a chaotic evening and everyone laughing. He then plays the underlying base voice: a fairly standard American delivery asking what problem they could solve. Replaying the Irish performance makes the change in delivery audible, although the two samples also use different text.

The second comparison uses Sapphire. Its base sample asks whether the listener is ready to build something, again in a standard American English delivery. A Singaporean scene produces dialogue about chicken rice and chili, with expressions including “lah,” “damn shiok” and “confirm plus chop.” Schaeff associates the result with a hawker center. The demonstration combines regional wording with delivery; he attributes that adaptation to the model’s understanding of what the scene sounds like.

7:247:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:24 · section reference included

A conversation without a text-only middle

Gemini 3.1 Flash Live moves from generating a performance to maintaining an interaction. Through the Live API, a WebSocket connection carries real-time text, audio and video inputs. The model returns audio responses and their text transcripts.

Schaeff cautions against relying too heavily on audio benchmarks and instead emphasizes the architectural distinction:

ArchitectureWhere reasoning happensResponse path
Cascading pipelineAn LLM receives transcribed textText response becomes speech
Native audio modelReasoning is integrated into the audio modelAudio response, with a transcript

A transcript can be an output without being the intermediate representation that drives the conversation. This is the distinction Schaeff draws between the native model and a cascade through text; he supplies no quantitative comparison here.

For an initial trial, he uses AI Studio’s Live interface. At the time of the talk, he describes it as a way to try the models without payment or entering a credit card. The next example tests what happens when audio understanding, visual input and performance instructions operate together.

11:5512:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:55 · section reference included

The camera works—and the accent instruction travels

Schaeff gives the model a system instruction to speak in a friendly Irish accent and enables his camera. He asks whether it can see him, then asks about his outfit. The response identifies his Gemini shirt and backwards hat while maintaining the requested conversational style. Visual input supplies the subject matter; the system instruction shapes its spoken presentation.

He then switches to German and asks for a poem. The model introduces its answer in English and recites German verse. Schaeff hears the Irish accent carry into the German, too. A global instruction has remained active across the language change: changing the language of the request did not narrow the scope of the accent instruction. His recommendation is to adjust the system instructions so the accent does not apply indiscriminately to every language.

Google AI Studio shows a webcam view, a German request for a poem, and a response containing English introductory text and German verse.Open full source frame
A German poem request and Gemini’s response during a live webcam conversation.

Screen sharing is another possible source of visual input. Schaeff states that Live video input is limited to one frame per second at the time of the talk. This is a limit on ingesting frames alongside audio, not a general limit on Gemini’s video understanding. The current Live documentation retains that frame-ingestion limit, but its SDK examples are a later API reference rather than a reconstruction of the on-stage application.

13:3213:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:32 · section reference included

Building the live connection

The implementation starting points follow where the connection lives:

  • Python: documentation-linked examples for a server-to-server connection.
  • JavaScript: examples for a client-to-server connection.

Schaeff also recommends the published Gemini API coding-agent skills, including guidance for the Live API. His reason is practical: real-time audio adds complexity, and an agent benefits from instructions specific to those APIs rather than having to infer the integration patterns.

Repository card for google-gemini/gemini-skills, described as skills for the Gemini API, SDK, and model/agent interactions, with a QR code.Open full source frame
The google-gemini/gemini-skills repository for Gemini development.
15:0915:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:09 · section reference included

Let the conversation request a song

The final component is Lyria 3. A short playback introduces music with sung lyrics, followed by two generation options:

ModelOutput described in the talk
Lyria 3 Clip30-second jingles
Lyria 3 ProFull-length songs

These are generated clips and songs. A live conversational interface around them does not itself imply streaming music synthesis; Lyria RealTime is a separate offering.

Live Jukebox combines the pieces around the familiar experience of calling a radio station to request a song. Schaeff gives the Gemini Live model a tool that invokes Lyria. The conversational model can gather the request, while the music model produces the song. The interface is live even though music generation is delegated to another model.

The DJ persona starts with an on-air greeting and asks what kind of vibe the caller wants. Schaeff interrupts to request German techno schlager about the UK startup scene. The DJ starts proposing a pounding techno beat and a Hans Zimmer-style element; he interrupts again to approve the direction and ask it to proceed.

Before generating, the DJ asks whether the lyrics should include particular startup buzzwords or stories, and whether the energy should be manic or more melodic. Schaeff selects manic energy and leaves the lyrics as a surprise. That exchange supplies a compact music brief: German techno schlager, a UK startup theme, manic energy and open-ended lyrics.

The DJ announces the result, and song playback follows. The German lyrics mention a pitch deck, growth, a Series A goal and coffee, then repeat a refrain celebrating startup life. This completes the demonstrated handoff: a spoken request becomes a clarified brief, a music-generation tool supplies the result, and the application plays it back.

Schaeff closes by offering to rewind to the slides so attendees can retrieve their links—the handoff from the working demonstrations to the resources for building with the same capabilities.

15:5516:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:55 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…