Speech-to-Speech Model Research at Google DeepMind — Valeria Wu Fon & Tom Ouyang, Google DeepMind

[music]

>> Hi everyone. Thanks for coming to our

talk. We're going to talk a lot about

how we are planning on voiceifying the

agentic future with speech-to-speech

research here at Google DeepMind. My

name is Valeria. I'm the product lead

for the speech-to-speech model in Gemini

and Tom.

>> Yeah, my name is Tom. I work on

speech-to-speech as an engineer and

Gemini. So, yeah, great to be presenting

our work.

>> Cool.

So, before we start diving into the

history of Google research and audio and

the latest research we've been doing, I

just want to take a step back and really

kind of like think about why this is

important to us in the team. And the

main reason is that voice is the most

natural way for humans to interact with

both the physical and the virtual world.

And we're already seeing so many

applications that are starting even

within the own Google products. We see

voice being used to ask questions and

you know, do homework or info seeking or

EDU in search live and Gemini live. All

the way to these same models being

deployed in cloud and the API for

enterprise or voice agent use cases.

Should I use this as a

microphone?

Hello. Okay.

Um

So, because of that, we only think that

the number of applications is going to,

you know, exponentially increase over

the next few years and we believe that

speech-to-speech models are the way to

go when we want to build robust

universal voice agents. And with that,

I'll hand it off to Tom.

>> All right. So, of course, like one way

to create a voice agent is to just add

an ASR like speech-to-text model in the

pipeline and then a text-to-speech model

on the other end, right? So, of course,

that's where a lot of the history of

speech come from. Of course, Google has

been working on this for a long time.

I'm going to give you a little bit of a

historical overview of what speech

modeling, especially speech-to-text

automatic

speech recognition, typically looks

like, right? So, up until around 2018,

this usually involved a lot of different

components. You have um you know of

course feature extraction that's fairly

general and then you have all these

different pieces like acoustic modeling,

pronunciation modeling, language

modeling, a second pass rescoring that

allows you to go from the audio input to

a text transcription. Right? And of

course like around 2018, these moved

more and more towards end-to-end

systems. You don't have to have so much

domain knowledge. You can actually have

mostly the the neural model learn this

pattern and mapping between acoustic

inputs and text. But these aren't really

the end-to-end models that we think

about when we think about LLMs. They're

only doing kind of one thing, which is

speech to a transcription of that

speech. They're not responding, they're

not translating. If you wanted to have

the model tell you about the tone or the

emotion or the speed of the speaker. If

you wanted to bias it towards words or

much less like images, those are all

things you have to build yourself as

part of the system and there's really

really a barrier to how easily you can

scale these systems. So,

fast forward to now, which is kind of of

course the LLM era, right? Like you

know, the first LLMs were mostly text,

but even there I think you could kind of

hack audio embeddings into these text

models and it kind of worked, but now of

course for a long time now, Gemini

models have been very natively

multimodal. So, what does that mean? It

means that when we train these models in

pre-training, which is where the bulk of

the data comes from, these are

multimodal interleaved examples, right?

So, the bottom uh diagram here gives you

kind of one example of what that might

look like. So, this is a task where

you're asking the model to summarize a

bedtime story and there's a text prompt

in the beginning, but then there's this

sequence of video and audio inputs that

the model gets

and then of course like what you expect

the model to do here is produce um both

the summary and also annotate timestamps

for where interesting things happen and

so forth. So, this example is teaching

the model to translate its understanding

of the audio and the video into text.

You might have other examples in

pre-training that ask you to caption a

video. So, you might have a video that

has audio and the model is learning to

bias towards both the video and the

audio signal to caption this well. And

of course like there's limitless like

YouTube videos with captions that you

can train these models on. Other models

might actually try to generate audio

from the video from the text, right? So,

you can have ASR, TTS, or any

combination of these plus all of these

sort of agentic tasks all kind of

learned under one unified token

embedding space. So, this becomes a

foundation for a lot of what we want to

do in audio because we already have a

model that understands audio, video,

text, and how these things relate and

transition from one to the next.

So, very quickly, like one of the

applications that this enables that

we've launched recently is live

translation, right? And this is a kind

of application that kind of only works

when you have all of these capabilities

working within the same model. You have

basically state-of-the-art translation

quality. Even though this model is

translating basically as the user or

speakers are speaking, you kind of ask,

"Hey, I I speak English. There's maybe

friends who are talking in Spanish and

Italian and Chinese." And it's

translating all of them to your language

as they talk. Um and then we're we're

finding is the translation quality for

this like streaming real-time

translation is about as good as you

would get with offline systems, right?

Where you kind of know the full

utterance

um from the very beginning. So, that's

something that has been classically very

hard to do with these cascaded systems,

but with LLMs, it actually just

a lot of it comes out of the

pre-training. So, of course, to do this

task it needs to do multilingual

switching because you could be

translating across different languages.

You don't know what those languages are

beforehand. It needs to preserve the

speaker voice of the source speaker and

be able to understand multiple speakers,

be robust to noise, and of course, like

do all this in real time, right? So,

again, it would be very hard to try to

engineer this, but then with the LLM and

Gemini models, this almost becomes a

prompting task.

And on that, like, I know, in this

diagram we're saying, "Hey, at the top

with these models, if you prompt it to

do the speaking like this streaming

translation task,

and you give it the audio, it will

produce the streaming translation

output, right? The same model, if you

ask it to act like an agent and respond

to maybe image and audio input, maybe

asking questions about that image, it

will give you an answer. And finally,

like very well we'll show examples of

this, you can also have it create this

embodied, you know, virtual agent that

has a face, that has, you know, things

and tools that it can show you, and it

will produce this sort of embodied agent

experience. So, with that, I'm going to

give it to Valeria to talk more about

the North Star and some of the key demo

products that we built.

>> Yeah, so to create this type of kind of

universal, versatile, uh kind of

model/product,

there's like three vectors that we think

about when we do research and product

for these models. Um and they also come

with some challenges, so I'll like walk

you through some of them. So, at the

core of a speech-to-speech model, the

first thing that people usually think

about is that it has very

conversational, right? It's low latency,

it's very conversational, very snappy,

very natural. But, I think within our

team, we really don't only want this

model to sound nice. We also have two

pillars at the top that we also really

care about like pulling all together

into one model, which is intelligence

and it being multimodal. So, when we

talk about intelligence, we talk about,

you know, task completion, instruction

following, reasoning, like capabilities

that the model needs to have natively in

order to complete tasks and to like do

things uh that have high customer

satisfaction, for example. And on the

other side of the Venn diagram, we also

have the idea that these models should

be very multimodal, both in audio in and

audio out, right? So, sometimes a user

doesn't only want to input audio in and

have that be the start of the

conversation. We need video, your screen

sharing, PDFs, whatever you would want

the model to interpret and understand,

we should be able to stream it in and

also produce output out of it. So,

that's kind of like the trifecta of

which we think about speech-to-speech

models. Um and I want to add a caveat

about ITNN. I think actually the

majority of our Gemini users are

non-English speakers. Um so, we put a

big focus on having and making sure that

all these capabilities work within not

only, you know, EN-US, but all the

languages that our customers care about.

Um

of course, this also our North Star also

becomes one of the biggest challenges in

our research because

once you move one of the knobs, it's

very easy for the other knobs to kind of

like mess up, right? Like a very quick

example, oh, how do we increase

intelligence in the model? Well, you can

turn thinking high or like the thinking

is high as possible to have the model

think a lot before calling a tool or

answering a question, which in eval's it

does show that it does improve the

model's intelligence. But when what does

that do to latency, right? And time to

first audio and the naturalness of the

conversation? So, within the Gemini

team, we're really trying to push

forward research initiatives that can

kind of blend in the three of them

without really sacrificing any of those

by a lot. Um,

but in the meantime, we're going to show

you some of the demos that we think are

hinting at how our speech-to-speech

model can combine all these three into

really cool application. So,

the first one is you kind of saw this as

a preview, but our live model, as you

know, uh, powers streaming translation

that supports 70+ languages. So, this is

a little bit around the core of

conversation quality and IT&N efforts

that we have in the team. I'll play a

quick video on how this works on Google

Meets to help two people or maybe

multiple people that are speaking

different languages still have a live

conversation.

Oh.

One sec. Okay.

>> Okay, let's turn on speech translation.

>> So, here the user can simply select the

language that they want the translation

to happen in and then the rest will be

done in real time in multiple speakers

kind of having a conversation back and

forth. So, I'll show you a snippet of

this video of what happens.

>> It's great to see you both. Cassie,

how's the weather in Shanghai?

>> It's nice to meet you. The weather here

is really nice, sunny and bright. I

spent all day Saturday in the park with

my family.

>> That sounds wonderful.

Anna, you mentioned last week that you

were celebrating your birthday. How was

it?

>> Oh, that was fantastic. I had dinner at

my favorite restaurant with some

friends. If you visit Sweden, you must

try this restaurant. It was

>> So, um a lot of things happening. Not

only there's, you know, real-time

translation. Whoops.

How do I

back to Okay.

Um not only there's real-time

translation that is happening, uh but we

see that it's in a multi-speaker

setting, you know, low latency, like

right after the user starts speaking,

the translation kicks off, so that it

doesn't feel like it's really turn by

turn and robotic and you're afraid to

interrupt because, you know, once you

start speaking, the model will start

catching your translation directly. So,

now I'll show some other applications

where we see that we have one single

speech-to-speech model for many

conversational frontiers. So, two

products that I want to highlight here

are the same model that we power search

live for everyday conversations in any

language. It's the same model that we

use to power developer experiences in

our live API. As you'll see in the

videos, both use cases are extremely

different, but because we try and make

this this model or this speech-to-speech

model very promptable and universal, you

can see how it adapts to the different

use cases. So, let me just play the

first video.

So, to highlight in this video, many

things are happening. First, the

multimodal, you know, it's real-time uh

video and audio streaming in, so that

you don't have to actually describe what

the sofa looks like. But then the

multimodal out, you see that there was a

tool called made to pull up the right

search cards if the user wanted to know

more information. And then finally, the

ITN component, you know, we're

localizing the speaker's Spanish to

Spain Spanish. And also, when we do the

kind of responding in Spanish, we're

also smart enough to know that some

words are better to be left in English,

like the mid-century type of word. It

didn't translate it directly to Spanish

cuz it knew that the term was commonly

known in English. Sorry, in Spanish. So,

this is one example of how users are

using search live for like info-seeking

questions. Now, let me show a very

different demo on how we do this for

conversational voice agents.

Oh.

Sorry.

>> Hi, I've blown a tire and had to pull

over to the side of the road. Can you

help?

>> Hello, you've reached Auto Transform

Insurance Roadside Assistance. My name

is Ace. I understand this is stressful,

but we'll take care of everything. To

get started, could I please have your

name and your policy number?

>> Uh my name is Richard Jenkins. I I don't

actually have my policy number to hand.

>> No worries, Richard. I can look you up

with your registration plate and

postcode instead. Could you provide

those for me, please?

>> Whoa, that was a bit close. Um yeah, my

registration plate is BD21

XYA

and uh my postcode is SN48ZX.

>> Policy details BD21

XYA. Thank you. For your safety, please

stay clear of the road. I've found your

details and I see you're in a blue Mini

Cooper F-Series. Is that the vehicle

you're in?

>> Yeah, yeah, that that's the vehicle.

>> Got it.

>> Okay, so I'll pause it here, but as you

can see, very different things are

happening under the hood with the same

model. You know, it's more about

alphanumeric accuracy for complex kind

of like postcode numbers or addresses.

There's like this feature that we have

called proactive audio, which is the

idea that the LLM knows when or when not

to respond to an external input. So, for

example, if you're in a conversation and

someone else is talking or there's

background noise happening in the back,

the model knows to not stop or cut

itself short because there's external

noise happening. Because we realized

that the majority of these conversations

are not happening in an office room with

like no noise, you know, in a sealed

environment. They're happening on the

go, on the train, while you're on a

walk. Um and that's the type of

experiences that we want to facilitate

with the speech-to-speech model.

Um lastly, um another thing we're really

excited about is not only voice out, but

also multimodal out. We believe that,

you know, the true AGI conversational

frontier will also require visual

presence. So, we were excited to launch

our first kind of pilot demo with City

um in Cloud Next, which supports

customized real-time avatars.

Um and you can kind of like

personalize anything from a

hyperrealistic human to a cartoon

appearance and everything in the middle.

And powered by the same speech-to-speech

model we've been showing, it allows to

have, you know, low-latency,

multilingual lip-syncing, and a really

kind of like uh ongoing fluid

conversation that has visual presence.

So, let me show you a demo that also

brings our Venn diagram together and

what we're excited about.

>> All this is reminding me of my

daughter's college fund. How are we

tracking on that?

>> You're tracking well, Jackson. I've also

identified a new opportunity that may

get you there even sooner.

Oh, and I can see Lisa just joined you.

Hi, Lisa. She must be so excited about

her college acceptance. Congratulations.

It's wonderful to see your savings goals

coming to life.

>> Excelente. Las cosas han estado tan

inestables últimamente.

>> Sí. Los servicios de tecnología y

comunicaciones han mostrado un desempeño

sólido en lo que va del año.

>> That's good news.

>> I always joke that the user's audio in

Spanish is worse than the audio model

speaking back, but um this is kind of

just to show how like our Venn diagram

of combining, you know, multimodality in

and out, you know, tool calling to like

pull up the relevant examples from the

user, and also conversational fluidity

with ITNN are starting slowly to come

together in these types of demos that

we're excited to keep pushing the

frontier of.

Um so with this parting thought, I guess

last

kind of thought that we have for you is

that we believe that AGI will not be

typed, that it will be spoken. Um and

for it to be spoken, there's a lot of

things that need to work together in a

single promptable, versatile model that

allows a user to switch between all the

sorts of conversation modes that we're

looking at, right? From translation to

taking action to brainstorming to

rambling, and we truly believe in the

power of these speech-to-speech models

to achieve that like seamless switching.

Um so we're excited to push the frontier

on that. So if you're excited or want to

learn more, please come talk to us, and

thank you so much for coming.

>> [music]