Voice Agents Can Just Do Things — Charlie Guo, OpenAI

[music]

So, my name is Charlie uh and I work on

the developer experience team at OpenAI.

And part of my job is talking to

developers to understand and see, you

know, what and how they're building with

our models. um whether that's text,

image or audio. And lately I've been

thinking about a misconception that I

have seen or maybe it's just a

misunderstanding

and it's the idea that voice agents

have to talk back.

>> And to some of you that might sound, you

know, absurd. It's a voice agent. What

do you mean it's not supposed to talk?

Uh, but I think if there's one thing

that you take away from this

presentation, I would like it to be the

idea that speech is not the only way

that a voice model has to respond.

And I think models are getting

intelligent enough and capable enough

that they're starting to open up some uh

new modes of design. I mean there's

actually three kind of modes that that

you know I kind of see emerging these

days right uh speech to speech speech to

action and event to speech and there's a

couple of things I think worth pointing

out about um these three categories. The

first is that they're not new, right? I

think as we've seen from from previous

talks, even just today, um there's a

long history of building these types of

systems in and around voice. Um if you

squint, you could make the argument that

the movie phone hotline where you called

in to get showtimes was an example of a

speech-to-pech system. Uh and I think

you could pretty reasonably make the

argument that you know GPS navigation in

your car which has existed since I was a

kid is an example of an event to speech

system. So, it's not that they are brand

new, but I think it is that we are able

to do some some much more interesting

things with them uh now that uh we're in

this era, right? And I think, you know,

the other thing I would mention here is

that um they're they're remixable,

right? They're not meant to be mutually

exclusive. Um and I think as we'll see

in a little bit, the best products uh

exist in a way that combines all of

these modes. So, speechto um everybody

knows it. Hopefully, everybody loves it.

the user talks uh and then the model

talks back and I think there are a few

examples that that I can give um for

this type of use case right you've got

things like live practice and coaching

especially around language learning

right I think the ability to um hear a

lot of emphasis or emotion and give

people that feedback is really powerful

um I think you can have you know what I

am sort of cheekily calling concierge

experiences which I think is just

another way to say customer support

plus+ Um the first voice tutorial that

most people you know try to build when

they have access to this technology is

some sort of customer support chatbot

for very good reasons. But I think with

you know when we add a richness and a

depth to the voice models as has been

happening in recent months in recent

years um we can build something that is

like much more enjoyable to use than

like talking your way through a phone

tree. Um, and so, you know, I have the

hope that like soon, if not like, you

know, now, we're capable of building

support experiences with agents that

actually feel much more enjoyable to

talk to than like arguably like the

median human support agent.

Um, and as we just saw if you hear the

last talk, uh, live translation, right?

The models have gotten good enough and

fast enough that we can just dynamically

translate content on the fly, um, with

like little to no latency. It wouldn't

shock me if at next year's keynote um

you know they live streamed it from the

main stage but also uh dubbed it in real

time across multiple languages.

The second category is speech to action.

Uh these are talks and the model uses

tools and I think this is one of the

most underexplored areas that we have.

Um I actually almost titled this talk uh

voice is the next capability overhang

because I think there is just a vast

vast amount of stuff um that we could be

doing in this category that we are not

currently doing. Uh for example um

there's a broad spectrum I don't have

you know there's way too many examples

even fit on this slide but three

categories that that I find particularly

interesting. Uh first is form filling

right um so much of the internet is just

filling out forms. Um, and there is, you

know, today no reason why you shouldn't

be able to just talk. You know, I would

love it if instead of spending an hour

filling out a government document, I

could just talk for five minutes and it

would get 90% of it for me and I would

do a quick check, you know, just to make

sure that everything looked good, right?

That is a vastly superior experience

than like having to type in every single

name and address that I've lived in the

last 5 years and, you know, all of my

previous identities. Um and so I think I

think that one is though it may seem

boring you know affects a a significant

GDP of the internet right the next

category is creative tools uh where I am

privileged enough that I can speak the

language of software and so I can tell

codeex you know here's exactly what I

want you to build and I can articulate

it in a way that um I get much more

leverage than sort of just like cludily

trying to iterate one thing at a time

but I can't do that when it comes to you

know using making music or painting um

and so if I don't have the ability to

articulate um the exact aesthetic that

I'm looking for. Um and if I don't know

how to use Photoshop or Ableton, I'm

left in this state where, you know, my

my taste exceeds my capability. Um and

so I'm really looking forward to

integrating voice into creative tools so

that I can just sort of cludgy go along

and, you know, vibe create, vibe

compose, vibe paint, um and make

something that that's really beautiful

to me. And I think the generalizable um

category here, right, then just starts

to become computer use. And we've

already seen some companies start to do

this. Um, you know, it raises the

question of like, look, if the models

are just getting good enough to do

everything on a computer that a human

can do, like why am I talking to an app?

Why am I talking to a terminal? Why am I

not just talking to the entire computer?

Uh, and so I think that's sort of a

really interesting uh way to start

exploring. But if you're a developer

today, right? Whoops. If you're a

developer today, um, what does that mean

for building your own software, right?

And I think it is like much easier than

you think to start adding audio as an

intelligence layer to the intelligence

layer to the apps that you already have.

Um, if you're building a modern web

application, you already expose so much

of it as like action as nouns and verbs,

right? And if you think about all the

verbs that you have, you have uh API

endpoints, you have, you know, React

hooks. Each of those things can like

pretty relatively easily be converted

into a tool that you expose to a model

and then you can give the user the

ability to just drive your existing

software um with their voice, right? And

yes, you still need guardrails, you

still need safety checks. Like many of

the talks today are going to talk about

securing and you know productizing this,

but um for this I just want you to think

about you know what would it mean to

take your existing software and just

talk to it.

Um and to go back to that misconception,

right? I think there are a lot of um you

know like if you're talking to the

software maybe it can talk back but

we've been developing other ways of

communicating with the user for decades

right we know these things we know we

can show notifications and popups we can

change state like the color of a button

or a drop shadow we can highlight text

um if you've used computer use in the

codeex app you know there's this amazing

like little ghost cursor animation that

goes around and clicks things for you so

we don't have to use words to actually

tell the user what is happening on

screen with their software

Uh and then the last bucket here is

event to speech, right? Um the model

receives an event and talks to the user.

Um and sort of the counterpoint from

speech to action. I think this one is

still very very exploratory, right? Um

you know, if you saw Quinn's talk, I

think there's a lot of uh space here of

like things we can do. Um and to me, we

haven't quite seen what AI native really

looks like in this vein yet. But um of

the things that I've seen, I think

there's a couple of through lines that I

tend to notice, right? The first is

hands-free or screen-free experiences.

There might be times where uh I need to

interact with software, interact with

objects and I can't use my hands or more

importantly my attention is diverted

elsewhere. Um that might be something

you know like uh a recipe app. Maybe I'm

cooking and I need to just say like

what's going on and and have something

else have something happen. Um the other

category is proactive outreach, right?

Where you the model needs to be able to

tell you something or get your attention

in a way um that you might not be

looking at, right? I think every

developer um has uh an endless amount of

notifications and events happening in

their software. But um no developer in

their right mind would sort of say I

should show all of these logs. Nor you

know would they say I should speak all

of these logs. But we can start to

conceive of voice as this like upper

level in this escalatory path of like

okay maybe you animate something and

then maybe you pop something up and then

if that doesn't work maybe you talk to

the user to get their attention.

And underlying both of these categories

and I think this this you know this

whole presentation is this broader theme

of accessibility. Um on a personal

personal note, I know like multiple

developers who um over the course of

their careers lost mobility in their

hands, lost dexterity in their fingers

and for many of them, they thought their

career as a programmer was more or less

over. Um and then came large language

models, right? Then came coding agents

and voice agents and now they generate

orders of magnitude more code than they

like previously did um you know on a

given given day or month. Um, and so I

think there's there's a lot that we can

unlock here uh for the broader world as

well.

Um, to go back to like I said, you know,

I think like when it comes to these

three modalities, you can mix and match

them and we already have some, you know,

rudimentary ways that we're seeing this.

I think there's things like, you know,

all of these pieces for in-car

assistants exist, though nothing has

quite like combined them into this

seamless way. You can talk to like the

CarPlay dashboard. Um, you can tell it,

"Hey, go play some Spotify music for

me." Um, and then it can come back and

tell you, Google Maps can come back and

tell you, hey, like, you know, there's

traffic on this route. We're going to

reroute you. But like we can now start

to think about what does it mean to

combine that into like a single voice

agent across multiple modes. [snorts]

Um, similarly, you know, there's a lot

of experimentation in the game space

with multimodality. Um, you can think

about a real life character where you're

talking to it to, you know, mine

information about the game, about the

world. um you can talk to it to execute

actions on your behalf and then it can

react to like world events right that

are happening and then give that

information to you rather than just like

a simple notification

um and I think the question you know

behind the question here right is like I

mentioned we've had all these things for

a while people have been prototyping

them for a while why focus on them now

why think about building with them now

um and I think that brings me to uh a

little bit of context here right as as

I'm hopefully most of you know

traditionally voice agents are built in

this chain Ed model, right? You uh talk,

you transcribe, you send that to a

language model. It calls tools.

Hopefully, it doesn't take too long to

respond. Um it then generates text

output. You make that into audio and

then you play that back to the user.

And some time ago, um OpenAI, you know,

decided on a different approach, right?

The real-time model family does not do

any transcription behind the scenes. It

is trained on native audio as tokens.

So, you send audio in and you get audio

back out. And the industry I think in

general has been, you know, trending

more in this direction. Um, and not just

making it native audio, but even just

letting go of the turnbased abstraction

that we've had, right? Um, and so making

it that it's just continuous streaming

audio in and out.

And the reason that opening I did this

was, you know, turns out there's a lot

of stuff that you lose when you

transcribe speech and when you

transcribe audio, right? Um there's the

old saying that when humans communicate

face to face, 55% of the information is

in body language, another 38% is in your

tone of voice and the last like 7% is

the actual words you are saying. Um and

so when you transcribe, you lose tone

and cadence and emotional uh you know

impact, you lose like whether they're

trying to interrupt you, you lose

background noise, all of this stuff

which is really important context for

the model to understand.

Um a much more quantitative reason to do

it is that you know the first two uh

voice modes in chat GBT were built with

this chained approach. Um and as you can

see had you know significantly higher

latency than using uh the native

approach with advanced voice mode.

And that brings me to GPT realtime 2. Um

and this is going to be the one part of

the talk where you know I make my

shameless plug. Um real time 2 is the

the latest model in the real time

family. We released it a couple of

months ago. Um, and the the really cool

thing about this model is that it brings

reasoning to the audio medium. Um, and

so much like our text models, it can now

think before it speaks. Um, I'm sure

many of us have seen some demos of voice

models saying things that are a little

bit less than intelligent. Um, and so

you can now, you know, try to ensure

that you give it more reasoning budget

uh to come up with a good answer. Part

of why that's also useful is that we

introduced tool calling a little while

ago. And so the model in addition to

thinking it can also delegate parallel

tool calls. Um you can start to bring

these together. Um though of course that

adds latency, right? And you know that's

why we also added preamles. Um preamles

are a way that you can prompt the model

to uh give the user a heads up if it's

going to be thinking or if it's going to

be calling tools. Um you know if you

think about the scenario of a travel

agent, right? If I called the travel

agent on the phone, uh you would want

the travel agent to say to say, "Hey,

like I'm going to go check flight

prices, right? give me a couple seconds

to do that. Um, and now with an AI

travel agent, um, and preamles, you can

actually have it communicate that to the

user while it's performing actions in

the background. Uh, there's a few other

things here, right? It's got longer

context, better domain understanding,

um, more natural voices, and it's much

more steerable. Um, there's some really

cool features, uh, that it can do when

it comes to like wake words and just

waiting for you to to tell it. You you

can give it a name. Uh, you can say

like, you know, hey, Marin, do you want

to say hi to the room? Um, and if you've

like prompted that into the model, then

it'll, you know, go ahead and and

respond to you, right? Uh, and of

course, you know, uh, obligatory

benchmark slide, uh, it does pretty well

on the the latest audio benchmarks, too.

So, TLDDR, uh, is a pretty good model.

Um,

but I think the the kind of final thing

that you know I want to leave you with

here is um when building voice agents uh

not to

start with the question of like what

kind of voice agent am I trying to

build, right? I think the thing I want

to leave you with is start with the

question of like what is the role of

voice and audio in this interaction? Um

and then how do I move forward from

there, right? Right? And often when I

ask that question, it leads to a bunch

more questions after that. Things like

what can the model perceive? What

context does it have? Right? Um what

tools are available to it and which of

those tools should it be, you know, uh

executing safely and correctly? Um

should it communicate now? Should it

wait? Uh you know, how should it

communicate? Should it be sending visual

notifications or using audio? Um, and so

taken together, yeah, I hope everybody

in here can can start to build some much

richer experiences with voice. Um,

because like others have said, uh, I do

believe that AGI will be spoken, not

typed.

Thank you very much. I'll be at the

OpenAI booth, uh, for any Q&A after. Um,

yeah, have a good event.

>> [music]