# Voice In, Visuals Out: The Agony and the Ecstasy

Allen Pike, Forestwalk Labs

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/65X0pQ6Lmbg-voice-in-visuals-out-agony-ecstasy) · [Video](https://www.youtube.com/watch?v=65X0pQ6Lmbg)

## Voice In, Visuals Out: The Agony and the Ecstasy

Speech carries intent quickly, but spoken replies demand tight timing. Visual responses give an agent more room to act while a conversation continues.

### Why type both sides of the conversation?

[0:01](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=1s)

If speaking is a natural way to tell an AI what we want, why do we usually type a prompt and read a text response? Allen Pike opens with a proposition he attributes to Andrej Karpathy the previous month: humans prefer audio as input to AI, but vision as output. That suggests a different interface from the familiar exchange of typed prompts and Markdown replies.

![Complete quotation attributed to Andrej Karpathy over a podcast image: “Audio is the human-preferred input to AIs, but vision is the preferred output from them.”](https://ai.engineer/api/talk-essay-media/662bfd520f157436ee56458054628c01d166b8b8ac15dfc5f939237e400815c4)

Audio as preferred input, vision as preferred output.

The output half is easy to imagine. Pike invokes the idea that a third of the brain processes visual information to emphasize our visual capacity. The practical change is what models can now produce: rich HTML and tool calls can support explanatory visualizations; interactive controls let users explore, modify, and direct a response; generated illustrations can communicate something a paragraph cannot. **Visual output can become the working interface**, rather than just a formatted answer.

### Speech carries more than words

[2:15](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=135s)

Voice input has a harder reputation to overcome. The imagined experience is a fluid conversation with an AI that understands and reacts appropriately. The familiar experience is Siri failing to turn on the lights, or **ChatGPT** voice mode becoming awkward and confused. When an interface both misunderstands the request and responds slowly, speaking to it starts to feel like extra work.

Yet speech offers two advantages: people can generally speak words faster than they can type them, and delivery adds information to those words. Pike demonstrates the second advantage with two versions of the same acknowledgment: a restrained “Okay” and an enthusiastic “Okay!” The text is nearly identical; the meaning conveyed by the voice is not. That is why an important or difficult exchange often moves from messages to a call or an in-person conversation. Voice gives the participants a richer channel for communicating intent.

### From a bug discussion to a filed issue

[4:08](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=248s)

At Forestwalk, Pike’s team built an agent that participates in calls and helps in real time. During one call, he mentioned a suspected bug in the Slack integration. His co-founder said she had seen it too. Pike then asked to file it as a Linear issue. In Pike’s account, the agent reported that it had filed the issue within one second. The request relied on the conversation already in progress: the participants had established what the bug was before asking the agent to act.

The useful response does not have to be another voice joining the conversation. An agent can take action on spoken intent and acknowledge it without interrupting the people talking. That applies both when someone deliberately addresses the AI and when useful intent emerges from the surrounding conversation. Getting the interaction to feel natural, however, depends on how quickly that acknowledgment arrives.

### The output determines the latency budget

[5:22](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=322s)

The difficulty is the entire response chain. A computer reaction around 100 milliseconds can feel immediate; around one second, it can still preserve the user’s train of thought. These are the interaction limits described in [Jakob Nielsen’s response-time guidance](https://www.nngroup.com/articles/response-times-3-important-limits/), rather than a claim that human attention lasts only one second. Networking and the work needed to fulfill a request make even those budgets challenging.

For fluid voice-in, voice-out conversation, Pike uses a target of 200 milliseconds or less. That target includes the timing needed for interruptions, interjections, and quick agreement. A conventional pipeline may have to move audio over the network, convert speech to text, run model inference, and send a response back. Fitting that serial work into a conversational gap is much harder than making a screen update feel responsive.

One approach is to change the architecture. Pike points to Thinking Machines’ [Interaction Models](https://thinkingmachines.ai/blog/interaction-models/), which continuously process input and output in 200-millisecond slices. Those slices describe the inference cadence, not a guarantee that a complete response arrives in 200 milliseconds. Continuous processing offers a way to participate in a conversation without treating every utterance as an isolated request followed by a reply.

A more immediately available approach is to change the output. **Visual feedback gives the system a more forgiving response envelope.** Something appearing on screen can keep pace with the user’s thought without having to land in the exact gap where a spoken interjection belongs.

| Interaction | Response target in Pike’s discussion |
| --- | --- |
| A reaction that feels instant | About 100 ms |
| Fluid spoken turn taking | 200 ms or less |
| A visual response that preserves flow | Within about 1 second |

That extra room accommodates speech recognition, inference, and variability in the response chain. It motivates three implementation choices: fast inference, frequent inference, and a stable cache.

![Green timeline labeled STT, first token, and variability, with markers for “100ms - feels instant” and “1000ms - seamless visuals.”](https://ai.engineer/api/talk-essay-media/d1a3ab5c8663b82c51c5f8076f2673952b5c86853ddd5d33d98a83de35031d2b)

The visual response budget includes room for variability.

### Keep the immediate response path fast

[8:47](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=527s)

The first requirement is a fast model, but model size alone does not settle the choice. The inference platform must also prioritize latency. When **GPT-5 Mini** appeared, Pike’s team expected its smaller size and greater intelligence to make it useful for this role. Pike reports GPT-5 Mini latencies of 5,000–7,000 milliseconds, with occasional observations of 10,000 milliseconds. He mentions P95 during that account without clearly assigning each number to a percentile or defining the measurement boundaries. The lower price did not make those responses fast enough for the interface.

**Haiku** performed better on P95 latency in the team’s experience; Pike does not specify the version. His recommendation is a Haiku-class model or a smaller open-source model for the immediate response path. Larger tasks can still use a more capable model, provided that work does not block the ongoing interaction:

1. Let the fast model handle the live exchange.
2. Send heavier work asynchronously to a larger model that can spend time thinking.
3. Interleave its result with the fast model’s continuing responses when the result becomes available.

The fast path also needs a short enough context to complete inference in a few hundred milliseconds. Choosing a small model while continually expanding its input can undermine the reason for choosing it.

### Start inference before the speaker stops

[10:22](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=622s)

Even a fast model cannot recover time spent waiting to invoke it. A traditional voice interface listens for several seconds of speech, waits for silence to establish that the turn has ended, and only then starts inference. Waiting for a full second of silence consumes the visual response budget before the model begins.

Pike recommends being willing to invoke inference every one or two seconds while the person is speaking. The system need not be certain that the utterance is finished before it begins interpreting it. The invocation interval and the desired response time are different: frequent calls create opportunities to recognize an actionable request, while the goal is still to show the relevant response promptly.

![Slide titled “Pillars of low latency” with two numbered cards: “1 Fast models” and “2 Short intervals.”](https://ai.engineer/api/talk-essay-media/17d131e6cf023083e1f20892e86e293ceaa8576b85d78a4765eaecc358f8befb)

Fast models and short intervals support low latency.

A person may ask for one change and then add another in the same sentence. Responding as those requests arrive can feel more seamless than holding everything until the speaker finally pauses. This requires both a model and infrastructure that can sustain rapid successive turns; fast inference in isolation is not enough if the surrounding application only dispatches work at the end of a long utterance.

### Keep the prefix stable and the output short

[11:32](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=692s)

Frequent inference makes **prefix caching** central to the design. When the beginning of the input stays identical across requests, a serving platform can reuse that prefix instead of processing it afresh every time. Pike cites savings of up to 90% under applicable caching conditions, alongside faster inference. The cost reduction concerns eligible cached input tokens, not the entire request bill, and he does not quantify the speed improvement.

Pike recommends keeping roughly the first 90% of the context unchanged and concentrating changes in the final 10%, where possible. That is a prompt-layout goal, not a required API ratio. The same arrangement helps both long-running agents and agents invoked frequently: stable instructions and background context come first; the newest information comes last.

For example, the Slack-bug discussion can remain in a stable prefix while the latest utterance changes. A small TypeScript prompt builder makes that ordering explicit:

```typescript
const stablePrefix = [
  "You assist a software team during calls.",
  "Use the call context to interpret the latest utterance.",
  "Call context: Both participants observed a Slack integration bug.",
].join("\n");

function buildPrompt(latestUtterance: string): string {
  return `${stablePrefix}\nLatest utterance: ${latestUtterance}`;
}

const firstPrompt = buildPrompt("Let's file that as a Linear issue.");
const nextPrompt = buildPrompt("Include that both of us saw it.");
```

Both strings begin with the same instructions and call context; only the suffix changes. Constructing these prompts does not itself file or update an issue. Cache eligibility and controls still depend on the serving platform, but putting changing material at the end preserves the shared beginning that prefix caching needs.

Finally, minimize output tokens. A fast, frequently invoked model should return only what the immediate interaction needs, so generation does not consume the time and money saved elsewhere. Stable input prefixes make repeated turns economical; short outputs help those turns finish quickly enough to keep the visual interface moving with the conversation.

## Resources

- [Building for Voice In, Visuals Out](https://allenpike.com/2026/voice-in-visuals-out/) — Allen Pike's companion explanation of visual responses, conversational latency, frequent inference, and stable prompt prefixes.
- [Thinking Machines' Interaction Models](https://thinkingmachines.ai/blog/interaction-models/) — Research preview explaining continuous audio, video, and text interaction through 200 ms micro-turns, with background models handling longer tasks.
- [Response Times: The 3 Important Limits](https://www.nngroup.com/articles/response-times-3-important-limits/) — Jakob Nielsen's guidance on response times for perceived immediacy, uninterrupted thought, and sustained attention.
- [Token-saving updates on the Anthropic API](https://claude.com/blog/token-saving-updates) — A March 2025 announcement explaining long-prompt caching savings and cache-aware rate limits for Claude 3.7 Sonnet.
- [Claude prompt caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — Implementation guidance for prompt caching, including cache lifetimes, pricing, and automatic or explicit cache controls.
- [Cedarloop](https://cedarloop.ai/) — Forestwalk Labs' meeting agent, with Linear ticket management and Slack follow-up features.

## Slides

### Voice In, Visuals Out

Allen Pike shares lessons from building AI experiences that take voice input and produce visual output.

[0:00](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=0s) [Title slide for Allen Pike’s talk, “Voice In, Visuals Out: The Agony and the Ecstasy.”](https://assets.ai.engineer/sites/aiecode2025/sha256/12369c10e8c3aa9749b5017af446c0577e22f952aaf0a82b9b1275ab4b3e036a.webp)

Voice In, Visuals Out
The Agony and the Ecstasy

Allen Pike, Forestwalk Labs
AI Engineering WF, 2026

AIE

### IBM PC DOS

A DOS command prompt illustrates the text-based exchange Pike describes: people type to AI, and it types back. He says audio input and visual output have recently become feasible.

[0:49](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=49s) [A historical IBM PC DOS screen with date and time prompts and a command prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/28b0e717c9fdaf9eb07abad966bdec65ad158057c75c60fd4fa9a7fc7260e02a.webp)

Current date is Tue 1-01-1980
Enter new date:
Current time is 21:35:24.18
Enter new time:

The IBM Personal Computer DOS
Version 2.00 (C)Copyright IBM Corp 1981, 1982, 1983

A>_

AIE

### Visuals Out

The speaker describes models using rich HTML and tool calling to return visualizations that explain and communicate their responses.

[1:20](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=80s) [“Visuals Out” over a background of example bar, area, circular and line charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/196079808a9b913c56c6ecf75bc87ec2522043ad84d5d18f94599142c9ddf8c1.webp)

Visuals Out

Bar chart labels: A B C D E F G H I J K

Area chart labels: A B C D E F G H I J
Legend: Lorem ipsum; Lorem ipsum; Lorem ipsum

Circular chart: Lorem ipsum dolor

Line chart: Lorem ipsum dolor
[illegible]
Chart labels: A B C D E F G H I J
Legend: Lorem ipsum; Lorem ipsum; Lorem ipsum

Interface text:
AIE

### Visualization, Interactivity, and Beauty

Models can respond with visualizations that explain, controls for exploring and directing them, and illustrations. The slide pairs each form with an example.

[1:32](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=92s) [A transformer architecture diagram labeled “Visualization.”](https://assets.ai.engineer/sites/aiecode2025/sha256/7f6d30bfbbfa490daa501ac6fb98b702f30dce25099980c4116e711119f46618.webp)

Visualization

Output Probabilities
Softmax
Linear

Encoder:
N×
Add & Norm
Feed Forward
Add & Norm
Multi-Head Attention
Positional Encoding
Input Embedding
Inputs

Decoder:
N×
Add & Norm
Feed Forward
Add & Norm
Multi-Head Attention
Add & Norm
Masked Multi-Head Attention
Positional Encoding
Output Embedding
Outputs

Figure 1: The Transformer - model architecture.

Interface text:
AIE

[1:57](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=117s) [Three cards show a transformer diagram, interactive controls, and a bird riding a bicycle.](https://assets.ai.engineer/sites/aiecode2025/sha256/9ce6cd97f42b9983d05dae4d57f4a8885cb41a96fa337802b31dd9d81f60b615.webp)

Visualization
Interactivity
Beauty

Output Probabilities
Softmax
Linear

Encoder:
N×
Add & Norm
Feed Forward
Add & Norm
Multi-Head Attention
Positional Encoding
Input Embedding
Inputs

Decoder:
N×
Add & Norm
Feed Forward
Add & Norm
Multi-Head Attention
Add & Norm
Masked Multi-Head Attention
Positional Encoding
Output Embedding
Outputs

Figure 1: The Transformer - model architecture.

Interface text:
AIE
THEME
DARK
LIGHT
BREAKPOINT
DESKTOP
TABLET
MOBILE
NETWORK
Arc color
Arc width 0.6
Arc glow 13
Arc density 100%
City size 1.0
Pulse speed 3.4s
Show cities

### Voice In

Voice as the preferred input evokes a fantasy of real-time conversation with AI. The speaker contrasts it with frustrating and awkward voice interfaces.

[2:28](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=148s) [Voice In overlays a film image of a man surrounded by a futuristic interface.](https://assets.ai.engineer/sites/aiecode2025/sha256/deb8511636702a3dc6cce9b51202ec0fd0e268962a001c2f5bbffdbab0198bbe.webp)

Voice In

AIE

### Two ways to say okay

The same “okay” can sound skeptical or enthusiastic. The contrast shows how voice carries meaning beyond the words.

[3:38](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=218s) [The word okay appears above a skeptical face emoji.](https://assets.ai.engineer/sites/aiecode2025/sha256/4aa1ada6a00707b0002470c7c72e36ee3aa2690537ecb840d1ee31aea31f0a4c.webp)

“okay”

Interface text:
AIE

[3:43](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=223s) [The word “okay” appears twice, paired with skeptical and smiling faces.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea409cc97f0a9346df0aaf1e8b3553851c7d8c8fc5f9c2de60e01e3c3d024be4.webp)

“okay”    “okay”

Interface text:
AIE

### Slack threading issue discussion and filing

Two people discuss a possible Slack threading bug and decide to file it. The speaker says their voice agent confirmed the Linear issue within a second.

[4:30](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=270s) [Two call messages describe a possible missed Slack thread response and a similar threading problem.](https://assets.ai.engineer/sites/aiecode2025/sha256/4a05d6c13c775a77758fc68e8bcae98d5298e112466fc62af4ce20182833cb82.webp)

I saw a weird thing where the Slack integration missed a response in a thread, I think.

Oh yeah, I’ve seen a problem with threading too.

Interface text:
AIE

[4:42](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=282s) [A conversation about missed Slack thread replies ends with confirmation that an issue was filed.](https://assets.ai.engineer/sites/aiecode2025/sha256/e798b4e1e0708aa2837b83eb1bfb7f9d75e0313d1ffc187b90ae5c4f06b86a0a.webp)

I saw a weird thing where the Slack integration missed a response in a thread, I think.

Oh yeah, I’ve seen a problem with threading too.

Okay, let’s file that in Linear.

Filed: CED-2817
Slack integration drops threaded replies

Interface text:
AIE

### The Tyranny of Latency

Getting a response through the whole chain quickly is difficult. The speaker says a reaction needs to take about 100 milliseconds to feel instant.

[5:23](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=323s) [The title “The Tyranny of Latency” in coral text.](https://assets.ai.engineer/sites/aiecode2025/sha256/d0beae4d0838821d2b5a75a6e3fcfb7aec57cee0ff991f8c3fc9141cc268db6c.webp)

The Tyranny of Latency

AIE

### Voice and visual latency timeline

The timeline contrasts a 200ms voice-response goal with a 1,000ms window for visuals. The longer window leaves room for speech-to-text, first-token processing and variability.

[7:11](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=431s) [A timeline shows speech-to-text and first-token processing against 100-millisecond and 200-millisecond response markers.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a3de238449a9e656dc57bc169369a345228aa006248c8f9e1ed921c4c0cb80a.webp)

200ms · seamless voice
STT
first token
100ms · feels instant

Interface text:
AIE

[7:59](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=479s) [Latency diagram showing speech-to-text, first-token processing, variability, and a 1,000 ms seamless-visuals marker.](https://assets.ai.engineer/sites/aiecode2025/sha256/bdf9c17852f7686afc4dd062765e7cd76afb1c5f5e796beb3527adcbdc467caf.webp)

1000ms · seamless visuals
STT
first token
variability
100ms · feels instant

Interface text:
AIE

### Pillars of low latency

Fast models and short intervals support quick turns. When the beginning of the context stays the same, prefix caching can make inference faster and cheaper, depending on conditions.

[11:36](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=696s) [Three pillars of low latency: fast models, short intervals, and stable cache.](https://assets.ai.engineer/sites/aiecode2025/sha256/490060749ab4cd3067a3691b73bd56afe72e44b03606b94d0ec700d7e863e993.webp)

Pillars of low latency

1 Fast models
2 Short intervals
3 Stable cache

AIE

[11:41](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=701s) [Three pillars of low latency: fast models, short intervals, and stable cache.](https://assets.ai.engineer/sites/aiecode2025/sha256/58f3d486251f4e5b02d3d0564378538694f9784054716873537031b498884e65.webp)

Pillars of low latency

1 Fast models
2 Short intervals
3 Stable cache

AIE

## Full transcript

[0:01](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=1s) All right. I'm Allen Pike, and today I'm gonna be sharing some of what we've learned building voice-in, visuals-out

[0:11](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=11s) experiences using AI. Here we go. So this

[0:18](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=18s) is Andrej Karpathy, and he made an argument last month

[0:24](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=24s) that voice is the human preferred input for AIs, but that we prefer visuals as the output.

[0:36](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=36s) And he knows a thing or two about this stuff, but that's not how we have been, for the most part, building or using AI. We've been typing to it.

[0:52](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=52s) It's been typing back maybe with some markdown. But there's been breakthroughs over the last few months where both this visuals-in and audio-- or the audio-in and visuals-out experience is now, um, really feasible, and we can create really delightful experiences with it.

[1:12](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=72s) The visuals-out side is pretty intuitive, right? Of course, a third of our brain is dedicated to processing visual information. We love looking at things.

[1:24](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=84s) And models have recently got to the point where they can generate rich HTML, tool calling, so we can have these experiences where there's visualizations that come back, explain things, help us understand, communicate the, the responses from these models.

[1:42](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=102s) They can give us interactive, um, controls that allow us to explore and understand and modify and change and, and, and direct the models. And they can even respond with beautiful illustrations and images, right?

[2:01](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=121s) And so the, the, the ceiling on the, the visuals-out piece has really lifted, and we have a lot more capability of what we can do in terms of responding from these models.

[2:15](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=135s) The more controversial half of Karpathy's argument, though, is this idea of voice as the preferred input.

[2:24](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=144s) We've long sort of idealized and fantasized about speaking to an AI and having this real-time conversation where it understands what we need and it, and it reacts, uh, appropriately in real time.

[2:39](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=159s) But the experiences that most people have had so far with voice interfaces have been more like trying to get Siri to turn the lights on, and it's not working, or, like, this guy [chuckles] that is, like, trying to get ChatGPT voice mode to do things, but it keeps being, like, awkward and confused, right?

[3:01](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=181s) The models that we have so far, the experiences that people-- most people have seen are both slow and dumb, which is, like, not a great combination. And so a lot of people are down on voice as an input.

[3:18](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=198s) But speaking is the ultimate way that as humans we communicate. When we're speaking, we have more words per minute than when we're typing,

[3:31](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=211s) but we also convey more with each word. There's a huge difference in between if you say to me, "Okay,"

[3:42](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=222s) versus if you say, "Okay!" And that's why when we have something that is really important that we need to communicate or get through to somebody, we will jump on a call,

[3:59](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=239s) and we will-- or we'll speak in person so that we can get through that high bandwidth, uh, communication.

[4:08](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=248s) Um, and so what we've done at Forestwalk is we have built an agent that is in our calls that can help us out in real time. And so the other day, I mentioned to my co-founder in a call-- on a call that I'd seen a bug with, uh, the Slack integration, or at least I thought I had.

[4:29](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=269s) And she responded that she had seen the same bug. And so I just said, "Okay, well, let's file that as a Linear issue." And the voice agent within a second responded that it had done so.

[4:46](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=286s) And that feels perfectly natural when you get it really dialed in, that you ha-- are speaking either incidentally or, you know, with a purpose, intentionally speaking to the AI, and it responds in a way that is not interruptive v- of you.

[5:04](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=304s) It doesn't need to be voice. It is just taking action on your intent. And so this is-- We're gonna see this experience more and more in the coming months and years.

[5:16](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=316s) But there's a huge barrier, a huge challenge to making this actually work and feel good.

[5:22](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=322s) And that is the tyranny of latency. [chuckles] It's really difficult to get a response through that whole chain fast enough. We, we've known since the sixties that

[5:40](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=340s) to have a computer react to us fast enough that it feels instant, it needs to react in about a hundred milliseconds, a tenth of a second.

[5:51](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=351s) And so of course, we are always aspiring to get our products to react that fast, and sometimes we can achieve that. But with networking and everything, it can be challenging depending on what work needs to get done.

[6:04](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=364s) And so sometimes we might flex up to a thousand milliseconds.

[6:08](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=368s) Right? You get a full second. It's about the limit before people start to lose their train of thought. They ask Siri to do something, takes more than a second, we're off to something else mentally, right?

[6:19](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=379s) And so we're always in that play in between those two kind of human limits of trying to respond within a second, ideally within a hundred milliseconds.

[6:30](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=390s) But if we wanna have a seamless voice conversation with a, an AI or another human, the limit is much more aggressive. We need to have a 200-millisecond latency or less if we want to have a fully conversive, people are verbalizing something, and they are interrupting or interjecting or

[6:54](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=414s) agreeing and forming that sort of connection for a full voice-in, voice-out, uh, conversation. Um, and so imagine trying to do that where you have a network request, uh, maybe you take-- turn speech to text, uh, and then you do some sort of model inference on that text and then send it back on the network.

[7:15](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=435s) It's like a ridiculous, uh, amount of, of work to do in 200 milliseconds. Um, there are some clever approaches of trying to work around that. Uh, Thinking Machines a new lab has, uh, a really thoughtful architecture where they were demoing recently, um, just a few weeks ago, a-- they time sliced into 200-millisecond chunks like that 200-millisecond goal

[7:40](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=460s) and, and have a model that's doing continuous inference in and out on 200-millisecond slices. Um, so there, there are ways around it for voice-in, voice-out, but there's also just we don't need to wait for novel architectures.

[7:57](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=477s) We can just switch to having voice-in, visuals-out, and then we benefit from the more forgiving visual response envelope that people have. That if you have something that appears on screen within a second of what you've, you've said, it feels like it has met that, it's keeping-- it's within your attention span.

[8:20](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=500s) It's reacting to you in a way that feels seamless. Um, and so we've been building for that over the last few months, and it's a lot of fun, and we've learned a lot doing that.

[8:30](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=510s) And so I wanna finish up just by sharing three of the things that we found, um, to be really important for getting within that latency envelope and making the experience feel really delightful, um, for, for people who are using it.

[8:47](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=527s) So the first one is that in order to respond in a way that feels seamless, you have to use a really fast model.

[8:57](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=537s) So that's not just a model that is so small, um, that it can respond fast enough in a few hundred milliseconds, so of course, it, it does need to be.

[9:05](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=545s) Um, but it also has to be on an inference platform that prioritizes latency. So when GPT-5 Mini came out, we were excited that, um, that, okay, well, yeah, we get this more intelligent model, and presumably it'll be fast.

[9:20](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=560s) But then in practice, we were seeing latencies of 5,000 milliseconds, 7,000, P95, sometimes 10,000 milliseconds [chuckles] latency for the small model. It was cheaper than the bigger model, but we weren't consistently or even ever seeing it respond fast enough.

[9:38](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=578s) Um, Haiku is much better, um, in terms of, of that, that P95 latency. So you re- but you really need, uh, kind of a Haiku class model as the one that's responding in this real time and, and, or one of the smaller open source ones.

[9:54](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=594s) And then if there's a larger chunk of work that needs to get done, then that model then hands off or sends off an asynchronous message to a larger model that can think, and then the continuous real-time model or soft real-time model that's responding quickly then can we interleave in those responses if it's doing something heavier.

[10:14](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=614s) Um, so that's number one, you need to have a fast model, obviously with a, a short enough context that you're providing it that it can respond in a few hundred milliseconds.

[10:22](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=622s) Um, the second thing that's really critical if we're going to have this feeling of it being really responsive is to have short intervals of how quickly we're sending out for inference.

[10:35](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=635s) And so traditionally for like a voice application, you might listen to the user for a few seconds, and then they stop. You listen for a second of silence, and then you have, uh, some sort of inference goes on, and then you're, you've blown your budget by a, a pretty wide margin just for waiting for silence.

[10:55](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=655s) So if we're gonna have our-- something up on screen within a second, um, then when we wanna have our inference pretty eagerly responding as the person is talking, even if we're not entirely sure they've stopped talking, being willing to send inference every one or two seconds as they speak, 'cause, you know, we sometimes have a sentence where,

[11:14](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=674s) you know, we ask, "Oh, hey, we're gonna wanna change this," and actually also let's do this other thing. It feels much more seamless if you're doing those things as people talk.

[11:24](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=684s) Um, so you have to have a, a model and, and an infrastructure where you're able to get these fast turns going.

[11:32](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=692s) And then finally, in order to actually make all of that work, you need to have a stable caching regimen. So we have this huge improvement in the way that we work with LLMs over the last year, where we have on the different platforms, different ideas of this prefix caching, where if the beginning of the context you send

[11:54](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=714s) to the model is the same each time, then you can get up to ninety percent cheaper, faster inference, um, depending on the conditions. And so you need to lean heavily into this architecture.

[12:06](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=726s) Really, I think from, for most applications, we're moving towards this, whether it's like a long-running agent or it's a frequently running agent. It's the same principle applies that we want the first ninety percent of the context window, if we can, to be the same from request to request and then just use that final ten percent, and then

[12:23](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=743s) also, of course, minimize the number of output tokens so that you can get these really fast and relatively affordable inference turns to create that, that de- delightful experience.

[12:36](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=756s) Um, so these are some of the, the techniques we've found really useful. Um, I would love to hear from anybody who, um, has been exploring and experimenting, whether it's with real time or any way that people are pushing the boundaries of creating delightful experiences, uh, with these models.

[12:55](https://www.youtube.com/watch?v=65X0pQ6Lmbg&t=775s) Uh, I'd love to chat, share what we've been learning. Uh, and I hope some of this inspires some of you to go out and, uh, build something great. Thanks.
