AI Engineer World's Fair 2026

Voice In, Visuals Out: The Agony and the Ecstasy

Loading slides…

Read the talk

Voice In, Visuals Out: The Agony and the Ecstasy

Speech carries intent quickly, but spoken replies demand tight timing. Visual responses give an agent more room to act while a conversation continues.

From a talk by Allen Pike

Why type both sides of the conversation?

If speaking is a natural way to tell an AI what we want, why do we usually type a prompt and read a text response? Allen Pike opens with a proposition he attributes to Andrej Karpathy the previous month: humans prefer audio as input to AI, but vision as output. That suggests a different interface from the familiar exchange of typed prompts and Markdown replies.

Complete quotation attributed to Andrej Karpathy over a podcast image: “Audio is the human-preferred input to AIs, but vision is the preferred output from them.”Open full source frame
Audio as preferred input, vision as preferred output.

The output half is easy to imagine. Pike invokes the idea that a third of the brain processes visual information to emphasize our visual capacity. The practical change is what models can now produce: rich HTML and tool calls can support explanatory visualizations; interactive controls let users explore, modify, and direct a response; generated illustrations can communicate something a paragraph cannot. Visual output can become the working interface, rather than just a formatted answer.

0:010:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Speech carries more than words

Voice input has a harder reputation to overcome. The imagined experience is a fluid conversation with an AI that understands and reacts appropriately. The familiar experience is Siri failing to turn on the lights, or ChatGPT voice mode becoming awkward and confused. When an interface both misunderstands the request and responds slowly, speaking to it starts to feel like extra work.

Yet speech offers two advantages: people can generally speak words faster than they can type them, and delivery adds information to those words. Pike demonstrates the second advantage with two versions of the same acknowledgment: a restrained “Okay” and an enthusiastic “Okay!” The text is nearly identical; the meaning conveyed by the voice is not. That is why an important or difficult exchange often moves from messages to a call or an in-person conversation. Voice gives the participants a richer channel for communicating intent.

2:152:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:15 · section reference included

From a bug discussion to a filed issue

At Forestwalk, Pike’s team built an agent that participates in calls and helps in real time. During one call, he mentioned a suspected bug in the Slack integration. His co-founder said she had seen it too. Pike then asked to file it as a Linear issue. In Pike’s account, the agent reported that it had filed the issue within one second. The request relied on the conversation already in progress: the participants had established what the bug was before asking the agent to act.

The useful response does not have to be another voice joining the conversation. An agent can take action on spoken intent and acknowledge it without interrupting the people talking. That applies both when someone deliberately addresses the AI and when useful intent emerges from the surrounding conversation. Getting the interaction to feel natural, however, depends on how quickly that acknowledgment arrives.

4:084:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:08 · section reference included

The output determines the latency budget

The difficulty is the entire response chain. A computer reaction around 100 milliseconds can feel immediate; around one second, it can still preserve the user’s train of thought. These are the interaction limits described in Jakob Nielsen’s response-time guidance, rather than a claim that human attention lasts only one second. Networking and the work needed to fulfill a request make even those budgets challenging.

For fluid voice-in, voice-out conversation, Pike uses a target of 200 milliseconds or less. That target includes the timing needed for interruptions, interjections, and quick agreement. A conventional pipeline may have to move audio over the network, convert speech to text, run model inference, and send a response back. Fitting that serial work into a conversational gap is much harder than making a screen update feel responsive.

One approach is to change the architecture. Pike points to Thinking Machines’ Interaction Models, which continuously process input and output in 200-millisecond slices. Those slices describe the inference cadence, not a guarantee that a complete response arrives in 200 milliseconds. Continuous processing offers a way to participate in a conversation without treating every utterance as an isolated request followed by a reply.

A more immediately available approach is to change the output. Visual feedback gives the system a more forgiving response envelope. Something appearing on screen can keep pace with the user’s thought without having to land in the exact gap where a spoken interjection belongs.

InteractionResponse target in Pike’s discussion
A reaction that feels instantAbout 100 ms
Fluid spoken turn taking200 ms or less
A visual response that preserves flowWithin about 1 second

That extra room accommodates speech recognition, inference, and variability in the response chain. It motivates three implementation choices: fast inference, frequent inference, and a stable cache.

Green timeline labeled STT, first token, and variability, with markers for “100ms - feels instant” and “1000ms - seamless visuals.”Open full source frame
The visual response budget includes room for variability.
5:225:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:22 · section reference included

Keep the immediate response path fast

The first requirement is a fast model, but model size alone does not settle the choice. The inference platform must also prioritize latency. When GPT-5 Mini appeared, Pike’s team expected its smaller size and greater intelligence to make it useful for this role. Pike reports GPT-5 Mini latencies of 5,000–7,000 milliseconds, with occasional observations of 10,000 milliseconds. He mentions P95 during that account without clearly assigning each number to a percentile or defining the measurement boundaries. The lower price did not make those responses fast enough for the interface.

Haiku performed better on P95 latency in the team’s experience; Pike does not specify the version. His recommendation is a Haiku-class model or a smaller open-source model for the immediate response path. Larger tasks can still use a more capable model, provided that work does not block the ongoing interaction:

  1. Let the fast model handle the live exchange.
  2. Send heavier work asynchronously to a larger model that can spend time thinking.
  3. Interleave its result with the fast model’s continuing responses when the result becomes available.

The fast path also needs a short enough context to complete inference in a few hundred milliseconds. Choosing a small model while continually expanding its input can undermine the reason for choosing it.

8:478:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:47 · section reference included

Start inference before the speaker stops

Even a fast model cannot recover time spent waiting to invoke it. A traditional voice interface listens for several seconds of speech, waits for silence to establish that the turn has ended, and only then starts inference. Waiting for a full second of silence consumes the visual response budget before the model begins.

Pike recommends being willing to invoke inference every one or two seconds while the person is speaking. The system need not be certain that the utterance is finished before it begins interpreting it. The invocation interval and the desired response time are different: frequent calls create opportunities to recognize an actionable request, while the goal is still to show the relevant response promptly.

Slide titled “Pillars of low latency” with two numbered cards: “1 Fast models” and “2 Short intervals.”Open full source frame
Fast models and short intervals support low latency.

A person may ask for one change and then add another in the same sentence. Responding as those requests arrive can feel more seamless than holding everything until the speaker finally pauses. This requires both a model and infrastructure that can sustain rapid successive turns; fast inference in isolation is not enough if the surrounding application only dispatches work at the end of a long utterance.

10:2210:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:22 · section reference included

Keep the prefix stable and the output short

Frequent inference makes prefix caching central to the design. When the beginning of the input stays identical across requests, a serving platform can reuse that prefix instead of processing it afresh every time. Pike cites savings of up to 90% under applicable caching conditions, alongside faster inference. The cost reduction concerns eligible cached input tokens, not the entire request bill, and he does not quantify the speed improvement.

Pike recommends keeping roughly the first 90% of the context unchanged and concentrating changes in the final 10%, where possible. That is a prompt-layout goal, not a required API ratio. The same arrangement helps both long-running agents and agents invoked frequently: stable instructions and background context come first; the newest information comes last.

For example, the Slack-bug discussion can remain in a stable prefix while the latest utterance changes. A small TypeScript prompt builder makes that ordering explicit:

typescript

const stablePrefix = [
  "You assist a software team during calls.",
  "Use the call context to interpret the latest utterance.",
  "Call context: Both participants observed a Slack integration bug.",
].join("\n");

function buildPrompt(latestUtterance: string): string {
  return `${stablePrefix}\nLatest utterance: ${latestUtterance}`;
}

const firstPrompt = buildPrompt("Let's file that as a Linear issue.");
const nextPrompt = buildPrompt("Include that both of us saw it.");

Both strings begin with the same instructions and call context; only the suffix changes. Constructing these prompts does not itself file or update an issue. Cache eligibility and controls still depend on the serving platform, but putting changing material at the end preserves the shared beginning that prefix caching needs.

Finally, minimize output tokens. A fast, frequently invoked model should return only what the immediate interaction needs, so generation does not consume the time and money saved elsewhere. Stable input prefixes make repeated turns economical; short outputs help those turns finish quickly enough to keep the visual interface moving with the conversation.

11:3211:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:32 · section reference included

Resources

From the talk

  • Research preview explaining continuous audio, video, and text interaction through 200 ms micro-turns, with background models handling longer tasks.

Updates since the talk

Transcript download
Full transcript

Loading transcript…