AI Engineer World's Fair 2026
Your Voice Agent Doesn't Need a Frontier Model
Loading slides…
Read the talk
Your Voice Agent Doesn’t Need a Frontier Model
Ace keeps its voice tutor responsive by moving lesson state, mastery tracking, and next-step decisions into code, leaving a small model to turn those decisions into speech.
From a talk by Joel Allou and Ornella Bahidika
The pause is the problem
A student asks a voice tutor a question. The tutor goes silent. Is it thinking, or has the app stopped working? For Ornella Bahidika and Joel Allou, who built the live AI voice tutor Ace, that ambiguity is the opening design problem. Bahidika describes even a one-second pause as enough to make the interaction feel broken.
When an answer feels wrong, reaching for a smarter, larger model is an understandable response. But a voice application also has to preserve the rhythm of conversation. Ace’s target is to start talking in about 950 milliseconds. In that budget, a model that spends a full second thinking has already waited too long, however good its eventual answer. The target is Ace’s design constraint, rather than a universal threshold for conversation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the model a turn summary, not the whole lesson
Ace makes a small model viable by taking responsibilities away from it. The surrounding system tracks what happened in the lesson, what the student knows, and what should happen next. Every turn, it hands the model a summary. The model’s remaining job is to express the prepared result conversationally.
Allou uses Anthropic’s Claude Opus 4.7 as the frontier-model example, initially calling it Claude 4.7. Such a model can reason through a lesson, interpret a student’s question, and work out an answer. But doing that reasoning during the conversational turn can consume the seconds the voice interface cannot spare. Ace moves that work outside the speaking model.
A state machine encodes anticipated lesson scenarios and coordinates transitions between steps. An additional intelligent layer derives the student’s mastery and helps determine whether the lesson is complete. Together, these components decide the next action, the content to display, and how a question should be answered. Their output goes to the model to verbalize. This separation makes the speaking model a consumer of lesson decisions instead of the owner of lesson progression.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The same question, two different systems
The recorded comparison starts with Opus 4.7 without Ace’s implementation. A simple question leaves the model visibly thinking before returning an answer. The second example asks the same question using the smaller Haiku 4.5 with the surrounding architecture in place.
| Configuration | Reported response |
|---|---|
| Opus 4.7 without Ace’s scaffolding | Allou describes a couple of seconds to return an answer. |
| Haiku 4.5 with Ace’s scaffolding | Allou reports an answer in about 900 milliseconds. |
Both the model and the scaffolding change, so the comparison illustrates the combined design rather than isolating either contribution. The recording does not specify the timing boundaries or repeated-trial conditions; the reported response time should not automatically be read as time to first text token or first audible speech.
Allou attributes the improvement to moving logic and reasoning into code before the speaking model is called. He also describes the smaller model as cost effective, without supplying cost figures. The visible tutor result shows a lesson titled “The Eight Parts of Speech,” alongside chat messages and a developer panel. The interaction feels almost instant to him because the decisions needed for the response have already been made.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Pay for structure in code
The smaller model creates its own engineering obligation. Allou reports that Haiku 4.5 without scaffolding tends to drift over long structures and needs strict rules to stay organized. Scaffolding is the price of narrowing the model’s job. He frames that price as development work paid in code, rather than reasoning performed afresh on every conversational turn. His selection rule is to choose the fastest model the latency budget allows, then spend the remaining effort on scaffolding.
That work means enumerating scenarios, defining transitions, and deciding how each situation should be handled. A small TypeScript teaching example can make the boundary concrete. Suppose a parts-of-speech lesson asks the student to identify a noun. An upstream evaluator supplies correct; the transition function updates lesson state and prepares the speaking instruction:
typescript
type LessonState = {
step: "noun-practice" | "verb-practice";
nounMastered: boolean;
};
type Evaluation = {
answer: string;
correct: boolean;
};
function handleNounAnswer(
state: LessonState,
evaluation: Evaluation,
) {
if (state.step !== "noun-practice") {
throw new Error("Expected noun practice");
}
const next: LessonState = evaluation.correct
? { step: "verb-practice", nounMastered: true }
: { ...state };
return {
state: next,
display: evaluation.correct ? "Verbs" : "Nouns",
speakingSummary: {
studentAnswer: evaluation.answer,
correct: evaluation.correct,
instruction: evaluation.correct
? "Confirm the noun answer, then introduce verbs."
: "Explain that a noun names a person, place, or thing; ask again.",
},
};
}
const preparedTurn = handleNounAnswer(
{ step: "noun-practice", nounMastered: false },
{ answer: "cat", correct: true },
);
Here, the example’s deliberately simple mastery rule advances after one correct answer. Ace’s actual mastery algorithm is not specified. The architectural point is the handoff: preparedTurn contains the next state, display choice, and speaking summary before any language generation occurs. The model receives the prepared instruction; it does not choose the transition or assign mastery.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small model inside a larger system
Allou extends the pattern beyond voice tutoring to other real-time applications where latency matters, and to high-volume workloads where repeated reasoning carries a recurring cost. The surrounding system owns the durable structure: state, scenarios, decisions, and progression. The model supplies the language at the point where language is needed. In that architecture, the model can become the smallest part of the system while the application remains capable of guiding a complete interaction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Anthropic's launch announcement for Opus 4.7, covering capabilities, availability, pricing, and reasoning-effort controls.
Anthropic's introduction to its small Haiku 4.5 model, including low-latency use cases and launch pricing.
Further reading
Metric definitions and instrumentation examples for separating model, speech synthesis, and end-to-end conversational latency.
Full transcript
Loading transcript…