AI Engineer World's Fair 2025
Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)
Loading slides…
Read the talk
Serving Voice AI at Scale
A voice agent must listen, reason, and speak before a pause becomes awkward. That shared deadline shapes model architecture, voice control, deployment, and failure diagnosis.
From a talk by Arjun Desai and Rohit Talluri
Before you start: Familiarity with speech-to-text, language models, and text-to-speech will help; no prior knowledge of state-space models is required.
When waiting changes the interaction
Why does a foundation model that works well for text still make an awkward voice agent? A text interface can tolerate an initial wait followed by fast output. A conversation makes that wait part of the experience. Cartesia’s starting point is to build interactive, multimodal models that can operate in real time and reach devices beyond the cloud.
Desai uses a 500–600 ms initial response delay as an example of what can be acceptable for text. His reference to 200 tokens per second concerns how quickly people read, not a measured model throughput result: users do not necessarily benefit from receiving text faster than they can consume it. Voice and video impose a different requirement. Quality is table stakes; the timing of delivery determines whether the interaction works.
That changes the target from a large model serving requests in the cloud to intelligence that responds continuously across modalities and locations. Voice is the immediate application, but real-time operation and deployment on different devices are separate parts of the goal.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Naturalness, first audio, and control
Imagine talking to the person beside you and waiting a full second for each response. Desai uses that pause to illustrate conversational awkwardness, not to establish a universal latency threshold. In customer-support triage, the same delay compounds the frustration of trying to resolve a problem. The agent must also handle interruptions, different accents, and background noise, while satisfying preferences that vary from listener to listener.
Three requirements follow:
- Quality: The generated voice needs to sound natural enough for the intended experience.
- Latency: The first audible response should arrive quickly. Reducing this delay leaves more time elsewhere in the agent for reasoning.
- Controllability: The voice should express the company’s identity and speak about its products in the intended style.
These requirements interact, but they are not interchangeable. A natural voice can still respond too late, and a fast voice can still sound wrong for the brand.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate from maintained state
Cartesia’s architectural approach uses state-space models, or SSMs, as an alternative to transformers. Desai motivates the choice through a quadratic-scaling comparison: as input sequences grow, conventional attention can require substantially more work and memory. That comparison needs a precise boundary. Quadratic attention costs are not a universal description of every transformer decoding implementation or its memory use.
The SSM mechanism is recurrent: maintain a state representing the processed sequence, then generate from that state. Desai’s O(1) claim means that a generation step need not become more expensive merely because more history has accumulated. It does not mean generating an entire response takes constant time; longer outputs still require more steps. This distinction matters for a sustained conversation, where the interaction keeps growing but the response deadline does not.
Recurrent efficiency alone would not be enough if it reduced voice quality. Desai says Cartesia has closed the quality gap historically associated with recurrent models and improved both latency and quality relative to transformers. The talk supplies no benchmark setup for that comparison, so it remains Cartesia’s reported result rather than a general ranking of the architectures.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast speech generation buys reasoning time
Sonic 2 provides voice generation, not the entire voice agent. A conventional agent connects speech-to-text, a language model, and text-to-speech. Each component consumes part of the time between a user’s turn and the audible answer. The language model may need substantial time to produce a useful response, especially when it was not designed for low-latency interaction.
Faster TTS creates slack for the language model; it does not eliminate the language model’s delay. Desai describes Cartesia’s TTS as the fastest available, although he supplies no comparative benchmark here. The engineering objective is more concrete than that ranking: spend less of the shared response budget turning an answer into sound.
Control extends beyond choosing a pleasant voice. Desai points to voice cloning, accents, and generated background sounds. A perfectly clean agent can feel uncanny on a phone call, while familiar phone noises and small imperfections can make the interaction sound more like what the caller expects. Naturalness therefore depends on context, not simply on removing every audible irregularity.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From support calls to licensed performances
The applications Desai names include healthcare, customer support, and real-time gaming. Gaming makes the interactive requirement particularly clear: a non-player character should respond dynamically to the player rather than only deliver fixed dialogue. He describes growing demand across these markets and connects it to AWS’s investment in infrastructure for voice workloads, without giving adoption measurements.
Narration raises a different question: what role remains for human performers? Desai describes a voice marketplace in which creators make their voices and personalities available for others to license. His stated aim is to amplify voice actors rather than replace them, and he says narration is already an important use case. The mechanism is licensing a performer’s voice through the platform; the discussion does not establish compensation terms or employment outcomes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The slow component still sets a constraint
An audience member using Cartesia through Pipecat reports that Claude introduces too much latency and asks which models pair well with the voice stack. This is a concrete integration problem: improving the speech generator can make the overall agent faster while leaving its largest remaining delay untouched.
Desai returns to the shared budget: time saved in TTS can be spent at the language-model stage. He suggests that dedicated instances may improve latency, but does not provide a configuration, a measured improvement, or a preferred model pairing. Some optimizations remain dependent on the language-model provider.
Talluri’s answer emphasizes model choice. SageMaker JumpStart and Amazon Bedrock offer different models for different workloads, and AWS’s stated strategy includes bringing specialized providers into that ecosystem. For voice applications, the relevant capabilities include real-time responses and deployment on edge devices. This broadens the available design choices; it does not resolve the audience member’s particular Claude configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Rich data must represent different preferences
The next question asks whether audio resembles a proposed pattern in video research: perhaps information density matters more than simply collecting more data. Desai’s answer preserves both sides. The familiar recipe of large-scale pretraining followed by alignment or preference fine-tuning remains relevant across modalities, including video, an area he worked on during his Stanford research.
Audio complicates the preference stage because listeners want different things. In Desai’s account, a single fine-tuning stage cannot fully capture that diversity. High-quality recordings and rich data are necessary, but they must be paired with information reflecting the range of experiences people want. The answer is application-dependent: it does not establish a formula in which data richness replaces scale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Integrated speech versus component control
Speech-to-speech models offer another way to organize the system. Asked about their future, Desai says they already have useful applications but, in his view at the time of this 2025 discussion, were not yet ready for the enterprise use cases he had in mind. The conversation mentions a recently released Amazon model without naming it.
| Approach | Principal advantage in the discussion | Design concern |
|---|---|---|
| Orchestrated STT → LM → TTS | Control over individual components | Coordinate their behavior and delays |
| Integrated speech-to-speech | Expected long-term latency advantage | Preserve enterprise control |
Separate components let developers determine how each stage behaves. Desai expects speech-to-speech models to win on latency over time, but argues that controllability must be designed into them from the beginning. A convincing interaction is not sufficient unless the system also functions reliably for its real-world task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare local execution with the whole cloud trip
Local models introduce a deployment tradeoff rather than making the cloud obsolete. Desai says Cartesia already has models running on edge devices. Cloud systems still support model sizes and capabilities that may not fit on small devices, particularly phones and hardware more constrained than laptops.
The relevant comparison is local execution against network latency plus cloud execution:
Desai reports that Cartesia’s edge models run approximately five times faster than a cloud round trip. This is a comparison with the network-inclusive path, not a claim that the device performs inference compute five times faster than a cloud accelerator. The model variant, device, network conditions, and measurement protocol are not specified.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When the agent says the instruction aloud
An audience member gives a useful failure case: the agent says “pause three seconds” instead of pausing. Where did the error originate—speech recognition, the language model, speech synthesis, or infrastructure? And how can the developer distinguish an integration problem from a provider regression?
Desai emphasizes evaluations and contacting the provider, but names no monitoring tool. He says many issues Cartesia encounters arise at the language-model stage. More specifically, the boundaries need attention: speech-to-text output may require formatting before the LM consumes it, and LM output may require formatting before TTS can speak it correctly.
One way to make that boundary explicit is to separate speech from actions. In this TypeScript example, the intended pause is an action record, so the TTS input contains only the words meant to be spoken:
typescript
type VoiceAction =
| { kind: "speak"; text: string }
| { kind: "pause"; durationMs: number };
const response: VoiceAction[] = [
{ kind: "speak", text: "Let me check that for you." },
{ kind: "pause", durationMs: 3000 },
];
async function deliver(
actions: VoiceAction[],
speak: (text: string) => Promise<void>,
pause: (durationMs: number) => Promise<void>,
): Promise<void> {
for (const action of actions) {
if (action.kind === "speak") {
await speak(action.text);
} else {
await pause(action.durationMs);
}
}
}
This illustrates a boundary contract, not automatic interpretation of arbitrary LM text. The producer must emit the intended action, and the orchestrator must route it correctly. Evaluating those records separately from the resulting audio makes the failure easier to locate.
Desai says Sonic 2 handles more formatting edge cases internally and has improved in quality as well as latency. The accompanying slide separates two variants: Sonic Turbo is listed below 40 ms time to first audio, while Sonic-2 is listed below 90 ms. It also shows a comparison with the next best alternative at P50, P90, P95, and P99.
Desai reports a 2.5× speed improvement over Cartesia’s initial model and approximately 40 ms model latency. The historical changelog associates the 40 ms figure specifically with Sonic Turbo, matching the slide’s distinction. Neither that figure nor the spoken improvement establishes total conversational latency; the talk does not supply the benchmark conditions or detailed baseline. His broader point is that the newer system also absorbs more of the edge cases developers would otherwise need to handle themselves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Beyond hearing, with the same real-time requirement
Talluri closes by asking for a five-year view from 2025 to 2030. Desai expects voice AI to become commonplace across industries, including triage, end-to-end support, and gaming. These are forecasts about adoption, not capabilities or market outcomes demonstrated in the session.
The longer-term direction extends beyond speech. Interactive models would help people interpret the world around them through multiple modalities, operating as assistants or copilots. Desai connects this ambition to world models, but the condition remains the one that made voice difficult in the first place: the system must work in real time to participate usefully in the user’s experience.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Historical Sonic 2 and Sonic Turbo release notes, including latency messaging and changes to voice controls.
Further reading
The 2024 announcement of Edge, the Rene language model, and Sonic On-Device private beta.
Open-source models and inference tools for running state-space language models locally with PyTorch, MLX, and Metal.
Original research on selective state-space models and efficient recurrent sequence processing.
Amazon's account of a unified speech-and-text model designed for streaming conversations, turn-taking, and interruptions.
Updates since the talk
Setup and configuration for streaming Cartesia speech through Pipecat, including text aggregation and interruption support.
Full transcript
Loading transcript…