AI Engineer World's Fair 2024
Giving a Voice to AI Agents
Loading slides…
Read the talk
Giving a Voice to AI Agents
A fast speech-to-text, language-model and text-to-speech loop can still miss what someone means. Scott Stephenson traces how context could connect those stages without sacrificing control.
From a talk by Scott Stephenson
Before you start: Basic familiarity with speech recognition, language models and speech synthesis is helpful; no Deepgram API experience is required.
Hearing through a noisy drive-through
How does a voice agent understand an order while cars idle and an Amazon truck reverses nearby? That is the acoustic environment Scott Stephenson brings into the discussion. Deepgram, which he describes as nine years into building end-to-end deep learning for speech recognition, had also introduced text-to-speech the preceding December. That early release belongs to the product’s preview period, before general availability. Its audio services could run in the cloud or on premises, with models automatically adapted to environments such as the drive-through he had just visited.
Deepgram’s research team investigates fundamental capabilities, and its product team turns those findings into services. Stephenson says many devices already use Deepgram behind the scenes. But deployment alone is not the quality target: the benchmark guiding the company’s conversational systems is what a human can do.
That benchmark becomes a way to formulate research problems. If neural networks learn from examples, what examples and learning task would let a machine acquire the required ability? Stephenson describes internal discussions comparing neural capacity with mice, cats and people—not as a deployment specification, but as part of thinking about what learning systems might become capable of after a decade of progress.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From narrow assistants to a repeating conversation
Earlier voice assistants such as Siri accepted broad questions but answered reliably only within particular domains. Stephenson characterizes this generation as slow, imperfectly accurate and lacking a real-time conversational feel. LLMs expand the possible exchange: they accept text and generate open-ended responses, within the capabilities established by their training.
The resulting conversation follows a repeating sequence:
- Speech-to-text turns incoming audio into words.
- Text-to-text transforms those words into a response.
- Text-to-speech turns the response into outgoing audio.
- The next spoken turn starts the loop again.
The engineering targets are speed, appropriate agent behavior and expressive speech. Calling the middle stage text-to-text keeps its function separate from its implementation: a transformer is one way to perform that transformation, not the definition of the stage itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The first cars in the street
Stephenson places this architecture near the beginning of adoption, comparing it with seeing the first cars in the street around 1910. His forecast is a 100× expansion in voice, text and related AI over the following decade. The analogy answers the suggestion that the industry has already reached its peak of hype: visible early products may represent only a small part of the eventual market.
His broader analogy compresses successive technological revolutions: agriculture over roughly 1,000–2,000 years, industry over 250–300 years and information technology over 75 years. He projects an intelligence revolution lasting 25–30 years and argues that AI companies consequently need to move three times faster than earlier technology companies. These are his historical framing and forecast, rather than measured adoption curves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast components can still miss the meaning
The three stages work largely independently, and each has improved. Stephenson describes speech recognition accuracy rising from roughly 75% five to eight years earlier to over 90% at the time of the talk. He also describes real-time speech recognition delay falling from 2–5 seconds to 100–200 milliseconds. No dataset or accuracy definition accompanies these comparisons, and speech recognition delay is a component measure, not the full conversational response time. Running services on premises and colocating them can also shorten the path between stages.
Stephenson cites Daily’s The World’s Fastest Voice Bot as an example using Deepgram speech recognition, Llama for text-to-text and Deepgram speech synthesis. He describes its round trip as below 500 milliseconds; the post more precisely reports a 500-millisecond best case and an achieved 800-millisecond median target, using colocated models, WebRTC and tuned endpointing. Stephenson compares this with a human turn-taking interval of 400–600 milliseconds. The distinction between best-case and median response matters when using the example as a latency target.
Fast text exchange does not preserve all conversational context. In the architecture Stephenson describes, the stages pass words along without carrying the surrounding conversational state. The system can therefore respond promptly while missing what the person means. He illustrates the problem with an estimate of misunderstandings in 10–20% of turns, explicitly refining his initial wording from interactions to turns. That estimate motivates the next design change; it is not presented as an evaluated failure rate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give speech recognition a conversation to work with
The proposed change begins at the recognizer’s input. Instead of accepting only audio, speech-to-text would accept audio plus context. Prompting need not mean a text instruction: the context might include other audio, images, documents or a previous conversational turn.
Stephenson compares conventional recognition to being thrown into a basketball game without knowing anything about the conversation and having to transcribe immediately. A recognizer may have only a few seconds from which to infer the setting. Giving it the conversation so far supplies evidence it otherwise has to reconstruct, which he argues makes transcription more accurate.
Context would also become an output. The recognizer could pass through the original context and add new information expressed as human-readable text, audio, images or vector embeddings. Those outputs would carry conversational state into later stages. Stephenson explicitly presents this as a proposed architecture, not the standard system design at the time of the talk; his expectation that it would become common within the following year is a forecast.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry intent into speech—and delivery into the next turn
Once the text-to-text stage receives context derived from audio, it can respond to more than a transcript. It may know whether someone sounds angry or happy, whether the conversation is moving quickly or slowly, and whether light music is playing in the background. That information helps it choose an appropriate response.
The response can then include delivery instructions for text-to-speech: speak softly, speak slowly, or use an authoritative tone. This separates what to say from how to say it, while keeping both in the same conversational state.
The proposed feedback loop continues after synthesis. Text-to-speech would report how it tried to deliver the response, how it sounded and its assessment of the result. That feedback would become context for the next turn. The following TypeScript expresses a small application-level record for this design; the response text and feedback values are a teaching example. A requested delivery and a report of actual delivery remain separate fields.
typescript
type Delivery = {
volume: "soft" | "normal";
pace: "slow" | "normal";
};
type Turn = {
responseText: string;
requestedDelivery: Delivery;
deliveryReport: {
observedDelivery: Delivery;
assessment: string;
} | null;
};
function recordDelivery(
turn: Turn,
report: NonNullable<Turn["deliveryReport"]>
): Turn {
return { ...turn, deliveryReport: report };
}
const pendingTurn: Turn = {
responseText: "Let’s take this one step at a time.",
requestedDelivery: { volume: "soft", pace: "slow" },
deliveryReport: null,
};
const reportedTurn = recordDelivery(pendingTurn, {
observedDelivery: { volume: "soft", pace: "slow" },
assessment: "Delivery matched the requested volume and pace.",
});
The assignment records supplied feedback; it does not synthesize or evaluate audio. In a working version of the proposed architecture, the synthesis stage would supply that report. Deepgram’s internal name for this exchange is contextual AI.
Stephenson treats speed and recognition accuracy as sufficiently advanced to make context the next major source of humanlike interaction. The change is not just a richer prompt for one model: each stage receives state, contributes to it and passes it onward.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep control where the business needs it
Multimodal or speech-to-speech models could combine some or all of these stages. Stephenson accepts that possibility but identifies reduced controllability as the tradeoff. He describes Deepgram as the largest speech-to-text API, attributing its scale to business applications including Spotify, food ordering and call centers. Those users need to constrain what their systems do.
A bank makes the requirement concrete. An open-ended instruction to a speech-to-speech model gives the bank less control than separately configuring recognition, response generation and vocal delivery. The workload may justify very different investments in each stage:
| Stage | Bank’s requirement | Allocation |
|---|---|---|
| Speech-to-text | Recognize the customer precisely | Substantial compute |
| Text-to-text | Handle a few tasks, such as password resets | Smaller model |
| Text-to-speech | Sound expressive but calm | One controlled voice |
The middle stage need not be especially large if its job is narrow, even when recognition quality and vocal delivery are demanding.
This is also a cost of goods sold, or COGS, decision. Separating the stages lets a business size each component for its own task and choose services that focus on operating cost. The expensive part of one application need not dictate the size of every other part of its voice stack.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An integrated agent with component choices
Deepgram’s platform pitch leaves room to adopt individual components. An application could use its low-latency speech recognition or speech synthesis without taking the entire stack. Stephenson describes TTS as a newer product and promises greater expressiveness in its next version. His subjective assessment places its then-current quality above Amazon’s and Microsoft’s neural voices, but below ElevenLabs.
The forthcoming Voice AI Agent API preview would bring the full stack together while allowing developers to use their own API keys and LLMs. Stephenson gives integrated-stack turn-taking examples of 300, 500 or 600 milliseconds, compared with 800 or 1,500 milliseconds when assembling components independently. He presents these as examples of the integration benefit, without a controlled measurement protocol.
The closing invitation is to apply for preview access using the displayed QR code or link, or speak with the staff at the back. Stephenson also points attendees to Damien, who had led a workshop roughly two days earlier.
He closes by offering $250 in Deepgram credit to try the platform. That is the offer made in the recording, rather than a statement of current signup terms.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Daily explains its 2024 voice bot architecture, including model colocation, WebRTC transport, endpointing, and latency optimization.
Further reading
- Introducing Deepgram AuraArticle
The original Aura general-availability announcement explains its role in Deepgram's speech recognition, language processing, and speech synthesis platform.
Updates since the talk
Configure managed LLM providers or custom model endpoints in the current Deepgram Voice Agent API.
Full transcript
Loading transcript…