AI Engineer World's Fair 2024

Building & Scaling an AI Agent Swarm of low latency real time voice bots!

Loading slides…

Read the talk

Building & Scaling an AI Agent Swarm of low latency real time voice bots!

A browser-based drive-through agent exposes the engineering behind real-time voice: streaming audio, backend tools, interruption handling, conversational memory, and regional scaling.

From a talk by Damien Murphy

Before you start: Familiarity with JavaScript, HTTP APIs, and WebSockets will help you follow the client, backend, and streaming-audio examples.

Audio in, an order out

How do you turn a caller’s speech into a useful backend action, then speak the result without making the caller wait? The problem begins before the LLM sees any text. Telephone audio can arrive at 8 kHz in mulaw or linear16, with much less detail than a clean microphone recording. Meeting transcription, drive-through ordering, and phone support therefore place different demands on speech recognition. Damien Murphy introduces this workshop from his work as a customer-facing applied engineer at Deepgram, helping companies build and operate voice bots.

The application to build is a voice-to-voice agent: microphone audio goes in, synthesized audio comes back, and an LLM calls a backend API to do useful work. A separate frontend displays the resulting application state. Deepgram’s role is to supply and operate the speech models as well as integrate the conversation pipeline; Murphy emphasizes that delivering models economically in production is a different problem from training them.

Slide titled “What you will learn today” with four bullets covering a voice-to-voice AI agent, a simple backend API, LLM function calling, and scaling a multi-agent swarm.Open full source frame
Workshop goals: voice-to-voice agents, a backend API, function calling, and agent swarms.

The workshop uses Node.js, a browser, a microphone, and speakers or headphones. Chrome is the recommended browser because the other browsers had not been tested. Keeping speaker volume low matters in a room full of agents: one conversation can become another agent’s microphone input. Although Murphy invites participants to create a Deepgram account, the workshop itself supplies access without either a Deepgram or an LLM API key.

0:270:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:27 · section reference included

The latency between three services

The conventional pipeline has three stages: speech-to-text produces a transcript, an LLM generates a textual reply, and text-to-speech turns that reply into audio. Each stage can be optimized, but the connections between them also matter. Self-hosting and co-locating the services can reduce delay; doing so makes infrastructure placement, capacity, and operation part of the application’s responsibilities. The integrated Voice Agent API packages these stages behind one audio interface and also handles endpointing—the decision that the user has finished a turn.

Model choice influences placement. Murphy suggests locating Claude deployments near AWS infrastructure and OpenAI deployments near Azure, while locally hosted Llama 3 or Phi gives the developer more control over location. These are placement examples: the objective is to keep the LLM, speech services, and function-calling backend close together.

Murphy cites 500 ms for speech-to-text and 700 ms for the LLM/TTS portion of the illustrated pipeline, without separating the latter stages. Reducing these delays eventually exposes a different constraint: a response can arrive so quickly that it sounds rude. The target is a timely conversational turn, not simply the earliest possible audio.

4:264:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:26 · section reference included

A Krabby Patty and a kelp shake

The first live attempt stops at a familiar browser boundary: microphone permission. After allowing access, Murphy receives a welcome from the Krusty Krab drive-through and asks for a Krabby Patty. The agent begins responding, but its output appears to feed back into the microphone. Murphy then requests a kelp shake, and the agent confirms that both items have been added. The interface displays the resulting order.

Browser demo with a menu on the left and a purple shape above voice buttons and an order containing one Kelp Shake and one Krabby Patty.Open full source frame
The ordering demo displays a kelp shake and a Krabby Patty in the order.

The next request exposes a recognition or interpretation problem. Murphy asks for a Krusty Combo; the agent treats the request as an unavailable Crispy Combo and offers a Krusty Combo instead. He agrees, then refreshes the page to stop the application from continuing to listen while he explains it.

The architectural result is more important than the fluent reply: the LLM calls the ordering API; the application does not parse the assistant’s prose to manufacture an order. Audio enters the service, the model selects a function, and that function changes backend state. The spoken confirmation and the visible order are separate outputs of that process.

7:077:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:07 · section reference included

Connecting the browser, agent, and backend

The implementation is split between the workshop client and workshop server. Participants can use Murphy’s shared backend or modify their own, but a replacement must be publicly reachable so the agent service can call it. The recording uses a pre-release, keyless sandbox and a Glitch-hosted backend. Those are historical setup details: Glitch hosting subsequently ended, and present-day integrations should follow the current Voice Agent API reference, rather than copy the workshop endpoint or assume its hosted services remain available.

The client is vanilla JavaScript and HTML. Its main.js opens a WebSocket, captures microphone audio, and sends it to the service. The rate at which the client delivers audio sets a lower bound on responsiveness: the server cannot recognize sound it has not yet received. Murphy suggests approximately 20 ms audio chunks for a telephone integration; the browser demonstration sends larger chunks.

Configuration supplies the backend base URL, the input encoding, the model, the instructions, and the available functions. This example sends linear16 audio and selects OpenAI’s GPT-4o. An add-item function tells the agent which API to call, while a unique call ID associates every item with the correct conversation’s order.

The resulting division of work is small and explicit:

  1. The browser sends microphone audio to the Voice Agent API.
  2. The integrated service performs recognition, model inference, and speech synthesis.
  3. The LLM invokes backend functions to add order items.
  4. The browser reads the backend order and displays it.

Both the browser and agent can access the backend, but they use it for different purposes: the agent changes the order, while the browser presents its state.

8:569:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:56 · section reference included

An add-only agent cannot fully modify an order

The starter agent exposes only add-item. The backend already has more operations, so the exercise is to expand the agent’s tool contract. Removing an item requires both a removal operation and access to the existing order; the model needs to know what is present before deciding what to remove. The server’s existence alone does not make those capabilities available to the LLM.

Slide titled “Backend - Menu, Calls & Orders API” groups GET, POST, and DELETE operations under Menu, Calls, and Orders, with the order-item POST marked “LLM Function Call.”Open full source frame
Backend operations for menus, calls, and orders.

The pre-release service also offers model and provider choices, including Claude, Llama 3, Mixtral, OpenAI, Anthropic, and Groq. Meanwhile, the application has two distinct representations of the menu: the browser retrieves one from the menu API when it creates the call, but the LLM’s menu is baked into its system prompt. A proposed get-menu tool would let the agent discover changing availability—for example, that chicken wings have gone out of stock—instead of continuing to rely on its initial prompt.

13:1413:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:14 · section reference included

Inspecting the conversation as it happens

Chrome DevTools’ Network tab provides a WebSocket inspector for watching the audio stream and returned messages. The initial settings message contains the agent configuration, including instructions and function definitions; the displayed settings payload is truncated, so it is not the complete contract. The service returns a session ID and conversation text. Displaying the user transcript gives immediate feedback that the application is hearing them.

Tool messages expose the requested function and arguments—the Krabby Patty in this example—followed by a success or failure result. Assistant text is also available independently of its audio, giving the application a place to inspect inappropriate offers such as free Krabby Patties. On the backend, logs show call creation, the order ID, item additions, and the updated order consumed by the frontend. Together, these views let a developer follow speech, a proposed action, the action’s result, and visible application state.

15:2615:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:26 · section reference included

Capturing, buffering, and interrupting audio

The client keeps its responsibilities in five files:

FileResponsibility
main.jsUI hookup and conversation lifecycle
config.jsAgent instructions and settings
services.jsBackend CRUD calls
audio.jsCapture, playback, and downsampling
animations.jsSpeech-responsive canvas bubble

The workshop downsamples browser audio from 48 kHz to 16 kHz to reduce bandwidth. The server is a small Express.js application whose index.js updates application state through CRUD operations; the animated bubble simply expands and contracts with speech.

audio.js contains receiveAudio, captureAudio, clearScheduledAudio, and downsample. The clearing function is essential because synthesized audio may already be scheduled for playback when the user interrupts. Stopping generation remotely is insufficient if the browser still has buffered speech to play. Murphy suggests an optional client-side Silero VAD to detect speech locally and stop playback sooner. A more advanced implementation also tells the LLM where playback stopped, so it does not assume the user heard the end of its previous answer.

Incoming WebSocket audio enters a playback buffer. In the opposite direction, browser media-device capture invokes a callback that sends audio over the socket. Page load prepares the configuration and order identifier; selecting Start Conversation opens the connection and begins transmission. Other handlers process connection errors, textual status messages, assistant text, and audio responses. Voice selection can also be updated during the application flow.

18:0818:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:08 · section reference included

A new voice does not add a new capability

In the second demonstration, Murphy orders a Krabby Patty and then a Krabby Meal. He asks to change that to two Krabby Patties. The agent responds that it has added another patty; the displayed order contains two patties and still contains the meal. This is the add-only contract made visible: the model can add an item, but the demonstration has not supplied the complete retrieval-and-removal flow needed to perform the requested replacement.

Ordering interface with a menu, purple shape, Athena and Angus voice buttons, and an order listing two Krabby Patties and one Krabby Meal.Open full source frame
The demo order now contains two Krabby Patties and one Krabby Meal.

Murphy identifies the second voice as his own, one of the voices available in the demonstration. That capability also has a social consequence: he recalls warning his parents about a call that might sound like him asking for money. The voice can change how an agent sounds without changing what its tools allow it to do.

An audience question makes the distinction explicit. config.js declares the functions and their backend location, but the LLM dynamically decides whether and when to use them. Murphy compares that choice to ChatGPT plugins: making a tool available does not guarantee its invocation. He favors GPT-4o, and tentatively Mistral, over GPT-3.5 for the function-calling behavior he has seen.

An attendee adds that clear system instructions and precise function definitions help newer models request the tool instead of inventing an answer. Murphy agrees, preferring Claude Sonnet and Claude Opus while expressing uncertainty about Claude Haiku. He describes Groq-hosted Llama 3 70B and Mixtral as more variable for tool use at the time. These are practical model-selection observations from the workshop, rather than a controlled tool-calling evaluation.

22:3422:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:34 · section reference included

A pause is not always the end of a turn

People think while speaking. They pause before completing a sentence, change their minds, and interrupt answers that are going in the wrong direction. A silence threshold alone cannot explain all of those events. Murphy introduces semantic endpointing, then planned for the API, as a way to use the meaning of the utterance when deciding what should happen next. If a caller says they need a moment to find an account number, the appropriate response is a brief acknowledgment and a wait, not another monologue.

Backchanneling supplies a different kind of timing signal. A caller reads four credit-card digits, hears an acknowledgment, then reads the next four. The acknowledgment confirms that the listener is keeping up; it need not seize the conversational turn. Repair also needs to resemble normal human conversation. An email address may require spelling with examples such as D for Damien and A for apple, just as it would with a human operator.

These details argue for specialized agents. Credit-card collection, account verification, and ordering have different edge cases, so each can have a focused set of instructions and tools. A large context window does not make a single prompt that handles every business process reliable. Longer prompts also increase the work before the LLM starts responding, making time to first token relevant to conversational design.

Murphy suggests starting TTS after roughly 5–10 generated tokens instead of waiting for token 250. He estimates many LLMs at about 30 tokens per second and notes that GPT-4o’s first-token latency can reach roughly 900 ms, with substantial variation. The practical mechanism is streaming: once enough text is available to begin speaking, later text can continue arriving while the first part plays. Waiting for the complete answer unnecessarily puts generation time on the caller’s critical path.

26:1726:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:17 · section reference included

Keeping the conversation moving

A GPT-style assistant with API tools and this voice agent share much of their application logic. The additional work is the real-time audio path: streaming sound into recognition and streaming speech back out. The discussion treats newly announced native-audio capabilities as forthcoming; its engineering comparison concerns the text-LLM-plus-speech-services architecture available to the workshop.

Murphy reports end-of-speech-to-transcript latency of about 50 ms self-hosted versus about 500 ms on the hosted API, associating the lower figure with roughly five times the compute. These are his deployment estimates; the workshop does not specify a matched model, hardware, workload, or endpointing configuration. The tool interface is a separate concern: he describes OpenAI-style function definitions and system prompts as broadly reusable across LLM providers.

Long-running functions threaten the conversational rhythm even when recognition is fast. Murphy proposes handing slow work to a secondary system, acknowledging that it has started, and continuing to help while it runs. The result can be offered when the work finishes.

It helps to separate three directions of data movement:

  • Order mutation: The LLM calls add-item and writes to the backend.
  • Display refresh: The browser polls the order and updates the interface.
  • Agent context refresh: A get-order or get-menu tool lets the LLM read current backend state.

The proposed remove-item tool completes the modification flow. Moving the menu out of the prompt and behind get-menu also allows another service—even another LLM—to update availability, with the next fetch bringing that change into the conversation.

The audience presses on a different question: can an external process proactively insert a completed result without waiting for another tool fetch? Murphy says he has not tried that flow and proposes a backend webhook handler:

  1. A tool starts the job and returns immediately.
  2. The agent acknowledges that the job has started.
  3. The background system calls a webhook when it finishes.
  4. The backend handler supplies the completion to the LLM, prompting a follow-up.

That is an architectural proposal, not a demonstrated conversation-injection contract for the workshop API.

30:4230:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

30:42 · section reference included

Stopping playback when the user speaks

Voice activity detection and deciding to answer are different operations. Murphy notes that vocal events such as coughing or humming can trigger a detector. An application can stop playback promptly, then use additional evidence—such as whether recognition produced a word—to decide whether it should respond. That avoids requiring every detected vocal sound to become a full conversational turn.

The workshop uses a server-side speech-start event to clear client playback. The relevant JavaScript branch is small because the audio module owns the queue:

javascript

function handleAgentMessage(message) {
  if (message.type === "UserStartedSpeaking") {
    clearScheduledAudio();
  }
}

The important action is clearScheduledAudio(): audio that has already arrived must stop along with any future response. Murphy estimates the variable speech-start notification latency at less than 100 ms.

VS Code displays a message handler checking “UserStartedSpeaking” and calling clearScheduledAudio(), with order-processing code below.Open full source frame
The message handler clears scheduled audio when the user starts speaking.
38:5439:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

38:54 · section reference included

Reconciling memory and checking commitments

For a long call, continually appending everything to the system context is an awkward memory strategy. Murphy points to AutoGen’s Teachable Agents as an example of using another LLM to analyze new information for memory. His concrete requirement is reconciliation: if a caller first gives 123 Street and later corrects it to 456 Street, the active memory should contain the corrected address rather than two competing facts. The correction below represents the desired memory update, not a demonstrated automatic behavior of the workshop agent.

Replace a superseded address in active memory

Constructed example: The memory field label and object ID are teaching structure. Both address values and the correction come from Murphy's example; the update is a proposed behavior, not an executed workshop result.

Caller correction — unchanged
Oh, actually I live at 456 Street.

Operation: Recognize that the new address corrects the existing address, then replace the active memory value.

Current address

Before: Before the proposed memory update
123 Street
After: Desired memory after reconciliation · Changed
456 Street
Keep one current address instead of two contradictory current facts.

Conversation text also creates a place for a separate moderation process. Asked about hostile input, Murphy recalls the story of a car chatbot making an invalid offer and proposes inspecting generated text for commitments the business cannot honor. He warns that repeated requests can undermine prompt-only rules, but supplies no measured attack success rate. The proposed protection is an output check that can detect and block an invalid offer; he does not present it as a complete solution to prompt injection.

40:1740:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

40:17 · section reference included

Voices, languages, and choosing what to automate

Murphy describes 12 publicly available English TTS voices at the time of the workshop. He positions their pricing near the large cloud providers and claims an approximately 20-fold price advantage over ElevenLabs, without specifying comparable plans or billing conditions. Promptable TTS—asking for an empathic delivery or a pirate voice—is still research work in this account. Other languages are to follow that work so the team does not have to retrain them around a new prompting capability. He then plays voice samples and mentions a customer preference he has observed among the voices.

For recognition, Murphy reports 36 supported languages in Nova-2, with some additional languages in older models. He describes an automatic training pipeline that identifies low-confidence words and uses poor performance to direct retraining. The useful distinction is between selecting a voice, expanding language coverage, and improving recognition on difficult audio: those are separate model-development tasks.

Before automating a call center, analyze the calls already being handled. Summarization, topic detection, intent recognition, and sentiment analysis help identify recurring work, including where topics occur within a call. Murphy’s example is a call population in which 40% concern phone issues: that concentration would make phone support an obvious candidate to investigate first. Agent assist can initially surface knowledge-base articles for humans, establishing useful retrieval before moving a task to an AI agent.

The handoff need not replace the human for the entire call. A human operator could delegate credit-card collection to an AI using a consistent voice, then return when assistance is needed. Murphy sketches a scenario with five simultaneous calls and approximately fivefold productivity, rather than presenting measured workshop results. The mechanism is selective delegation of routine segments while preserving human escalation.

43:0543:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

43:05 · section reference included

Monitoring an agent and absorbing peak demand

The workshop places the agent connection in a browser, but a production telephone integration commonly runs on a server connected to a provider such as Twilio. That server is a natural place to inspect activity, apply moderation, and block behavior outside the application’s rules. Adding more instructions to the system prompt helps only up to a point.

Asked whether to evaluate the complete voice flow or individual components, Murphy recommends observability tools rather than a specific metric suite. He names AgentOps for temporal debugging through the LLM flow and notes that conventional monitoring tools remain useful. The answer establishes the need to reconstruct what happened over time, but does not supply an end-to-end evaluation protocol or acceptance thresholds.

Operating the pipeline yourself means solving that problem in every region. Routing European users to the US adds latency and can raise data-governance concerns; APAC callers also need sensible placement. Regional clusters then need their own autoscaling. Murphy uses an intentionally large outage scenario—ten routine customer-service agents suddenly facing demand for a million agents for an hour—to illustrate the difference between ordinary load and a surge. Disaster-related emergency calls offer another example of demand that can exceed available human capacity. These examples motivate burst capacity, not a demonstrated capacity result for this application.

46:4446:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

46:44 · section reference included

Recognizing speech more frequently

A follow-up question removes networking and regional scale from the discussion: what makes inference itself faster? Murphy describes hosted transcription as updating at one-second intervals, while a self-hosted configuration can run five times per second. More frequent inference revisits an expanding audio context, so a partial recognition can evolve from so to some to something as more of the word arrives.

Murphy describes a streaming context window of roughly 3–5 seconds, with end-of-speech detection triggering finalization of the available transcript. The system need not wait until the end of a whole utterance to begin recognition; it has already processed overlapping, incomplete evidence. Increasing the update cadence costs compute. He mentions a Tesla T4 as able to serve many streams but supplies no concurrency count.

Self-hosting uses Deepgram’s Docker images and models on local compute. Murphy offers several GPUs in one machine as a way to place the speech and language components close together. When asked how a move from AWS Transcribe would compare, he names speed, quality, and price, but gives no comparative measurements. That leaves the operational choice dependent on the actual audio, workload, and serving configuration.

50:1750:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

50:17 · section reference included

Building the swarm from narrow responsibilities

The ordering backend can be replaced with another business workflow, such as table booking. A routing agent can send a new reservation to a booking agent and a cancellation request to a cancellation agent, while preserving the same phone line and voice. Similar flows apply to call centers with human escalation, wearables and toys, or drive-through workers who are also preparing food.

Single responsibility is the organizing principle of the swarm. Build a narrow agent that works reliably, keep its boundary clear, and reuse it in larger flows. Use the smallest and cheapest model that still performs the task; Murphy acknowledges that the larger models may be necessary for the capabilities under discussion. His expectation of rapidly falling model prices supports revisiting that choice over time, rather than weakening the task contract to fit an inadequate model.

The hierarchy in the slide makes reuse concrete:

AgentSpecialized dependency
RoutingSelect support or booking
Technical supportRetrieve information through RAG
Account supportVerify account ownership
Existing bookingVerify ownership before access
New bookingInvoke credit-card payment

Account verification can serve more than one branch, and payment can remain a focused capability instead of becoming part of every agent’s prompt. The complete diagram connects those shared services to the support and booking specialists.

Agent hierarchy diagram branches from Routing Agent into Support Agent and Booking Agent, then technical support, account support, existing booking, and new booking agents, connected to RAG, account verification, and credit card payment.Open full source frame
A routing agent connects support and booking specialists to shared services.

The deployment still needs regional capacity and availability. Murphy calls for horizontal scaling in regions such as US East, US West, and EMEA, plus redundancy within the deployment. Redundancy raises cost, but an always-available phone service needs spare capacity and failure tolerance. He describes Kubernetes support and autoscaling Helm charts as available infrastructure building blocks. The choice is whether to operate those clusters or let a managed service own them while the application team concentrates on agent behavior.

52:4453:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

52:44 · section reference included

Where the models run and where the audio enters

On-device inference changes the deployment problem but does not eliminate it. Models must still be distributed and downloaded; Murphy uses Gemma as an example of a substantial browser or mobile download. A device handling one stream is also a different workload from a centralized service handling many simultaneous calls. He cites the open-source Friend wearable as a device experience integrated with remote Deepgram recognition: using an agent on a device does not require every model to run there.

Asked specifically about embedded speech recognition, Murphy says the model can run on a Raspberry Pi CPU, but slowly and without handling multiple concurrent requests. That distinction separates basic executability from a useful latency and concurrency envelope.

For telephony, the interface requirement is access to the audio stream. Murphy names Twilio and Vonage, while emphasizing that the agent architecture is provider agnostic. An integration can connect through a provider API or use a SIP trunk to hand call processing to another server.

The next question returns to native audio models. Murphy welcomes the voice flexibility of the announced GPT-4o audio demonstration, but treats production price and latency as open questions in the workshop’s timeframe. His focus remains scalable, inexpensive inference: an impressive conversation demo and an economically operable voice service must satisfy different constraints.

56:5157:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

56:51 · section reference included

Custom speech models need carefully prepared data

Murphy describes speech-recognition fine-tuning as requiring approximately 20–50 hours of audio. The preparation uses three labeling passes by different people: an initial transcription, correction by another person, and a final review. He reports about 12% word error rate after the first pass and about 99% label accuracy after the third. Those figures describe the preparation of training labels, not the accuracy of the resulting speech model.

Murphy estimates that model training itself could take under a day once collection and labeling are complete. Gathering suitable audio and producing reliable labels therefore remain important parts of delivery time. On the synthesis side, voice cloning is not generally offered in the workshop’s account because of misuse concerns, although selected customers can commission a brand voice through a business engagement.

The commercial-access discussion reinforces that the workshop sandbox is temporary. Murphy describes subsequent wait-list access requiring an API key, with speech recognition, the LLM, and speech synthesis intended to sit under a single usage fee. That unified billing boundary mirrors the integrated audio-in/audio-out interface.

1:00:381:00:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:00:38 · section reference included

Diarization needs enough speech to distinguish people

Multiple speakers introduce another inference problem. With a single audio channel, diarization assigns labels such as speaker A and speaker B. Separate channels allow the system to process those streams independently. For the single-channel case, Murphy explains that the system accumulates speaker embeddings from the available speech; a few short sentences may not yet distinguish one person reliably.

Murphy describes 512-dimensional speaker embeddings and recommends roughly 30 seconds of audio per speaker to establish a solid representation. Lowering the evidence requirement risks assigning multiple labels to the same person. Brief overlapping acknowledgments such as “yeah” or “mm-hmm” are especially difficult because they provide little isolated audio. He warns that the first 60 seconds of a call may not reliably establish who is who.

An attendee asks whether similarity or confidence scores could be exposed so applications could make their own decisions. Murphy’s answer shifts to speaker identification: recognizing that a voice belongs to a known person across calls. He says that capability is not exposed at the time and raises legal concerns about voice fingerprinting. The answer leaves the confidence-score question unresolved; assigning within-call labels and identifying a persistent person are distinct features.

1:02:501:02:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:02:50 · section reference included

The lower limit of useful latency

The workshop closes by checking whether anyone has added removal or other APIs. No completed extension is demonstrated. The closing exchange instead reports that the starter demo works and accepts back-to-back additions, while the noisy room causes it to pick up surrounding speech, including the presenter. The basic function-calling loop is useful, but its acoustic environment remains part of its behavior.

Murphy says the hosted setup has not been optimized for minimum latency. In a higher-compute sandbox, responses arrived so quickly after pauses that the agent felt interruptive. That brings the implementation back to its central design constraint: a voice agent must be ready to respond quickly and still know when to wait. Streaming, co-location, and compute make the response available; endpointing, interruption handling, and conversational judgment determine whether it should be spoken yet.

1:05:491:06:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:05:49 · section reference included

Resources

From the talk

  • Launch-era account of Aura's 12 English voices, character-based pricing and intended conversational applications.

Updates since the talk

Transcript download
Full transcript

Loading transcript…