AI Engineer World's Fair 2025

Your realtime AI is ngmi — Sean DuBois (OpenAI), Kwindla Hultman Kramer (Daily)

Loading slides…

Read the talk

Your realtime AI is ngmi

From conference Wi-Fi to a talking Raspberry Pi, natural voice interaction depends on a tight latency budget, the right media transport, and room to experiment.

From a talk by Sean DuBois and Kwindla Hultman Kramer

Before you start: Basic familiarity with client-server applications and streaming APIs is useful; no prior WebRTC experience is required.

An anxious stuffed animal has a point

Squabbert is nervous about the live demo: an unscripted conversation with a nondeterministic LLM, running on conference Wi-Fi. Then there is the stage audio, the room full of echoes, and text-to-speech that sometimes turns its own name into Squibbly. A good prompt cannot remove those problems. Before a voice agent can be charming, it has to hear, respond, and speak reliably in the environment where people actually use it.

The introductions bring together both sides of that engineering problem. Sean DuBois works on WebRTC at OpenAI, including the Realtime API and 1-800-CHATGPT; previously, he worked on Pion, the Go implementation of WebRTC. Kwindla Hultman Kramer—Kwin—works on real-time audio and video infrastructure at Daily and on Pipecat, an open-source voice-agent framework. Their starting point is building voice experiences that feel natural, fast, and human-like.

0:220:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:22 · section reference included

Measure the gap the listener experiences

Much of the engineering behind a multi-turn agent carries over to voice. The difference is how little time the system has to respond. Sean uses around 500 milliseconds as a natural conversational response target. He associates response delays much above one second with low completion rates, low NPS, and hang-ups. These are his design heuristics, rather than a measured universal threshold for AI usability. The relevant measurement is voice-to-voice latency: the interval between the human finishing an utterance and hearing the first returned audio. A fast text response is insufficient if the listener is still waiting for sound.

Two colored audio waveforms separated by blue dots and a bracket labeled Voice-to-Voice latency; the heading states that 500 milliseconds is typical during human conversation.Open full source frame
A 500-millisecond gap illustrates voice-to-voice latency in human conversation.

The example application runs in a browser on macOS and communicates over the internet with a cloud-hosted Pipecat agent. Sean reports voice-to-voice latency just under one second for that application. He calls that good, but not great: reducing it further introduces quality or cost trade-offs. It is also easy to do worse—a slower LLM adds delay, and the device's audio path can add more. Bluetooth earns a particularly emphatic warning from both speakers. The latency budget belongs to the whole application, not just its model.

1:471:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:47 · section reference included

Choose the transport for its job

A familiar implementation path starts with a reasonable observation: audio needs a long-lived connection, and WebSockets provide long-lived connections. But that resemblance hides a mismatch. A connection suited to delivering small data messages does not automatically provide the timing behavior interactive media needs. Sean and Kwin identify this transport choice as one of the biggest avoidable mistakes in voice applications.

TransportBest fit in the talkMain trade-off
WebSocketsPrototypes, server-to-server connections, small structured payloadsEasy to implement across platforms
WebRTCLive audio and video from web or native clientsMedia quality and low latency, with more implementation complexity

An application can use both. The recommendation is to separate the jobs: use WebSockets where their simple data connection fits, and WebRTC for interactive audio and video traveling between a user's device and the cloud.

Red and blue panels recommend WebSockets for server-to-server connections and WebRTC for mobile apps and web apps, with examples beneath each.Open full source frame
Use WebSockets for server-to-server connections and WebRTC for mobile and web apps.
3:253:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:25 · section reference included

Reliable delivery can make audio arrive too late

A WebSocket uses TCP, which provides reliable, ordered delivery. Once the application puts data into the operating system's send queue, the networking stack keeps trying to deliver it until the other side acknowledges it or the connection times out. That is useful for a web request: missing bytes should not silently disappear. For conversational audio, however, waiting for missing data can make the rest of the stream late.

With a voice-to-voice budget below a second, old audio quickly loses its value. Recovering a missing packet is not helpful if the recovery makes the conversation fall behind. WebRTC combines timely packet delivery with buffer management and techniques for hiding missing audio. Its media machinery can disregard packets that miss their useful playback window, preserving the flow of the conversation instead of insisting that every piece of audio arrive.

The important distinction is timely media versus ordered completeness. Application code above a WebSocket cannot remove TCP's underlying obligation to recover and order the bytes already in its stream. Packet loss and substantial network delay can therefore hold up subsequent data. Kwin reports audio glitches, high latency, or unexpected socket disconnections in 10–15% of real-world network connections affected by this transport problem. He does not specify the measurement population or methodology, so that figure is an attributed operational observation.

4:575:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:57 · section reference included

Let the media stack handle the media

Packet timing is only the beginning. Sending audio over a WebSocket leaves the application responsible for resampling, packetization, and bandwidth estimation. Since network capacity changes during a session, a fixed sending bitrate is not enough. WebRTC supplies that media machinery together with standard statistics and observability APIs. The capability slide also groups encoding, playout timing, buffer flushing, and media quality metrics into the work handled by the stack.

Five illustrated capability groups cover audio and video encoding, audio resampling, bandwidth estimation, playout timing and buffer flushing, and media quality metrics.Open full source frame
WebRTC includes media processing, bandwidth adaptation, interruption handling, and observability.

The next slide compares code for one bidirectional audio stream: WebSockets on the left, WebRTC on the right. Sean's point is that application developers should spend less time managing sample rates. At the browser boundary, the essential WebRTC pattern is to capture a media stream, attach its tracks to a peer connection, and route incoming media to an audio element:

javascript

async function attachAudio(pc, audioElement) {
  const microphone = await navigator.mediaDevices.getUserMedia({
    audio: true
  });
  const incoming = new MediaStream();
  audioElement.srcObject = incoming;

  pc.addEventListener("track", ({ track }) => {
    incoming.addTrack(track);
  });

  for (const track of microphone.getAudioTracks()) {
    pc.addTrack(track, microphone);
  }

  return () => {
    microphone.getTracks().forEach(track => track.stop());
    pc.close();
    audioElement.srcObject = null;
  };
}

This media-wiring function takes an RTCPeerConnection; the surrounding application still handles connection negotiation and playback. Audio goes through media tracks rather than through application-managed audio chunks.

The onstage comparison uses OpenAI's Realtime API, which offers both transport options. The linked Realtime API WebRTC guide documents the current integration, rather than reproducing that historical code slide. The architectural lesson is the same: choosing WebRTC brings jitter buffers, packet management, and bandwidth shaping into the media layer instead of making them separate application projects.

6:426:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:42 · section reference included

An established transport for an unsettled interface

Once the transport is in place, real-time audio can live in a website, an iOS or Android application, or an embedded device. Kwin describes the goal as good audio across devices and platforms on almost any real-world network connection. Sean points to Facebook Messenger, WhatsApp, Zoom, and Discord as familiar applications using WebRTC, then extends the range to internet-mediated surgery and vehicle teleoperation. Conversational intelligence can build on infrastructure already used for demanding real-time interactions.

The interface conventions are less mature than the transport. Kwin describes people spending hours talking to computers: driving development environments, brainstorming, and using them as assistants, coaches, therapists, or researchers. He expects voice to become a core building block of generative-AI interfaces, while acknowledging that their forms still need iteration. His analogy is late 2007: the first iPhones exist, but pull-to-refresh has not yet been invented. A working capability can precede the interaction patterns that make it feel obvious.

Sean calls voice the next “bicycle for the mind.” It adds a way to use computing when eyes and hands are occupied, and it allows a small nearby device to provide access to computing elsewhere. That possibility leads directly back to the stuffed animal.

7:457:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:45 · section reference included

Squabbert's local stack and uneven poetry

Squabbert takes a moment to become ready. Kwin asks about its stack, Sean pauses the setup, and Kwin explains that Chad built the animal and Chad's daughter Ella named it. After the question is repeated, Squabbert describes a local web interface on a Raspberry Pi connected directly to a Python process on a laptop. That process uses MLX Whisper, Gemma 3, and a custom logit sampler written by Kwin. Squabbert characterizes the sampler as mildly buggy.

Kwin's reason for writing the sampler is to give the agent a specific generation capability: counting syllables, something he contrasts with the abilities of large cloud LLMs. He likens the challenge to counting the Rs in strawberry, then asks for a four-line poem about computer programming using only two-syllable words. The constraint is precise: the subject, the form, and the word choice all matter.

The first response includes:

Logic, coding, people, knowing. Systems, working, future, growing.

Those words follow the two-syllable restriction, though the response does not clearly deliver the requested four-line form. Kwin asks for another attempt. Squabbert then produces a more recognizably four-line poem, but uses words such as bright, feels, day, and night, breaking the syllable constraint. Kwin explicitly acknowledges that the second response forgot the restriction before saying goodbye. The live exchange demonstrates both the intended control over generation and its uneven success; it does not reveal the sampler's implementation.

9:5910:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:59 · section reference included

Keep the device, change where the intelligence runs

The demonstrated connection is peer-to-peer WebRTC between the Raspberry Pi and Kwin's laptop on the same local network. Sean calls it serverless: Squabbert talks directly to the laptop. Here, that describes the connection topology; the laptop still runs the Python process that supplies the intelligence.

The same transport supports different application shapes:

  • Local: Keep the direct device-to-laptop connection used in the demonstration.
  • Cloud: Connect a device such as Squabbert to a remote server that performs the AI processing.
  • Multiparty: Use a setup such as Pipecat to bring LLMs into meetings and other shared conversations.

The location of the processing and the number of participants can change without abandoning the real-time media foundation.

12:2312:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:23 · section reference included

A first prototype for bilingual families

The closing builder video shifts from infrastructure to a concrete need. Sean introduces Yaxin, who is in the audience, as someone who had never written code before building her project. Making voice tools accessible matters because people with useful ideas should be able to try them themselves.

Yaxin is a mother of two bilingual children. She wants them to connect with her cultural roots, but describes bilingual education as expensive, time-consuming, and often a chore for both children and parents. Her goal is to make language learning feel more natural and fun. The prototype clip starts with a friendly greeting and moves into an invitation to say hello in Mandarin. It turns language practice into a conversational exchange.

Yaxin describes it as an early first version, brought to life with guidance from someone in the community. She already has an eager group of prospective testers, mostly parents like her, and invites others to connect. The result shown is a working starting point for testing an idea with families, not a finished language-learning product.

13:0913:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:09 · section reference included

Make the next experiment easier to build

The closing resource slide includes a QR code for Yaxin's project, and Kwin invites people interested in multilingual applications to find her. He also identifies Sean as the community member who helped her build it. That practical assistance is part of the speakers' invitation: experienced programmers and people just getting started should both be able to explore voice AI.

Sean names Pipecat and libpeer as tools that can make these projects easier to create. The speakers offer to continue the conversation in the hallway or through Discord, Twitter, and LinkedIn, then point to the scannable resources and Kwin's book, which Sean believes attendees received in their bags. The desired next step is another builder taking a specific idea—like playful language practice—and getting it far enough to put in front of people.

Resources slide with six QR codes labeled webrtcforthecurious.com, github.com/espressif/esp-webrtc-solution, Yaxin Duan building something new, voiceaiandvoiceagents.com, discord.gg/pipecat, and pipecat.ai.Open full source frame
Resources for WebRTC, Pipecat, and further community projects.
15:1315:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:13 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…