AI Engineer World's Fair 2025
Realtime Conversational Video with Pipecat and Tavus — Chad Bailey and Brian Johnson, Daily & Tavus
Loading slides…
Read the talk
Building realtime conversational video with Pipecat and Tavus
A responsive video replica needs more than fast rendering: it needs streaming orchestration, sensible turn timing, synchronized media, and infrastructure that connects each user to a bot.
From a talk by Chad Bailey and Brian Johnson
Before you start: Familiarity with LLM message history, streaming output, and basic Python will help with the pipeline walkthrough.
What would make the robot concierge work?
A visitor approaches a robot concierge at a counter. The promise is obvious: ask a question and have a useful conversation. The familiar experience is less convincing. Chad Bailey opens with that gap—these interfaces ought to work, but too often they are frustrating to use. Building a good version is now possible; making the interaction work requires more than choosing a model.
There are three responsibilities to solve in sequence: models, which interpret input and generate responses; orchestration, which coordinates the conversation; and deployment, which makes a bot available when someone wants to talk. A compelling demo starts with the first. A usable service needs all three.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From speech pipelines to video replicas
The conventional voice pipeline separates transcription, reasoning, and speech synthesis. Voice-to-voice models offer another route, but neither approach removes the additional work involved in generating realtime video.
| Approach | Processing path |
|---|---|
| Cascading voice pipeline | Speech-to-text → LLM → text-to-speech |
| Voice-to-voice model | Spoken input → model → spoken output |
The choice depends on the application. Video adds another output modality that must participate in the same live interaction.
Tavus began with a rendering model. Brian Johnson describes the next requirement as speed: rendering had to work in a realtime context to become conversational. Once it did, the missing pieces became apparent—turn detection, response timing, interpreting signals, and orchestration. Tavus initially built its own system without knowing about Pipecat, then began partnering with the project.
Brian points viewers to the Tavus website instead of running a live demo. The product he describes is an end-to-end conversational video interface: create a replica of yourself, put it online, and converse with it. Brian reports a response time of around 600 milliseconds for the Tavus interface described in the talk. That is a reported figure, not a benchmark with specified measurement conditions—and he immediately adds that it can be too fast. Sometimes the system needs to delay its response.
Alongside the basic conversational stack, Tavus includes proprietary models Sparrow-0 and Raven-0. At this point in the recording, they are offered inside Tavus's stack, with access through systems such as Pipecat described as a future direction. The developer overview brings those pieces together as perception, thinking, and responding around the user's input and output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Understanding why the bot behaves that way
A first conversation can feel remarkable. Production raises less glamorous questions: why did the bot pause, which part of the system was slow, and what caused it to respond that way? Observability and control over conversation flow mean being able to investigate those questions and change the behavior, rather than merely observe the final reply. Capturing metrics becomes part of building the interaction.
Pipecat is the open-source, vendor-neutral orchestration framework Bailey introduces for this work. Its job is to receive a user's audio or video, coordinate processing, and deliver audio or video back with low latency. The conference website's Talk to AIE bot provides a concrete example: Bailey says it uses Pipecat with Gemini Live, a voice-to-voice model.
An integrated model still leaves application behavior to coordinate. Bailey invokes Gemini's documentation to distinguish experimentation from production orchestration. That recommendation should not be read as a Google requirement to use Pipecat: the current Gemini Live API overview supports multiple integration approaches. The engineering issue is the work surrounding the model, even when the model handles the voice exchange itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Frames, processors, and pipelines
Pipecat's work divides into three stages:
- Input: Receive microphone audio and, where needed, camera video. A replica can respond to what it sees as well as what it hears.
- Processing: Run multiple models in sequence, or delegate much of the interaction to an integrated service such as Gemini Live or Tavus.
- Output: Deliver synthesized speech, generated video, or application UI updates. For a video replica, the audio and video must remain synchronized.
That last requirement is separate from generating plausible speech or imagery: the user has to receive them as one coherent interaction.
The implementation rests on frames, processors, and pipelines. A frame is a typed container for data. Bailey's examples include small audio snippets, such as 10 or 20 milliseconds of sound, and camera video frames. Events also travel as frames: voice activity detection, or VAD, can produce a user-started-speaking frame. Media and information about the conversation therefore move through the same processing structure.
A processor receives frames and emits frames. An LLM processor, for example, receives conversation context and produces a stream of text frames. A pipeline composes these processors in the order needed for the bot's behavior. Pipecat runs the work asynchronously to minimize the delay each piece of information experiences as it moves through the system. Streaming output can keep progressing instead of waiting for an entire response to become one finished object.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From incoming audio to a completed turn
Rather than live coding, Bailey walks through a Tavus example from the repository, with QR links to the example and the pipeline documentation. The first processor is transport input. It turns media arriving over WebRTC, WebSockets, or Twilio WebSockets into frames that enter the pipeline.
The speech-to-text processor collects incoming audio frames. The example uses 20-millisecond chunks to explain the flow, but the transcription service decides how much audio it needs before emitting text. Deepgram and Whisper are examples of transcription choices. The important boundary is that incoming audio frames and outgoing transcription fragments do not have a one-to-one relationship.
A transcription fragment is not yet a conversational turn. The STT processor emits text according to its own processing behavior. A context aggregator combines those fragments with other signals moving through the pipeline: the user started speaking, microphone activity dropped, and the user may now have stopped. A drop in activity is a cue to consider ending the turn, rather than proof that the person has finished their thought.
Once the aggregator decides to release the turn, the rest of the sequence follows:
- Group the accumulated transcription fragments and emit a context aggregation frame.
- Supply the LLM with the conversation context, including message history and tools.
- Stream the LLM's output as text frames, accompanied by start and end frames.
- Let text-to-speech accumulate text and generate speech.
Aggregation is therefore a control point: it determines when a stream of partial observations becomes a request for the model to answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turning synthesized speech into synchronized video
The bot file shown uses an explicitly older Tavus integration. Pipecat generates speech first, then sends the audio to Tavus to generate video. Tavus returns the audio synchronized with that video, and those media frames leave through transport output. The transport is the delivery boundary in both directions: input brings the user's media into the bot; output sends the replica's media back.
In Python, the composition can be expressed as an ordered pipeline of configured processors:
python
from pipecat.pipeline.pipeline import Pipeline
def build_replica_pipeline(
transport, stt, context_aggregator, llm, tts, tavus
):
return Pipeline([
transport.input(),
stt,
context_aggregator.user(),
llm,
tts,
tavus,
transport.output(),
])
This captures the audio-to-video arrangement being explained: Tavus sits after speech synthesis because generated audio is its input. Service construction and transport setup depend on the integration version; the current Tavus integration guide is the setup reference, rather than the historical example's configuration. The useful architectural property is that additional processors can be inserted around the renderer without replacing the whole conversation system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Running analysis beside the conversation
A pipeline need not be a single straight path. Pipecat supports parallel pipelines, so another LLM can perform sentiment analysis while the main conversation continues. That makes analysis a concurrent activity instead of an obligatory serial step before every reply.
Bailey describes a user applying Gemini Live Multimodal to an outbound-call problem: did a person answer, or is the bot hearing a voicemail greeting? One model makes that determination and sends a signal back to the pipeline. The signal selects a human branch or a voicemail branch. The classifier decides what was heard; orchestration decides which behavior follows.
Even when Tavus handles most of what makes the interaction feel convincing, peripheral tasks still need coordination. The runnable example makes that surrounding machinery visible. Bailey describes signing up for Tavus, supplying a key, and running the example unchanged at the time of the talk. Its UI supports conversation with the replica while a debug panel exposes activity inside Pipecat. The result is both an interaction surface and a way to inspect how the interaction is being produced.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Moving model capabilities into the orchestration layer
Tavus's own experience illustrates the cost of rebuilding orchestration. Brian says the team spent the preceding year learning lessons Pipecat had already addressed in aggregation, communication, and coordinating the conversation. He estimates that reusing those capabilities could save months of work. This is an engineering experience claim, not a measured development-time comparison.
At the time of the talk, Tavus was not using Pipecat internally. Enterprise customers were already using it, however, and wanted Tavus's models available inside their existing pipelines. Phoenix, the rendering model, had already been integrated. Turn-taking, response timing, and perception were planned next; adopting Pipecat inside Tavus was also a future intention. Brian grounds that intention in a concrete frustration: he had just spent days debugging a problem Pipecat already solved. These plans are distinct from the rendering integration already available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Finishing a turn and choosing when to answer
Tavus's multilingual turn-detection model addresses a deceptively difficult question: has the person finished speaking? A fast pipeline that answers too early talks over the user. A slow one leaves the user wondering whether it is broken. Better turn detection aims to avoid both outcomes by making the decision about completion more accurate.
Response timing is a separate decision, made after the person has finished. Brian's example is someone describing a grandmother moving into a home and feeling sad. An immediate answer can feel inappropriate; a considered pause fits the exchange better. Casual chitchat calls for a quicker response. This explains why the earlier fast-response capability sometimes needs deliberate slowing.
| Decision | Question |
|---|---|
| Turn detection | Has the user finished speaking? |
| Response timing | How soon should the bot answer afterward? |
Minimum processing latency and appropriate conversational timing are different goals. The response-timing model described here was still being developed, with Pipecat integration planned.
Multimodal perception adds information beyond the words themselves: emotions, surroundings, and what the person is wearing. The planned next step is to feed those observations into turn-taking and response timing, making the conversational flow sensitive to more of the situation. Perception becomes useful not just because the replica can describe what it sees, but because it can influence how the replica participates.
The closing integration example returns to control over the pipeline boundary. Some functions can remain inside Tavus; others can be exposed as components the application coordinates. As models change, a differently composed Pipecat pipeline lets the application retain control where it needs it while delegating other work to Tavus.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Starting a bot and connecting its media
Deployment completes the system. Pipecat leaves substantial freedom in how bots are hosted, but the application still needs a concrete startup path:
- A client calls a REST API to request a conversation.
- Infrastructure starts a bot instance promptly and connects it to the requesting user.
- A media transport carries audio and video in both directions.
Bailey favors WebRTC for that transport. Starting the process alone is insufficient: the infrastructure must also configure the connection that lets the user and bot exchange media.
Pipecat Cloud is presented as the paid, managed option for teams that do not want to operate infrastructure such as Kubernetes themselves. Operating the infrastructure independently remains an option. Bailey closes by pointing to his colleague Mark's follow-up talk on deploying bots at scale, covering both the managed service and doing it yourself. Either route has the same responsibility: turn a request to talk into a running bot with a working media connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open-source framework and examples for building real-time voice and multimodal agents.
Managed infrastructure for deploying agents, scaling sessions, and connecting users through Daily WebRTC.
Further reading
Explains frames, processors, context aggregation, processing order, and parallel pipeline branches.
The March 2025 introduction of Phoenix-3, Raven-0, and Sparrow-0 explains their rendering, perception, and conversational timing roles.
Updates since the talk
Current setup instructions for adding Tavus rendering as a room participant or a video service within a Pipecat pipeline.
Current integration approaches for Gemini's real-time audio and vision API, including Pipecat.
Full transcript
Loading transcript…