AI Engineer World's Fair 2025

Milliseconds to Magic: Real‑Time Workflows using the Gemini Live API and Pipecat

Kwindla Hultman Kramer· CEO, DailyShrestha Basu Mallick· Product lead, Gemini Developer API, Google DeepMind21:43

Loading slides…

Read the talk

Milliseconds to Magic: Real-Time Workflows using the Gemini Live API and Pipecat

A voice-driven personal assistant turns grocery lists, book requests, and animated cats into a practical exploration of turn detection, tool behavior, and conversational state.

From a talk by Kwindla Hultman Kramer and Shrestha Basu Mallick

Before you start: Familiarity with LLM tool calling and the distinction between application code, frameworks, and APIs will help with the implementation discussion.

What makes talking to a computer feel natural?

People think aloud. We learn to speak before we learn to read, usually talk faster than we type, and convey emotion through our voices. A conversational interface can work with those habits: the computer becomes something to think with, rather than another form to fill out.

Making that interface work requires cooperation across layers. Shrestha Basu Mallick approaches the problem through models and APIs; Kwindla Kramer through applications and agent frameworks. For early adopters, voice agents already serve as sounding boards, coaches, and interfaces to information on devices and in the cloud. The use cases extend further: translation between patients and doctors, directed learning for a fourth grader, speech therapy, and help navigating enterprise software.

Slide titled “Voice AI today” listing applications, with the two presenters inset at bottom left.Open full source frame
Voice AI today: translation, customer support, healthcare, learning, speech therapy, brainstorming, and software copilots.

Kramer reports that Daily customers sometimes encounter callers who do not realize they are speaking to an agent, even after an initial AI disclosure. That apparent ease conceals substantial engineering. The magic is the interface: underneath it are problems ranging from real-time responsiveness—the foundation of a workable voice conversation—to generating a different visual interface for each conversational turn.

0:000:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Locate the responsibility, then assess its maturity

The voice stack has four broad layers. Read from the bottom upward, each supplies capabilities that the next layer can compose:

LayerResponsibility
Model weightsLanguage and multimodal capabilities
Real-time APIsLive access to inference and session behavior
OrchestrationCoordination of real-time application components
Application codeThe particular workflow and interface

The Gemini Live API occupies the API layer. Pipecat occupies the orchestration layer, helping manage the complexity of real-time multimodal applications.

For any difficult behavior, ask two questions: Where does the code responsible for it live, and how solved is the problem? The slide combines stack position with a right-to-left maturity axis. Kramer puts none of the listed challenges beyond about 50% solved, explicitly describing that as an arbitrary personal estimate. The map is a way to discuss unfinished work, not a measured performance scale.

“Layers, capabilities, maturity” slide with Application, Orchestration, APIs, and Model weights beside capabilities arranged along a 10% to 50% axis.Open full source frame
Voice capabilities mapped across technology layers and a “How solved is this?” axis.

As a problem becomes common, its solution tends to move down the stack. An application developer first writes a specialized implementation. Once enough applications need it, a framework can generalize it; later, an API may absorb it. Meanwhile, models become more capable independently of that software migration.

Turn detection makes this progression concrete. Following the discussion of semantic voice activity detection in the preceding talk, Kramer traces his own work: a little over two years earlier, he implemented turn detection inside an application because suitable tools did not yet exist. A few months later, generalized implementations moved into Pipecat. By this talk, detection was available inside the same live API surface that handled inference. Having models handle it themselves remained the expected next step. New use cases, meanwhile, keep creating new responsibilities at the application layer.

Built-in detection is also a choice, not a requirement. Basu Mallick explains that developers can disable the server-side detector and use alternatives from ecosystems such as Daily or LiveKit. In that arrangement, the client supplies speech-boundary signals; Daily and LiveKit are not themselves model names. This option already existed in the historical Live API, with activityStart and activityEnd events, rather than being a later capability to project backward onto the demo.

2:573:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:57 · section reference included

Start with an asparagus-pizza grocery list

Kramer’s personal assistant lives somewhere between a to-do list and a brainstorming partner. After roughly a year of using voice to manage priorities, he finds that loose, conversational workflow particularly useful. The version onstage is deliberately experimental: no unit tests or evals, the latest models, and a bleeding-edge Pipecat branch. Basu Mallick will use it live to expose where it breaks.

Both presenters have already seen substantial variation between runs. The deeper adjustment for Kramer is that the model drives the application cycle in ways his conventional programming expectations do not anticipate. An unexpected action may be a failure, but it may also be a useful behavior he never explicitly implemented.

The first task is straightforward: create a grocery list for asparagus pizza. The agent initially announces that it has added pizza ingredients without naming them. Basu Mallick asks it to spell out the ingredients, and it reports pizza crust, mozzarella cheese, tomato sauce, and asparagus. Asked whether it would add anything else, it adds garlic and olive oil. The workflow progresses from a broad intention to an explicit list, then to an open-ended suggestion.

6:536:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:53 · section reference included

When incidental speech becomes application state

Switching to a reading list exposes the conversational boundary problem. The agent begins responding while Basu Mallick is still forming her request, prompting her to remark on turn detection. She then asks to add Dream Count and look up its author. The agent instead announces that it has added Quick.

She repeats the request, then asks whether the agent is still there. Kramer says “Segmentation fault,” and the agent treats the aside as a book title, announcing that it has added Segmentation Fault. Another attempt to request Dream Count stalls; Kramer says “Segmentation,” and that becomes another claimed entry. These are spoken jokes absorbed into the workflow, not evidence that the application process crashed. Basu Mallick jokes that the agent prefers his voice, then tries once more to add the intended book.

The next request is for the second and third books in the Three Body Problem trilogy. Initially, the agent continues discussing segmentation faults and talks over a repeated request. Basu Mallick says it worked much better in ten trials the previous night. That is her account of rehearsal behavior, rather than an evaluation with recorded success criteria.

After another clear request, the agent repeats the intended task, and Basu Mallick reports that it did find the book names. The conversation turns to Google search, including a compound search, but she distinguishes two possible sources of knowledge: the older trilogy might be in the model’s training data, whereas the newer Dream Count had triggered Google search during the previous night’s successful run. The onstage result therefore does not establish that every book lookup used retrieval.

9:5610:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:56 · section reference included

Adding a task is not the same as displaying it

Basu Mallick creates a work list and asks for an item: create the H2 roadmap by end of day Friday. The agent says it has added the task, but she cannot see it. Only after an explicit request to show the list does it announce a display update. A spoken acknowledgment and a visible interface update are separate parts of the workflow.

The next item is to finish writing a podcast script by end of day Thursday, including the exact date. The agent initially acknowledges the task without the date, so the repair takes two more requests:

  1. Ask for the exact date corresponding to Thursday. The agent states that today is Wednesday, June 4, 2025, making Thursday June 5, 2025.
  2. Ask it to add that date to the list. The agent then reads back the task with June 5 included.

The interaction separates interpreting a relative date from putting the resolved value into the task text.

13:0213:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:02 · section reference included

Grouping, duplication, and ownership

The requests now become transformations of existing information. Basu Mallick asks the agent to combine groceries, reading, and work into two lists: personal tasks and work tasks. Its response says the lists have been combined and displayed, but the result needs repair. Garlic and olive oil are separate from the pizza ingredients, so she asks for the grocery list to be fleshed out. She then points out two work lists and asks for consolidation. Although the agent claims it has consolidated them, the accompanying frame still shows two Work List groups alongside Tasks & Errands and Reading List.

Browser demo with a black list panel containing two Work List headings, Tasks & Errands, and Reading List, above a Debug Info panel.Open full source frame
The list demo still displays two Work List groups alongside Tasks & Errands and Reading List.

Next comes an ownership rule with an exception:

ItemsRequested owner
GroceriesKwin
Reading, except Segmentation FaultShrestha
Segmentation FaultKwin
WorkShrestha

The agent first restates the assignments. Basu Mallick has to ask again for them to appear on the screen. After the announced update, she observes that it has spelled Shrestha correctly but Kwin incorrectly. The workflow now requires the model to preserve items, regroup them, apply an exception, and present the result with the right names.

14:1414:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:14 · section reference included

Ask for an interface, not just another list

A request for a nicer background becomes something more ambitious. Basu Mallick asks for an app with Hello World jumping up and down in Google colors, plus two animated neon-green ASCII cats. The agent announces that generation has started and that the app may take a moment to appear. After a wait, the audience applauds, and she confirms the Google colors and the cats, connecting them to Pipecat.

The voice request initiates the workflow; it does not imply that the live audio model directly produces the rendered interface. The companion demo repository describes an asynchronous tool that calls the standard, non-Live Gemini API to generate embedded UI. It identifies the demonstrated model as Gemini 2.5 Flash Preview. That is historical preview context, not a claim about current model availability or the exact deployed configuration.

15:5716:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:57 · section reference included

What the application leaves for the model to infer

The display problems become easier to understand when Kramer explains the tool contract. His code gives the model a function that can add text to the screen, with an optional clear argument, but almost no instructions about how to use it. The model has to infer from the conversation when it should append content and when it should replace what is already there.

A small Python example makes that distinction explicit. Here, two existing work groups remain visible when a consolidated group is merely appended; clearing first produces a replacement view:

python

screen = [
    "Work List: Create H2 roadmap by end of day Friday",
    "Work List: Finish writing podcast script by June 5, 2025",
]


def add_text(text: str, clear: bool = False) -> None:
    if clear:
        screen.clear()
    screen.append(text)


consolidated = (
    "Work List\n"
    "- Create H2 roadmap by end of day Friday\n"
    "- Finish writing podcast script by June 5, 2025"
)

add_text(consolidated)  # Appends; previous groups remain.
add_text(consolidated, clear=True)  # Replaces the displayed content.

This illustrates the display operation Kramer describes, rather than reproducing his implementation. Correctly composing the consolidated text does not by itself choose the right operation on the existing screen.

Kramer describes a jagged frontier between what the model can infer and what application code must scaffold. Basu Mallick adds that Kwin’s name can remain misspelled even when she supplies the letters explicitly. Turn detection still needs work, and grocery completeness and list consolidation vary from one run to another. The successful animated app and the awkward list repairs coexist in the same system.

The application also delegates much of its contextual understanding to conversation history. Each session loads many previous conversations as alternating user and assistant messages. Some versions summarize that history; others do not. The model must work out what a list means and which conversational context applies to it. This explains the assistant’s flexibility: the workflow is voice-driven and multimodal, with much of its interpretation carried by the LLM rather than explicit application rules.

17:1617:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:16 · section reference included

Remembering what the reminder was for

The purpose of this loose personal assistant comes into focus through two family stories. Basu Mallick’s grandmother in India tied knots in her sari to remind herself to do things. Kramer’s grandmother in North Carolina tied strings around her fingers. Across Calcutta and North Carolina, they arrived at a similar reminder mechanism.

Both mechanisms preserve the fact that something needs remembering, but not what that something is. The wearer still has to supply the missing context from memory. A conversational assistant offers the possibility of preserving that context too: the task, the surrounding discussion, and the meaning that makes the reminder useful. Basu Mallick connects her work at Google to expanding those creative possibilities across continents.

Her outlook is that most interaction with language models will eventually happen through voice. That is a prediction about the interface, while the underlying models remain broader: Gemini is trained multimodally to ingest text, voice, images, and video. She closes by inviting builders to bring their Gemini questions through X, LinkedIn, or email—the next applications will need both the conversational interface and the engineering underneath it.

19:3319:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:33 · section reference included

Resources

From the talk

  • A contemporary introduction to session management, configurable voice activity detection, interruptions, and Pipecat and LiveKit integrations.

  • Gemini API I/O updatesArticle

    May 2025 release context for native audio preview models and asynchronous function calling, coauthored by Shrestha Basu Mallick.

Updates since the talk

Transcript download
Full transcript

Loading transcript…