AI Engineer World's Fair 2026

Voice agents with Realtime Video — Sidney Primas, LemonSlice

Loading slides…

Read the talk

Building a real-time visual layer for voice agents

LemonSlice’s avatar demonstrations expose the engineering behind live video agents: expressive audio conditioning, causal generation, long-session stability, and a harness that keeps everything synchronized.

From a talk by Sidney Primas

Before you start: Familiarity with voice-agent pipelines, attention, and diffusion denoising will help with the architecture discussion.

What makes an avatar feel human?

What would make an avatar indistinguishable from a human on a video call? A photorealistic face is only the beginning. Sidney Primas, introducing himself as LemonSlice’s CTO and founder, describes a longer list: appropriate emotions, convincing object interactions, microexpressions, and an internal state that makes the avatar’s behavior feel coherent. The target is a believable interaction, not just a believable image. Those requirements span both the product LemonSlice has built and the capabilities it is still trying to develop.

A rendered face fills the slide behind the words “Break the avatar Turing Test,” with the presenter inset at left.Open full source frame
“Break the avatar Turing Test” mission slide.
0:000:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

A conversation with Theodore Roosevelt

The first example is the Theodore Roosevelt presidential-library installation. Primas describes a Microsoft partnership launched the previous day: inside a replica Oval Office, visitors can converse with a Teddy Roosevelt avatar in real time. He explicitly says this does not yet solve the avatar Turing test. To show the experience, he plays what he identifies as authentic footage of President Trump interacting with the installation.

The question is whether the Panama Canal was Roosevelt’s greatest achievement. The avatar initially calls the canal one of his proudest battles, then broadens the answer from a large engineering project to lives improved: parks preserved, food and drugs made safer, and the Square Deal. It closes by returning to the image of steam shovels digging a passage that would change the world’s map. The demonstration combines a sustained spoken answer with an embodied historical character.

Slide with vertical footage of an office installation, a Newsweek headline about Trump talking with AI Theodore Roosevelt, and a large image of the Roosevelt avatar behind a desk.Open full source frame
The Roosevelt avatar installation shown alongside a news headline and demonstration footage.

Primas reports that Trump stayed for ten minutes instead of the planned brief interaction, then left and returned to speak with the avatar again. The avatar is full-body, with hands, actions, and movement generated in real time. That combination matters to LemonSlice’s product thesis: the visual presence is meant to help sustain an interaction, rather than merely decorate the answer.

Primas connects that experience to a broader attention problem. Sustained reading can be difficult, and voice alone does not always hold someone’s attention. His bet is that people understand and engage more readily when a visual component accompanies the exchange, and that most human–AI interactions will eventually include such a layer. This is the motivation for building video into the interface.

1:341:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:34 · section reference included

One image, a moving person, and a moving world

LemonSlice’s modeling bet is to focus world models on humans. Primas acknowledges that this makes the initial model harder to train and deploy. The expected return is a shared foundation for full-body movement, object interactions, movement through a scene, microexpressions, and emotion. These capabilities do not come free, but he expects them to be easier to obtain together than through approaches that must solve each behavior separately.

The next demonstration begins with an avatar offering to listen, help, chat, or play a game. Primas attributes an apparent lip-sync mismatch to the venue’s AV setup, then points to the avatar’s waving, hands, and movement. The starting input is a single image. It can depict a photorealistic person, a Pixar-style character, or a cartoon; the same basic product turns that reference into an animated conversational presence.

Within the same video call and inference setup, the demonstration changes the avatar’s clothes and surroundings. As the avatar responds to something it interprets as a request about a story, Primas draws attention to moving earrings and water. He presents those details as evidence that the video model has learned physical regularities: realism depends on the surrounding scene responding plausibly, as well as the face moving correctly.

The product boundary is an API. Customers bring their own LLM and usually their own voices; LemonSlice supplies the visual layer. Language learning and AI sales calls are examples, but the integration idea is broader: an existing voice agent used on a screen can become a video agent without LemonSlice taking over its reasoning or voice stack.

5:025:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:02 · section reference included

Expression starts with the audio representation

The disclosed architecture begins with an in-house video diffusion transformer, or video DiT. Primas does not expose every training detail, but emphasizes audio as a critical input for facial expression and emotion. Expressive training data matters, and so does an encoder whose embeddings preserve the distinctions the video model needs.

Primas’s criticism of audiobook-trained audio encoders is that relatively monotone speech gives them too little emotional variation for this task. He does not identify particular encoders or datasets. The engineering point is that an audio representation useful for recognizing speech is not automatically sufficient for generating an expressive face: the conditioning must retain how something is said. In his terminology, the resulting world model is fundamentally a video model with learned understanding of physical behavior.

8:489:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:48 · section reference included

Making the model interactive, fast, and persistent

A video model that produces a finished clip is not yet an interactive model. Conventional bidirectional generation lets different video latents attend to both earlier and later parts of the clip. In a live conversation, later inputs have not arrived. LemonSlice trains with an attention mask that restricts access to past context, so inference does not depend on future audio or other future inputs.

Causality solves the input-availability problem; it does not by itself make generation fast enough. Primas illustrates the speed objective by reducing denoising from thirty steps to one, mapping noise to video in a single step. This is his explanation of the desired acceleration, rather than a disclosed production configuration.

RequirementChangeWhat it addresses
InteractivityRestrict attention to past contextFuture inputs do not exist yet
Real-time generationReduce repeated denoisingEach video block must arrive promptly

Both changes are needed: a fast model that requires future speech cannot participate in a live exchange, and a causal model that generates too slowly cannot keep up.

Causal generation introduces another problem: error accumulation. Primas plays a failure clip whose subject introduces an ophthalmology professorship at Stanford School of Medicine. As generation continues, the result deteriorates. Each generated block contains some error; that block then becomes context for the next block. The model conditions on its own imperfect output and can add further errors, allowing drift to compound over time.

Primas reports that the Roosevelt avatar generates continuously for eight hours without a reset; a sixteen-hour installation is planned. Those durations make drift a practical constraint rather than an occasional defect in a short clip. He says the team developed a different solution, novel to its knowledge, but does not disclose the method. His stated outcome is no noticeable error accumulation in long videos; the talk provides neither an endurance protocol nor a quantitative visual-quality criterion.

Hardware optimization addresses the economics of sustaining that stream. A shareable AI clip might last five seconds, while an avatar session requires minutes or hours of generation. Primas attributes feasibility to making the models smaller and more efficient, and compares their cost level with voice models despite video’s much larger data volume. He connects that affordability to consumer and entertainment customers; the later Q&A clarifies the comparison in terms of customer charges.

Finally, the model harness must turn those capabilities into uninterrupted playback. Separate threads manage streaming data while CPU and GPU work proceed together. Interruptions, buffering, queues, and queue cleanup all have to be coordinated without making the video stutter. A model can generate good frames and still deliver a poor conversation if that orchestration fails. Primas describes the harness as a substantial part of the production engineering effort and expects it to account for an increasing share of the value in real-time systems.

“Phase A: World Models for Avatars” slide with a triangular attention grid, generation timeline, rolling KV cache eviction diagram, model harness diagram, and black embedded-video panel.Open full source frame
World models for avatars: interactive attention, real-time generation, and supporting infrastructure.
10:1110:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:11 · section reference included

Choosing the right expression at the right moment

The next development target is an emotion engine that makes avatars aware of what is happening in the conversation. The requirement is more specific than generating a happy or sad face: an emotion or action must occur at the right moment and last for the right duration. Primas acknowledges that current interactions still feel awkward, partly because the avatars do not respond emotionally while listening or consistently emote appropriately while speaking.

A real-time demonstration makes the expressive difference obvious through a goal celebration. The avatar repeatedly celebrates winning the game, with excitement carrying the interaction. Primas then shows a next-generation model that is explicitly not yet real-time. Its dialogue moves through taking a breath, deciding whether to stay somewhere, and a question about an apartment, while he points toward more natural interactions with the character’s own body and environment.

Being able to generate an action is different from being able to schedule it reliably in a conversation. Primas says the model already has many of the desired capabilities, but they are not controllable or deterministic enough for live use. The proposed emotion engine takes the avatar’s audio and the text it is going to say, then predicts the action it should perform at the current moment. At the talk, the target for the next model was one to two months away.

16:0116:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:01 · section reference included

An EQ model in front of an IQ model

The long-term proposal is a single EQ model for human interaction. It would consume the user’s video and audio directly, as in a FaceTime call, and emit the avatar’s video and audio. Audio understanding, response generation, and the avatar’s internal emotional state would be trained together inside that model. Primas mentions early proof-of-concept papers without naming them.

That interaction model would not need to contain all of the system’s reasoning ability. A separate IQ model would supply tool use, deep thinking, and other intelligent work to the EQ layer.

Proposed componentResponsibility
EQ modelPerceive the person, maintain emotional state, produce audiovisual responses
IQ modelReason, call tools, supply task intelligence

The separation preserves a specialized front end for human interaction while allowing another model to perform the underlying work. Primas predicts that end-to-end EQ models will reach the market within two or three years of the talk.

19:0819:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:08 · section reference included

Controlling emotional state and measuring human likeness

The first audience question asks how finely an EQ model’s output could be controlled, given the enormous variety of human emotions. Primas expects the future model to maintain an internal latent state representing emotions or goals. That state could be implicit and need not be human-interpretable. Today’s controls are more explicit: actions and emotions are described with words.

The follow-up clarifies that this state should represent the avatar’s appropriate response to what it perceives, and persist across the conversation. Primas agrees: the model should track that state over a history of time. The proposed mechanism is therefore more than selecting an independent expression for each utterance; it carries a changing emotional condition through the interaction.

A joke about whether Trump recognized the Roosevelt avatar returns the discussion to the opening demonstration. Primas recounts online debate and a Gavin Newsom joke, but the exchange does not establish that the visitor mistook the avatar for a real person. A subsequent question asks how platforms could distinguish humans from avatars. Primas answers in terms of evaluation, rather than a platform identification mechanism.

LemonSlice had not yet run its proposed avatar Turing test. Primas says the team was developing a test involving real people, intended to run and publish it during the talk’s calendar year, and did not expect to pass it that year. The purpose was to track progress over time, with possible participation by others. The installation is a product demonstration; a repeatable assessment of human likeness remained work to be done.

21:0821:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:08 · section reference included

Product boundaries and the next cost reductions

Asked about digital twins, Primas draws a product boundary: building a complete digital twin is not LemonSlice’s goal. Another team could combine its visual technology with suitable context, a harness, and an IQ model to build such an application. LemonSlice intends to supply the infrastructure rather than own that entire problem.

The final technical question concerns hourly cost and whether the product is affordable enough for real-world use. Primas says LemonSlice’s customer charges are at roughly the same level as an audio model, without naming a comparator or giving an hourly price. He still wants costs to fall: consumer applications need very low prices, and cheaper generation would also make higher resolution practical.

He identifies algorithmic and hardware improvements as two paths forward, while expressing hope that their recent pace continues. A further possibility is architectural: moving from diffusion toward token-based generation could make this kind of video substantially cheaper. That remains a proposed direction, not a demonstrated result. Primas closes by inviting people interested in these technical problems to speak with him; LemonSlice is hiring.

23:5224:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:52 · section reference included

Resources

From the talk

  • The library's account of its conversational Roosevelt exhibit and the partners behind its live avatar.

Updates since the talk

Transcript download
Full transcript

Loading transcript…