AI Engineer Europe 2026
Building Conversational Agents
Loading slides…
Read the talk
Building Conversational Agents: From Stateful Tool Loops to Live Audio
Build a small coding agent, give a conversational model tools and vision, and follow the deployment and reliability tradeoffs exposed by the live demonstrations.
From a talk by Thor Schaeff and Philipp Schmid
Before you start: You should be comfortable reading Python, using environment variables, and following JSON tool schemas and WebSocket connections.
Start with a key—and a device that can use it
What does it take to connect an application to Gemini? The first practical step in Thor Schaeff and Philipp Schmid’s workshop is a secret API key. Both work on developer experience at Google DeepMind, spanning the Gemini API and Google AI Studio. AI Studio provides a place to try models and create credentials; using Antigravity itself does not require this same key setup.
The setup proceeds through AI Studio, reached through ai.dev:
- Open Get API key, also available through
ai.dev/api-key. - Create or import a project.
- Select the project and create an API key.
Philipp describes the core hands-on exercises as accessible with a Google account and the free tier, without a credit card. The selected frame shows the project-created notification and the key-creation dialog, before a credential is exposed.
An existing key works too. The workshop mentions shell configuration such as .bashrc or .zshrc, as well as inline configuration for the demo. The essential constraint is that the key remains secret: do not share it or commit it to GitHub. Philipp deletes the key he exposes during setup.
While credentials are being created, Wolfram Rebenwolf from Thursday AI describes using Gemma and Gemini for coding and agents, including an experiment on glasses. A brief Chrome tip—right-click the tab bar to move tabs vertically—recovers some laptop screen space. The glasses discussion then becomes concrete: VisionClaw connects Gemini with OpenClaw tools. The glasses connect to a phone, and the phone maintains the Gemini Live WebSocket connection. The Meta Ray-Ban SDK makes that integration possible; the attendee describes his setup as experimental. Another attendee, Michael from GetYourGuide in Zurich, is building an initial AI customer-support agent. These are two distinct destinations for the same ingredients: conversational input, retained context, and tools that act outside the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One interaction surface for models and agents
The first build is a small coding agent that can read files, write files, and run shell commands. Live audio comes afterward. Philipp introduces the Interactions API, launched in public beta in December 2025, as the intended direction beyond generateContent. That is a direction of travel, not a claim that the established API had already been retired. The new interface follows conventions familiar from Chat Completions, Anthropic, and OpenAI.
A request selects either a model, such as Gemini 3 Flash, or an agent, such as Deep Research. The latter can plan and conduct an extended research task rather than return a single immediate model response. Philipp describes runs lasting roughly 10–15 minutes and visiting hundreds of sites. Bringing or defining custom agents is presented as future work. The broader goal is to chain outputs across models: Nano Banana for images, Flash for further processing, Lyria for audio, and potentially Veo for video.
The structural simplification is typed content blocks. Inputs and outputs use a common pattern with an explicit type, covering text, images, audio, video, function calls, and thought signatures. Compared with a protobuf-style structure in which fields such as text or inline data imply the content type, this gives application code a more direct way to inspect and dispatch content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep history on the server without surrendering context control
With server-side state, the client sends new input together with a previous interaction ID. The server attaches that input to the retained conversation. Long-running agent work can use background: true and polling instead of holding a connection open; webhooks are described as forthcoming. Streaming uses server-sent events. Built-in tools, remote MCP, and the recently introduced combination of Google Search with custom functions extend the same surface.
| Concern | Server-managed history | Client-managed history |
|---|---|---|
| Follow-up request | New input plus previous ID | Explicit conversation input |
| Context editing | Continue retained history | Remove or rewrite selected content |
| Reuse | Preserve existing content | Preserve it carefully when rebuilding |
Server-side state is optional. If an application needs to remove parts of its context, it can still manage and submit the history itself. Background execution addresses a different concern: Philipp’s examples involve agent tasks taking minutes, for which polling or notification avoids tying completion to a long-lived HTTP request.
Retaining history also helps implicit caching. A model must process the input tokens before generating a response; cached processing can be reused on a follow-up request. Philipp estimates that cached input tokens are 90% cheaper and reports two to three times better cache-hit rates among startups using Interactions API. His explanation is that rebuilding history on the client invites small changes—removed line breaks, stripped whitespace, or deleted content—that can disrupt reuse. The reported improvement concerns those startups, not a controlled comparison supplied with the workshop. The ensuing code comparison reinforces the API’s other simplification: explicit typed blocks instead of oneof-style input fields.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The loop connects reasoning to action
An agent has four working parts: a model chooses what to do; tools interact with its environment; context describes the task, capabilities, preferences, and constraints; and a loop keeps those parts working until the model stops requesting tools. Server-side state simplifies history management, but the client still runs the tool loop.
For ordinary chat, interactions.create receives a model and input—for example, a question about the capital of France. A follow-up supplies the returned ID, so the server already has the earlier user input and model response. The same continuation mechanism can cross model boundaries: the slide example researches AI agents in 2026 with Deep Research, then asks Nano Banana to generate a visual from that research. Applications that need manual control can instead provide an array of turns with user and model roles.
A function declaration supplies its type, name, description, and parameter schema. When an interaction requires action, the application must do the work:
- Inspect the interaction’s outputs.
- Find function-call blocks.
- Execute the corresponding local functions.
- Submit their results in another interaction.
- Repeat until the model produces its final response.
A function call is a request to act; the function result records what the application actually did. The slide places that distinction inside the complete request–execute–respond cycle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the coding agent documentation it can use
The workshop uses a preferred IDE agent or CLI to write the implementation. Before asking it to build anything, install the relevant Gemini API skill. The documentation’s coding-agent setup distinguishes general Gemini development, Live API, and Interactions API skills. For this first exercise, choose Interactions. A project-local install is sufficient; the demonstrated location is .agents/skills/gemini-interactions-api.
Philipp has tested discovery with Cursor, Antigravity, and Gemini CLI. The discussion of Claude Code’s .claude versus .agents discovery is less settled, so the useful check is behavioral: ask the coding agent which skills it can use. Antigravity identifies the installed Interactions skill. Context7 from Upstash and Vercel’s skills.sh provide installation routes backed by the same GitHub repository. The deliverable itself is a Python script, which Philipp expects can also run inside an attendee’s sandbox-based workflow.
The skill is deliberately smaller than a copied API manual. It addresses things the model may do unreliably, such as choosing an obsolete model, and workflow preferences, such as using Bun for tests. Available-model guidance prevents stale Gemini 1.5 defaults. For detailed API behavior, the skill points to Markdown documentation that the agent can fetch. When a feature changes, the documentation can be updated without requiring every user to reinstall a large skill. Asked about the extra fetch, Philipp compares it with reading a local reference file: either approach requires retrieving the material before using it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the smallest stateful agent
The implementation request specifies an Agent class in workshop.py, rather than simply asking for an agent. Its constructor creates a GenAI client, selects a model, and initializes the retained previous interaction ID. Its run method sends input and updates that ID. A main function exercises the class. During generation, Philipp corrects the coding agent to use the workspace’s uv environment; the required package is google-genai, installable with pip or uv pip install.
This Python version follows the recording’s beta-era outputs response shape. The current Interactions documentation uses execution steps and model_output instead; it also specifies that tools, system instructions, and generation configuration must be supplied again on subsequent requests. A previous ID retains conversation history, not those request settings.
python
import os
from google import genai
class Agent:
def __init__(self, model: str):
self.client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
self.model = model
self.previous_interaction_id = None
def run(self, text: str) -> str:
request = {"model": self.model, "input": text}
if self.previous_interaction_id is not None:
request["previous_interaction_id"] = self.previous_interaction_id
interaction = self.client.interactions.create(**request)
self.previous_interaction_id = interaction.id
return "\n".join(
output.text
for output in interaction.outputs
if output.type == "text"
)
if __name__ == "__main__":
agent = Agent(model=os.environ["GEMINI_MODEL"])
print(agent.run("My name is Phil."))
print(agent.run("What is my name?"))
In the generated workshop code, the model defaults to Gemini 3 Flash. The important state transition is the assignment of interaction.id: the next call points to the interaction that just completed. The screenshot shows that update and the name-recall test awaiting execution.
The first run produces an experimental-API warning. A moment of confusion follows because one prompt asks for the model’s name rather than the user’s; when the question asks for the user’s name, the response recalls it. Philipp credits precise instructions, the installed skill, useful context, and actually running the script for the coding workflow’s success. Participants also report working TypeScript implementations and even phone-based participation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add file tools, then close the execution loop
The next request adds readFile and writeFile. Each tool needs two things: a JSON schema that tells the model which arguments to generate, and a Python function that executes those arguments locally. A tools map associates the name with both. During generation, the coding agent discovers an existing solution folder and uses its implementation as guidance—an instructive reminder to inspect what context an agent actually consulted.
The basic file implementation is small enough to see the entire model-to-code contract:
python
from pathlib import Path
def read_file(file_path: str) -> str:
return Path(file_path).read_text(encoding="utf-8")
def write_file(file_path: str, content: str) -> str:
Path(file_path).write_text(content, encoding="utf-8")
return content
TOOLS = {
"readFile": {
"function": read_file,
"schema": {
"type": "function",
"name": "readFile",
"description": "Read a UTF-8 file and return its contents.",
"parameters": {
"type": "object",
"properties": {"file_path": {"type": "string"}},
"required": ["file_path"],
},
},
},
"writeFile": {
"function": write_file,
"schema": {
"type": "function",
"name": "writeFile",
"description": "Write UTF-8 text to a file.",
"parameters": {
"type": "object",
"properties": {
"file_path": {"type": "string"},
"content": {"type": "string"},
},
"required": ["file_path", "content"],
},
},
},
}
def execute_tool(name: str, arguments: dict) -> str:
if name not in TOOLS:
raise ValueError(f"Unknown tool: {name}")
return TOOLS[name]["function"](**arguments)
The model receives the schemas; the application retains the Python callables. These file functions directly access the process’s filesystem, so the environment in which the exercise runs determines what they can read or overwrite.
The agent’s input now accepts either text or a list of content blocks, because a follow-up may contain function results. Each model request includes the tool schemas. The client inspects interaction.outputs, which can include reasoning-related content and thought signatures as well as function calls. With server-managed state, those signatures do not have to be reconstructed and sent back manually. Before dispatching a call, the application checks that the requested tool exists: a request for an unsupported edit-file tool must not become an arbitrary invocation.
Each executed function produces a corresponding function result. The generated implementation recursively calls self.run when it has tool results to submit; otherwise it returns the interaction’s text. The test asks the agent to write Hello from the agent to hello.txt and read it back. The demonstrated trace contains a writeFile call, its result, a readFile call, and the final response. That trace is stronger evidence of filesystem work than a sentence claiming that a file was created.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use conversation to expose missing instructions
An outer stdin loop turns the one-shot example into an interactive application. It waits for user input, calls agent.run, and prints the result. Inside that call, the tool loop continues until the model finishes. A greeting correctly produces a conversational response without a file operation. A request for a thumbs-up SVG is more revealing: the model initially supplies content but hesitates to write it to disk, even though the tools are available.
Philipp proposes making the agent’s role explicit: it is a coding assistant that can use tools to interact with the local filesystem. A write call eventually appears in the existing conversation, and he then asks the coding agent to add system instructions to the implementation. The skill remains useful here because it tells the coding agent how this API accepts those instructions. Philipp says Gemini 3 Flash’s training predates the Interactions API, attributing the implementation to supplied documentation and coding infrastructure rather than memorized API examples.
The generated configuration permits instructions at construction or when running a request. After restarting with a coding persona and explicit filesystem access, the same thumbs-up SVG request produces a writeFile call, and the resulting icon is displayed. Tool availability and instructions about when to use tools are separate parts of the application. This exercise improves the latter without establishing the cause of every earlier hesitation.
The final coding capability is a shell-command tool implemented with Python’s subprocess, returning standard output. An attendee reports hallucinated tools during their own testing, while Philipp explicitly sets security aside for this shell example. The generated change also updates the system prompt. A proposed Nano Banana image test is set aside because the agent has no corresponding skill or guidance; instead, asking for the time invokes date and returns Wednesday, April 8. A suggestion to delete all files is rejected, not executed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Branch history, but account for expiry and context limits
Keeping interaction IDs on the client enables branching. Instead of always continuing from the latest turn, select an earlier ID and start a different continuation. Philipp suggests using an initial web-research result as the common base for several parallel requests. To export history, interactions.get retrieves an interaction and its previous ID; following those links walks backward through the conversation.
Philipp states default interaction retention of one day on the free tier and 55 days for paid usage. Storage duration is distinct from context capacity: he describes a million-token Gemini context limit and says exceeding it produces an error. Compaction remains the client’s responsibility in this demonstration. An eight-hour pause can still leave a free-tier interaction available; after its retention period expires, the old ID can no longer be found. This is expiry of stored interactions, not a global daily conversation reset. Vertex support and additional flexibility are prospective, and no dedicated context-reworking API is offered in the discussion. An audience suggestion to speak an API joke through eSpeak prompts a mention of Gemini text-to-speech before the handoff to live audio.
A follow-up caching question sharpens the earlier explanation. Philipp describes reuse at an object level rather than treating an entire interaction as indivisible: a 4,000-token PDF might be cached while a ten-token accompanying text input is not; later, the PDF and some follow-up turns might both be reusable. The example is conditional. Whitespace changes can still disrupt reuse, and request routing and the timing of follow-ups affect whether a cache hit occurs. Preserving the conversation exactly improves the opportunity for reuse without guaranteeing it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A live DJ with a music-generation tool
Thor opens the audio portion with Gemini 3.1 Flash Live, a newly released native-audio model. He describes architectural work aimed at latency and scalability. The Live API remains a separate interface in the workshop; bringing it into Interactions is work in progress.
The first application combines Live conversation with Lyria 3. Thor describes its clip model as generating 30-second songs, including lyrics, alongside a separate full-song model. Live Jukebox gives the conversational model a DJ persona and a music-generation tool. Its inspiration is the BBC Radio 1 phone-in experience: talk to the host, request a genre and mood, and hear the result.
Thor requests high-energy German techno schlager about the UK AI scene. The DJ asks for more detail, accepts a request to be surprised, and introduces the generated track. The lyrics bring together London, robots in bowler hats, builder’s tea, and German flourishes. A second audience request asks for a nursing song in Swahili; Thor adds hardcore techno. Music plays, but he acknowledges that its vibe is wrong. The demonstration exercises conversational clarification and tool use while also exposing a mismatch between the requested style and the result.
Live Jukebox was built and published in AI Studio. Unlike the core free-tier exercises, Thor says this demo requires a paid API key with billing enabled because it generates music.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What travels over the Live connection
Thor treats live-audio benchmarks as an incomplete guide to application behavior and encourages trying the interaction itself. He identifies the model as powering Gemini Live on phones and is less certain about Search Live. The Live API provides a stateful WebSocket connection accepting text, streamed audio chunks, and video frames. The demonstrated Live API accepts video at a maximum of one frame per second. A frame can come from a camera, a canvas, or screen sharing. Thor’s Shopify Sidekick example uses screen context to guide a merchant through setting up a custom domain.
The return stream contains audio buffers and can include transcription. Tool calling adds external capabilities, including Google Search grounding. Underneath the conversation, the model processes sound tokens directly rather than requiring a speech-to-text → text-model → text-to-speech cascade. Thinking levels adjust how much reasoning precedes a response. Thor estimates support for 97 languages in preview and highlights mixed-language conversation, such as German and English within the same exchange. Automatic voice activity detection enables barge-in: the user can interrupt while the model speaks.
The direct transport is WebSocket-only in this demonstration. Thor contrasts that with GPT Realtime’s direct WebRTC infrastructure. Applications that want WebRTC can use integrations from LiveKit, Pipecat, Fishjam by Software Mansion, Vision Agents, or Voximplant. That choice determines how much media transport infrastructure the application must assemble around the model connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompt the voice, then test its access to facts
In AI Studio’s Live view, Thor enables the webcam and types a question about his outfit. The response identifies a green jacket, blue T-shirt, and black cap. He then changes the delivery: the base voice is Puck, and the system instruction asks for a friendly Irish accent. Thor describes roughly 30 base voices, with prompts supplying additional style. The subsequent outfit conversation uses the requested Irish manner of speaking.
Current weather requires more than audio understanding. Thor turns to Google Search grounding and explains the thinking control: the interface’s no-thinking setting means minimal thinking, while additional reasoning increases response latency. He then asks for London’s current weather. The model first claims it is about nine degrees and cloudy. Challenged to identify the date and look again, it identifies April 8, 2026, then gives a forecast of five to thirteen degrees with rain. Thor disputes the answers and suspects a grounding or UI problem. No repair is established in this sequence. A fluent spoken answer does not establish that the intended information source was used.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a generated app or a working example
AI Studio’s real-time voice option supplies one starting point. Thor asks it to build a multilingual interview-practice assistant for languages including German, English, and Spanish. He describes this builder as using Gemini 3.0 Flash Preview and supporting JavaScript full-stack environments such as Next.js and Angular, with XR building blocks for glasses and web VR experiences. The generation starts in the background; the recording does not return to a completed interview app.
The alternative is to start from the Live API documentation, which links AI Studio, coding-agent skills, and GitHub example apps. Thor clones the examples into a directory named AINGEU and opens them in Antigravity with agy. The repository presents two architectures worth comparing: a server that proxies media, and a server that issues credentials for a direct client connection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Route browser media through a Python server
The Python SDK example uses a browser → application server → Gemini path. The browser captures microphone and camera input. A FastAPI WebSocket receives audio, video, and text queues, and the backend forwards them into a server-side Live session. A separate Gemini Live module contains the model integration, keeping it apart from the browser-facing transport.
LiveConnectConfig holds system instructions, guardrails, and tool definitions. After setting up the session, the server forwards the client’s queued media through it. To launch the example, Thor creates and activates a uv virtual environment, installs dependencies, copies .env.example to .env, and supplies GEMINI_API_KEY. Running main.py starts the demo at localhost:8000.
The running application opens with an Irish-accented greeting and responds to the camera view. Thor requests German, then Chinese, and asks about eating. The exchange appears to be understood, but he points out missing transcription. He also notices worse latency and attributes it to the extra server hop. That observation motivates removing the application server from the media path.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect directly using an ephemeral token
A direct browser connection must not expose the long-lived secret API key. The second example keeps that key on a small backend, which creates an ephemeral token and returns it to the browser or phone. The client uses that short-lived credential to establish its own WebSocket connection to the Live API. The token server still needs its virtual environment, dependencies, and API-key configuration; it no longer forwards the media stream.
In the demonstrated version, token creation and the bidirectional WebSocket endpoint use v1alpha. An expiry limits how long a leaked token remains useful. The frontend uses raw WebSockets without an SDK, passes the token as its access credential, and handles the incoming events itself.
| Architecture | Backend responsibility | Media path |
|---|---|---|
| Server proxy | Hold key and run Live session | Browser → backend → Live API |
| Ephemeral token | Hold key and mint expiring token | Browser → Live API |
The network inspector makes the direct sequence visible: first a response containing a token and its expiry, then the WebSocket connection, then a setup message with the model, real-time input configuration, and tools. The Live model’s published identifier is gemini-3.1-flash-live-preview. Connection establishment and session configuration are separate steps; having a socket open does not by itself show that the desired tools were configured correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate successful actions from unresolved failures
The direct-connection demo does not fix the weather problem. Asked whether an umbrella is needed in London, the model says it cannot obtain the current weather and suggests checking elsewhere. Explicitly asking it to use Google Search does not recover the result. Thor moves on to test custom tools.
Those tools produce visible effects: a request displays a Hello World alert, and another changes the page background to green. After further configuration changes, however, a request to restore white receives a refusal—the model says it cannot control display or page settings. Thor revisits the grounding and custom-tool toggles without establishing a diagnosis. Another attendee also reports that Google Search grounding is not working. The successful browser actions remain useful demonstrations, but neither the search failure nor the later loss of tool behavior receives a confirmed fix.
The implementation entry points remain the same after the failures: the Live documentation links AI Studio, the example repository, and coding skills. The useful artifact is a connection and event-handling structure to inspect and extend, not an assumption that every configured capability has already passed an end-to-end test.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Longer sessions require a policy for forgetting
Asked about Live context, Thor states uncompressed session limits of 15 minutes for audio-only input and two minutes for audio-video input. He describes a go-away notification around session termination, then introduces context-window compression as a way to extend conversation.
Context-window compression trades older context for continued interaction. In Thor’s explanation, a sliding window retains a configured amount of recent material and drops what precedes it. The quantity of video matters: adding frames consumes context that would otherwise remain available for conversation. Audio-only sessions can therefore retain more conversational history than sessions continually ingesting images. Extending a session does not mean preserving everything said or seen within it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose conversational flow and control deliberately
The business-use question brings several applications into view. Thor describes Shopify Sidekick as already in production. Stitch supports voice-directed design, though he points out that it is a Google-built product. Waymo integration is prospective. He then shows Ato, an elder-companion example combining voice interaction with an app for caregivers or family members. He describes the startup as Argentina-based and emphasizes multilingual conversation; the product page shows a fabric-covered companion device beside an AI care-companion headline. A brief case-study playback introduces a grandmother named Maria.
| Approach | Principal benefit | Intervention point |
|---|---|---|
| Native audio | Natural conversational flow | Less fine-grained control before speech |
| Cascading pipeline | Observable intermediate stages | Inspect or rewrite text before synthesis |
Thor suggests that a cascading pipeline may fit some current business requirements better. Its separate transcription, model, and speech stages make it easier to see what happened and modify a response before it is spoken. Native audio offers a more immediate conversational experience, but that does not make it the right fit for every workflow.
Transcripts are another application responsibility. Thor says the Live session does not provide a transcript archive that can simply be retrieved afterward; the application must store the events it needs. He points to LiveKit and Pipecat for surrounding capabilities such as audio and transcript storage and additional observability.
For recruitment, he suggests selected interview stages or screening rather than replacing the entire interview pipeline. The audience raises criteria, conversational guardrails, and Google Meet integration with existing transcripts. The answer supplies no concrete Meet integration. Thor instead emphasizes evaluating the use case and building the required surrounding controls, including requirements such as SOC 2. He characterizes the preview as ready for experimentation, with production suitability depending on that work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recognizing speech is not authorizing its speaker
An attendee describes a practical hazard: during a voice-agent demo, someone nearby could tell a coding agent to delete a file. Thor does not claim a reliable capability that authorizes only one enrolled speaker. Proactive audio can filter out irrelevant conversation, but he says it is not a dependable way to accept only the owner’s voice. The attendee reports experimenting with NVIDIA Parakeet and a short voice sample for speaker differentiation; that remains an attendee-reported approach, not a demonstrated Live API feature.
A separate question asks what the model’s thinking looks like. Thor describes opt-in thinking information as text events on the WebSocket. It is not spoken aloud as part of the audio response.
For coding dictation, the relevant problem may be grounding rather than continuous conversation: the transcription should use the class names visible in the editor. If fully real-time interaction is unnecessary, Thor recommends Gemini 3 Flash with contextual inputs. The discussion gives a concrete multimodal request: supply main.py and util.py as text alongside the audio recording. The model can use both the spoken request and the exact source identifiers. Stitch’s spoken edits to a visual mockup illustrate the same principle: visual context disambiguates which button the user means.
Live interaction can also ingest text or images, but those inputs consume context. For a static editor view, the suggested sequence is to send one image, then several seconds of audio, without resending the unchanged image every second. Philipp estimates roughly 1,200 tokens per image. The optimization is driven by information change: send a fresh frame when its contents or relevance change, rather than treating continuous image streaming as mandatory.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Personalization, evaluation, and storage have separate boundaries
A civil-engineering example initially sounds like a question about supplying specialist knowledge. Thor suggests putting that knowledge into the initial context or providing function calls that retrieve it during conversation. The questioner then clarifies: can the agent gradually adapt to what it learns about the user’s expertise? Thor’s answer is yes within the active context window. The model can remember names and information from the conversation while that material remains in context. That conversational adaptation is different from durable memory or trusted-speaker authorization.
The evaluation question receives a requirements-oriented answer rather than a testing protocol. Thor names HIPAA, SOC 2, function-call behavior, and guardrails as factors that change what a business application needs. He recommends looking at partner infrastructure such as LiveKit for the surrounding voice-agent capabilities. The workshop’s enjoyable demos are therefore starting points for evaluation, not substitutes for defining what the application must reliably do.
Returning to Interactions, an attendee asks when it will reach Vertex. No date is given; the timing is outside the presenters’ control. They expect the same API surface, so experimentation can begin on the Gemini API, with requests for higher rate limits handled separately. Applications that require Vertex-specific enterprise features may need to wait.
A question about PII and data sovereignty exposes the cost of opting out of interaction storage. store=false disables storage of the interaction resource, which removes server-side continuation and background-execution features. It should not be read as a universal guarantee about every aspect of processing or retention. Philipp expects future Vertex data-sovereignty behavior to resemble generateContent, but presents that as an expectation rather than an established capability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Return to the failed weather answers
The final substantive question returns directly to the weather demonstrations: entertaining failures are one thing, but how should an application handle hallucinations in production? Thor acknowledges that the demos lack best-practice system instructions. He opens guidance on defining the agent’s persona, conversational rules, tool behavior, and examples.
His proposed next step is to structure the system prompt with explicit guardrails, guidelines, and tool definitions so the agent better follows its intended operating boundaries. That is a concrete implementation task, but the recording does not demonstrate a repaired weather lookup or elimination of hallucinations. The remaining work is to make the instructions and tool configuration explicit, then establish through the application’s behavior that they are being followed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A voice and vision assistant for Meta Ray-Ban glasses using Gemini Live and optional OpenClaw actions.
Installable coding-agent skills for Gemini models, the Interactions API, and real-time Live API applications.
Technical specifications and guides for server-proxied or direct-client voice and vision applications.
Python SDK and raw WebSocket examples, including a direct-client application using ephemeral tokens.
Further reading
- Interactions API launchArticle
Philipp Schmid's coauthored introduction to the public beta, server-managed state, background execution, and Deep Research.
Thor Schaeff's coauthored launch overview, including voice-agent examples and integration partners.
Updates since the talk
Current interaction schema, state management, storage controls, supported models, and API limitations.
Full transcript
Loading transcript…