AI Engineer World's Fair 2026
An Interaction Is All You Need — Ivan Leo, Google DeepMind
Loading slides…
Read the talk
An Interaction Is All You Need
Ivan Leo introduces Gemini’s Interactions API and Managed Agents through the work an application used to manage itself: conversation state, multimodal handoffs, tool execution, persistent workspaces and credentials.
From a talk by Ivan Leo
At a glance
Ideas worth remembering
Use
previous_interaction_idto continue server-managed conversation context, including the state associated with thought signatures.Multimodal chaining keeps the calling pattern familiar while changing the model and requested medium. A photograph can branch into location-specific image interactions, and a linked image interaction can supply context for video generation.
Conversation continuity and filesystem continuity are separate: retain the interaction ID for context and the environment ID for the sandbox’s files and packages.
A single Managed Agent request can launch substantial internal work. The repository analysis consuming more than two million tokens illustrates delegation scale, not a fixed context-window size or guaranteed analysis quality.
Proxy-side credential injection keeps a token value out of the agent’s working context. Safe use of the authenticated capability remains a separate concern.
Skills and named agents package different reusable assets: skills carry procedures and supporting files, while named agents save a prepared configuration. Maintained API skills also help coding agents avoid stale model and method choices.
A weather lookup grows into an execution loop
“Tell me a joke” fits comfortably into a request and a response. A weather question adds a dependency: the model requests weather data, receives the result, and uses it to answer. An agent takes that pattern further, choosing among several tools, inspecting what happened, and deciding what to do next before returning to the user. Ivan Leo, a developer experience engineer at Google DeepMind, opens with this progression to explain why Gemini needs an API designed for more than a single completion.
Function calling supplies an important intermediate step: predictable JSON objects that application code can consume. Leo compares this to a registration form sending a username and password to a backend. The server can act because it knows the structure of the payload. Similarly, a model’s structured output can support extraction, parsing and tool requests without asking the application to interpret arbitrary prose. Once tool results feed subsequent decisions, however, the application must manage a continuing process rather than just parse one object.
A language model running in a loop is a useful shorthand for an agent, but it leaves out what the loop needs. The model interprets intent; functions and resources let it act; memory carries information forward; an environment executes the work. A coding agent without somewhere to execute code cannot complete the execution part of its assignment. Leo also observes that more capable models can need less specialized scaffolding, though that does not remove the need for tools and a runtime.
The first practical example is small and personal. Unable to use presenter view, Leo asked AI Studio for a mobile-responsive website that could display his speaker notes. Brief instructions became a deployed application he could access during the talk—he says he “vibed it out” two minutes beforehand. AI Studio serves here as the low-friction place to try model capabilities and tune parameters before moving into API integration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One interface must accommodate very different workloads
An image-generation request and a research agent do not have the same shape of work. In Leo’s example, Deep Research can take three minutes or more: the user interacts with a research plan, the agent investigates independently, and a detailed report comes back later. Yet an application may want to combine that research with image generation and other agent tasks. Separate endpoints make those combinations harder to assemble.
Deeply nested response objects add another integration cost. Each capability can require its own extraction logic before its result becomes useful to the next call. The Interactions API is presented as a common interface for models and agents, including the combinations Google itself wants to build across its products. Its first concrete simplification is to move conversation-state handling onto the server.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep an interaction ID instead of reconstructing state
Gemini’s thought signatures make state handling more consequential than remembering the visible chat. Leo describes them as opaque values returned alongside model activity, including function calls and responses, that must be preserved for later turns. He reports two practical failure modes: missing signatures can reduce model performance, and changes as small as an accidentally inserted whitespace character have caused customers to lose cache reuse. Manually rebuilding the conversation therefore creates opportunities to alter information the model expects to receive intact.
What changes in the request? The demonstration starts with “Hi, my name is Phil.” The response supplies an interaction ID. The next request passes that value as previous_interaction_id, allowing the service to recover the earlier context. The important code distinction is between sending a new prompt alone and sending a new prompt linked to the earlier interaction; the client no longer has to reconstruct that history and its signatures by hand.
The follow-up asks “What is my name?” The name appears only in the first request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Carry a photograph through image and video generation
The next demo turns state preservation into a multimodal handoff. Starting from one photograph, Nano Banana generates variations of the same person in different countries and locations. Each generated photograph is its own interaction, branching from the initial input. A later video-generation call uses an earlier interaction’s ID to continue with its image and context. The location variants are separate branches; they do not have to become a serial chain in which one destination depends on the previous one.
Which parts of the call change when a still image becomes a video? Leo’s concrete example first requests an image of himself in Venice. The application then calls client.interactions.create again, selects a video-capable model, requests video output and passes the image interaction’s ID. The new prompt asks the model to use the generated image as the first frame.
Audio and image outputs follow the same effort to simplify handling. Instead of navigating a different deeply nested object for each medium, the response exposes a type discriminator such as output.type. Application code can inspect whether an item is audio or an image and send it to the appropriate handler. For image generation, Leo changes the requested response modality and generation configuration while keeping client.interactions.create. The benefit is shared calling and dispatch logic across media, rather than identical model capabilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine tools, then represent the work as typed steps
The security-report example combines three distinct capabilities for investigating a React application:
- Google Search: Discover relevant security reports through the search index.
- URL context: Retrieve information from web pages for closer inspection.
- File Incident: Expose a custom application action alongside the built-in research tools.
The model can decide what information it needs and use the available tools to develop a response. Search and page retrieval extend the information available beyond what was present at training time.
Leo describes making these tools available in a single API call so the model can decide what information it needs. File Incident is the action offered to the model. The example ends with Leo describing the model’s response, without showing an incident being filed.
How can a response represent all this activity without pretending it was one assistant message? The Steps Data model replaces the legacy outputs array with typed execution steps. A step’s type distinguishes model output, thoughts and function calls. A thought step can carry a signature in its own field, while content types identify media such as audio and video. This gives asynchronous tool activity and work involving multiple models a more explicit representation. A consumer can tell what kind of event it is handling rather than infer that from position in an alternating user–model transcript.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Conversation state and workspace state need different IDs
Managed Agents adds an execution environment to the interface. The Antigravity agent runs remotely using the harness—the machinery around the model that manages its work—also used in the Antigravity product and AI Studio. Leo says Gemini was co-trained with this harness and expects benefits from using it. The immediate engineering proposition is concrete: developers do not first have to tune a harness, choose a sandbox provider, manage infrastructure and invent persistence between runs.
The first Managed Agent demo asks what is in a GitHub repository. An API request starts a remote sandbox; the agent lists workspace files, reads individual files, and produces a report. Those operations happen inside the hosted environment without Leo intervening between them. One caller request can therefore contain a substantial internal sequence of model decisions and filesystem operations.
What must a follow-up preserve to resume that work? Two IDs carry different kinds of continuity:
| Handle | What it preserves or selects |
|---|---|
interaction.id, passed as previous_interaction_id | Earlier conversation context and the state needed to continue it |
| Environment ID | The original sandbox, including its files and installed packages |
The interaction ID tells the service which conversation to continue. The environment ID tells it where the work should run. Keeping one is not a substitute for keeping the other.
Sources populate the workspace with material from GCS buckets, GitHub repositories and inline files. These can supply helper files and other project context. A continuation demo passes the environment ID and finds the same files where the earlier call left them. Installed packages and newly created files remain available across turns when the conversation and environment are resumed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A two-million-token investigation still starts with one request
The larger repository example came from a hackathon project: a programming language for creating reinforcement-learning environments, described through the analogy of Scratch’s drag-and-drop programming. Its repository combined a custom domain-specific language, a web page, schemas and an RL training loop running inside HUD. Leo reports that one analysis run consumed more than two million tokens. This is total work during an agent run, not evidence of a two-million-token context window or a measured accuracy result.
One API call delegated the investigation. Inside that call, the harness could inspect files and follow up on what it found. The reported token count gives a sense of how much computation that single request set in motion.
The local-to-cloud path reuses more than a prompt. Workflows and skills can first be developed in the Antigravity IDE, then packaged in a folder and uploaded through sources into .agents. Leo presents the shared harness and prompt as the reason this can carry familiar behavior into the remote agent. The useful unit of customization is the skill package and its supporting files, rather than a separately rebuilt agent loop for each deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let code use a credential without giving it the secret
A remote agent may need authenticated access to the Gemini API or a private GitHub repository. Giving the agent a raw credential creates a second problem: malicious instructions could induce it to disclose the value. Managed Agents introduces a credential-injecting proxy between sandbox traffic and the external service. The proxy inspects outbound request headers and applies configured replacements.
Where does the GitHub token enter the request? In Leo’s example, code in the sandbox makes a call to the GitHub API, and the proxy injects the token on its way out. The slide shows the proxy configuration. The agent can use the authenticated connection without seeing the token value. What it can do with that access remains a separate question.
The request and the credential meet at the proxy
Read the diagram as text
- Agent code in sandbox. request-path — Constructs and sends a GitHub API request without needing the real token value.
- Credential-injecting proxy. request-path — Inspects outbound headers and applies the configured token injection.
- GitHub token. credential-path — Added by the proxy rather than exposed to the agent.
- GitHub API. request-path — Receives the forwarded request with authentication injected.
- Agent code in sandbox → Credential-injecting proxy: Outbound API request.
- GitHub token → Credential-injecting proxy: Injected into request headers.
- Credential-injecting proxy → GitHub API: Authenticated request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Save a prepared configuration as a named agent
Once the agent behaves as intended, a named agent makes its setup reusable. Leo describes two ways to prepare one:
- Define the starting environment: Specify configured files and network sources. His example creates a data analyst from the Antigravity preview base agent with a GCS-supplied skill.
- Prepare an environment interactively: Chat with an agent to install the dependencies and packages it needs, then freeze that prepared environment as a specific agent.
Both approaches save the configuration that future work should start from.
The slide labels the second creation path “Fork a Snapshot.” Once the dependencies are in place, the developer supplies the prepared environment as the base for a named agent. Earlier, the environment ID served a different purpose. It routed a follow-up back to the workspace where the agent had been working.
At the July 2, 2026 presentation, Leo reported support for up to a thousand named agents. He said users paid for the model and that sandbox and storage were included.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Iterate locally—and give coding agents the API knowledge they lack
The closing tools shorten the path from experimenting to deployment. The open-source Gemini API CLI lets developers try model calls through shell commands. Leo also describes using it to iterate on an Antigravity agent locally, package the result as a folder, upload it and create an agent. This continues the earlier local-to-cloud approach: develop the workflow where it is easy to inspect, then transfer the prepared files to the managed runtime.
Migration has a different failure mode. A coding agent asked to implement a Gemini integration may keep choosing older model names and familiar methods, even when the developer intends to use the new API. Leo introduces an Interactions API skill to supply the agent with updated model lists, methods and implementation guidance. The team maintains that material and regularly evaluates whether the skills help agents build with Gemini as intended.
That is a practical ending for an API talk: changing the interface is only part of making it usable. The developer’s coding agent also needs enough current procedural knowledge to choose the interface correctly. Interaction IDs reduce hand-managed conversation state; environment IDs preserve the workspace; the migration skill addresses the knowledge gap in the tool writing the integration. Each takes over a different piece of work the application builder previously had to keep straight.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Gemini Interactions API overview.
Getting-started resource for the Managed Agents portion of the talk.
Guide to the Antigravity agent used in the repository-analysis examples.
Command-line tool for trying Gemini models locally and packaging a prepared agent for upload.
Related talks
- Why AI Agents Should Have Their Own Sandbox — Philipp Schmid, Google DeepMind
Philipp Schmid discusses why an AI agent should have its own sandbox.
- Can Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff
Thor Schaeff discusses building live voice agents with Gemini.
- The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder
Michael Patterson discusses the “lethal trifecta” on laptops.
Full transcript
Loading transcript…