AI Engineer World's Fair 2026

An Interaction Is All You Need — Ivan Leo, Google DeepMind

Loading slides…

Read the talk

An Interaction Is All You Need

Ivan Leo introduces the Interactions API and managed execution with “An Interaction is all you need.”Open full source frame
Ivan Leo introduces the Interactions API and managed execution with “An Interaction is all you need.”

Ivan Leo introduces Gemini’s Interactions API and Managed Agents through the work an application used to manage itself: conversation state, multimodal handoffs, tool execution, persistent workspaces and credentials.

From a talk by Ivan Leo

At a glance

Ideas worth remembering

  • Use previous_interaction_id to continue server-managed conversation context, including the state associated with thought signatures.

  • Multimodal chaining keeps the calling pattern familiar while changing the model and requested medium. A photograph can branch into location-specific image interactions, and a linked image interaction can supply context for video generation.

  • Conversation continuity and filesystem continuity are separate: retain the interaction ID for context and the environment ID for the sandbox’s files and packages.

  • A single Managed Agent request can launch substantial internal work. The repository analysis consuming more than two million tokens illustrates delegation scale, not a fixed context-window size or guaranteed analysis quality.

  • Proxy-side credential injection keeps a token value out of the agent’s working context. Safe use of the authenticated capability remains a separate concern.

  • Skills and named agents package different reusable assets: skills carry procedures and supporting files, while named agents save a prepared configuration. Maintained API skills also help coding agents avoid stale model and method choices.

A weather lookup grows into an execution loop

“Tell me a joke” fits comfortably into a request and a response. A weather question adds a dependency: the model requests weather data, receives the result, and uses it to answer. An agent takes that pattern further, choosing among several tools, inspecting what happened, and deciding what to do next before returning to the user. Ivan Leo, a developer experience engineer at Google DeepMind, opens with this progression to explain why Gemini needs an API designed for more than a single completion.

A presenter-notes application built in AI Studio gives the speaker a page for following his notes.Open full source frame
A presenter-notes application built in AI Studio gives the speaker a page for following his notes.

Function calling supplies an important intermediate step: predictable JSON objects that application code can consume. Leo compares this to a registration form sending a username and password to a backend. The server can act because it knows the structure of the payload. Similarly, a model’s structured output can support extraction, parsing and tool requests without asking the application to interpret arbitrary prose. Once tool results feed subsequent decisions, however, the application must manage a continuing process rather than just parse one object.

A language model running in a loop is a useful shorthand for an agent, but it leaves out what the loop needs. The model interprets intent; functions and resources let it act; memory carries information forward; an environment executes the work. A coding agent without somewhere to execute code cannot complete the execution part of its assignment. Leo also observes that more capable models can need less specialized scaffolding, though that does not remove the need for tools and a runtime.

The first practical example is small and personal. Unable to use presenter view, Leo asked AI Studio for a mobile-responsive website that could display his speaker notes. Brief instructions became a deployed application he could access during the talk—he says he “vibed it out” two minutes beforehand. AI Studio serves here as the low-friction place to try model capabilities and tune parameters before moving into API integration.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

One interface must accommodate very different workloads

An image-generation request and a research agent do not have the same shape of work. In Leo’s example, Deep Research can take three minutes or more: the user interacts with a research plan, the agent investigates independently, and a detailed report comes back later. Yet an application may want to combine that research with image generation and other agent tasks. Separate endpoints make those combinations harder to assemble.

Deeply nested response objects add another integration cost. Each capability can require its own extraction logic before its result becomes useful to the next call. The Interactions API is presented as a common interface for models and agents, including the combinations Google itself wants to build across its products. Its first concrete simplification is to move conversation-state handling onto the server.

4:134:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:13 · section reference included

Keep an interaction ID instead of reconstructing state

Gemini’s thought signatures make state handling more consequential than remembering the visible chat. Leo describes them as opaque values returned alongside model activity, including function calls and responses, that must be preserved for later turns. He reports two practical failure modes: missing signatures can reduce model performance, and changes as small as an accidentally inserted whitespace character have caused customers to lose cache reuse. Manually rebuilding the conversation therefore creates opportunities to alter information the model expects to receive intact.

What changes in the request? The demonstration starts with “Hi, my name is Phil.” The response supplies an interaction ID. The next request passes that value as previous_interaction_id, allowing the service to recover the earlier context. The important code distinction is between sending a new prompt alone and sending a new prompt linked to the earlier interaction; the client no longer has to reconstruct that history and its signatures by hand.

The second request supplies previous_interaction_id to continue the first interaction’s conversation.Open full source frame
The second request supplies previous_interaction_id to continue the first interaction’s conversation.

The follow-up asks “What is my name?” The name appears only in the first request.

5:135:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:13 · section reference included

Carry a photograph through image and video generation

The next demo turns state preservation into a multimodal handoff. Starting from one photograph, Nano Banana generates variations of the same person in different countries and locations. Each generated photograph is its own interaction, branching from the initial input. A later video-generation call uses an earlier interaction’s ID to continue with its image and context. The location variants are separate branches; they do not have to become a serial chain in which one destination depends on the previous one.

Which parts of the call change when a still image becomes a video? Leo’s concrete example first requests an image of himself in Venice. The application then calls client.interactions.create again, selects a video-capable model, requests video output and passes the image interaction’s ID. The new prompt asks the model to use the generated image as the first frame.

An image interaction for Venice is linked to a later video-generation request.Open full source frame
An image interaction for Venice is linked to a later video-generation request.

Audio and image outputs follow the same effort to simplify handling. Instead of navigating a different deeply nested object for each medium, the response exposes a type discriminator such as output.type. Application code can inspect whether an item is audio or an image and send it to the appropriate handler. For image generation, Leo changes the requested response modality and generation configuration while keeping client.interactions.create. The benefit is shared calling and dispatch logic across media, rather than identical model capabilities.

6:136:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:13 · section reference included

Combine tools, then represent the work as typed steps

The security-report example combines three distinct capabilities for investigating a React application:

  • Google Search: Discover relevant security reports through the search index.
  • URL context: Retrieve information from web pages for closer inspection.
  • File Incident: Expose a custom application action alongside the built-in research tools.

The model can decide what information it needs and use the available tools to develop a response. Search and page retrieval extend the information available beyond what was present at training time.

Leo describes making these tools available in a single API call so the model can decide what information it needs. File Incident is the action offered to the model. The example ends with Leo describing the model’s response, without showing an incident being filed.

How can a response represent all this activity without pretending it was one assistant message? The Steps Data model replaces the legacy outputs array with typed execution steps. A step’s type distinguishes model output, thoughts and function calls. A thought step can carry a signature in its own field, while content types identify media such as audio and video. This gives asynchronous tool activity and work involving multiple models a more explicit representation. A consumer can tell what kind of event it is handling rather than infer that from position in an alternating user–model transcript.

Typed execution steps distinguish model output, thoughts, function calls and multimodal content.Open full source frame
Typed execution steps distinguish model output, thoughts, function calls and multimodal content.
8:138:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:13 · section reference included

Conversation state and workspace state need different IDs

Managed Agents adds an execution environment to the interface. The Antigravity agent runs remotely using the harness—the machinery around the model that manages its work—also used in the Antigravity product and AI Studio. Leo says Gemini was co-trained with this harness and expects benefits from using it. The immediate engineering proposition is concrete: developers do not first have to tune a harness, choose a sandbox provider, manage infrastructure and invent persistence between runs.

The first Managed Agent demo asks what is in a GitHub repository. An API request starts a remote sandbox; the agent lists workspace files, reads individual files, and produces a report. Those operations happen inside the hosted environment without Leo intervening between them. One caller request can therefore contain a substantial internal sequence of model decisions and filesystem operations.

What must a follow-up preserve to resume that work? Two IDs carry different kinds of continuity:

HandleWhat it preserves or selects
interaction.id, passed as previous_interaction_idEarlier conversation context and the state needed to continue it
Environment IDThe original sandbox, including its files and installed packages

The interaction ID tells the service which conversation to continue. The environment ID tells it where the work should run. Keeping one is not a substitute for keeping the other.

Separate interaction and environment IDs preserve conversation context and workspace contents; sources populate the sandbox.Open full source frame
Separate interaction and environment IDs preserve conversation context and workspace contents; sources populate the sandbox.

Sources populate the workspace with material from GCS buckets, GitHub repositories and inline files. These can supply helper files and other project context. A continuation demo passes the environment ID and finds the same files where the earlier call left them. Installed packages and newly created files remain available across turns when the conversation and environment are resumed.

9:4310:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:43 · section reference included

A two-million-token investigation still starts with one request

The larger repository example came from a hackathon project: a programming language for creating reinforcement-learning environments, described through the analogy of Scratch’s drag-and-drop programming. Its repository combined a custom domain-specific language, a web page, schemas and an RL training loop running inside HUD. Leo reports that one analysis run consumed more than two million tokens. This is total work during an agent run, not evidence of a two-million-token context window or a measured accuracy result.

An agent-generated repository report illustrates work performed inside the managed environment.Open full source frame
An agent-generated repository report illustrates work performed inside the managed environment.

One API call delegated the investigation. Inside that call, the harness could inspect files and follow up on what it found. The reported token count gives a sense of how much computation that single request set in motion.

The local-to-cloud path reuses more than a prompt. Workflows and skills can first be developed in the Antigravity IDE, then packaged in a folder and uploaded through sources into .agents. Leo presents the shared harness and prompt as the reason this can carry familiar behavior into the remote agent. The useful unit of customization is the skill package and its supporting files, rather than a separately rebuilt agent loop for each deployment.

11:5612:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:56 · section reference included

Let code use a credential without giving it the secret

A remote agent may need authenticated access to the Gemini API or a private GitHub repository. Giving the agent a raw credential creates a second problem: malicious instructions could induce it to disclose the value. Managed Agents introduces a credential-injecting proxy between sandbox traffic and the external service. The proxy inspects outbound request headers and applies configured replacements.

A configured network proxy supplies credentials to outbound requests, using placeholder token references in the example.Open full source frame
A configured network proxy supplies credentials to outbound requests, using placeholder token references in the example.

Where does the GitHub token enter the request? In Leo’s example, code in the sandbox makes a call to the GitHub API, and the proxy injects the token on its way out. The slide shows the proxy configuration. The agent can use the authenticated connection without seeing the token value. What it can do with that access remains a separate question.

The request and the credential meet at the proxy

Agent code in sandboxrequest-path — Constructs and sends a GitHub API request without needing the real token value.
Credential-injecting proxyrequest-path — Inspects outbound headers and applies the configured token injection.
GitHub tokencredential-path — Added by the proxy rather than exposed to the agent.
GitHub APIrequest-path — Receives the forwarded request with authentication injected.
Sandbox code supplies the GitHub request. The proxy supplies authentication before forwarding it, so the agent does not need the token value in its working context.
Read the diagram as text
  • Agent code in sandbox. request-path — Constructs and sends a GitHub API request without needing the real token value.
  • Credential-injecting proxy. request-path — Inspects outbound headers and applies the configured token injection.
  • GitHub token. credential-path — Added by the proxy rather than exposed to the agent.
  • GitHub API. request-path — Receives the forwarded request with authentication injected.
  • Agent code in sandbox → Credential-injecting proxy: Outbound API request.
  • GitHub token → Credential-injecting proxy: Injected into request headers.
  • Credential-injecting proxy → GitHub API: Authenticated request.
13:2613:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:26 · section reference included

Save a prepared configuration as a named agent

Once the agent behaves as intended, a named agent makes its setup reusable. Leo describes two ways to prepare one:

  • Define the starting environment: Specify configured files and network sources. His example creates a data analyst from the Antigravity preview base agent with a GCS-supplied skill.
  • Prepare an environment interactively: Chat with an agent to install the dependencies and packages it needs, then freeze that prepared environment as a specific agent.

Both approaches save the configuration that future work should start from.

The slide labels the second creation path “Fork a Snapshot.” Once the dependencies are in place, the developer supplies the prepared environment as the base for a named agent. Earlier, the environment ID served a different purpose. It routed a follow-up back to the workspace where the agent had been working.

Named agents can be configured from sources or saved from a prepared environment.Open full source frame
Named agents can be configured from sources or saved from a prepared environment.

At the July 2, 2026 presentation, Leo reported support for up to a thousand named agents. He said users paid for the model and that sandbox and storage were included.

14:2614:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:26 · section reference included

Iterate locally—and give coding agents the API knowledge they lack

The closing tools shorten the path from experimenting to deployment. The open-source Gemini API CLI lets developers try model calls through shell commands. Leo also describes using it to iterate on an Antigravity agent locally, package the result as a folder, upload it and create an agent. This continues the earlier local-to-cloud approach: develop the workflow where it is easy to inspect, then transfer the prepared files to the managed runtime.

The Gemini skills repository provides maintained API guidance for coding agents.Open full source frame
The Gemini skills repository provides maintained API guidance for coding agents.

Migration has a different failure mode. A coding agent asked to implement a Gemini integration may keep choosing older model names and familiar methods, even when the developer intends to use the new API. Leo introduces an Interactions API skill to supply the agent with updated model lists, methods and implementation guidance. The team maintains that material and regularly evaluates whether the skills help agents build with Gemini as intended.

That is a practical ending for an API talk: changing the interface is only part of making it usable. The developer’s coding agent also needs enough current procedural knowledge to choose the interface correctly. Interaction IDs reduce hand-managed conversation state; environment IDs preserve the workspace; the migration skill addresses the knowledge gap in the tool writing the integration. Each takes over a different piece of work the application builder previously had to keep straight.

15:2915:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:29 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…