# An Interaction Is All You Need — Ivan Leo, Google DeepMind

Ivan Leo, Google DeepMind

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/8aVbXXvJUY4-interaction-is-all-you-need) · [Video](https://www.youtube.com/watch?v=8aVbXXvJUY4)

## An Interaction Is All You Need

Ivan Leo introduces Gemini’s Interactions API and Managed Agents through the work an application used to manage itself: conversation state, multimodal handoffs, tool execution, persistent workspaces and credentials.

![Ivan Leo introduces the Interactions API and managed execution with “An Interaction is all you need.”](https://ai.engineer/api/talk-essay-media/6ca3dca839fdccdc8d34f8ae6ead9b0493ed04c06c5c57a7385885a153ccf8c9)

Ivan Leo introduces the Interactions API and managed execution with “An Interaction is all you need.”

### A weather lookup grows into an execution loop

[0:12](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=12s)

“Tell me a joke” fits comfortably into a request and a response. A weather question adds a dependency: the model requests weather data, receives the result, and uses it to answer. An agent takes that pattern further, choosing among several tools, inspecting what happened, and deciding what to do next before returning to the user. Ivan Leo, a developer experience engineer at Google DeepMind, opens with this progression to explain why Gemini needs an API designed for more than a single completion.

![A presenter-notes application built in AI Studio gives the speaker a page for following his notes.](https://ai.engineer/api/talk-essay-media/cf73d02871561154e9494373850214a581effcd98205e9a0bb4fbadb7972fede)

A presenter-notes application built in AI Studio gives the speaker a page for following his notes.

Function calling supplies an important intermediate step: predictable JSON objects that application code can consume. Leo compares this to a registration form sending a username and password to a backend. The server can act because it knows the structure of the payload. Similarly, a model’s structured output can support extraction, parsing and tool requests without asking the application to interpret arbitrary prose. Once tool results feed subsequent decisions, however, the application must manage a continuing process rather than just parse one object.

A language model running in a loop is a useful shorthand for an agent, but it leaves out what the loop needs. The model interprets intent; functions and resources let it act; memory carries information forward; an environment executes the work. A coding agent without somewhere to execute code cannot complete the execution part of its assignment. Leo also observes that more capable models can need less specialized scaffolding, though that does not remove the need for tools and a runtime.

The first practical example is small and personal. Unable to use presenter view, Leo asked AI Studio for a mobile-responsive website that could display his speaker notes. Brief instructions became a deployed application he could access during the talk—he says he “vibed it out” two minutes beforehand. AI Studio serves here as the low-friction place to try model capabilities and tune parameters before moving into API integration.

### One interface must accommodate very different workloads

[4:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=253s)

An image-generation request and a research agent do not have the same shape of work. In Leo’s example, **Deep Research** can take three minutes or more: the user interacts with a research plan, the agent investigates independently, and a detailed report comes back later. Yet an application may want to combine that research with image generation and other agent tasks. Separate endpoints make those combinations harder to assemble.

Deeply nested response objects add another integration cost. Each capability can require its own extraction logic before its result becomes useful to the next call. The [Interactions API](https://ai.google.dev/gemini-api/docs/interactions-overview) is presented as a common interface for models and agents, including the combinations Google itself wants to build across its products. Its first concrete simplification is to move conversation-state handling onto the server.

### Keep an interaction ID instead of reconstructing state

[5:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=313s)

Gemini’s thought signatures make state handling more consequential than remembering the visible chat. Leo describes them as opaque values returned alongside model activity, including function calls and responses, that must be preserved for later turns. He reports two practical failure modes: missing signatures can reduce model performance, and changes as small as an accidentally inserted whitespace character have caused customers to lose cache reuse. Manually rebuilding the conversation therefore creates opportunities to alter information the model expects to receive intact.

What changes in the request? The demonstration starts with “Hi, my name is Phil.” The response supplies an interaction ID. The next request passes that value as `previous_interaction_id`, allowing the service to recover the earlier context. The important code distinction is between sending a new prompt alone and sending a new prompt linked to the earlier interaction; the client no longer has to reconstruct that history and its signatures by hand.

![The second request supplies previous_interaction_id to continue the first interaction’s conversation.](https://ai.engineer/api/talk-essay-media/683f697f37b11be1984a04b0079ce3d3c175328105a75164227b210bd66ac662)

The second request supplies previous_interaction_id to continue the first interaction’s conversation.

The follow-up asks “What is my name?” The name appears only in the first request.

### Carry a photograph through image and video generation

[6:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=373s)

The next demo turns state preservation into a multimodal handoff. Starting from one photograph, **Nano Banana** generates variations of the same person in different countries and locations. Each generated photograph is its own interaction, branching from the initial input. A later video-generation call uses an earlier interaction’s ID to continue with its image and context. The location variants are separate branches; they do not have to become a serial chain in which one destination depends on the previous one.

Which parts of the call change when a still image becomes a video? Leo’s concrete example first requests an image of himself in Venice. The application then calls `client.interactions.create` again, selects a video-capable model, requests video output and passes the image interaction’s ID. The new prompt asks the model to use the generated image as the first frame.

![An image interaction for Venice is linked to a later video-generation request.](https://ai.engineer/api/talk-essay-media/82bc6b0bfe674835aa9bace0a3c9fcbb5dffc516dd4fe57c972aae3c0fa755a1)

An image interaction for Venice is linked to a later video-generation request.

Audio and image outputs follow the same effort to simplify handling. Instead of navigating a different deeply nested object for each medium, the response exposes a type discriminator such as `output.type`. Application code can inspect whether an item is audio or an image and send it to the appropriate handler. For image generation, Leo changes the requested response modality and generation configuration while keeping `client.interactions.create`. The benefit is shared calling and dispatch logic across media, rather than identical model capabilities.

### Combine tools, then represent the work as typed steps

[8:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=493s)

The security-report example combines three distinct capabilities for investigating a React application:

- **Google Search:** Discover relevant security reports through the search index.
- **URL context:** Retrieve information from web pages for closer inspection.
- **File Incident:** Expose a custom application action alongside the built-in research tools.

The model can decide what information it needs and use the available tools to develop a response. Search and page retrieval extend the information available beyond what was present at training time.

Leo describes making these tools available in a single API call so the model can decide what information it needs. File Incident is the action offered to the model. The example ends with Leo describing the model’s response, without showing an incident being filed.

How can a response represent all this activity without pretending it was one assistant message? The **Steps Data model** replaces the legacy outputs array with typed execution steps. A step’s type distinguishes model output, thoughts and function calls. A thought step can carry a signature in its own field, while content types identify media such as audio and video. This gives asynchronous tool activity and work involving multiple models a more explicit representation. A consumer can tell what kind of event it is handling rather than infer that from position in an alternating user–model transcript.

![Typed execution steps distinguish model output, thoughts, function calls and multimodal content.](https://ai.engineer/api/talk-essay-media/a83efa2fd27ba4f1140322ea512aa1fc1221f7478321dde478233773851034f0)

Typed execution steps distinguish model output, thoughts, function calls and multimodal content.

### Conversation state and workspace state need different IDs

[9:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=583s)

**Managed Agents** adds an execution environment to the interface. The [Antigravity agent](https://ai.google.dev/gemini-api/docs/antigravity-agent) runs remotely using the harness—the machinery around the model that manages its work—also used in the Antigravity product and AI Studio. Leo says Gemini was co-trained with this harness and expects benefits from using it. The immediate engineering proposition is concrete: developers do not first have to tune a harness, choose a sandbox provider, manage infrastructure and invent persistence between runs.

The first Managed Agent demo asks what is in a GitHub repository. An API request starts a remote sandbox; the agent lists workspace files, reads individual files, and produces a report. Those operations happen inside the hosted environment without Leo intervening between them. One caller request can therefore contain a substantial internal sequence of model decisions and filesystem operations.

What must a follow-up preserve to resume that work? Two IDs carry different kinds of continuity:

| Handle | What it preserves or selects |
| --- | --- |
| `interaction.id`, passed as `previous_interaction_id` | Earlier conversation context and the state needed to continue it |
| Environment ID | The original sandbox, including its files and installed packages |

The interaction ID tells the service which conversation to continue. The environment ID tells it where the work should run. Keeping one is not a substitute for keeping the other.

![Separate interaction and environment IDs preserve conversation context and workspace contents; sources populate the sandbox.](https://ai.engineer/api/talk-essay-media/e1de1f79596f1e70ff2a4c947d1d2f971fcc01c989cdda2582ce286376874128)

Separate interaction and environment IDs preserve conversation context and workspace contents; sources populate the sandbox.

Sources populate the workspace with material from GCS buckets, GitHub repositories and inline files. These can supply helper files and other project context. A continuation demo passes the environment ID and finds the same files where the earlier call left them. Installed packages and newly created files remain available across turns when the conversation and environment are resumed.

### A two-million-token investigation still starts with one request

[11:56](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=716s)

The larger repository example came from a hackathon project: a programming language for creating reinforcement-learning environments, described through the analogy of Scratch’s drag-and-drop programming. Its repository combined a custom domain-specific language, a web page, schemas and an RL training loop running inside HUD. Leo reports that one analysis run consumed more than two million tokens. This is total work during an agent run, not evidence of a two-million-token context window or a measured accuracy result.

![An agent-generated repository report illustrates work performed inside the managed environment.](https://ai.engineer/api/talk-essay-media/c5f6cd3eb5e239f33bd5c40e425ed36ed60549095b796e039acb821e2cc5186d)

An agent-generated repository report illustrates work performed inside the managed environment.

One API call delegated the investigation. Inside that call, the harness could inspect files and follow up on what it found. The reported token count gives a sense of how much computation that single request set in motion.

The local-to-cloud path reuses more than a prompt. Workflows and skills can first be developed in the Antigravity IDE, then packaged in a folder and uploaded through sources into `.agents`. Leo presents the shared harness and prompt as the reason this can carry familiar behavior into the remote agent. The useful unit of customization is the skill package and its supporting files, rather than a separately rebuilt agent loop for each deployment.

### Let code use a credential without giving it the secret

[13:26](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=806s)

A remote agent may need authenticated access to the Gemini API or a private GitHub repository. Giving the agent a raw credential creates a second problem: malicious instructions could induce it to disclose the value. Managed Agents introduces a credential-injecting proxy between sandbox traffic and the external service. The proxy inspects outbound request headers and applies configured replacements.

![A configured network proxy supplies credentials to outbound requests, using placeholder token references in the example.](https://ai.engineer/api/talk-essay-media/839d7ad756dfd8a1e3e02474416c9bbf004d8cad3cccbe73bee48e9672ef1c20)

A configured network proxy supplies credentials to outbound requests, using placeholder token references in the example.

Where does the GitHub token enter the request? In Leo’s example, code in the sandbox makes a call to the GitHub API, and the proxy injects the token on its way out. The slide shows the proxy configuration. The agent can use the authenticated connection without seeing the token value. What it can do with that access remains a separate question.

The request and the credential meet at the proxy
Sandbox code supplies the GitHub request. The proxy supplies authentication before forwarding it, so the agent does not need the token value in its working context.
- Agent code in sandbox — Constructs and sends a GitHub API request without needing the real token value.
- Credential-injecting proxy — Inspects outbound headers and applies the configured token injection.
- GitHub token — Added by the proxy rather than exposed to the agent.
- GitHub API — Receives the forwarded request with authentication injected.
- Agent code in sandbox → Credential-injecting proxy (Outbound API request)
- GitHub token → Credential-injecting proxy (Injected into request headers)
- Credential-injecting proxy → GitHub API (Authenticated request)

### Save a prepared configuration as a named agent

[14:26](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=866s)

Once the agent behaves as intended, a named agent makes its setup reusable. Leo describes two ways to prepare one:

- **Define the starting environment:** Specify configured files and network sources. His example creates a data analyst from the Antigravity preview base agent with a GCS-supplied skill.
- **Prepare an environment interactively:** Chat with an agent to install the dependencies and packages it needs, then freeze that prepared environment as a specific agent.

Both approaches save the configuration that future work should start from.

The slide labels the second creation path “Fork a Snapshot.” Once the dependencies are in place, the developer supplies the prepared environment as the base for a named agent. Earlier, the environment ID served a different purpose. It routed a follow-up back to the workspace where the agent had been working.

![Named agents can be configured from sources or saved from a prepared environment.](https://ai.engineer/api/talk-essay-media/ef60ca0e10adbe52c2aa945f59ed286744e20624221d88af4a1eb7909797da1a)

Named agents can be configured from sources or saved from a prepared environment.

At the July 2, 2026 presentation, Leo reported support for up to a thousand named agents. He said users paid for the model and that sandbox and storage were included.

### Iterate locally—and give coding agents the API knowledge they lack

[15:29](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=929s)

The closing tools shorten the path from experimenting to deployment. The open-source [Gemini API CLI](https://github.com/google-gemini/gemini-api-cli) lets developers try model calls through shell commands. Leo also describes using it to iterate on an Antigravity agent locally, package the result as a folder, upload it and create an agent. This continues the earlier local-to-cloud approach: develop the workflow where it is easy to inspect, then transfer the prepared files to the managed runtime.

![The Gemini skills repository provides maintained API guidance for coding agents.](https://ai.engineer/api/talk-essay-media/4c1b88ca58043c1f6069b8ea1f3c6f5147f4a4541812cdf3c5da9b5896e088ac)

The Gemini skills repository provides maintained API guidance for coding agents.

Migration has a different failure mode. A coding agent asked to implement a Gemini integration may keep choosing older model names and familiar methods, even when the developer intends to use the new API. Leo introduces an Interactions API skill to supply the agent with updated model lists, methods and implementation guidance. The team maintains that material and regularly evaluates whether the skills help agents build with Gemini as intended.

That is a practical ending for an API talk: changing the interface is only part of making it usable. The developer’s coding agent also needs enough current procedural knowledge to choose the interface correctly. Interaction IDs reduce hand-managed conversation state; environment IDs preserve the workspace; the migration skill addresses the knowledge gap in the tool writing the integration. Each takes over a different piece of work the application builder previously had to keep straight.

## Key takeaways

- Use `previous_interaction_id` to continue server-managed conversation context, including the state associated with thought signatures.
- Multimodal chaining keeps the calling pattern familiar while changing the model and requested medium. A photograph can branch into location-specific image interactions, and a linked image interaction can supply context for video generation.
- Conversation continuity and filesystem continuity are separate: retain the interaction ID for context and the environment ID for the sandbox’s files and packages.
- A single Managed Agent request can launch substantial internal work. The repository analysis consuming more than two million tokens illustrates delegation scale, not a fixed context-window size or guaranteed analysis quality.
- Proxy-side credential injection keeps a token value out of the agent’s working context. Safe use of the authenticated capability remains a separate concern.
- Skills and named agents package different reusable assets: skills carry procedures and supporting files, while named agents save a prepared configuration. Maintained API skills also help coding agents avoid stale model and method choices.

## Resources

- [Interactions API overview](https://ai.google.dev/gemini-api/docs/interactions-overview) — Gemini Interactions API overview.
- [Managed Agents quickstart](https://ai.google.dev/gemini-api/docs/managed-agents-quickstart) — Getting-started resource for the Managed Agents portion of the talk.
- [Antigravity agent guide](https://ai.google.dev/gemini-api/docs/antigravity-agent) — Guide to the Antigravity agent used in the repository-analysis examples.
- [Gemini API CLI](https://github.com/google-gemini/gemini-api-cli) — Command-line tool for trying Gemini models locally and packaging a prepared agent for upload.

## Related talks

- [Why AI Agents Should Have Their Own Sandbox — Philipp Schmid, Google DeepMind](https://ai.engineer/talks/oWTEiYpxl80-why-ai-agents-should-have-their-own) — Philipp Schmid discusses why an AI agent should have its own sandbox.
- [Can Your Agent Hear You Now? Building Live Voice Agents with Gemini — Thor Schaeff](https://ai.engineer/talks/xfPVBzDYjH4-your-agent-hear-you-now-building-live) — Thor Schaeff discusses building live voice agents with Gemini.
- [The Lethal Trifecta Is Already on Your Laptops — Michael Patterson, Coder](https://ai.engineer/talks/abw_m-DRi1k-lethal-trifecta-is-already-on-your-laptops) — Michael Patterson discusses the “lethal trifecta” on laptops.

## Slides

### An Interaction is all you need

Ivan introduces the Interactions API and Managed Agents, setting up a look back at how interactions with models have changed.

[0:20](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=20s) [Large cream text on black reads “An Interaction is all you need.”](https://assets.ai.engineer/sites/aiecode2025/sha256/91434b15cd297c43a350c8e660ac1369ef616eaada699af454a4979b155ff18a.webp)

An Interaction is
all you need

### Brief History

The diagram traces a shift from text completion and chat to function calling, then goal-driven work: a model uses multiple tools, reflects on their results, and compiles a response rather than answering in one step.

[0:24](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=24s) [Diagram traces completion, instructions, user/model conversation, instructions with functions, and goals with thinking and functions.](https://assets.ai.engineer/sites/aiecode2025/sha256/9fd06bed37a28126c849310df21e0d58e024a269a940a5a74fa4c67da17b6428.webp)

Brief History

Completion
In a shocking finding, researchers discovered that a common household chemical drastically accelerates aging in lab mice.

Instructions
Tell me a joke. Why don't scientists trust atoms? ... Because they make up everything!

User / Model
User: Hi!
Assistant: Hello! How can I assist you?

Instructions + Functions
Functions: get_weather(location)
User: What's the weather in ny?
Model: call get_weather("New York City")
System: It's 72° and sunny in New York

Goals + Thinking + Functions
Functions: get_weather(), search(), browser()
User: Find adoption rates, % who want to learn another language...
Model: thinking ..., search(...), browse...
Model: Compiling comprehensive...

### What is an Agent

An agent combines a model, tools, short- and long-term memory, and an execution environment. Tool calls loop back into model inference, letting the model use their results before responding.

[2:01](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=121s) [Slide defines agents through reasoning, tools, context, memory, and environment, beside a diagram linking user input, model inference, responses, and tool calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/c76485818b5e20f7feb0aa374afd51042ffc907d1d6c5a6d130ad9cf94a86f9c.webp)

What is an Agent

Agents = Reasoning + Tools + Context
(Memory) + Environment

✓ LLM used to understand the users request
✓ Functions or external resources that the agent can use to interact with the world.
✓ Short-term memory keeps track
✓ Long-term memory to recall past
✓ Environment to execute functions

User Input
Model Inference
Agent Response
Tool Calls

### Get started with just a few clicks

The checklist pairs no-setup, free-tier experimentation in AI Studio with billing and spend management. Ivan recommends it for trying Google’s models and says he built his speaker-notes app there.

[3:01](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=181s) [Seven-step AI Studio onboarding checklist covers visiting the site, obtaining an API key, experimenting, billing, and spend management, beside a playground screenshot.](https://assets.ai.engineer/sites/aiecode2025/sha256/38bc83278d705484f255269c1fb022a3f92bc30f4d4df4cc237999ff0f40f982.webp)

Get started with
just a few clicks

01 Visit Google AI Studio (ai.studio)
02 Get your API key
03 No setup required
04 Generous free-tier
05 Start experimenting
06 Set up billing
07 Manage spend

Interface text:
https://aistudio.google.com/api-keys
Google AI Studio
Playground
History
Build
Apps
Gallery
Dashboard
Documentation
Search
What's new
Get API key
Settings

Playground
Explore Google models
Models
Agents

Featured
Test out our most advanced and newest models.

Code and Chat
Build chatbots, agents, and code with Gemini 3.

Image Generation
Create and edit images with Nano Banana and Imagen.

Video Generation
Generate videos with Veo models, our state of the art video generation models.

Speech and Music
Explore our text to speech and music [illegible]

Real-time
Real-time voice and video with Live API

Start typing a prompt to see what our models can do

Run settings
Gemini 3.1 Pro Preview
gemini-3.1-pro-preview
Our latest SOTA reasoning [illegible]
unprecedented depth and [illegible]
powerful multimodal [illegible]
coding capabilities

System instructions
Optional tone and style [illegible]
model

Temperature
Thinking level
High
Tools
Structured outputs
Code execution

### Presenter Notes Reader demo

Ivan shows the speaker-notes app he built in AI Studio after learning he could not use presenter view. The preview advances from page 1 to 2 of 21, pairing large white-on-black notes with timing estimates and navigation.

[3:17](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=197s) [AI Studio displays an app-building conversation beside a dark presenter-notes preview with introductory notes, a timer, and slide navigation.](https://assets.ai.engineer/sites/aiecode2025/sha256/17babff2e2adb53445650d6710df66cad1eb74379dea1d68aca3772c28028aa4.webp)

SLIDE 1

Hi, I'm Ivan, and I'm on the Developer Experience team here at Google DeepMind.

Today, we're covering the brand new Interactions API - an interaction is all you need

Interface text:
Presenter Notes Reader
Back to start
Remix
Share
Publish
Preview
Code
Gemini
Gemini 3.5 Flash · Ran for 187s
Edited 7 files
metadata.json
src/types.ts
src/utils/parser.ts
src/index.css
src/components/NotesInput.tsx
src/components/NotesPresenter.tsx
src/App.tsx
Built

I have successfully designed and built the Presenter Notes Reader, a mobile-responsive stage reader designed specifically for high-legibility and touch navigation under spotlight environments.

Design & Functionality Overview
• Dark Stage Aesthetic: Styled in a pitch-black theme to minimize blinding backlights and preserve professional stage presence. It utilizes highly legible typography pairing Space Grotesk headings with Inter body text.

AI Features
Add presentation timer
Add font size togg[illegible]
Make changes, add new features, ask for anything

EXIT
STAGE MODE
Slide: 1 / 21
Est. Slide speaking: 12s
Left: ~16 mins
01:35
Outline
1 / 21
Touch Helper: Off

Chrome
File
Edit
View
History
Bookmarks
Profiles
Tab
Window
Help
Thu Jul 2 3:23 PM
Presenter Notes Reader | [illegible]
aistudio.google.com/apps/[illegible]?showPreview=true&showAssistant=true
Ask Gemini
Action Required
JULY
S M T W T F S
Ghostty
Screen Studio
demo.mp4
Ghostty.dm[illegible]
Mirror for
[illegible] Display (P3-1600 nits)
ProMotion
Wallpaper
Notifications
Advanced…
Night Shift…
Arrange…

[3:30](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=210s) [Browser-based presenter-notes reader showing its second notes page, navigation controls and timing indicators alongside the app-building conversation.](https://assets.ai.engineer/sites/aiecode2025/sha256/ce8a2ac1133bc6e1d45221dd4e071fb16c7a60331ca39cbc9a449f37dcada453.webp)

SLIDE 2

Before we look at the new syntax, let's trace how drastically developer expectations have shifted.

Originally, we had simple instructions. A strict, predictable turn. You ask for a joke; the model generates a punchline. Linear and contained.

Then, we moved to instructions plus functions. You ask for the weather. The model recognizes the intent, pauses, generates a tool call, swallows the external data, and synthesizes a final

Interface text:
Presenter Notes Reader
Back to start
Remix
Share
Publish
Preview
Code
Gemini
Gemini 3.5 Flash · Ran for 187s
Edited 7 files
metadata.json
src/types.ts
src/utils/parser.ts
src/index.css
src/components/NotesInput.tsx
src/components/NotesPresenter.tsx
src/App.tsx
Built

I have successfully designed and built the Presenter Notes Reader, a mobile-responsive stage reader designed specifically for high-legibility and touch navigation under spotlight environments.

Design & Functionality Overview
• Dark Stage Aesthetic: Styled in a pitch-black theme to minimize blinding backlights and preserve professional stage presence. It utilizes highly legible typography pairing Space Grotesk headings with Inter body text.

AI Features
Add presentation timer
Add font size togg[illegible]
Make changes, add new features, ask for anything

EXIT
STAGE MODE
Slide: 2 / 21
Est. Slide speaking: 49s
Left: ~15 mins
01:44
Outline
2 / 21
Touch Helper: Off

Chrome
File
Edit
View
History
Bookmarks
Profiles
Tab
Window
Help
Thu Jul 2 3:23 PM
Presenter Notes Reader | [illegible]
aistudio.google.com/apps/[illegible]?showPreview=true&showAssistant=true
Ask Gemini
Action Required
JULY
S M T W T F S
Ghostty
Screen Studio
demo.mp4
Ghostty.dm[illegible]
Mirror for
[illegible] Display (P3-1600 nits)
ProMotion
Wallpaper
Notifications
Advanced…
Night Shift…
Arrange…

### Introducing the Interactions API

The model and agent examples use the same client.interactions.create call. The API aims to unify model inference and longer-running agent work, with optional server-side state and combined tools.

[4:08](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=248s) [Interactions API introduction listing context management, background tasks, optional server-side state, composability and combined tools, beside model and agent Python examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/e86e85becfa8e07c7d3e0ef20c6591ca37b07ccab08d9d75bfae6e063b85bd26.webp)

Introducing the Interactions API

A unified API to support inference of models and agents. Built for:

Complex context management
Long horizon background tasks
Optional server side state management
Easy interpretability and composability
Combined-tool use (Google Search + Custom Functions)

For Models

from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-3.5-flash",
    input="Who won the last euro?",
    tools=[{"type": "google_search"}],
)

For Agents

from google import genai

client = genai.Client()

interaction = client.interactions.create(
   agent="antigravity-preview-05-2026",
   input="Research the history of Google
TPUs.",
    environment="remote"
)

### Server-side State

The example introduces Phil, then asks for his name. Passing the first response’s ID as previous_interaction_id lets the second request reuse server-side context instead of resending the conversation.

[5:26](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=326s) [Server-side State slide with Python code that introduces a name in one request and asks for it in a second request using previous_interaction_id.](https://assets.ai.engineer/sites/aiecode2025/sha256/fb58e221afd7376aed0f66e55e4b018c41b5ebde7690c6ebb2528817e846aa65.webp)

Server-side
State

from google import genai

client = genai.Client()

# 1. First turn
interaction1 = client.interactions.create(
    model="gemini-3-flash-preview",
    input="Hi, my name is Phil."
)
print(f"Model: {interaction1.output_text}")

# 2. Second turn (passing previous_interaction_id)
interaction2 = client.interactions.create(
    model="gemini-3-flash-preview",
    input="What is my name?",
    previous_interaction_id=interaction1.id
)
print(f"Model: {interaction2.output_text}")

### Venice Photo Demo

The demo turns one photo into portraits in different locations, including Venice. Ivan explains that each image is an interaction; reusing its interaction ID lets the demo generate a video.

[6:40](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=400s) [Demo card showing a smiling person beside a Venice canal and bridge, with the heading Venice Canals, Italy and gray placeholder lines.](https://assets.ai.engineer/sites/aiecode2025/sha256/3baeb87297cc9d940276c8e76dbc4f564bbc32a93b93f946ddf1e6b4090dbf5d.webp)

Venice Canals, Italy 🇮🇹

Interface text:
×
‹ 7 ›
⋮

### Audio Generation

The Python example requests speech through client.interactions.create, then checks output.type for audio. It decodes the returned data and saves a WAV file, replacing nested response handling with a typed-output check.

[7:35](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=455s) [Audio Generation slide with Python code requesting speech output and saving decoded audio to a WAV file.](https://assets.ai.engineer/sites/aiecode2025/sha256/c1b7d605c688779b884232e276b07f2dba7f5281cbbfdcaac829d34af51afcc1.webp)

Audio Generation

import base64
from google import genai
import wave

# Set up the wave file to save the output:
def wave_file(filename, pcm, channels=1, rate=24000, sample_width=2):
    with wave.open(filename, "wb") as wf:
        wf.setnchannels(channels)
        wf.setsampwidth(sample_width)
        wf.setframerate(rate)
        wf.writeframes(pcm)

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-3.1-flash-tts-preview",
    input="Say the following: WOOHOO This is so much fun!. [laughs]",
    response_modalities=["AUDIO"],
    generation_config={
        "speech_config": {
            "language": "en-us",
            "voice": "kore"
        }
    }
)

for output in interaction.steps:
    if output.type == "audio":
        print(f"Generated audio with mime_type: {output.mime_type}")
        # Save the audio as wave file to the current directory.
        wave_file("generated_audio.wav", base64.b64decode(output.data))

### Image Generation

Image generation uses the same client.interactions.create call as audio. The example requests a 9:16, 2k city image, then decodes the image-typed output and saves it as a PNG.

[8:11](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=491s) [Image Generation slide showing Python code requesting a futuristic city image, setting image options, and saving the returned image.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4a92cfe8f396630ef7d4097025a24b57c3849d3a152dd762f98749e6913aff9.webp)

Image Generation

import base64
from google import genai

client = genai.Client()

interaction = client.interactions.create(
    model="gemini-3-pro-image-preview",
    input="Generate an image of a futuristic city.",
    generation_config={
        "image_config": {
            "aspect_ratio": "9:16",
            "image_size": "2k"
        }
    }
)

for output in interaction.steps:
    if output.type == "image":
        print(f"Generated image with mime_type: {output.mime_type}")
        # Save the image
        with open("generated_city.png", "wb") as f:
            f.write(base64.b64decode(output.data))

### Tool Use

The example combines Google Search, URL context, and a custom file_incident tool in one request to investigate a React CVE. It then inspects typed steps for function calls and prints their names and arguments.

[8:20](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=500s) [Tool Use slide with Python code defining an incident-reporting function and configuring search, URL context, and function tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/8b68f3f5103a19abdc3746890f3d3a2115f7cd5534dcb9a2390dc3872cc76e75.webp)

Tool Use

from google import genai

client = genai.Client()

file_incident = {
    "type": "function",
    "name": "file_incident",
    "description": "Files a security incident in the internal tracking system.",
    "parameters": {
        "type": "object",
        "properties": {
            "cve_id": {"type": "string", "description": "CVE identifier"},
            "severity": {"type": "string", "description": "Critical, High, Medium, or Low"},
            "summary": {"type": "string", "description": "Brief description of the vulnerability"},
        },
        "required": ["cve_id", "severity", "summary"],
    },
}

interaction = client.interactions.create(
    model="gemini-3-flash-preview",
    input="Search for the latest critical CVE affecting react, read the full advisory page, and file an incident for it.",
    tools=[
        {"type": "google_search"},
        {"type": "url_context"},
        {"type": "function", "name": "file_incident", "parameters": ...},
    ],
)

for output in interaction.steps:
    print(f"{output.type}...")
    if output.type == "function_call":
        print(f"Function: {output.name}")
        print(f"Arguments: {output.arguments}")

### The Steps Data Model

Steps replaces the outputs array with a typed timeline of model activity. The JSON separates a thought from a model output; content holds raw modalities, while typed fields carry names, arguments, and signatures.

[9:02](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=542s) [The Steps Data Model slide describing typed steps, raw modality content, structured fields, and statuses beside a JSON example.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d5c8dbda0760a8feee61c89edaf7f14d18ba6c0c25fb7291c436ffcef0c4ef4.webp)

Part 1: Interactions API & Agents

The Steps Data Model

A typed timeline of everything the model did

✓ Steps [ ] replaces the legacy Outputs [ ] array
✓ Type discriminator
  Model_output, thought,
  Function_call,
  google_search_call
✓ Content [ ] is reserved for raw modalities (text, image, audio, video)
✓ Structured data lives in typed fields (name, arguments, signature)
✓ Step Status: done, waiting, in_progress

{
  "id": "int_456",
  "status": "COMPLETED",
  "steps": [
    {
      "type": "thought",
      "status": "done",
      "summary": [{"type": "text", "text": "Analyzing request..."}],
      "signature": "abc123..."
    },
    {
      "type": "model_output",
      "status": "done",
      "content": [
        {"type": "text", "text": ”Here are the results…”}
      ]
    }
  ]
}

### Antigravity as Remote Agent

One API call runs the Antigravity IDE's harness in a managed Linux sandbox with code, filesystem and web tools. The example requests a CSV dashboard, then reads the text from the final interaction step.

[9:44](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=584s) [Antigravity as Remote Agent slide listing harness, toolset, sandbox, and API features beside a remote-agent invocation example.](https://assets.ai.engineer/sites/aiecode2025/sha256/0c598ecf2aa951f3e41b48b7f466f74102fe5102c28f514ea7ae1850730c81bf.webp)

Part 1: Interactions API & Agents

Antigravity as Remote Agent

General purpose agent running in a managed sandbox

Same harness that powers the Antigravity IDE
Rich Toolset: code execution, filesystem, web search, URL context, Agent Skills
Managed Linux sandbox ensures secure and isolated execution
One API call: no complex orchestration to build yourself

from google import genai
client = genai.Client()

interaction = client.interactions.create(
  agent="antigravity-preview-05-2026",
  input="Analyze this CSV and build a dashboard.",
  environment="remote",
)

# Access the result
print(interaction.steps[-1].content[0].text)

### Sample Starter Kit: Project Submission and Repository Review

The demo moves from a project-submission form to a repository review: Antigravity lists and reads files in a remote sandbox, then generates an evaluation report without manual intervention.

[10:40](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=640s) [Browser demo showing a project-submission dialog with project name, repository, and description fields plus a Submit for evaluation button.](https://assets.ai.engineer/sites/aiecode2025/sha256/97029ad118b087a5e6c2b626124d6ecbff5fe9078c7f9a78d83b57e26d950058.webp)

Submit a project

Project name
Sample Starter Kit

GitHub repo
https://github.com/octocat/Spo

Description
This is a project which demonstrates how to get started with Git

Cancel
Submit for evaluation

Interface text:
TanStack Start Starter
octocat/Spoon-Knife: This [illegible]
Ask Gemini
14

[10:47](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=647s) [Browser-based repository review with file-reading tool calls, a file list, and the beginning of a project evaluation report.](https://assets.ai.engineer/sites/aiecode2025/sha256/e5daad8c93b6fa065431264bd4a5bfe58b6a90f86c0553fc013fd72b22c33090.webp)

Sample Starter Kit

STATUS
Verified

SUBMITTED
Submitted June 22, 2026

REPO
octocat/Spoon-Knife

INTERACTION
[illegible]

ENVIRONMENT
[illegible]

[illegible] tokens

/workspace/repo/README.md

tool_call read_file
Read HTML file
/workspace/repo/index.html

tool_call read_file
Read CSS file
/workspace/repo/styles.css

tool_call list_files
List directory
/workspace/repo/.git

Generating the Hackathon Review

I'm starting with a review of the "Sample Starter Kit" hackathon submission. The project name is clear, taken directly from the user's prompt. The GitHub repository is accessible, and the description provides basic context. I'll need to assess its functionality and completeness based on the description, and consider what the user might expect in a final report.

Assessing the Repository Content

I've examined the repository contents: README.md, index.html, and styles.css. The project's simplicity is intentional, serving as a GitHub fork/pull request practice example. It perfectly fulfills its stated objective with minimal HTML, CSS, and a guide in the README. Now, I'm ready to craft the markdown report, ensuring it's valid and adheres to the supplied formatting guidelines.

Project Evaluation Report: Sample Starter Kit

FILES
3 indexed
index.html
README.md
styles.css

Select a file
Linked report paths and repository files open here.

Interface text:
Back to submissions
Message the review agent...
14 trace events captured.
TanStack Start Starter
octocat/Spoon-Knife: This [illegible]
Ask Gemini
14

### Network Proxies

The proxy injects a GitHub token into outbound request headers so the agent can clone a private repository without seeing the credential. The code scopes the header transformation to github.com.

[12:58](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=778s) [Slide titled Network Proxies, subtitled Protect your secrets, with a code example configuring an outbound authorization-header transformation.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca5c07a7f9e0342bbbf149c6c324a52ef5538edd0fef8478d986907be51f413c.webp)

Part 1: Interactions API & Agents
Network Proxies
Protect your secrets

Network Proxies

client = genai.Client()
interaction = client.interactions.create(
    agent="antigravity-preview-05-2026",
    input="Clone my private repo test/private-repo and analyze its contents.",
    environment={
        "type": "remote",
        "network": {
            "allowlist": [
                {
                    "domain": "github.com",
                    "transform": {
                        "Authorization": f"token {github_token}"
                    },
                }
            ]
        },
    },
)

Interface text:
18

### Build your own managed agent

Two paths create a reusable named agent: define instructions and skill files from source, or prepare an environment through conversation and save its snapshot. The examples load GCS skills or install pandas before saving.

[14:22](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=862s) [Slide comparing two ways to create a managed agent: define it from source configuration or save an iteratively prepared environment as a snapshot.](https://assets.ai.engineer/sites/aiecode2025/sha256/37dc0ad10ffb35b2162894d81835899ecc7a239738f59888865561fd0fcd45eb.webp)

Part 1: Interactions API & Agents
Build your own managed agent
Bundle instruction, skills and environment into a named agent

From Source
Define from Scratch

agent = client.agents.create(
  name="data-analyst",
  base_agent="antigravity-preview-05-2026",
  instructions="You are a data analyst.",
  base_environment={
    "config": {
      "sources": [
        {"type": "gcs", "source": "gs://skills/", "target": "/.agents/skills/"}
      ]
    }
  },
)

From Environment
Fork a Snapshot

# Step 1: Iterate with base agent
r = client.interactions.create(
  agent="antigravity-preview-05-2026",
  input="Install pandas. Create template.",
  environment="remote",
)
# Step 2: Save environment as named agent
agent = client.agents.create(
  name="my-data-analyst",
  base_agent="antigravity-preview-05-2026",
  instructions="You are a data analyst.",
  base_environment=r.environment_id,
)

Interface text:
‹ 19 › ⋮

### The Gemini API CLI

The Gemini API CLI connects local testing to managed-agent deployment: run prompts, scaffold and test an agent, then create it. Ivan describes packaging the agent as a folder and uploading it when local iteration is done.

[15:27](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=927s) [Slide listing prompt execution, image and speech generation, and managed-agent workflows, alongside CLI command examples, skill links, and a QR code.](https://assets.ai.engineer/sites/aiecode2025/sha256/95db275db1225d900c99e00c2ff2530f42d9adbf72bbdfccf8c689adb7b63e9f.webp)

Part 1: Interactions API & Agents
The Gemini API CLI

Structured, agent-friendly interactions from the command line

Run prompts against any model
Generate images and text-to-speech
Scaffold, test, and deploy managed agents

Gemini API CLI Skill
Interactions API Skill

# Run a prompt
gemini-api run "What is the capital of France?"

# Scaffold, test, and deploy an agent
gemini-api agents init my-agent
gemini-api agents test --prompt "Analyze Q1 data"
gemini-api agents create

# Run against a deployed agent
gemini-api run "Summarize quarter" --agent
my-agent

Interface text:
20

### Gemini API and Interactions Skills

The command installs the Interactions API skill for coding agents migrating to the API. Ivan says the team keeps model and method lists current and regularly evaluates the skills.

[15:53](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=953s) [Gemini skills repository slide with a description of API, SDK, and model or agent skills, repository statistics, a QR code, and an Interactions API skill installation command.](https://assets.ai.engineer/sites/aiecode2025/sha256/d7f80f169f8814fb579d805fd16d47770a19d8cc26694eb526129b95706994bc.webp)

google-gemini/gemini-skills

Skills for the Gemini API, SDK and model/agent interactions

11 Contributors
4 Issues
4k Stars
356 Forks

npx skills add google-gemini/gemini-skills --skill gemini-interactions-api -y

Interface text:
21

### AI Engineer World's Fair

[16:57](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=1017s) [Black end card with white AI Engineer World's Fair lettering inside a rectangular outline and the event website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ee71568f145ebb71043c874b89203b406b0737530279e0e9d8dbf13f669a386.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:12](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=12s) Hey guys, my name's Ivan. I'm on the developer experience team at Google DeepMind, and today we're covering the brand new Interactions API and our new offering, Managed Agents. And so to take a quote from a very famous paper, "An interaction is all you need." Before we move on to what the Interactions API is and what the new Managed Agents offering is, let's take a trip down memory lane, and let's kinda trace how models have changed and how we've worked with models. So when we first started working with models, a lot of times you would have a single interaction. You would send a message, "Tell me a joke," and then you

[0:42](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=42s) might get a response. For example, "Why don't scientists trust atoms? Because they make up everything." After a while, as models got more and more capable and they understood more about the world, this became a bit more complicated. We wanted to take these models, put them in different applications, but to do so, we needed them to be reliable. And so to do so, we invented function calling, where models could create individual JSON objects with predictable structures. Think like how when you wanna register for a website, your webpage sends a JSON payload to the back end

[1:12](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=72s) with a username and a password, and that enables the back-end server to say, "Okay, we have a new user, and now we can actually create a new account." That made it possible for us to actually use models in different parts of this application, whether it was extracting user information, parsing data, and more. But right now we're seeing this incredible cusp where models are getting more capable. We're seeing models being able to reason, go for very extended periods of time before they come back to people. And so you're moving from a place where we went from completions, instruction and

[1:42](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=102s) functions where we might ask, "Hey, what's the weather in New York City?" And the model has a call weather tool that it can get that data, ingest it, and generate a response to now a case where models have multiple tools under their belt. They're able to call tools, think and reflect on what the response meant, interact with the environment, and ultimately generate a response. And so this just brings us back to what is an agent? Fundamentally, if you look at what an agent is, it's really a model that powers a huge chunk of it. A lot of

[2:12](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=132s) people have reduced this to just a language model running in a loop. And for most cases, this is true. If you have a really capable model, you'll find that as we've got models that become more and more capable, a lot of the scaffolding has fallen away. A lot of models, for example, the latest Opus models or Fable models, are now just immediately just using a batch tool instead of individually specialized tools like a edit file or a read file tool like we used to do. And so really what we have is a language model that's used to understand the context, the user's intent,

[2:42](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=162s) that's given a bunch of different functions, all different resources that it then interacts with its environment in. It has some short-term memory and some long-term memory to keep track of what it's encountered, and then an environment to execute functions in. If you're building a coding agent, but you don't give it an environment where it can actually execute code, for example, it's not gonna be able to do anything. And so if you'd like to try any of our models, the easiest way to do so is actually with AI Studio. And so I actually like to point out that my speaker notes today were actually coded on AI Studio.

[3:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=193s) I vibed it out at two minutes before this talk. Uh, you see the, um, as... You can see the conversation history over here. Um, I literally said, "I need a mobile responsive website that I can pass in speaker notes," because I found out I wasn't gonna be able to use presenter view. And so what I did, I just gave it some brief instructions, and before you know it, you have a simple application that you can use without any hassle deployed on a website that's accessible to all. So we have a really generous, uh, free tier. For most of my presentations, I've just been using the free tier.

[3:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=223s) I haven't been using my Google unlimited tokens, thankfully. And I think it's the best way for you to try our models, whether it's our new speech-to-speech translation models where you take in some... You speak in, let's say, English, and it can translate to any language, or, you know, you wanna try our latest models like the Gemini 3.5 Flash, et cetera. It makes it very easy for you to do so with simplified UI toggles to just be able to tune all of the different parameters. So let's talk about the Interactions API now. One of the things that we've seen, especially as we've moved from

[4:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=253s) models to a dynamic where now we have models and agents, is that the kind of workloads that we want to support have changed significantly, right? Through the Gemini API, you have Nano Banana, you have, you know, Gemini 3.5 Flash, but you also have very capable agents like the Deep Research agent that's gonna run for anywhere from three minutes and more. You interact with it. You have some sort of research plan. It's gonna go out and do its own thing, and it comes back with very deep and detailed research reports. Now, it's pretty difficult then to

[4:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=283s) think about consuming models and agents through a whole bunch of these different endpoints. Additionally, for a lot of the older kind of API endpoints, a lot of the data was in, in very deeply nested objects. And so you can imagine the kind of difficulties that we had ourselves when we were building in first party integrations, and we wanted these use cases where you might be able to generate a Nano Banana image, do some research, generate a VO image from there, or spawn out all these different agentic, kind of like agents that go out and do their own thing. And so the

[5:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=313s) Interactions API is our sort of solution to this as we look to build an API that can support all of these different use cases and ones that we can build on ourselves as we look to bring Gemini across the entire surface of the Google ecosystem. So the best place to get started is just with server-side state. One of the things that you realize, especially as you use the new Gemini series of models, is that we essentially have this thing called thought signatures, where if you interact with the model, if it calls a function, if it gives a response, it's gonna pass back

[5:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=343s) a very opaque series of these numbers. For a lot of people, manually managing this was very difficult, and sometimes when we talk to different startups, we found that they would lose their cache just with a single white space that they mistakenly added in. So now with the Responses API, you can see on the first turn, "Hi, my name is Phil." And in the response, we give you a interaction ID. And as long as you give us back that same interaction ID under the parameter previous_interaction_id that you see in the second turn, all that context is preserved. And this is

[6:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=373s) really important with the new Gemini series of models because if you don't give us back those thought signatures, you're gonna see a decrease in performance. But what else does this unlock? Here you see a demo built by Vamsi from our team, that did an incredible demo where using a single photo you could generate with Nano Banana all these different variations of yourself in different countries and different locations. Each of these individual photos are a single interaction, but you can think of it from an initial API call, it spawns all these different interaction calls. Using those interactions, the

[6:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=403s) same interaction ID, we then generate a video using the interaction ID and the new Omni model that we just launched. Something like this would not have been possible without the Interactions API built by Philip over there.  So actually, it's a really incredible piece of work, I will say. So to give you an example, an idea of what it would take to build something similar, this is all it would take. First, you pass in Gemini 3.1 Flash-Lite Image, generate an image of me in Venice. You take that interaction ID, you chunk it

[7:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=433s) back into the client.interactions.create, and this time you specify that you want the Omni Flash model instead that we launched on Wednesday. And that allows you to now use the same Omni model with the same information, the same kind of images, and the same context throughout all these calls, making it possible to build more complex, rich, and just these very complex agentic applications. The same thing occurs with audio generation. If you look at the bottom, you see that now we have output.type. In the past, you would have to

[7:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=463s) take this very nested object, like .completions.audio, and it would be very tricky for you to nest it out. I can't even remember it off the top of my head. But now every single bit of the outputs that come out are clearly demarcated by a strong type. If it's an audio, as you see over here, all you gotta do is just parse it and handle it accordingly, right? But what happens if now I wanna generate an image? It's the same method over there, client.interactions.create. All you gotta do is just modify the response modality and the

[8:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=493s) generation config, and now I have an output.type of image, and this makes it very easy for me to start building and experimenting with Gemini. The same thing comes with tool use. You can mix and match different inbuilt tools that wasn't possible before. For example, over here, we're trying to get an agent to look up the internet to find the latest security reports for the React application. We've given it access to the same index that Google has with type Google Search. We've given it the ability to be... essentially retrieve information from web pages using the URL context tool that we ship with, and

[8:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=523s) we've given it a custom tool here called File Incident. The model can reason autonomously on its own, figure out what information it needs, and then actually generate a response to you, all in a single API call. This makes it very powerful because now you have models that not only can work with information at the training cut-off date, but beyond, because now they have access to the entirety of the internet. And all this boils down to the new steps data model that we have, where instead of a world where we have a single message sent to a model and a response, we're gonna have very

[9:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=553s) complex things like async tool calls, models working together, right? And we wanna make sure we have a strong kinda data structure to support this. And so we introduced the Steps Data model, which replaces the original legacy outputs array, with type discriminator that makes it very clear at each step what the model output, what the thought signatures were, what the functions called were, and especially when it comes to the content that's being generated. Earlier you saw a content.type where it was audio and video, and this makes it easy to build these multimodal pipelines in a way that only the Gemini

[9:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=583s) model families can. The second thing that we've launched is Antigravity as a remote agent, as what we like to call Managed Agent. So we've standardized around using the Antigravity harness across all of our Google Suite products. In AI Studio, which you saw just now, websites are built with the Antigravity agent. In the Antigravity application, you're also using the Antigravity agent. Now, this is a very capable harness that we've co-trained Gemini on. And moving forward, I think you're gonna see many gains if you use this harness out of the box. If you wanted to

[10:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=613s) build a coding agent in the past, you would first have to tune the harness, then you had to find a sandbox provider, manage the infrastructure, and then kind of figure out a way how to preserve the context, especially in between runs. But with the Antigravity Managed Agent, you get a persistent sandbox out of the box. With a single API call, you get a sandbox that you can keep hitting and treat as a personal claw without any sort of modifications. You can see an example over here where we wanna analyze what's in a GitHub repository. We then pass it over in an

[10:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=643s) API call over to the Antigravity agent, and it then boots up a remote sandbox and then starts investigating what the repository's about. You can see it calls a list file to get what's in the workspace. It then reads each individual file, and then it generates a final report. All of this is happening in a remote sandbox without any sort of intervention from our side. The agent's able to research, reason, and handle all of this autonomously. We also handle multi-state persistent sandboxes, like what we mentioned earlier, where as long

[11:13](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=673s) as you preserve the environment ID that we return to you, you're able to route it right back to the original sandbox that you had. So these are two primitives that we've given you, the interaction.id, where you're able to preserve and work with context and ensure you're not busting your cache, and the environment ID that enables you to route it to the exact same context. And so this way you can use the same sandbox. You don't need to manage any sort of persistence of state, and all you need to do is just add a few lines of code. The other way we've tried to make it easy is for you to be able to load in sources. A lot of times you might have custom

[11:43](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=703s) dependencies, you might have .EMV files, helper files that you wanna show on the, on this specific sandbox. So we support Google's GCS buckets, we support GitHub repositories, and inline files.

[11:56](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=716s) And so what you see over here is a demo whereby it continues off the previous call that we did. We're passing in the environment ID, and the model has the exact same files right where we left it. Any sort of packages you install, any sort of files you throw and create, the model has access to it through each ev- and every turn as long as you pass us back the interaction ID and the environment ID. This isn't just the sort of constraint to very simple tasks with the Managed Agent. What you're seeing over here is a run where the model burnt over two million tokens trying to analyze a repository that we got for a

[12:26](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=746s) hackathon, where someone built from scratch an entire programming language to code reinforcement learning environments. If you've ever used a scratch programming language, it's a drag and drop language where you can kind of get simple characters to move. But over here with a single API call, we're able to analyze an entire repository that has a custom DSL, a webpage, a whole bunch of like schemas, and an RL training loop running inside HUD. And so it really opens up this kind of opportunities for you to build these very custom and performant agents without you having to handle the underlying

[12:56](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=776s) infrastructure or tune the harness. The best part about using the Antigravity agent is that fundamentally, it's the same agent running in your Antigravity IDE, which means that if you have certain workflows that, uh, you've tuned locally, you have skills that you've made sure are great, you can... Once you're happy with those skills, you just package it into a single folder. We allow you to upload the same way using the sources into s- into a .agents folder, and that gives your agent the exact same skills with the same agent harness and the same prompt running locally and in the cloud.

[13:26](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=806s) When you're ready or when you're comfortable with what your agent is doing locally, you can then push it up, and everything works as intended. The next question we often get from companies are, "Hey, I need my agent to be secure. What am I gonna do if my agent leaks a network credential?" And to do so, we've implemented a man-in-the-middle proxy. So what does this mean? A lot of times you need to make API calls to different providers, say the Gemini API, you might need to download private GitHub repositories, et cetera. We have a proxy that stands in the middle, and so

[13:56](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=836s) any call your model makes, we will look at the header, and we can replace items at will. What you see over here is a case where we'll transform every single outbound call to the GitHub API with... and inject in the token dynamically. And so even if the model is somehow prompt injected and leaks your GitHub API token, it's never actually gonna see the token. It's only gonna execute code that will run against the GitHub API, and your token is never exposed. Once you're comfortable with how the agent is performing, you've

[14:26](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=866s) set it up well, we give you two different ways for you to basically configure and set it up as a named agent, much like you've been using the Antigravity preview agent. The first one is to freeze it with a certain set of configured files and network sources. That's what you're seeing on the left. You... I have a data analyst agent that I'm creating from the Antigravity preview base agent with some base environment configurations, here just a GCS skill. The other way that we allow you to do so is by chatting iteratively with an agent to set up an environment as you like. For example,

[14:56](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=896s) you might want certain dependencies. You might want certain packages. And you can freeze an environment as a specific agent, and all your subsequent calls kinda go through that. Once you're happy with that, it makes it very easy for you to scale up your workloads. At this point, we support up to a thousand named agents, which means that you can have all these different configurations, and you don't pay for any of the storage. You don't pay for, you don't pay for the sandbox. You only pay for the model. We've optimized our infrastructure to be very performant to be able to handle a workload like this, and we're pretty excited to see what people are gonna

[15:26](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=926s) build.

[15:29](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=929s) Recently, we've also out- open sourced a new Gemini API CLI. This is a simple CLI that you can use to get started working with the Gemini models to test out the models locally using simple CLI commands. The other benefit of this is that you're gonna be able to iterate on your co- on your coding agent or your Antigravity agent locally. And once you're ready, it packages it as a single folder and uploads it straight and creates an agent for you.

[15:55](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=955s) Lastly, if you're thinking of switching over to the Interactions API, which is where all of our new models will be on, we've also created a new Interactions API skill that you can feed to your favorite coding agent, and that will then basically allow you to do that migration without any issues. I'm sure you guys have faced issues where you've asked Gemini to implement a specific kind of, uh, implementation, and it keeps using Gemini 2.5 Flash or Gemini 2.0. And so we... what we do is we make sure we keep up, up to date the list of all the different models. We keep up to date all the different

[16:25](https://www.youtube.com/watch?v=8aVbXXvJUY4&t=985s) methods. We provide a good bunch of skills. And like Philip mentioned yesterday, we regularly evaluate these and make sure, make sure that they're actually doing what we want them to do, which is allowing you to use Gemini and build on Gemini well. So that's the end of my talk. Uh, thank you so much for coming by and staying around for the... and the, uh, conference, and yeah. I hope you guys get started. I'll be on the side if you guys have any questions about the new API.
