[
  {
    "startMs": 12440,
    "text": "Hey guys, my name's Ivan. I'm on the developer experience team at Google DeepMind, and today we're covering the brand new Interactions API and our new offering, Managed Agents. And so to take a quote from a very famous paper, \"An interaction is all you need.\" Before we move on to what the Interactions API is and what the new Managed Agents offering is, let's take a trip down memory lane, and let's kinda trace how models have changed and how we've worked with models. So when we first started working with models, a lot of times you would have a single interaction. You would send a message, \"Tell me a joke,\" and then you"
  },
  {
    "startMs": 42380,
    "text": "might get a response. For example, \"Why don't scientists trust atoms? Because they make up everything.\" After a while, as models got more and more capable and they understood more about the world, this became a bit more complicated. We wanted to take these models, put them in different applications, but to do so, we needed them to be reliable. And so to do so, we invented function calling, where models could create individual JSON objects with predictable structures. Think like how when you wanna register for a website, your webpage sends a JSON payload to the back end"
  },
  {
    "startMs": 72340,
    "text": "with a username and a password, and that enables the back-end server to say, \"Okay, we have a new user, and now we can actually create a new account.\" That made it possible for us to actually use models in different parts of this application, whether it was extracting user information, parsing data, and more. But right now we're seeing this incredible cusp where models are getting more capable. We're seeing models being able to reason, go for very extended periods of time before they come back to people. And so you're moving from a place where we went from completions, instruction and"
  },
  {
    "startMs": 102240,
    "text": "functions where we might ask, \"Hey, what's the weather in New York City?\" And the model has a call weather tool that it can get that data, ingest it, and generate a response to now a case where models have multiple tools under their belt. They're able to call tools, think and reflect on what the response meant, interact with the environment, and ultimately generate a response. And so this just brings us back to what is an agent? Fundamentally, if you look at what an agent is, it's really a model that powers a huge chunk of it. A lot of"
  },
  {
    "startMs": 132160,
    "text": "people have reduced this to just a language model running in a loop. And for most cases, this is true. If you have a really capable model, you'll find that as we've got models that become more and more capable, a lot of the scaffolding has fallen away. A lot of models, for example, the latest Opus models or Fable models, are now just immediately just using a batch tool instead of individually specialized tools like a edit file or a read file tool like we used to do. And so really what we have is a language model that's used to understand the context, the user's intent,"
  },
  {
    "startMs": 162480,
    "text": "that's given a bunch of different functions, all different resources that it then interacts with its environment in. It has some short-term memory and some long-term memory to keep track of what it's encountered, and then an environment to execute functions in. If you're building a coding agent, but you don't give it an environment where it can actually execute code, for example, it's not gonna be able to do anything. And so if you'd like to try any of our models, the easiest way to do so is actually with AI Studio. And so I actually like to point out that my speaker notes today were actually coded on AI Studio."
  },
  {
    "startMs": 193060,
    "text": "I vibed it out at two minutes before this talk. Uh, you see the, um, as... You can see the conversation history over here. Um, I literally said, \"I need a mobile responsive website that I can pass in speaker notes,\" because I found out I wasn't gonna be able to use presenter view. And so what I did, I just gave it some brief instructions, and before you know it, you have a simple application that you can use without any hassle deployed on a website that's accessible to all. So we have a really generous, uh, free tier. For most of my presentations, I've just been using the free tier."
  },
  {
    "startMs": 223500,
    "text": "I haven't been using my Google unlimited tokens, thankfully. And I think it's the best way for you to try our models, whether it's our new speech-to-speech translation models where you take in some... You speak in, let's say, English, and it can translate to any language, or, you know, you wanna try our latest models like the Gemini 3.5 Flash, et cetera. It makes it very easy for you to do so with simplified UI toggles to just be able to tune all of the different parameters. So let's talk about the Interactions API now. One of the things that we've seen, especially as we've moved from"
  },
  {
    "startMs": 253820,
    "text": "models to a dynamic where now we have models and agents, is that the kind of workloads that we want to support have changed significantly, right? Through the Gemini API, you have Nano Banana, you have, you know, Gemini 3.5 Flash, but you also have very capable agents like the Deep Research agent that's gonna run for anywhere from three minutes and more. You interact with it. You have some sort of research plan. It's gonna go out and do its own thing, and it comes back with very deep and detailed research reports. Now, it's pretty difficult then to"
  },
  {
    "startMs": 283760,
    "text": "think about consuming models and agents through a whole bunch of these different endpoints. Additionally, for a lot of the older kind of API endpoints, a lot of the data was in, in very deeply nested objects. And so you can imagine the kind of difficulties that we had ourselves when we were building in first party integrations, and we wanted these use cases where you might be able to generate a Nano Banana image, do some research, generate a VO image from there, or spawn out all these different agentic, kind of like agents that go out and do their own thing. And so the"
  },
  {
    "startMs": 313700,
    "text": "Interactions API is our sort of solution to this as we look to build an API that can support all of these different use cases and ones that we can build on ourselves as we look to bring Gemini across the entire surface of the Google ecosystem. So the best place to get started is just with server-side state. One of the things that you realize, especially as you use the new Gemini series of models, is that we essentially have this thing called thought signatures, where if you interact with the model, if it calls a function, if it gives a response, it's gonna pass back"
  },
  {
    "startMs": 343860,
    "text": "a very opaque series of these numbers. For a lot of people, manually managing this was very difficult, and sometimes when we talk to different startups, we found that they would lose their cache just with a single white space that they mistakenly added in. So now with the Responses API, you can see on the first turn, \"Hi, my name is Phil.\" And in the response, we give you a interaction ID. And as long as you give us back that same interaction ID under the parameter previous_interaction_id that you see in the second turn, all that context is preserved. And this is"
  },
  {
    "startMs": 373856,
    "text": "really important with the new Gemini series of models because if you don't give us back those thought signatures, you're gonna see a decrease in performance. But what else does this unlock? Here you see a demo built by Vamsi from our team, that did an incredible demo where using a single photo you could generate with Nano Banana all these different variations of yourself in different countries and different locations. Each of these individual photos are a single interaction, but you can think of it from an initial API call, it spawns all these different interaction calls. Using those interactions, the"
  },
  {
    "startMs": 403856,
    "text": "same interaction ID, we then generate a video using the interaction ID and the new Omni model that we just launched. Something like this would not have been possible without the Interactions API built by Philip over there.  So actually, it's a really incredible piece of work, I will say. So to give you an example, an idea of what it would take to build something similar, this is all it would take. First, you pass in Gemini 3.1 Flash-Lite Image, generate an image of me in Venice. You take that interaction ID, you chunk it"
  },
  {
    "startMs": 433876,
    "text": "back into the client.interactions.create, and this time you specify that you want the Omni Flash model instead that we launched on Wednesday. And that allows you to now use the same Omni model with the same information, the same kind of images, and the same context throughout all these calls, making it possible to build more complex, rich, and just these very complex agentic applications. The same thing occurs with audio generation. If you look at the bottom, you see that now we have output.type. In the past, you would have to"
  },
  {
    "startMs": 463956,
    "text": "take this very nested object, like .completions.audio, and it would be very tricky for you to nest it out. I can't even remember it off the top of my head. But now every single bit of the outputs that come out are clearly demarcated by a strong type. If it's an audio, as you see over here, all you gotta do is just parse it and handle it accordingly, right? But what happens if now I wanna generate an image? It's the same method over there, client.interactions.create. All you gotta do is just modify the response modality and the"
  },
  {
    "startMs": 493596,
    "text": "generation config, and now I have an output.type of image, and this makes it very easy for me to start building and experimenting with Gemini. The same thing comes with tool use. You can mix and match different inbuilt tools that wasn't possible before. For example, over here, we're trying to get an agent to look up the internet to find the latest security reports for the React application. We've given it access to the same index that Google has with type Google Search. We've given it the ability to be... essentially retrieve information from web pages using the URL context tool that we ship with, and"
  },
  {
    "startMs": 523636,
    "text": "we've given it a custom tool here called File Incident. The model can reason autonomously on its own, figure out what information it needs, and then actually generate a response to you, all in a single API call. This makes it very powerful because now you have models that not only can work with information at the training cut-off date, but beyond, because now they have access to the entirety of the internet. And all this boils down to the new steps data model that we have, where instead of a world where we have a single message sent to a model and a response, we're gonna have very"
  },
  {
    "startMs": 553436,
    "text": "complex things like async tool calls, models working together, right? And we wanna make sure we have a strong kinda data structure to support this. And so we introduced the Steps Data model, which replaces the original legacy outputs array, with type discriminator that makes it very clear at each step what the model output, what the thought signatures were, what the functions called were, and especially when it comes to the content that's being generated. Earlier you saw a content.type where it was audio and video, and this makes it easy to build these multimodal pipelines in a way that only the Gemini"
  },
  {
    "startMs": 583436,
    "text": "model families can. The second thing that we've launched is Antigravity as a remote agent, as what we like to call Managed Agent. So we've standardized around using the Antigravity harness across all of our Google Suite products. In AI Studio, which you saw just now, websites are built with the Antigravity agent. In the Antigravity application, you're also using the Antigravity agent. Now, this is a very capable harness that we've co-trained Gemini on. And moving forward, I think you're gonna see many gains if you use this harness out of the box. If you wanted to"
  },
  {
    "startMs": 613376,
    "text": "build a coding agent in the past, you would first have to tune the harness, then you had to find a sandbox provider, manage the infrastructure, and then kind of figure out a way how to preserve the context, especially in between runs. But with the Antigravity Managed Agent, you get a persistent sandbox out of the box. With a single API call, you get a sandbox that you can keep hitting and treat as a personal claw without any sort of modifications. You can see an example over here where we wanna analyze what's in a GitHub repository. We then pass it over in an"
  },
  {
    "startMs": 643156,
    "text": "API call over to the Antigravity agent, and it then boots up a remote sandbox and then starts investigating what the repository's about. You can see it calls a list file to get what's in the workspace. It then reads each individual file, and then it generates a final report. All of this is happening in a remote sandbox without any sort of intervention from our side. The agent's able to research, reason, and handle all of this autonomously. We also handle multi-state persistent sandboxes, like what we mentioned earlier, where as long"
  },
  {
    "startMs": 673136,
    "text": "as you preserve the environment ID that we return to you, you're able to route it right back to the original sandbox that you had. So these are two primitives that we've given you, the interaction.id, where you're able to preserve and work with context and ensure you're not busting your cache, and the environment ID that enables you to route it to the exact same context. And so this way you can use the same sandbox. You don't need to manage any sort of persistence of state, and all you need to do is just add a few lines of code. The other way we've tried to make it easy is for you to be able to load in sources. A lot of times you might have custom"
  },
  {
    "startMs": 703156,
    "text": "dependencies, you might have .EMV files, helper files that you wanna show on the, on this specific sandbox. So we support Google's GCS buckets, we support GitHub repositories, and inline files."
  },
  {
    "startMs": 716596,
    "text": "And so what you see over here is a demo whereby it continues off the previous call that we did. We're passing in the environment ID, and the model has the exact same files right where we left it. Any sort of packages you install, any sort of files you throw and create, the model has access to it through each ev- and every turn as long as you pass us back the interaction ID and the environment ID. This isn't just the sort of constraint to very simple tasks with the Managed Agent. What you're seeing over here is a run where the model burnt over two million tokens trying to analyze a repository that we got for a"
  },
  {
    "startMs": 746384,
    "text": "hackathon, where someone built from scratch an entire programming language to code reinforcement learning environments. If you've ever used a scratch programming language, it's a drag and drop language where you can kind of get simple characters to move. But over here with a single API call, we're able to analyze an entire repository that has a custom DSL, a webpage, a whole bunch of like schemas, and an RL training loop running inside HUD. And so it really opens up this kind of opportunities for you to build these very custom and performant agents without you having to handle the underlying"
  },
  {
    "startMs": 776164,
    "text": "infrastructure or tune the harness. The best part about using the Antigravity agent is that fundamentally, it's the same agent running in your Antigravity IDE, which means that if you have certain workflows that, uh, you've tuned locally, you have skills that you've made sure are great, you can... Once you're happy with those skills, you just package it into a single folder. We allow you to upload the same way using the sources into s- into a .agents folder, and that gives your agent the exact same skills with the same agent harness and the same prompt running locally and in the cloud."
  },
  {
    "startMs": 806664,
    "text": "When you're ready or when you're comfortable with what your agent is doing locally, you can then push it up, and everything works as intended. The next question we often get from companies are, \"Hey, I need my agent to be secure. What am I gonna do if my agent leaks a network credential?\" And to do so, we've implemented a man-in-the-middle proxy. So what does this mean? A lot of times you need to make API calls to different providers, say the Gemini API, you might need to download private GitHub repositories, et cetera. We have a proxy that stands in the middle, and so"
  },
  {
    "startMs": 836724,
    "text": "any call your model makes, we will look at the header, and we can replace items at will. What you see over here is a case where we'll transform every single outbound call to the GitHub API with... and inject in the token dynamically. And so even if the model is somehow prompt injected and leaks your GitHub API token, it's never actually gonna see the token. It's only gonna execute code that will run against the GitHub API, and your token is never exposed. Once you're comfortable with how the agent is performing, you've"
  },
  {
    "startMs": 866664,
    "text": "set it up well, we give you two different ways for you to basically configure and set it up as a named agent, much like you've been using the Antigravity preview agent. The first one is to freeze it with a certain set of configured files and network sources. That's what you're seeing on the left. You... I have a data analyst agent that I'm creating from the Antigravity preview base agent with some base environment configurations, here just a GCS skill. The other way that we allow you to do so is by chatting iteratively with an agent to set up an environment as you like. For example,"
  },
  {
    "startMs": 896644,
    "text": "you might want certain dependencies. You might want certain packages. And you can freeze an environment as a specific agent, and all your subsequent calls kinda go through that. Once you're happy with that, it makes it very easy for you to scale up your workloads. At this point, we support up to a thousand named agents, which means that you can have all these different configurations, and you don't pay for any of the storage. You don't pay for, you don't pay for the sandbox. You only pay for the model. We've optimized our infrastructure to be very performant to be able to handle a workload like this, and we're pretty excited to see what people are gonna"
  },
  {
    "startMs": 926584,
    "text": "build."
  },
  {
    "startMs": 929164,
    "text": "Recently, we've also out- open sourced a new Gemini API CLI. This is a simple CLI that you can use to get started working with the Gemini models to test out the models locally using simple CLI commands. The other benefit of this is that you're gonna be able to iterate on your co- on your coding agent or your Antigravity agent locally. And once you're ready, it packages it as a single folder and uploads it straight and creates an agent for you."
  },
  {
    "startMs": 955644,
    "text": "Lastly, if you're thinking of switching over to the Interactions API, which is where all of our new models will be on, we've also created a new Interactions API skill that you can feed to your favorite coding agent, and that will then basically allow you to do that migration without any issues. I'm sure you guys have faced issues where you've asked Gemini to implement a specific kind of, uh, implementation, and it keeps using Gemini 2.5 Flash or Gemini 2.0. And so we... what we do is we make sure we keep up, up to date the list of all the different models. We keep up to date all the different"
  },
  {
    "startMs": 985564,
    "text": "methods. We provide a good bunch of skills. And like Philip mentioned yesterday, we regularly evaluate these and make sure, make sure that they're actually doing what we want them to do, which is allowing you to use Gemini and build on Gemini well. So that's the end of my talk. Uh, thank you so much for coming by and staying around for the... and the, uh, conference, and yeah. I hope you guys get started. I'll be on the side if you guys have any questions about the new API."
  }
]
