7 Things Your AI Coworker Needs - Philipp Schmid, Google DeepMind

Loading slides…

Read the talk

Building Bug, a Gemini coworker for the 8:30 portfolio audit

Philipp Schmid of Google DeepMind builds a finance agent that can run a Python audit on its own computer, update a Google Sheet, and explain three portfolio positions when someone follows up in Slack.

From a talk by Philipp Schmid

At a glance

Ideas worth remembering

  • Write the audit skill through to its destination. Tell the agent when to select it, which script to run, how to interpret the result, and where to record it.

  • Choose the execution location per operation. Bug can run Python in its sandbox, use MCP for a spreadsheet, and return a function call to the application for work that belongs on another machine.

  • Configure credential destinations and command approvals separately. The proxy determines where a real token can be used. A protected hook determines when a matching tool call needs review.

  • Exercise a scheduled job by sending its real input on demand. Follow it through the skill read, service calls, and destination update, then test a follow-up with the thread context the user will actually have.

What a coworker needs to finish a job after you leave

Philipp Schmid starts with a change in what it means to give an AI work. Two years earlier, his examples were questions a model could answer in a chat box, such as naming the capital of France or the winner of the last European Championship. Coding assistants took a further step. They could use tools and write code on the user's computer. The finance coworker Schmid wants to build should be able to take an assignment on its own computer and have a result ready when the person who assigned it comes back hours later.

Schmid borrows seven requirements from onboarding a human colleague. The first two are a brain to do the work and a computer on which to save artifacts and iterate. A capable new hire still needs a playbook for the team's particular workflows. Tools and credentials come separately because knowing which piece of software to use only gets someone as far as its login. House rules specify what the coworker may do independently and where someone else must approve a decision. A schedule tells it when to start.

Seven illustrated requirements appear in two rows. The top row covers a brain, a computer, a playbook and tools. The bottom row covers logins, house rules and a schedule.Open full source frame
What a coworker needs to do the job

The assignment that ties those requirements together is a recurring portfolio audit. An agent has to recognize the procedure, check what changed, and leave its findings somewhere the team already works. Someone should then be able to ask about the findings. Producing the morning spreadsheet and answering the subsequent question are related jobs, but they need different instructions, tools, and context.

0:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Gemini 4 supplies the reasoning, with room for longer outputs

Schmid introduces Gemini 4 Argon, which he says Google announced the previous day. He singles out its one-million-token unified context window. The detail he emphasizes is on the output side. Gemini can generate up to one million tokens, giving it room to work beyond 64K tokens on a long reasoning problem. He reports improvements in mathematics, where working through a proof and putting the result into the required form can both draw on that capacity. His entry point for trying the model is the Interactions API. Get an API key in AI Studio, then send a request, an example he describes as four lines of code.

A Python example sends Gemini 4 Argon a question about portfolio drift and prints the response. Three notes beside the code cover thinking levels, long outputs and token pricing.Open full source frame
One request to Gemini 4 Argon

Google employees had a few weeks of internal access. Schmid credits the model with helping the Chrome team accelerate work to decouple parts of Chromium, helping the Gemma team make its open model more efficient to run, and helping the Cloud ADK team improve its framework. He also reports improvements on Google's internal codebase, without giving a benchmark score or a timing comparison for those projects.

A colleague's experiment makes the kind of assignment Schmid has in mind easier to picture. The colleague prompted Gemini to build a custom 3D ray tracer, and Gemini produced the renderer entirely in Python. The demo paired the renderer with an explanatory voiceover, using Gemini text-to-speech and Lyria for the audio. For the portfolio job, the next question is where a model that can write a program actually gets to run one.

1:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:50 · section reference included

Move execution into a sandbox, then give Bug an agent ID

In a conventional tool-calling loop, an application sends a request to the model and may receive a JSON function call. At that point the application has a description of an operation. It must find the corresponding function, execute it on a machine it controls, and send the result back to the model. If the operation writes a file, the application's execution environment determines where that file goes. The application sits in the middle of the next tool call as well.

The Gemini API's Managed Agents service gives Gemini a remote sandbox where it can execute code and write files directly. Schmid first describes invoking an existing agent. Where an ordinary model request identifies the model, this request identifies an agent, supplies a prompt, and selects the managed sandbox as its remote environment. The caller can initiate work with an API request while the agent uses its own computer for the operations that follow.

Arrows connect the user, Gemini API, Gemini 4 Argon and an Ubuntu sandbox. The API sends context to the model, receives a tool call, executes it in the sandbox and receives results. Three sources feed the sandbox.Open full source frame
How a managed agent works

Schmid then defines his own coworker, Bug. The agent creation API takes an ID, the name callers will later use to address Bug. The definition selects an underlying agent harness, a model, and system instructions. Schmid chooses the same harness used in Antigravity and the new Gemini model. These choices have distinct jobs. Gemini reasons about what to do, the harness lets it act on the remote computer, and the ID gives callers a way to request work from that configured combination.

4:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:26 · section reference included

Divide the playbook by when Bug needs to read it

Skills, hooks, and system instructions all have a file representation in the managed environment. A developer can give Bug a skill they already use with a local agent. Bug can also use its filesystem for knowledge it acquires, saving a lesson from an earlier session or creating a skill for a procedure it expects to repeat. A file on the computer can remain available between tasks without its entire contents having to accompany every request.

Schmid divides the knowledge into three categories. First come system instructions, the material Bug needs on every request. Brevity here has a recurring payoff because the caller pays for those tokens each time. Second come skills that Bug can select for a particular assignment. A short description gives it enough information to recognize when a skill applies. Once it recognizes an audit request, it can open the relevant skill and read the procedure.

Third come dedicated references and scripts for individual steps. Selection alone solves only part of the context problem. If every audit requires opening one dense skill, the model still incurs that cost every time it selects the audit. A shorter procedure can direct Bug to the particular reference it needs or to a script it can execute. That also leaves a Slack follow-up free to use the thread history without carrying the full audit procedure on every turn.

The portfolio-audit skill shows what belongs in the procedure. Its description tells Bug when to use it. Its instructions tell Bug how to run a supplied Python script, what to do with the result, and where to store it. Those last instructions connect a successful calculation to the team's deliverable. The script can finish on the remote computer while the work of recording the result in the spreadsheet is still outstanding.

A portfolio audit skill file highlights the description that tells Buck when to use it and the instruction to run the audit script. A callout reports about 50% better triggering after rewriting the description.Open full source frame
An example skill
6:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:56 · section reference included

Choose among built-in tools, MCP servers, and application functions

Schmid extends the agent creation request to declare Bug's tools. The first option is a built-in capability. Google Search is available through the Gemini API, so enabling it requires no separate search integration. Code execution gives Bug a way to run the audit script on its own computer. These capabilities cover looking something up or doing local computation, while writing the findings into the team's spreadsheet requires access to an external service.

The second option is a Model Context Protocol (MCP) server. Schmid points to Google Workspace's MCP server, which exposes operations for services including Gmail and spreadsheets. For this job, the relevant capability is spreadsheet access. A developer can also supply an MCP server of their own to expose another service or data provider to the agent.

The third option is an application-owned function. Bug generates the function call, the application runs it, and the application sends the function result back. This preserves a useful part of the conventional loop. If a particular operation belongs on a computer the developer controls, the developer can take responsibility for that operation alone. Bug can still run Python and work with its files in the managed sandbox.

Python code shows Buck requesting an MSFT quote. The server runs get_quote and returns the result with the matching call ID, continuing in the same conversation and environment.Open full source frame
Your own functions
8:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:47 · section reference included

Let Bug authenticate without putting the real token on its computer

Credentials are the onboarding step Schmid is most interested in because agents make a familiar integration practice easy to get wrong. Managed Agents accommodates three interfaces. A normal API call can use a bearer token. A consumer-facing integration, including an MCP tool, can use OAuth. A CLI that expects to discover its login in an environment variable can keep using that interface. The developer creates the credential outside the sandbox and defines where and how the agent may use it.

The credential store is write-only from the developer's perspective. Once a value has been written, the interface allows it to be overwritten for rotation but offers no readback. In Schmid's Slack example, the developer supplies a credential and configures its permitted domain and how it will be presented. That configuration matters at request time, when an external network proxy inspects traffic crossing the sandbox boundary.

Consider the environment-variable case from the CLI's perspective. The expected variable is set, so the CLI can read it and build its request as usual. Inside the sandbox, its value is a dummy. When the request reaches the proxy, the proxy checks the destination against the developer's configuration. For a matching domain it substitutes the real credential and forwards the request. The service authenticates it and sends back a result that Bug can use. The CLI has followed its ordinary login path without needing access to the real value.

Schmid illustrates the boundary with two attempts to steal a key. A prompt injection might persuade an agent to print the contents of its .env file. With the proxy arrangement, inspecting the configured variable exposes the dummy. A malicious MCP server might try to collect a Gmail token by getting the agent to send it there. The proxy substitutes the Gmail credential only on requests to its configured domain. Bug can still make authenticated requests to permitted services, and some of those operations may warrant approval.

Inside the sandbox, audit.py sends a placeholder to the egress proxy. The proxy takes the secret from the credential store. A green arrow carries the real token to slack.com, and a dashed red arrow marks the dropped request to attacker.example.Open full source frame
The token never enters the sandbox
9:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:57 · section reference included

Enforce command approvals with protected hook files

For an operation that needs monitoring or human approval, Schmid uses a hook. The developer defines a matcher as a regular expression. The harness checks each proposed tool call against the matcher and runs the associated hook on a match. That gives the developer a place to require a human decision outside the agent's ordinary conversation. Enforcement depends on the harness seeing the specified call, so the matcher must describe the operation the developer intends to guard.

A code panel shows .agents/hooks.json. Two highlighted lines identify the tool-name pattern and the guard.py command. Annotations explain how the command receives the call and what the post hook can do.Open full source frame
One file declares the rules

A hook is a file, just as a skill is, but Bug's permissions differ. Bug may write a new skill as it learns how to carry out a procedure. Its hook files are protected against edits by the agent. That separation allows Bug to improve the instructions it follows while leaving control over which commands need review with the person who configured it.

12:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:54 · section reference included

Attach an input and an environment to a cron schedule

The final requirement is a way to start work without waiting for a person to type a prompt. At the time Schmid demonstrates Bug, Managed Agents supports cron triggers. The developer specifies a schedule and the input the agent should receive. The trigger runs the assignment in the same environment, where the relevant files and tools are available. A periodic due-diligence check and a daily audit can use this mechanism with different inputs and cadences.

In schedule.py, four highlighted settings schedule Buck’s pre-market audit for 08:30 on weekdays in New York, pause it after three consecutive failures, and set a 900-second timeout. A note below says files persist between runs.Open full source frame
Buck clocks in at 8:30

Schmid also sketches an event-driven version under development. An external provider could call a webhook, the agent could perform an initial audit, and it could report to Slack or Microsoft Teams. He identifies external webhooks as work in progress. The portfolio demonstration uses the cron design, with a manual way to exercise it in front of an audience.

13:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:36 · section reference included

Follow one audit from the simulated timer to the Slack thread

Schmid asked Gemini to help him build a browser replica of Google Sheets for the demonstration. He clicks a control in that interface to simulate the scheduled trigger. The click sends a real request to the Managed Agents API. This lets the audience follow the work that an 8:30 a.m. trigger would initiate without waiting for the scheduled time.

Bug's first visible decision is to read the portfolio-audit skill. Once the instructions come back, it works through calls to financial services to determine whether the market or a stock price has changed. Schmid points out an initial result while the run is still in progress, then reports that Bug updates the Google Sheet. The sequence matters for reconstructing the job. The trigger supplies the assignment, the skill supplies the procedure, the services supply the observations, and the spreadsheet is where the result becomes available to the team.

With Slack access, Bug could also post what changed to a designated trading channel. Schmid demonstrates the follow-up by asking for a recommendation. After a short wait, the reply discusses three positions. He attributes that specificity to the previous turns in the Slack thread, which are available to the agent when it answers. The recording gives the count but does not identify the holdings or their price changes. Bug explains what changed and takes no further action, behavior Schmid attributes to its system instructions.

A sequence diagram follows an analyst’s question about NVDA through Slack to the server. The server passes the previous interaction ID and environment to Buck, who reads audit.json and returns an answer to the server.Open full source frame
Every thread is one conversation

The sheet update and the Slack reply show two scopes of autonomy in the same coworker. Bug has a standing assignment to record audit results. A question about those results gives it conversational context for an explanation, while its instructions govern what it does next. The scheduled run, remote execution, and access to the configured services all come through the Managed Agents environment.

14:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:31 · section reference included

Transcript download
Full transcript

Loading transcript…