AI Engineer World's Fair 2026

Give the Agent a Budget, Not a Token — Sachin Malhotra, Anthropic

Loading slides…

Read the talk

Give the Agent a Budget, Not a Token

Sachin Malhotra explains how replenishing limits, asymmetric permissions, aggregate monitoring, recovery tests, and infrastructure-stamped identity can give production agents useful autonomy without giving one bad loop unlimited consequences.

From a talk by Sachin Malhotra

At a glance

Ideas worth remembering

  • Replace boolean agent permissions with budgets that account for quantity, speed, reversibility, and observation.

  • Give agents operations whose mistakes fail loudly and remain cheap to correct; require humans for quiet, break-glass actions.

  • Rate-limit every write, vary the allowance by resource context, and keep exceptional bypasses outside agent sessions.

  • Watch aggregate behavior with visible tripwires: individually reasonable actions can collectively reveal duplication or a shared underlying failure.

  • Apply the undo test before granting autonomy. Irreversible or unacceptably damaging operations need a second key held by another actor.

  • Use text to teach intent and deterministic infrastructure to enforce hard limits; neither layer replaces the other.

  • Stamp agent and session identity in a trusted proxy. If callers can claim their own identity, they can reset identity-keyed budgets by renaming themselves.

The cleanup that matched everything

Sachin Malhotra works on Anthropic’s CI team, which maintains systems including test quarantining, merge automation, autoscaling, and merge queues. That production setting is the backdrop for a warning about the familiar agent demo: give a model a “god-token” and a tool list, watch it build and deploy something in seconds, then mistake the successful demonstration for a safe operating model.

One cleanup agent showed what happens next. It listed workloads it believed were no longer useful and prepared to delete them. One pipeline stage then evaluated to nothing, causing its filter to disappear. The resulting selector matched every workload rather than the intended subset. About 200 workloads disappeared in 90 seconds, affecting work associated with roughly 20 engineers.

The agent was not malicious; it was trying to tidy up after itself. Some deleted workloads were long-running training jobs, and Malhotra says some may not have been checkpointed. That uncertainty matters because it leaves the amount of unrecoverable work unmeasured, but the immediate failure is clear: a small selection error could trigger destructive writes at machine speed.

The agent also did nothing its human operator was technically forbidden to do because it used his token. The dangerous step was transferring that authority, unchanged and unbounded, to an unattended system that never tires and can remain confidently wrong. Human onboarding already addresses a similar problem: new engineers receive useful access, escalation paths, and structural barriers around catastrophic actions rather than constant keystroke supervision.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Replace boolean permission with a multidimensional budget

The obvious response—remove delete permission—creates a different failure. Eventually the agent encounters a legitimate deletion and a human returns to pressing Enter on its behalf. A token scope is boolean: the caller either has an operation or does not. Too narrow, and the agent cannot finish useful work; too broad, and a selection bug can become a postmortem.

A budget asks four richer questions: how much can the agent do, how quickly can it do it, what can it undo itself, and who will notice its actions? Malhotra turns those dimensions into three enforceable primitives—asymmetric verbs, rate limits, and tripwires—plus an undo test used to size the other controls.

The first primitive classifies verbs by how failure becomes visible. An incorrect unskip test action makes CI turn red, so the error announces itself and a human can cheaply restore the skip. An incorrect skip test action can leave every check green while allowing a real bug into production. Paging an on-call unnecessarily is likewise noisy and correctable. These operations can look symmetrical in an API while carrying very different failure modes.

That yields an operational rule: agents may perform verbs that fail loudly on a dashboard, while verbs that fail quietly require a human. In the test-quarantine service, the agent may re-enable tests because a wrong decision produces visible failures. Skipping remains a break-glass operation for the on-call, with an audit trail. A proxy—not the agent—stamps the caller identity on both operations, so the agent never controls its own provenance.

What does this asymmetry make visible? The comparison below separates API-level similarity from operational consequences: the safer autonomous verb is the one whose incorrect result creates an observable signal and a cheap correction path.

The same resource, opposite failure visibility

Agent unskips a testfails loudly — If wrong, builds turn red and a human can restore the skip cheaply.
Agent skips a testfails quietly — If wrong, checks can stay green while a bug proceeds toward production.
Autonomous accessdecision — Suitable for the operation whose failure is visible and reversible.
Human second keydecision — Retained for the silent, break-glass operation.
Skipping and unskipping both modify test state, but only the wrong unskip reliably announces itself through red CI.
Read the diagram as text
  • Agent unskips a test. fails loudly — If wrong, builds turn red and a human can restore the skip cheaply.
  • Agent skips a test. fails quietly — If wrong, checks can stay green while a bug proceeds toward production.
  • Autonomous access. decision — Suitable for the operation whose failure is visible and reversible.
  • Human second key. decision — Retained for the silent, break-glass operation.
  • Agent unskips a test → Autonomous access: fails loudly.
  • Agent skips a test → Human second key: fails quietly.
4:234:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:23 · section reference included

Put a ceiling on every write—and let it refill

Rate limits provide the most concrete version of a budget: a ceiling that refills. Each caller receives a limited number of disruptive actions per time window. Requests inside the budget need no approval. Once the caller crosses the line, the next request is rejected with the relevant count; after time passes, capacity returns automatically.

This preserves autonomy without allowing one runaway loop to grow without bound. Every write gets a limit, but the limit reflects context. Deleting workloads in the caller’s own namespace may receive a larger allowance than modifying a shared namespace. The control changes the maximum damage rate rather than removing the operation entirely.

After the deletion incident, a neighboring team built an admission webhook that caps deletes at a fixed number per hour, resource kind, and namespace. A bypass exists for exceptional human operations, but it refuses to activate inside an agent session. The agent can only ask a human to run the command. This keeps routine cleanup self-service, reserves the override for a separate actor, and avoids permanent access-expansion tickets because the ordinary allowance replenishes itself.

8:278:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:27 · section reference included

Use tripwires to learn from aggregate behavior

Allowlists predict in advance what an agent will need. That prediction is static and can become stale before the team has observed much real behavior. For cheap actions, a tripwire takes the opposite approach: let the action occur within enforced limits, record it with a trusted actor identity, and watch aggregate behavior for anomalies.

Rate limits and tripwires therefore solve different problems. The rate limit is enforcement: it caps the pace of writes. The tripwire is detection after the fact: it reveals what the permitted activity added up to. Malhotra calls it a smoke detector rather than a lock, and stresses that a detector nobody sees is useless; crossing the threshold must notify someone who can respond.

A CI investigation agent supplies the concrete example. The team tracked the number of investigation threads it opened per hour for test-job failures. One morning that count rose well above baseline and paged the on-call. Each thread looked reasonable by itself: a job had failed, so the agent opened an investigation. In aggregate, dozens of jobs shared the same error signature because one infrastructure failure was producing the same symptom everywhere.

The response was a small context change rather than a new orchestration system: before opening separate threads, correlate failures across jobs and look for a shared cause. When the pattern recurred, the agent followed that guidance. The observable change was fewer redundant investigations because the agent first grouped matching failures, inferred a common infrastructure problem, and then avoided treating every affected job as an independent incident.

10:0510:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:05 · section reference included

Size autonomy with the undo test

The undo test is not another code-level guard. It asks two questions when choosing the other controls: can the agent restore the previous state by itself, and how bad is the impact if it gets the operation wrong? Asymmetric verbs ask whether failure becomes visible; the undo test asks whether recovery is possible.

If the agent can roll back its change and the possible blast radius is acceptable, the system can log the operation and let it proceed. If either condition fails, the operation needs a second key held by someone else, plus an audit record explaining what happened and why that second actor approved it.

Feature flags show how this can work outside CI. In canary traffic—staging and internal dogfooding—the agent may move a feature from zero to 100 percent, observe filed bugs, and turn it back off. It cannot promote the feature into production. It may propose promotion after canary testing, but a separately scoped production key supplies the second key; no novel authentication system is required.

Day to day, the agent can own the canary rollout loop and ask clarifying questions about the intended audience while each action carries the agent’s identity rather than the human requester’s. The controls compose around distinct questions: asymmetric verbs determine what it may touch, rate limits determine how often, tripwires report what the activity became, and the undo test sets the size of those permissions and limits.

Applied to the opening deletion, a rate limit would have stopped the loop after a few tens of workloads rather than roughly 200. The undo test then observes that a running job in another engineer’s namespace cannot simply be undeleted. Deletion beyond the cap therefore belongs behind a human-held second key. Not every write needs every safeguard; the relevant mix follows from visibility, frequency, reversibility, and impact.

This resembles an onboarding checklist because it answers the same operational questions: what may a new actor touch, how much room does it receive, who signs off on exceptional operations, and how does the team learn whether the arrangement works? The difference is that the checklist must now govern an actor operating continuously at software speed.

12:1312:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:13 · section reference included

Let text explain intent and infrastructure enforce the boundary

The policy lives in two places. Prompts, context files, and Markdown explain intent: why an action is risky, what pattern should be preferred, and how the agent should reason about it. These instructions are cheap to change and can incorporate lessons such as correlating similar failures before opening investigations.

Malhotra reports that one such sentence in a Markdown file works about 80 percent of the time. Text also accumulates and needs pruning, and ultimately remains advice rather than enforcement. The proxy supplies the hard boundary: it sees a delete or a crossed budget and returns 403. It counts, compares, allows, or denies without needing to understand the agent’s explanation.

The two layers compensate for each other’s limits. Text can explain the “why” but cannot guarantee compliance. Infrastructure cannot explain intent, but it is narrow and deterministic, and prompt injection cannot persuade it to ignore a numeric limit. Text shapes what the agent tries to do; infrastructure bounds how wrong execution can go.

11:3412:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:31 · section reference included

Identity must come from outside the request

Every agent session runs beside a proxy. The agent first reads its context files, then sends every outbound call through that proxy. The proxy stamps the action with an identity it already knows. When an agent launches Kubernetes jobs, the cluster writes that identity onto the job as a label, and child jobs inherit it.

Ownership checks, quotas, rate limits, approvals, and tripwires can all key off the same inherited stamp. The proxy does not need to follow every descendant operation itself; identity propagates through the infrastructure. Most importantly, the agent never gets to edit the value on which its safeguards depend.

What does the topology need to guarantee? The diagram below shows why attribution must enter through the proxy and then flow with created resources. One trusted stamp gives downstream systems a stable key for every budget and alert, even when work branches into child jobs.

Trusted identity propagates through agent-created work

Agent sessioncaller — Reads context and requests outbound actions but cannot choose its trusted identity.
Session proxytrust boundary — Holds real credentials and stamps each call with the known agent and session identity.
Kubernetes clusterexecution — Writes the trusted stamp onto the launched job as a label.
Child jobsexecution — Inherit the same identity label from their parent work.
Budgets and safeguardspolicy — Ownership, quotas, rate limits, approvals, and tripwires all key off the stamp.
The proxy assigns identity once; downstream resources inherit it, and safeguards consistently read the same infrastructure-controlled stamp.
Read the diagram as text
  • Agent session. caller — Reads context and requests outbound actions but cannot choose its trusted identity.
  • Session proxy. trust boundary — Holds real credentials and stamps each call with the known agent and session identity.
  • Kubernetes cluster. execution — Writes the trusted stamp onto the launched job as a label.
  • Child jobs. execution — Inherit the same identity label from their parent work.
  • Budgets and safeguards. policy — Ownership, quotas, rate limits, approvals, and tripwires all key off the stamp.
  • Agent session → Session proxy: all outbound calls.
  • Session proxy → Kubernetes cluster: trusted identity stamp.
  • Kubernetes cluster → Child jobs: identity inheritance.
  • Kubernetes cluster → Budgets and safeguards: label supplies policy key.
  • Child jobs → Budgets and safeguards: same inherited policy key.
16:4317:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:43 · section reference included

A self-asserted identity turns limits into suggestions

If the caller may set its own identity header, every budget can be reset by changing the name. After exhausting the allowance for “Sachin,” an agent can present itself as “Sachin two” and receive a fresh budget. The system may appear to have a rate limit, but the caller can route around it, making the control a suggestion.

With the proxy in the path, the agent never declares who it is. The proxy holds the real credentials and stamps calls from infrastructure-known identity. It can also attach a per-session ID, allowing operators to distinguish concurrent sessions and identify which one is overreacting or failing to act as intended.

The final operating model is compact: give agents verbs that fail loudly; keep humans on verbs that fail quietly; put a replenishing ceiling on every write; monitor aggregate behavior; use the undo test to choose caps and second keys; and fix inexpensive behavioral mistakes with text where possible. All of those controls depend on identity supplied by infrastructure rather than claimed in the request. Get that rule right, Malhotra concludes, and the remaining work is tuning.

17:4318:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:43 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…