Don't Vibe Code Your Agent Infrastructure - Akshat Bubna, Modal

Loading slides…

Read the talk

What It Takes to Put an Agent Behind Every Spreadsheet Cell

Modal CTO Akshat Bubna follows a spreadsheet query down to the sandbox, then shows how coding, accounting and scientific agents change what that sandbox needs to do.

From a talk by Akshat Bubna

At a glance

Ideas worth remembering

  • Draw the execution boundary and the sharing boundary separately. A tool can run in its own container while selected agents exchange files through a volume.

  • Choose persistence by what must survive. A root filesystem snapshot retains the environment, a directory snapshot retains a selected dependency or work tree, and a volume gives collaborators ongoing access.

  • Measure resource demand within individual runs as well as across them. Size for bursts with startup rate and time to interaction alongside concurrency.

  • Give the harness access to failure logs, utilization and spend so it can make an informed decision about a retry or a running job.

One portfolio query can launch thousands of agents

Consider a financial analyst who wants to run the same complex analysis on every company in a fund’s portfolio. The question may be the same across companies, but the analyst still has to connect to each company’s data sources, inspect the data and perhaps run Python against it. Bubna estimates that doing the portfolio by hand could take days. Modal customer Hebbia puts that work behind a spreadsheet. The analyst can run a query across the sheet, with a different agent doing the work for each cell.

In the architecture Bubna describes, each agent gets a Modal sandbox, a secure environment for running untrusted code. A sandbox used to correspond one-to-one with a container or microVM. Modal now allows multiple containers or microVMs inside one sandbox. That gives the developer a place to draw a security boundary within a single cell’s work. One container runs the harness, the loop that calls the model API for inference, dispatches a tool call and takes the result into the next iteration. The code invoked by the tool can execute in a separate container.

Storage has its own boundaries. Bubna suggests assigning a volume to each spreadsheet row so agents in that row can exchange files while running in separate sandboxes. Within one sandbox, the main container and a sidecar can also share files when an operation requires it. The developer chooses which participants get access. The spreadsheet imposes one more requirement on the whole arrangement. A single query may need thousands of these environments to start quickly, do their work and shut down.

A task starts the agent harness, which calls the model and sends tool calls to a sidecar that returns results. Both containers mount shared volumes, as do agents in separate sandboxes.Open full source frame
Agent harness, tool execution and shared volumes
0:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:24 · section reference included

The job determines what belongs in a workspace

Modal built its own filesystems, scheduler and runtime for AI workloads. Those components support language-model inference, training, functions and batch processing, as well as sandboxes. For an agent developer, the first question about the sandbox is concrete. What does one agent need within reach to complete its assignment?

Ramp’s internal coding agent, Inspect, needs a GitHub repository, its dependencies and the development services used to work on it. Each agent needs that combination in its workspace. Bubna says he believes Inspect already pushes almost every pull request at Ramp. At Basis, an accounting agent needs the incoming documents, the accounting workpapers and the scripts written to process them. A game-development customer needs its game engine alongside its assets. In each case, providing an interpreter addresses only one part of the job. The code has to run where it can use the services or material it operates on.

3:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:15 · section reference included

An accountant can take days to reply

At Basis, the arrival of documents can trigger an agent to begin work. It processes what it can, then reaches a point where it needs a response from the accountant. The wait might last many days. When the reply arrives, the agent needs to pick up with the documents, workpapers and scripts in the state it left them. Its working state may have to survive indefinitely, although the customer needs compute only while the agent is working.

A timeline shows compute pausing while the agent waits for an accountant’s input, then resuming. A continuous state bar spans all four stages.Open full source frame
Agents need their work to persist

Bubna distinguishes three persistence choices. A snapshot of the entire root filesystem captures the workspace’s filesystem as a unit for later use. A snapshot of a single directory lets the developer retain a narrower piece, such as node_modules, while upgrading the base image separately. A volume provides a shared filesystem when several participants need ongoing access to the same files. Choosing among them depends on what happens after the pause. Recovering an environment, carrying a directory into a newer environment and letting another participant work on the same files are three different requirements.

5:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:04 · section reference included

Ramp’s CPU and memory peaks vary even within one sandbox

With Ramp’s permission, Bubna shares observations from randomly selected Inspect sandboxes. CPU and memory use differ substantially from one sandbox to the next. They also change over time inside the same sandbox. A compilation can drive CPU demand up for part of a coding session. A different operation, such as an analysis that holds a lot of data in memory, can change the RAM requirement.

A second comparison takes the peak usage of each sandbox. Even those peaks vary widely. Bubna estimates that provisioning every Inspect sandbox to cover the observed peaks would mean assigning roughly 15 CPU cores and 15 GB of RAM to each. That figure is a sizing decision someone might make from the peak data. The time series shows why it would be expensive to carry that allocation through every operation. Modal has worked on letting a sandbox obtain resources reliably when demand rises, without requiring the user to request the peak allocation up front. The user pays for those resources for the interval they are needed.

CPU and memory traces for 57 sandboxes show spikes early in the first 30 minutes. A scatter plot shows how peak CPU and peak memory vary across sandboxes from one app on one day.Open full source frame
Agents need elastic resources
6:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:20 · section reference included

A scientific agent can choose hardware for each experiment

Bubna describes a scientific application with a built-in Modal integration. When its agent needs intensive compute, it can offload data preprocessing to a CPU sandbox or start a GPU sandbox to run a scientific model. The operation determines which hardware the agent requests. Bubna also sees demand for GPUs in game design, video editing and CAD. He raises TPUs and AMD hardware as possible future needs.

An experiment in recursive self-improvement illustrates what the harness can do with that choice. Modal gave an agent access to sandboxes to hill-climb toward a model-training target. The agent could change GPUs between runs and launch ablations in parallel to compare experimental variations. It could therefore choose the hardware for an individual trial and decide which trials could proceed at the same time. Bubna reports that the agent reached its target quickly, without giving a timing or a speedup figure.

A research article heading sits above two charts on a shared time axis. GPU use rises and falls in bursts, peaking above 50. Best BPB drops sharply around 02:00 and declines in smaller steps later.Open full source frame
Autoscaling Autoresearch: Give your agents elastic GPUs on Modal
7:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:41 · section reference included

Shared files carry both unfinished work and agent memory

A subagent may need to work on the same files as the agent that launched it. A human may also be working with those files. In the spreadsheet example, assigning a volume to a row answers who shares the work. The filesystem’s consistency behavior determines when one participant can expect to see an update made by another. Bubna identifies speed and consistency that developers can reason about as priorities for Modal Volumes.

A similar storage requirement appears when an agent maintains memory. It can save files while it works and run background tasks later to revise what it has retained, a practice Bubna calls “dreaming.” The background writer and the agent that consults the revised memory both depend on the filesystem, even when the two activities happen at different times.

9:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:03 · section reference included

A sidecar proxy can restrict individual GitHub or Datadog operations

Bubna imagines an agent accidentally leaking sensitive data to a German forum. The example motivates controls in both directions. On the way in, the operator needs to decide which IP addresses may reach the agent and how a caller authenticates. On the way out, the operator needs to decide which destinations the agent can contact and how it can use credentials. Injecting a credential into an authorized request lets the agent use a service while keeping the secret out of the agent’s reach for a later transfer.

A domain allowlist answers only the destination question. An agent may have a legitimate reason to call GitHub or Datadog, yet need access to only particular paths or operations there. One pattern Bubna sees in production is for developers to write a lightweight proxy with custom rules and run it in a sidecar. The proxy can examine the requested operation and apply the developer’s policy at that finer level. Ramp has used this pattern successfully.

An agent connects to a sandbox’s main container. A proxy sidecar applies separate egress rules for Datadog, GitHub and Modal. A quote from Ramp’s Zach Bruggeman explains why the team runs its proxy beside the agent.Open full source frame
Secure resource access — Ramp’s proxy sidecar rules
9:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:57 · section reference included

Training logs, GPU utilization and spend belong in the agent loop

Suppose an agent launches a training run and the run fails. A failure signal gives it a reason to try again, but little basis for choosing new parameters. If the harness can read the failure logs, it can investigate what happened and use that evidence to choose parameters for another iteration.

Other observations matter before a run ends. The agent can inspect GPU utilization to investigate whether training has stalled, or consult spend to see whether continuing fits the budget. Humans also need access to those observations. As agents dispatch more compute themselves, Modal is working to expose the same evidence to the process making the next execution decision.

An agent reads logs after a GPU runs out of memory, reduces the batch size, then checks GPU usage and spend on a second run. Beside the diagram, a Modal dashboard shows sandbox counts, startup timing and resource charts.Open full source frame
Agent observability — diagnosing a GPU training failure and checking spend
11:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:20 · section reference included

Why Bubna advises against vibe coding a sandbox platform

Bubna gives three reasons. The first is reliability. Starting a sandbox on the expected path leaves open what the system will do during a rare failure, including how its own response feeds back into the failure. The second is security. Someone has to keep examining the boundaries around untrusted execution as the runtime and the ways people use it evolve. That responsibility continues well after the first implementation works.

The third is scale. Modal has described the engineering required to reach a million sandboxes. For a developer evaluating a platform, a successful launch demonstrates a different property from being able to launch at the rate an application demands. A personal assistant makes the difference particularly clear.

12:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:12 · section reference included

A personal assistant needs 10,000 starts a second

Bubna first gives the platform-wide storage figures. Modal sees 20 million snapshots per week, processes about eight petabytes of snapshot data per day, and sees users create almost 2.6 million volumes per week. The personal-assistant example is one customer workload with its own compute requirements. It calls for 10,000 sandbox starts per second and close to a million sandboxes running. Each lives for only two or three minutes, but needs to become interactive in less than a second. Startup rate, concurrent capacity, lifetime and time to interaction each describe a different constraint on the platform.

Finding enough CPUs is itself part of the engineering. Bubna says the capacity crunch extends to CPUs as well as GPUs, and Modal cannot source all the CPUs it needs from one cloud provider. It pools capacity from hyperscalers and neoclouds so users can get CPUs across regions when they need them.

13:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:32 · section reference included

From pets and cattle to an ocean of microbes

Bubna expects a clearer separation between the harness, which decides what to do, and the runtime, which performs the work. A harness can start an environment suited to the immediate operation, work across several environments, and snapshot selected ones as artifacts it may need later. In the training example it can create several GPU experiments at once. In the accounting example the valuable artifact can outlive the compute by days.

A harness installs numpy and uploads input.txt in Sandbox A, then takes a snapshot. Later, the harness creates Sandbox B from the snapshot and runs a Python import. Both sandboxes show numpy and input.txt.Open full source frame
How a harness uses a snapshot across two sandboxes

He extends the familiar progression from pets to cattle. A “pet” was a bespoke service whose operator had to reason about that particular deployment. Containerization and Kubernetes made it practical to run homogeneous servers from templates, scale their numbers up or down and recover when an instance failed. Bubna thinks many agent deployments are at this second stage, scaling copies of one environment.

Bubna borrows the image of an ocean of microbes, with organisms evolving and working together. The practical difference is who chooses the environments and when. Given dependable compute, storage and networking, the harness can decide during execution which environments to create, which should share work and which are worth preserving for the next operation.

14:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:54 · section reference included

Transcript download
Full transcript

Loading transcript…