# Don't Vibe Code Your Agent Infrastructure - Akshat Bubna, Modal

Akshat Bubna, Modal

[Talk page](https://ai.engineer/talks/_0SQh8a1Osk-dont-vibe-code-your-agent-infrastructure) · [Video](https://www.youtube.com/watch?v=_0SQh8a1Osk)

## What It Takes to Put an Agent Behind Every Spreadsheet Cell

Modal CTO Akshat Bubna follows a spreadsheet query down to the sandbox, then shows how coding, accounting and scientific agents change what that sandbox needs to do.

### One portfolio query can launch thousands of agents

[0:24](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=24s)

Consider a financial analyst who wants to run the same complex analysis on every company in a fund’s portfolio. The question may be the same across companies, but the analyst still has to connect to each company’s data sources, inspect the data and perhaps run Python against it. Bubna estimates that doing the portfolio by hand could take days. Modal customer Hebbia puts that work behind a spreadsheet. The analyst can run a query across the sheet, with a different agent doing the work for each cell.

In the architecture Bubna describes, each agent gets a Modal sandbox, a secure environment for running untrusted code. A sandbox used to correspond one-to-one with a container or microVM. Modal now allows multiple containers or microVMs inside one sandbox. That gives the developer a place to draw a security boundary within a single cell’s work. One container runs the harness, the loop that calls the model API for inference, dispatches a tool call and takes the result into the next iteration. The code invoked by the tool can execute in a separate container.

Storage has its own boundaries. Bubna suggests assigning a volume to each spreadsheet row so agents in that row can exchange files while running in separate sandboxes. Within one sandbox, the main container and a sidecar can also share files when an operation requires it. The developer chooses which participants get access. The spreadsheet imposes one more requirement on the whole arrangement. A single query may need thousands of these environments to start quickly, do their work and shut down.

![A task starts the agent harness, which calls the model and sends tool calls to a sidecar that returns results. Both containers mount shared volumes, as do agents in separate sandboxes.](https://ai.engineer/api/talk-essay-media/f056d6da0fc4e640a7231cfbd393ce433efd942058b25e6bd733ba0243617024)

Agent harness, tool execution and shared volumes

### The job determines what belongs in a workspace

[3:15](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=195s)

Modal built its own filesystems, scheduler and runtime for AI workloads. Those components support language-model inference, training, functions and batch processing, as well as sandboxes. For an agent developer, the first question about the sandbox is concrete. What does one agent need within reach to complete its assignment?

Ramp’s internal coding agent, Inspect, needs a GitHub repository, its dependencies and the development services used to work on it. Each agent needs that combination in its workspace. Bubna says he believes Inspect already pushes almost every pull request at Ramp. At Basis, an accounting agent needs the incoming documents, the accounting workpapers and the scripts written to process them. A game-development customer needs its game engine alongside its assets. In each case, providing an interpreter addresses only one part of the job. The code has to run where it can use the services or material it operates on.

### An accountant can take days to reply

[5:04](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=304s)

At Basis, the arrival of documents can trigger an agent to begin work. It processes what it can, then reaches a point where it needs a response from the accountant. The wait might last many days. When the reply arrives, the agent needs to pick up with the documents, workpapers and scripts in the state it left them. Its working state may have to survive indefinitely, although the customer needs compute only while the agent is working.

![A timeline shows compute pausing while the agent waits for an accountant’s input, then resuming. A continuous state bar spans all four stages.](https://ai.engineer/api/talk-essay-media/478db926f1f7fe8d71225b993dfbbda466d85a904e8f3a0986ab9dea922bbc36)

Agents need their work to persist

Bubna distinguishes three persistence choices. A snapshot of the entire root filesystem captures the workspace’s filesystem as a unit for later use. A snapshot of a single directory lets the developer retain a narrower piece, such as `node_modules`, while upgrading the base image separately. A volume provides a shared filesystem when several participants need ongoing access to the same files. Choosing among them depends on what happens after the pause. Recovering an environment, carrying a directory into a newer environment and letting another participant work on the same files are three different requirements.

### Ramp’s CPU and memory peaks vary even within one sandbox

[6:20](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=380s)

With Ramp’s permission, Bubna shares observations from randomly selected Inspect sandboxes. CPU and memory use differ substantially from one sandbox to the next. They also change over time inside the same sandbox. A compilation can drive CPU demand up for part of a coding session. A different operation, such as an analysis that holds a lot of data in memory, can change the RAM requirement.

A second comparison takes the peak usage of each sandbox. Even those peaks vary widely. Bubna estimates that provisioning every Inspect sandbox to cover the observed peaks would mean assigning roughly 15 CPU cores and 15 GB of RAM to each. That figure is a sizing decision someone might make from the peak data. The time series shows why it would be expensive to carry that allocation through every operation. Modal has worked on letting a sandbox obtain resources reliably when demand rises, without requiring the user to request the peak allocation up front. The user pays for those resources for the interval they are needed.

![CPU and memory traces for 57 sandboxes show spikes early in the first 30 minutes. A scatter plot shows how peak CPU and peak memory vary across sandboxes from one app on one day.](https://ai.engineer/api/talk-essay-media/93bd4bcdf19cdb2b4f495e646606425ab163796dfbb40922ff133f5609343e4b)

Agents need elastic resources

### A scientific agent can choose hardware for each experiment

[7:41](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=461s)

Bubna describes a scientific application with a built-in Modal integration. When its agent needs intensive compute, it can offload data preprocessing to a CPU sandbox or start a GPU sandbox to run a scientific model. The operation determines which hardware the agent requests. Bubna also sees demand for GPUs in game design, video editing and CAD. He raises TPUs and AMD hardware as possible future needs.

An experiment in recursive self-improvement illustrates what the harness can do with that choice. Modal gave an agent access to sandboxes to hill-climb toward a model-training target. The agent could change GPUs between runs and launch ablations in parallel to compare experimental variations. It could therefore choose the hardware for an individual trial and decide which trials could proceed at the same time. Bubna reports that the agent reached its target quickly, without giving a timing or a speedup figure.

![A research article heading sits above two charts on a shared time axis. GPU use rises and falls in bursts, peaking above 50. Best BPB drops sharply around 02:00 and declines in smaller steps later.](https://ai.engineer/api/talk-essay-media/dc70ebbeca2316096d5c76cb3b8e4a4415cc53ca64e83b161fc015f3450c6c84)

Autoscaling Autoresearch: Give your agents elastic GPUs on Modal

### Shared files carry both unfinished work and agent memory

[9:03](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=543s)

A subagent may need to work on the same files as the agent that launched it. A human may also be working with those files. In the spreadsheet example, assigning a volume to a row answers who shares the work. The filesystem’s consistency behavior determines when one participant can expect to see an update made by another. Bubna identifies speed and consistency that developers can reason about as priorities for Modal Volumes.

A similar storage requirement appears when an agent maintains memory. It can save files while it works and run background tasks later to revise what it has retained, a practice Bubna calls “dreaming.” The background writer and the agent that consults the revised memory both depend on the filesystem, even when the two activities happen at different times.

### A sidecar proxy can restrict individual GitHub or Datadog operations

[9:57](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=597s)

Bubna imagines an agent accidentally leaking sensitive data to a German forum. The example motivates controls in both directions. On the way in, the operator needs to decide which IP addresses may reach the agent and how a caller authenticates. On the way out, the operator needs to decide which destinations the agent can contact and how it can use credentials. Injecting a credential into an authorized request lets the agent use a service while keeping the secret out of the agent’s reach for a later transfer.

A domain allowlist answers only the destination question. An agent may have a legitimate reason to call GitHub or Datadog, yet need access to only particular paths or operations there. One pattern Bubna sees in production is for developers to write a lightweight proxy with custom rules and run it in a sidecar. The proxy can examine the requested operation and apply the developer’s policy at that finer level. Ramp has used this pattern successfully.

![An agent connects to a sandbox’s main container. A proxy sidecar applies separate egress rules for Datadog, GitHub and Modal. A quote from Ramp’s Zach Bruggeman explains why the team runs its proxy beside the agent.](https://ai.engineer/api/talk-essay-media/03ac01d6fca7d82fa81f8870cc11e8fa89e513ea0df66ed184f4d8baa61a9ad7)

Secure resource access — Ramp’s proxy sidecar rules

### Training logs, GPU utilization and spend belong in the agent loop

[11:20](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=680s)

Suppose an agent launches a training run and the run fails. A failure signal gives it a reason to try again, but little basis for choosing new parameters. If the harness can read the failure logs, it can investigate what happened and use that evidence to choose parameters for another iteration.

Other observations matter before a run ends. The agent can inspect GPU utilization to investigate whether training has stalled, or consult spend to see whether continuing fits the budget. Humans also need access to those observations. As agents dispatch more compute themselves, Modal is working to expose the same evidence to the process making the next execution decision.

![An agent reads logs after a GPU runs out of memory, reduces the batch size, then checks GPU usage and spend on a second run. Beside the diagram, a Modal dashboard shows sandbox counts, startup timing and resource charts.](https://ai.engineer/api/talk-essay-media/1a550c77d385efbd9e897583d6d6d63f6fe9c092b04581577e6327f88d616aff)

Agent observability — diagnosing a GPU training failure and checking spend

### Why Bubna advises against vibe coding a sandbox platform

[12:12](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=732s)

Bubna gives three reasons. The first is reliability. Starting a sandbox on the expected path leaves open what the system will do during a rare failure, including how its own response feeds back into the failure. The second is security. Someone has to keep examining the boundaries around untrusted execution as the runtime and the ways people use it evolve. That responsibility continues well after the first implementation works.

The third is scale. Modal has described the engineering required to reach a million sandboxes. For a developer evaluating a platform, a successful launch demonstrates a different property from being able to launch at the rate an application demands. A personal assistant makes the difference particularly clear.

### A personal assistant needs 10,000 starts a second

[13:32](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=812s)

Bubna first gives the platform-wide storage figures. Modal sees 20 million snapshots per week, processes about eight petabytes of snapshot data per day, and sees users create almost 2.6 million volumes per week. The personal-assistant example is one customer workload with its own compute requirements. It calls for 10,000 sandbox starts per second and close to a million sandboxes running. Each lives for only two or three minutes, but needs to become interactive in less than a second. Startup rate, concurrent capacity, lifetime and time to interaction each describe a different constraint on the platform.

Finding enough CPUs is itself part of the engineering. Bubna says the capacity crunch extends to CPUs as well as GPUs, and Modal cannot source all the CPUs it needs from one cloud provider. It pools capacity from hyperscalers and neoclouds so users can get CPUs across regions when they need them.

### From pets and cattle to an ocean of microbes

[14:54](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=894s)

Bubna expects a clearer separation between the harness, which decides what to do, and the runtime, which performs the work. A harness can start an environment suited to the immediate operation, work across several environments, and snapshot selected ones as artifacts it may need later. In the training example it can create several GPU experiments at once. In the accounting example the valuable artifact can outlive the compute by days.

![A harness installs numpy and uploads input.txt in Sandbox A, then takes a snapshot. Later, the harness creates Sandbox B from the snapshot and runs a Python import. Both sandboxes show numpy and input.txt.](https://ai.engineer/api/talk-essay-media/299f174d4db72bd1bdcfb9accb5f6da20e1cce8f894958bc2c4a299f31da9a5a)

How a harness uses a snapshot across two sandboxes

He extends the familiar progression from pets to cattle. A “pet” was a bespoke service whose operator had to reason about that particular deployment. Containerization and Kubernetes made it practical to run homogeneous servers from templates, scale their numbers up or down and recover when an instance failed. Bubna thinks many agent deployments are at this second stage, scaling copies of one environment.

Bubna borrows the image of an ocean of microbes, with organisms evolving and working together. The practical difference is who chooses the environments and when. Given dependable compute, storage and networking, the harness can decide during execution which environments to create, which should share work and which are worth preserving for the next operation.

## Key takeaways

- Draw the execution boundary and the sharing boundary separately. A tool can run in its own container while selected agents exchange files through a volume.
- Choose persistence by what must survive. A root filesystem snapshot retains the environment, a directory snapshot retains a selected dependency or work tree, and a volume gives collaborators ongoing access.
- Measure resource demand within individual runs as well as across them. Size for bursts with startup rate and time to interaction alongside concurrency.
- Give the harness access to failure logs, utilization and spend so it can make an informed decision about a retry or a running job.

## Slides

### What is Modal?

[3:12](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=192s) [Four green icons represent inference, sandboxes, training and functions: a hollow cube, stacked layers, a stepped block and a cluster of small cubes.](https://assets.ai.engineer/sites/aiecode2025/sha256/ff0d9dee10e92a54814d8e5543330a3a4c54658ee61804642b02154b1f2fd467.webp)

What is Modal?
Infrastructure primitives built for AI

Inference
Sandboxes
Training
Functions

### Agent harness, tool execution and shared volumes

[3:52](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=232s) [A task starts the agent harness, which calls the model and sends tool calls to a sidecar that returns results. Both containers mount shared volumes, as do agents in separate sandboxes.](https://assets.ai.engineer/sites/aiecode2025/sha256/f056d6da0fc4e640a7231cfbd393ce433efd942058b25e6bd733ba0243617024.webp)

MODEL API
TASK / CELL
START TASK

SANDBOX
MAIN CONTAINER
AGENT HARNESS
PLANS THE TASK
CALLS THE MODEL
CHOOSES THE NEXT TOOL

TOOL CALLS
RESULTS

SIDECAR CONTAINER
TOOL EXECUTION
SHELL / PYTHON
FILE OPERATIONS
TASK-SPECIFIC TOOLS

MOUNT
MOUNT

OTHER AGENTS
SEPARATE SANDBOXES
MOUNT

SHARED VOLUMES
DOCUMENTS · WORKING FILES · GENERATED ARTIFACTS
PERSISTENT STATE
COLLABORATION ACROSS AGENTS

### Agent workspaces for coding, accounting and game development

[4:01](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=241s) [Three columns pair Ramp Inspect with a coding workspace, Basis with accounting documents and scripts, and a game controller with a game engine and assets.](https://assets.ai.engineer/sites/aiecode2025/sha256/97a7dcb12e62581730e85595e8a42c15b837edf61a4e900062b27227925f4741.webp)

Agents needs a workspace
SOME EXAMPLES

ramp INSPECT
CODING
Code repo
Dependencies
Dev services (Postgres, Redis, etc)

Basis
ACCOUNTING
Documents
Work papers
Scripts

GAME DEV
Game engine
Assets

### Agents need their work to persist

[5:02](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=302s) [A timeline shows compute pausing while the agent waits for an accountant’s input, then resuming. A continuous state bar spans all four stages.](https://assets.ai.engineer/sites/aiecode2025/sha256/478db926f1f7fe8d71225b993dfbbda466d85a904e8f3a0986ab9dea922bbc36.webp)

Agents need their work to persist

DOCUMENTS ARRIVE
AGENT WORKS
ACCOUNTANT RESPONDS
AGENT CONTINUES

COMPUTE
WAITING FOR INPUT
STATE

### Agents need elastic resources

[6:18](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=378s) [CPU and memory traces for 57 sandboxes show spikes early in the first 30 minutes. A scatter plot shows how peak CPU and peak memory vary across sandboxes from one app on one day.](https://assets.ai.engineer/sites/aiecode2025/sha256/93bd4bcdf19cdb2b4f495e646606425ab163796dfbb40922ff133f5609343e4b.webp)

Agents need elastic resources

57 random sandboxes from the same app, same day
CPU (CORES)
15  10  5  0
MEMORY (GIB)
15  10  5  0
0  10  20  30
MINUTES SINCE SANDBOX START

One app, one day: each dot is a sandbox’s peak CPU and memory
PEAK MEMORY (GIB)
10  1  0.1
1  10
PEAK CPU (CORES)

DATA COURTESY OF ramp INSPECT

### Autoscaling Autoresearch: Give your agents elastic GPUs on Modal

[8:27](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=507s) [A research article heading sits above two charts on a shared time axis. GPU use rises and falls in bursts, peaking above 50. Best BPB drops sharply around 02:00 and declines in smaller steps later.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc70ebbeca2316096d5c76cb3b8e4a4415cc53ca64e83b161fc015f3450c6c84.webp)

Research
Tony Chen   Leon Li   Jason Mancuso   Charles Frye   Rahul Chalamala
Apr 14, 2026

Autoscaling Autoresearch: Give your
agents elastic GPUs on Modal

GPUs in Use
60  50  40  30  20  10  0

Best BPB (log)
1.40  1.35  1.30  1.25  1.20  1.15
1.422  1.402  1.367  1.340  1.159  1.145  1.130
00:00  02:00  04:00  06:00  08:00  10:00  12:00  14:00
Time (EST) →

### Agents need to collaborate — shared workpaper

[9:00](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=540s) [Arrows from an accountant and a parent agent converge on a shared workpaper stored in Modal Volumes. An arrow leads from the workpaper to a subagent.](https://assets.ai.engineer/sites/aiecode2025/sha256/514c1f2cf7b44c6195430cd48ff4a3855d5bd2c2e44e90c84bc12f2bb068d05f.webp)

Agents need to collaborate

ACCOUNTANT
PARENT AGENT

SHARED WORKPAPER
MODAL VOLUMES

SUBAGENT

### Agents need to self-improve — reading and consolidating memories

[9:29](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=569s) [Solid arrows show an agent run reading and updating memories in a Modal Volume. Dashed arrows show a background dreaming process reading those memories and saving consolidated memories.](https://assets.ai.engineer/sites/aiecode2025/sha256/9f51d8d32880f3a48e0c4d78029b95aaa96de6011d6a215875525a432c1bb9b4.webp)

Agents need to self-improve

AGENT RUN
READ MEMORIES
UPDATE MEMORIES

MEMORIES
MODAL VOLUME

DREAMING
BACKGROUND
SAVE CONSOLIDATED MEMORIES
READ MEMORIES

### Secure resource access — network allowlists and credential injection

[9:54](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=594s) [The upper diagram shows an inbound CIDR allowlist and TLS tunnel leading to a sandbox, with a domain allowlist for outbound traffic. Below, the Modal egress proxy replaces a placeholder bearer token for a request to api.example.com.](https://assets.ai.engineer/sites/aiecode2025/sha256/b7e9d42f246b6622ff9c18c6132b62fa619ed987b1fadc79f70e4588dbbf4e1e.webp)

Agents need access to resources, securely

NETWORKING
ALLOWED    BLOCKED

INBOUND
CLIENT
203.0.113.42
CLIENT
192.0.2.42

INBOUND CIDR ALLOWLIST
203.0.113.0/24
TLS TUNNEL
:443 → :8080
HTTP

SANDBOX
AGENT
HTTP SERVER · :8080

OUTBOUND
DOMAIN ALLOWLIST
api.example.com

api.example.com
other.example

CREDENTIAL INJECTION

SANDBOX
GET https://api.example.com/data
Authorization: Bearer placeholder
HTTPS · :443

MODAL SECRET · API-TOKEN
API_TOKEN = ••••••••

MODAL EGRESS PROXY
Match: api.example.com
Replace with Bearer $API_TOKEN
HTTPS · :443

api.example.com
GET /data

### Secure resource access — Ramp’s proxy sidecar rules

[10:37](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=637s) [An agent connects to a sandbox’s main container. A proxy sidecar applies separate egress rules for Datadog, GitHub and Modal. A quote from Ramp’s Zach Bruggeman explains why the team runs its proxy beside the agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/03ac01d6fca7d82fa81f8870cc11e8fa89e513ea0df66ed184f4d8baa61a9ad7.webp)

Agents need access to resources, securely

AGENT
SANDBOX
MAIN CONTAINER
PROXY SIDECAR
EGRESS RULES

DATADOG RULES
ALLOW GET AND POST
TO ALLOWLISTED ENDPOINTS

GITHUB RULES
BLOCK GRAPHQL MUTATIONS
NOT ON ALLOWLIST

MODAL RULES
BLOCK GRPC CALLS
NOT ON ALLOWLIST

“We need to know exactly what our
coding agent is reaching out to, and
be able to stop it when it’s accessing
something it shouldn’t. Sidecars let
us run our own egress proxy, with our
own rules, right next to the agent.”

ZACH BRUGGEMAN
PRINCIPAL SOFTWARE ENGINEER, RAMP

### Agent observability — diagnosing a GPU training failure and checking spend

[11:18](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=678s) [An agent reads logs after a GPU runs out of memory, reduces the batch size, then checks GPU usage and spend on a second run. Beside the diagram, a Modal dashboard shows sandbox counts, startup timing and resource charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/1a550c77d385efbd9e897583d6d6d63f6fe9c092b04581577e6327f88d616aff.webp)

Agents (and humans) need great observability

AGENT
CREATE GPU SANDBOX
START MODEL TRAINING
TRAINING FAILED
READ SANDBOX LOGS
GPU OUT OF MEMORY

GPU SANDBOX A
MODEL TRAINING
FAILED

DEBUG LOGS · REDUCE BATCH SIZE
CREATE A NEW GPU SANDBOX
START MODEL TRAINING
READ HEALTH METRICS
GPU 91% · GPU MEMORY 62%

GPU SANDBOX B
MODEL TRAINING
RUNNING

CHECK TOTAL SPEND AGAINST BUDGET
WITHIN BUDGET
BILLING
$42 SPENT / $100 BUDGET

modal-labs / main
Apps / highfreq-sandbox-synthetic-monitor
Sandboxes
Concurrent sandboxes
224
Live
Total sandboxes
1,637,984
10k
03:00    06:00    09:00

Metrics    Sandboxes
Add filter
Created (EDT)    Time to Started
Oct 13, 00:01:20    –
Oct 13, 00:01:20    –
Oct 13, 00:01:20    –
Oct 13, 00:01:20    425ms
Oct 13, 00:01:20    408ms
Oct 13, 00:01:19    –

Sandbox
sb-01M4SPQQA68ESV2WHSX7CBMHDZ
Internal    Done
Created    Scheduled    Started    Done
0ms    439ms    1.61s
00:01:09    00:01:09    00:01:09    00:01:11

Logs    Metrics    Details    Events
CPU cores used
0.1    0.05
Used    Reserved
Memory used
102M    51M
Used    Reserved
Network
2/s    1/s
Egress    Ingress
CPU pressure
2%    1%

### Why not vibecode your sandboxes? Reliability, security and scale

[12:21](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=741s) [Three panels pair reliability with a GitHub uptime chart, security with a post describing a patched gVisor sandbox escape, and scale with Modal’s post about one million concurrent sandboxes.](https://assets.ai.engineer/sites/aiecode2025/sha256/5700bbf79319841c741c2edb52a51d072477c73fecf3f879b5b741cb198f3803.webp)

Why not vibecode your sandboxes?

Last 90 days uptime
90 days    All time
Updated Oct 7, 2026
71 incidents
GitHub Platform    93.09% uptime
90 days ago    Today
Operational    Maintenance    Minor    Major    Critical
by Marek Šuppa · @mareksuppa
Copy card
RELIABILITY

Asiid Kalam
@odinshell
Sandboxing is now more important than ever.

Here’s a guest to host sandbox escape for gVisor release-20260928

This was our planned submission for Wiz zeroday-cloud-2026. Releasing
it today because it’s now patched in release-20261005

enjoy :)
0:20
5:28 PM · Oct 8, 2026 · 30.9K Views
SECURITY

Modal
@modal
Modal
Scaling to 1 million concurrent
sandboxes in seconds
24    146    921    515K
SCALE

### Agents need massive scale — snapshots and volumes at Modal

[13:30](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=810s) [Three green charts span May through October. Weekly snapshots rise from 4.3M to 20.3M, the snapshot-data curve reaches 182 PB, and weekly volume creations rise from 241K to 2.6M.](https://assets.ai.engineer/sites/aiecode2025/sha256/2496e313057f87c76dceb85105f3b57cff89d20c53b0ab99a957109c28b489ac.webp)

Agents need massive scale
TODAY, AT MODAL

~20M
SNAPSHOTS / WEEK
20M  10M  0M
4.3M  20.3M
MAY  JUN  JUL  AUG  SEP  OCT

~8.4 PB
SNAPSHOT DATA PROCESSED / DAY
200 PB  100 PB  0 PB
182 PB
MAY  JUN  JUL  AUG  SEP  OCT

~2.6M
VOLUMES CREATED / WEEK
3M  2M  1M  0M
241K  2.6M
MAY  JUN  JUL  AUG  SEP  OCT

### Agents need massive scale — one customer's personal assistant workload

[13:58](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=838s) [Four green figures show 10k sandbox creations per second, 800k+ concurrent sandboxes at peak, an average lifetime of about three minutes, and p99 time to interactive under one second. A wireframe cube fills the right side.](https://assets.ai.engineer/sites/aiecode2025/sha256/966f45d89d6a1d951679b1819b835c727f3bec075f54fcdec4fc7f27fa6f13bd.webp)

Agents need massive scale
ONE CUSTOMER WORKLOAD · PERSONAL ASSISTANT

10k
SANDBOX CREATIONS / S

800k+
CONCURRENT SANDBOXES AT PEAK

~3 min
AVERAGE LIFETIME

<1s
P99 TIME TO INTERACTIVE (TTI)

### You can’t rely on a single cloud for CPUs anymore

[14:25](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=865s) [A dark, dotted world map beneath the headline has a few brightly colored points across Europe, Asia, and Africa.](https://assets.ai.engineer/sites/aiecode2025/sha256/16fc29be1d2f018cf6a70aec907b4fe27cca8c7fa561d2d3aef7efaeb7c7d804.webp)

You can’t rely on a single cloud for CPUs anymore

### How a harness uses a snapshot across two sandboxes

[14:52](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=892s) [A harness installs numpy and uploads input.txt in Sandbox A, then takes a snapshot. Later, the harness creates Sandbox B from the snapshot and runs a Python import. Both sandboxes show numpy and input.txt.](https://assets.ai.engineer/sites/aiecode2025/sha256/299f174d4db72bd1bdcfb9accb5f6da20e1cce8f894958bc2c4a299f31da9a5a.webp)

Harnesses are good at making
use of dynamic environments
↓
Separation of “brains” from
“hands” over time

HARNESS

python:3.13-slim
SANDBOX A
CREATE
sb.exec
pip install numpy
STDOUT / STDERR
sb.filesystem.copy_from_local(…)
upload input.txt
numpy
input.txt
SNAPSHOT

SNAPSHOT
LATER

SANDBOX B
CREATE
sb.exec
python -c "import numpy"
STDOUT / STDERR
numpy
input.txt

### Pets → Cattle → Ocean of self-evolving organisms

[16:40](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=1000s) [Green arrows connect three panels. Pets shows three distinct dashboard windows, Cattle shows a regular grid of purple squares, and Ocean shows scattered purple shapes of varying sizes.](https://assets.ai.engineer/sites/aiecode2025/sha256/0804627695f93b6916ae9dedd3e52c8d6a7e6ae791ac3a69fd97dfe6c438c0bd.webp)

Pets → Cattle → Ocean of self-evolving organisms

PETS → CATTLE → OCEAN

## Full transcript

[0:12](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=12s) Hey everyone, I'm Akshat, CTO of Modal, and I'm gonna talk about what do agents really need from their infrastructure.

[0:24](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=24s) So I, I'm told this audience is, uh, very finance adjacent, so I thought I'd talk about what if every cell in a spreadsheet could be backed by an agent. And, uh, imagine you're a financial analyst at a fund. You wanna run a complex analysis on every company in your portfolio. Uh, normally in the past, this would take days. You'd have to

[0:54](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=54s) connect every portfolio company, look at their data sources, uh, run some Python for each of them. Uh, and now obviously with agents, this should be much easier. Well, actually, one of our customers, Hebia, built something that's exactly this. You can go in right now in, in their product. You can, uh, you have a spreadsheet, and you can, uh, run a query where literally every cell is backed by a different

[1:24](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=84s) agent. And I wanted to talk about what does building this really look like. Well, uh, first, every agent is backed by a modal sandbox. Uh, so a sandbox is a, uh, secure environment for running untrusted code. And, uh, it used to be that every sandbox, uh, was one-to-one with a container or

[1:54](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=114s) microVM. That's actually no longer the case. Uh, today actually, uh, Modal has this, uh, feature where you can run multiple containers or microVMs inside a sandbox. And you actually need to do that for doing something like this, where, um, in this architecture, you run the harness. Uh, the harness is the agent loop that does things like running the, calling the model API for inference, uh, running the tool calls, and then looping over

[2:23](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=143s) it. Uh, you wanna separate that from the place where the tool calls actually run, uh, for security and isolation and a bun- bunch of other things. Uh, so you have the sandbox that's running multiple containers, including a sidecar. Uh, you also want shared storage, uh, because you want your agents to be able to talk to each other, and it's up to you how you, uh, stripe it. Like, maybe you have every row, uh, gets the same volume, and agents in that row in the spreadsheet

[2:53](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=173s) can collaborate. Uh, and you also wanna share storage between your main container and the sidecar for some cases. A- and finally, you need all of this to spin up really, really fast because your spreadsheet can have lots of cells. If someone runs a query, you wanna launch thousands of agents, and then you wanna spin them back down when it's over.

[3:15](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=195s) So before I talk more about, uh, that, I just wanted to take a quick aside to talk about what is Modal. Uh, Modal is a platform of primitives that is built for AI applications. Uh, so we've basically built this platform for, uh, from scratch. We've built our own file systems, the scheduling, and, uh, the runtime, uh, to give you really great performance and DevX for use cases like inference, including LM inference, sandboxes, training,

[3:44](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=224s) and, uh, functions, running a lot of batch processing and whatnot. Uh, and today in particular, I'm gonna talk about sandboxes, what this primitive is, and how it's evolving. Uh, so if you wanna build something like this, uh, or even further, and we'll talk more about what that means, uh, what are the primitives you need for that? So the first thing is if you're building an agent, you need a workspace, and I have some examples from some of our current customers. Uh, you might have heard of Ramp Inspect. Uh, Ramp Inspect is

[4:15](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=255s) Ramp's internal coding agent that, uh, I think at this point pushes almost every single PR at Ramp. Uh, and if you're building a coding agent, you need your, uh, repo, uh, your GitHub repo, you need your dependencies, you need your dev services all in the same workspace, uh, for each agent. Uh, another example is Basis. They're an, uh, accounting agent, and if you're building that, you need your, uh,

[4:45](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=285s) documents, you need your accounting work papers, you need scripts you've written, uh, all in one work- workspace. Uh, and we have another, uh, very interesting company. They're a game dev company, and if you're doing game dev with agents, you need your game engine, you need your assets, uh, all in one place.

[5:04](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=304s) The next step is you need these workspaces to persist. Uh, so going back to the accounting example with Basis, what it looks like is your documents might arrive, your, uh, agent is then triggered to work on them, but then it's waiting for a response from the accountant, and you don't know how long this could take. Could take many days. Uh, when the accountant responds, you want the agent to start up again and continue from that last place where things were left

[5:33](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=333s) off. Uh, so you want to persist the state over time, uh, potentially indefinitely. But you don't wanna run the compute the entire time because compute is expensive, so, um, you only want compute to run when the agent is actually working. Uh, this is something we support at Modal, and we've realized that there's a bunch of different flavors of persistence that people need. Uh, sometimes you might want to snapshot your whole root file system. Uh, sometimes you might want to

[6:03](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=363s) snapshot just a single directory. And, uh, maybe it's like your node modules folder, uh, but then you want to upgrade your base image over time. Uh, and sometimes you might need something that looks more like a shared file system, like a volume.

[6:20](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=380s) Um, a very interesting thing, uh, that's been showing up is agents actually need a lot of elasticity and resource consumption. Uh, so we got permission from Ramp to share some of the data from Ramp Inspect. And, um, if you look at random sandboxes running in their, um, workload, they actually all have a huge amount of variance, uh, both, uh, versus each other, but also over time within the same sandbox for how much CPU and memory the agent is

[6:50](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=410s) using. Uh, and it makes sense. I mean, what if you're compiling, um, something that's very resource-intensive or doing some analysis that needs a lot of memory? Your usage is gonna change. And in the second chart, you can see peak usage for each sandbox, and, uh, there's definitely some consistent trends, but it's all over the place. So if you were provisioning this for the peak, you would end up provisioning, uh, like fifteen CPU cores and fifteen gigs of RAM for every

[7:20](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=440s) sandbox, which is just, uh, doesn't make sense economically. Uh, and something we've worked on a lot at Modal is h- um, letting users basically not have to request these resources upfront and then reliably get them when the sandbox needs them, uh, and only pay for them for, for that duration.

[7:41](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=461s) The other interesting emerging thing is agents need a lot more, um, interesting compute. Like, they need GPUs. Over time, they might need TPUs, AMD, who knows? And a great example of this is, uh, Cloud Science, where, um, Cloud Science actually ships with a modal integration out of the box. And, uh, the way it's used is when Cloud needs to do some intensive compute, like preprocess some data on CPUs, uh, or

[8:11](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=491s) spin up GPUs for running a scientific model, uh, it will offload that to a modal sandbox. And we're seeing this show up in things like game design, video editing, uh, three D dev, like CAD. All those things need GPUs, uh, for your agent. And of course, the, the thing that everyone's really excited about right now is RSI or recursive, uh, self-improvement, and, um, th-that's something that GPU sandboxes are really good at and agents are really good at using

[8:41](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=521s) them. Uh, so we have this blog post about what happens when you give an agent, uh, sandboxes to hill climb over training a model, and it turns out they're very good. They can swap GPUs. They can run ablations in parallel, run lots of experiments, and then really get, uh, to the target very quickly.

[9:03](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=543s) The next emerging thing is agents are collaborating a lot more, and, um, that means that you share a file system with your sub-agent. You share file systems and files with your, uh, human, where, uh, you're all collaborating with the same thing. And this needs file systems that are fast, have consistency and, uh, DevX you can reason about. Uh, it's something we're working on with Modal Volumes. Related thing is

[9:33](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=573s) we're seeing a lot more self-improvement. Uh, so agents will, uh, save and update memory. They'll run background tasks, uh, which is now called dreaming, to update this memory over time. And actually, this also just looks like files in a file system. But again, you want really great performance from this file system.

[9:57](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=597s) And the scary thing about agents is they can mess up, and, uh, they don't really know when they might leak your sensitive data to a German forum or something. So you need to have a lot of control over what they can and cannot do, uh, and you need things like controlling what IPs can even talk to the agent, how do you authenticate it, uh, what domains can it talk to. Uh, and also if you're using

[10:27](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=627s) credentials for talking to services, um, basically you want to inject them in a way that the agent cannot actually see them and then exfiltrate them somewhere later.

[10:39](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=639s) And the interesting evolution of this is we've realized that domain, uh, related allow lists are actually, um, not enough. U-- Uh, a lot of our users actually want very specific control over, uh, even, like, what paths can I talk to, uh, on, let's say, Datadog or GitHub. Like, maybe I want to, uh, reason about specific operations that the agent's doing. And going back to sidecars, we've seen this emerging as a pattern where users actually write their own lightweight

[11:08](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=668s) proxy that has custom rules about what exactly it can do. And Ramp's another example of a company that's been using this successfully.

[11:20](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=680s) Uh, and the final underrated thing is it's really important to have really great observability. Uh, so going back to the example of auto research and an agent doing a lot of training, uh, what happens when the training run fails? Uh, your... You, you want your agent to be able to easily look at the logs of the failure, uh, reason about what happened, and do another iter-iteration with a different set of parameters. And while training is running, you want it to be able to look at the metrics. You want to look at GPU

[11:50](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=710s) utilization and make sure it's not stalled. Uh, maybe you want to look at the budget and make sure you're not spending too much in your sandbox. Uh, and obviously, all of this is really important for humans. Uh, but as agents are driving a lot more compute, uh, it's... we're spending a lot of time exposing all of this to the agents themselves so they can reason about all of this.

[12:12](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=732s) All right. I think, uh, we know what we need. Uh, why don't we have Astra build us an agent platform? Well, I, I don't think it's a good idea. I don't think you should vibe code a sandbox platform. Uh, and, and there's three main reasons for that. Uh, number one is reliability. I think we're all painfully aware how hard it is to get right. Uh, certain websites we use every day keep going down. Uh, and the reason for that is even with AI, it's actually really hard

[12:42](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=762s) to reason about the feedback loop of, like, what happens in the case of this tail event when the system breaks. Uh, and this is why you need to be very careful in building systems. Uh, security is another one. Uh, you could vibe code a sandbox platform, but who is gonna do all the work to, like, continually reason about the security boundaries and make sure it's, um, defensible in, in an age when everything is evolving very fast? Uh, and finally, scale. Uh, we have a blog

[13:12](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=792s) post about how we scale to a million sandboxes, which involves a lot of very interesting engineering design and reasoning about systems. And this is, again, something that you kind of don't wanna leave up to a vibe coded platform because when you get to scale, you want it to work. Uh, and on that point, we're, we're actually seeing massive scale for, for a lot of things I mentioned. So, uh, for example, for the snapshots I mentioned, um, it's growing quite fast. We're, we're seeing twenty million

[13:42](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=822s) snapshots a week. Uh, we're processing, uh, about eight petabytes of data, uh, a day, uh, for, for each of these snapshots. And, um, our users are creating, uh, almost two point six million volumes a week, uh, and all of this is growing super fast. And just to take a customer workload for personal assistant, uh, you need to create ten thousand sandboxes a second. You need to run, um, almost a million sandboxes,

[14:12](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=852s) and they're not gonna last very long. They'll just last for, like, two or three minutes. Uh, but when they come up, you want them to be interactive in, in less than a second. That's, that's what this workload really demands.

[14:26](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=866s) And the other interesting thing on scale is you actually cannot get all of the scale from a single cloud platform. The, the world is actually getting crunched on CPUs as well, not just GPUs. Uh, so one thing we spent a lot of time doing is actually aggregating cloud capacity from a bunch of hyperscalers and neo clouds. Uh, so our users have access to CPU capacity across the world when they need it, uh, but we cannot rely on one cloud provider to do that.

[14:54](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=894s) Okay, so where is all this going? I, I hinted at this earlier, but the harnesses are really good at making use of dynamic environments. And what this is leading to is a lot of separation of the brains, which is the harness, uh, from the hands, which is the sandbox or the runtime where stuff is running. And, um, what this means, I have an example here, is the harnesses are going to be able to spin up environments based on what they

[15:24](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=924s) need right now. They're, they, they're good at snapshotting them and keeping them around for, uh, a-as artifacts that they need later, uh, and again, adjusting all of this as necessary. So I, I think this points at an interesting evolution of the pets to cattle, uh, analogy we're all familiar with. Uh, to recap, i-in the olden days, uh, people thought of services like pets. Uh, each one was bespoke. You had to manage

[15:54](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=954s) it. Uh, you had to reason about this individual deployment, uh, specifically. Uh, and then over time, it, um, with containerization and Kubernetes, uh, people started to think about it as, um, homogenous, uh, templates, uh, of, uh, these servers are all the same. You can scale them up and down. If one of them fails, it's not an issue because our system can heal from that. And I think this is where a lot of people deploying agents are today. They're kind of using

[16:24](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=984s) these homogenous environments and scaling up these agents. But, but I think that's really limiting because the models are so good at reasoning about dynamic environments, snapshotting, uh, creating lots of environments when needed, and working across them. Uh, so what, what does the new world look like? One analogy here is it kind of looks like an ocean of microbes, of all these organisms that are self-evolving and working with each other. Uh, and credit Suspix for the

[16:54](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=1014s) analogy here. Um, but one thing I'm really excited about is as you give agents these really rock-solid primitives for compute and storage and networking, they will be able to actually do a lot more, uh, by doing the stuff I talked about, like spinning up lots of environments, working with each other, and being very dynamic. So yeah, I and us at Modal really think that great primitives let your agents do more.

[17:24](https://www.youtube.com/watch?v=_0SQh8a1Osk&t=1044s) And, uh, yeah, if you're interested in talking about this more, reach out to us. Thank you.
