AI Engineer Summit 2025
Keynote: The AI developer experience doesn't have to suck – why and how we built Modal
Loading slides…
Read the talk
Keynote: The AI developer experience doesn't have to suck – why and how we built Modal
Modal’s edit–run loop connects Python functions to remote GPUs, but making that loop fast requires rethinking container storage, initialization, and shared capacity.
Before you start: Familiarity with Python functions, decorators, and basic container concepts will help you follow the examples.
Write the application, not its infrastructure
How do you deploy models, scale them out, and run large batch jobs without spending your development time managing infrastructure? Erik Bernhardsson, who previously built Spotify’s music recommendation system, describes that as the founding problem for Modal. He started the company during the pandemic, four or five years before this talk, to make building data and machine-learning applications enjoyable again.
The intended user wants to write code: perhaps a custom model, perhaps a workflow around an existing one. Modal is closer to Kubernetes or AWS Lambda than to a fixed-model inference API: it runs arbitrary code and containers, while managing the underlying pool of thousands of GPUs and CPUs. The tradeoff is straightforward. Developers retain control over the application, but must implement more of it themselves. Python is the supported language in the platform described here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast feedback starts below the SDK
A fast edit–run loop depends on fast container startup. Cloud infrastructure provides more powerful machines, but putting those machines behind image builds, uploads, and remote execution can make development feel slower than working locally. Modal’s goal is to take locally authored code, execute it remotely in the right image and on the right hardware, and return feedback quickly enough to preserve the rhythm of local development.
That requirement led to a custom scheduler and file system, part of a multiyear infrastructure effort Bernhardsson describes as still unfinished. The resulting platform has two faces: a resource pool containing H100, A100, L4, and T4 GPUs, and a Python SDK that exposes that pool. Python was the starting point because of its dominance in AI, machine learning, and data work. At the application boundary, a decorator turns a Python function into a serverless function.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One execution platform, many workloads
Bernhardsson identifies generative inference as probably Modal’s largest use case, but the workload range extends well beyond serving language models:
- Diffusion and media: generating music, video, and images.
- Batch processing: processing medical images and applying computer vision to video frames.
- Computational biology: protein folding and other workloads on GPUs or CPUs.
- Language models: fine-tuning, batch embeddings, and inference.
Suno is the concrete customer example: Bernhardsson says it runs much of its AI music inference on Modal. The common requirement is flexible compute under application code, rather than a single prescribed model API.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A square function crosses the cloud boundary
The live demonstration starts with a deliberately small application: square returns a number’s square and writes a message to standard error. The app.function decorator makes it remotely executable; a local entrypoint triggers it from the laptop. The essential structure is:
python
import sys
import modal
app = modal.App("square-demo")
@app.function()
def square(x: int) -> int:
print(f"Squaring {x}", file=sys.stderr)
return x * x
@app.local_entrypoint()
def main():
print(square.remote(12))
The arithmetic stays ordinary Python. The decorator and .remote() call establish where it runs.
With that file saved as square_demo.py, the interactive command is:
bash
modal run square_demo.py
Modal packages the code into a container, executes it in the cloud, and streams output back to the terminal. Bernhardsson then changes a print statement and runs it again. The new code is picked up and the container rebuilt automatically, removing the manual cycle of rebuilding an image, pushing it, and retrieving logs. He describes the experience as nearly local in speed; the demonstration emphasizes iteration rather than presenting a timed comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Request hardware and define its environment
The next change requests an H100 through the function decorator. A100s and T4s are other available choices. Bernhardsson reports getting access to an H100 in a couple of seconds in this demonstration. Merely attaching a GPU does not move Python arithmetic onto it: x * x still does not use the accelerator.
To inspect the device, the function needs PyTorch installed. Modal accepts a Dockerfile or an existing Docker image, but the demonstration defines the environment directly in Python: start with modal.Image.debian_slim(), install torch, and attach the image to the function. Extending the same example gives:
python
import sys
import modal
app = modal.App("square-demo")
image = modal.Image.debian_slim().pip_install("torch")
@app.function(image=image, gpu="H100")
def square(x: int) -> int:
import torch
print(torch.cuda.get_device_name(), file=sys.stderr)
print(f"Squaring {x}", file=sys.stderr)
return x * x
@app.local_entrypoint()
def main():
print(square.remote(12))
torch.cuda.get_device_name() returns the CUDA device’s name. It establishes hardware access without pretending that squaring an integer is useful GPU work. Bernhardsson estimates roughly one second of Torch initialization overhead in this run. That remaining initialization cost becomes important later.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Map the function, then inspect the actual scale
Once a function can run remotely, batch parallelism becomes a change in how it is called. After considering smaller batches, Bernhardsson settles on 10,000 invocations. Replace the local entrypoint with a mapping call and consume its result iterator:
python
@app.local_entrypoint()
def main():
for result in square.map(range(10_000)):
print(result)
Modal fans the work out across containers. During the ramp, Bernhardsson points out five, then six, then eight running containers. He describes hundreds or thousands of GPU containers as possible with a longer run; that larger scale is not reached in this demonstration. The mechanism is the useful part: independent batch inputs become parallel function invocations without application code explicitly provisioning each worker.
The console also prints a link to the application dashboard. There, the run can be inspected through app details, container metrics, and logs; the UI also includes user management. The demonstrated run peaks at 18 containers. Bernhardsson then examines CPU and GPU utilization, a GPU temperature reading of 33°C, and power consumption. These views connect the convenient mapping call to the resources it actually created.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the function available, not the container
Interactive runs are initiated from the laptop. Deployment makes the function persistently addressable from another Python context:
bash
modal deploy square_demo.py
In the recording, Bernhardsson opens a REPL, imports modal, and uses a lookup operation to obtain a remote function handle. The important distinction is between a persistent callable endpoint and a permanently running container. The deployed function remains available even when its execution container has shut down.
The first deployed call incurs a cold start, including Torch import in an H100-backed container. Bernhardsson describes a default 60-second idle retention period before shutdown. A subsequent call can reuse the warm container and is typically faster. The tiny arithmetic workload still wastes the GPU’s compute capacity; it demonstrates deployment and lifecycle behavior, not acceleration. More invocations can create more containers, followed by scale-down.
The same platform also supports distributed file systems mounted into containers for exchanging information, web endpoints, and cron jobs. Remote function calls are therefore one interface to a broader application environment, rather than the only way to organize work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start containers by moving less data
The infrastructure explanation begins with the speed requirement. Bernhardsson says Kubernetes and Docker could not deliver the desired startup behavior for Modal’s internal execution stack, so the team built its own scheduler and surrounding machinery. That does not conflict with accepting Docker images as inputs. The execution system uses gVisor for isolation, alongside existing Linux facilities and cloud tools.
For this explanation, a container can be reduced to two pieces: a root file system, and process isolation that prevents workloads from interfering with one another. The file-system side contains two opportunities to avoid work. First, an image may contain data the application never reads—Perl, manual pages, locale information, or Uzbekistan time-zone data. Second, different images contain many identical files. Bernhardsson’s comparison of three otherwise different images finds extensive overlap. Moving every byte of every image independently wastes both bandwidth and startup time.
Modal uses content-addressed storage to exploit that overlap. An image becomes metadata pointing to blobs; each blob is identified by a checksum or hash, allowing identical content to be deduplicated. Bernhardsson notes that AWS Lambda uses the same broad technique.
| Observation | Storage response | Startup benefit |
|---|---|---|
| Images share identical content | Address blobs by content hash | Reuse cached blobs across images |
| Many files are never read | Load data on access | Avoid unnecessary transfers |
| Images can reference shared blobs | Store image metadata separately | Keep the image description small |
Caching and lazy loading solve different parts of the problem: caching avoids fetching content already nearby, while lazy loading avoids fetching content that is not needed at all.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Predict reads, then bypass initialization
Lazy loading introduces another problem: Python startup can require reading several thousand modules, with many file accesses occurring sequentially. A small delay on each read accumulates. Bernhardsson explains that millisecond-scale accesses repeated a thousand times can add seconds to startup. Saving transfer volume is not enough if each required file introduces another remote round trip.
Modal therefore uses prefetching and task tracing. Historical runs reveal which files were accessed, allowing the system to anticipate those reads on a later start. Bernhardsson describes these techniques as part of the effort to get startup below a second. They preserve demand loading’s avoidance of unnecessary data while reducing the latency of predictable reads.
The next step goes beyond fetching files efficiently. Modal builds its own container images, but also snapshots CPU memory. Instead of loading data and repeating initialization, it can restore the container’s RAM state through gVisor’s checkpoint/restore support. Image caching and memory restoration reinforce one another: one makes the underlying data available cheaply; the other avoids recreating already initialized process state.
Bernhardsson reports starting Stable Diffusion in a couple of seconds despite roughly 5–10 GB of model weights. The talk does not specify the model version, hardware, cache conditions, or latency percentile for that claim. GPU snapshotting is still under investigation at this point in the recording. CPU memory restoration should therefore not be read as a claim that the demonstrated system restores GPU state.
These optimizations sit on a distributed storage system that Bernhardsson says uses R2 and a CDN across multiple regions. Together, the scheduler, file system, caching, prefetching, and snapshots bring the discussion back to the original goal: making remote execution fast enough to support an enjoyable development loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fast startup changes the economics of shared GPUs
Fast cold starts also make serverless resource allocation more practical. Bernhardsson frames the promise as provisioning only what is needed and paying for used capacity. With expensive GPUs, pooling many users and assigning resources dynamically can improve utilization and reduce cost. Pooling also reduces relative demand variance, making aggregate demand more predictable than individual customers’ needs.
The capacity-planning problem moves to the provider. Modal operates thousands of GPUs across cloud vendors and regions, scaling its fleet up and down. Bernhardsson says the system solves a mixed integer programming problem to minimize total capacity spend. A request for 100 GPUs may require acquiring capacity somewhere in the world and starting machines; a maintained buffer can instead satisfy the request immediately. The simple function interface rests on continuous procurement and allocation work beneath it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From an infrastructure platform to an install command
The resulting contract is to run your own code, iterate quickly, and let the provider manage capacity. Bernhardsson describes billing as covering the time containers are active, and says hundreds or thousands of GPUs can become available within seconds or minutes. Active-container billing is not the same as billing only while useful computation runs: retained idle resources can still incur charges. The developer avoids operating an internal platform built around Kubernetes, Docker, and GPU capacity planning.
The talk closes with pip install modal and describes the client as configuring its connection automatically. For a present-day setup, the installation guide also requires an account and authentication:
bash
pip install modal
modal setup
The recording’s Python-only access and REPL lookup belong to its SDK era; current documentation also supports JavaScript/TypeScript and Go clients for calling Functions and managing resources, while Python remains primary for implementing Functions. At the talk, Bernhardsson offers $30 per month in free credits and up to $50,000 in startup credits. Those are the recorded offers, rather than a statement of current startup eligibility. The long infrastructure journey ends at a deliberately small entry point: install the client, write a function, and let the platform supply its execution environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Official documentation for saving and restoring container execution state in gVisor.
Further reading
Engineering explanation of content-addressed images, FUSE, shared caches, and the latency costs of fetching container data.
Explains CPU snapshot restoration, its interaction with lazy loading, and measured startup improvements for example workloads.
A July 2023 systems paper describing Lambda's container storage, caching, deduplication, and demand-loading architecture.
Modal's customer case study describes how Suno moved from batch preprocessing to deployed inference and autoscaling.
Updates since the talk
Current setup instructions and a complete Python example defining an image, requesting a GPU, and running code remotely.
Current snapshot configuration, lifecycle hooks, and limitations, including alpha GPU snapshot support.
Explains queueing, initialization, warm-container settings, and the cost of retaining idle resources.
Full transcript
Loading transcript…