AI Engineer Summit 2025

Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

21:38

Loading slides…

Read the talk

Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Modal’s edit–run loop connects Python functions to remote GPUs, but making that loop fast requires rethinking container storage, initialization, and shared capacity.

Before you start: Familiarity with Python functions, decorators, and basic container concepts will help you follow the examples.

Write the application, not its infrastructure

How do you deploy models, scale them out, and run large batch jobs without spending your development time managing infrastructure? Erik Bernhardsson, who previously built Spotify’s music recommendation system, describes that as the founding problem for Modal. He started the company during the pandemic, four or five years before this talk, to make building data and machine-learning applications enjoyable again.

The intended user wants to write code: perhaps a custom model, perhaps a workflow around an existing one. Modal is closer to Kubernetes or AWS Lambda than to a fixed-model inference API: it runs arbitrary code and containers, while managing the underlying pool of thousands of GPUs and CPUs. The tradeoff is straightforward. Developers retain control over the application, but must implement more of it themselves. Python is the supported language in the platform described here.

0:000:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Fast feedback starts below the SDK

A fast edit–run loop depends on fast container startup. Cloud infrastructure provides more powerful machines, but putting those machines behind image builds, uploads, and remote execution can make development feel slower than working locally. Modal’s goal is to take locally authored code, execute it remotely in the right image and on the right hardware, and return feedback quickly enough to preserve the rhythm of local development.

That requirement led to a custom scheduler and file system, part of a multiyear infrastructure effort Bernhardsson describes as still unfinished. The resulting platform has two faces: a resource pool containing H100, A100, L4, and T4 GPUs, and a Python SDK that exposes that pool. Python was the starting point because of its dominance in AI, machine learning, and data work. At the application boundary, a decorator turns a Python function into a serverless function.

Slide titled “We wanted fast feedback loops” links making cloud development feel local to fast container cold starts and a custom container engine and scheduler.Open full source frame
Fast feedback loops require fast container cold starts.
1:502:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:50 · section reference included

One execution platform, many workloads

Bernhardsson identifies generative inference as probably Modal’s largest use case, but the workload range extends well beyond serving language models:

  • Diffusion and media: generating music, video, and images.
  • Batch processing: processing medical images and applying computer vision to video frames.
  • Computational biology: protein folding and other workloads on GPUs or CPUs.
  • Language models: fine-tuning, batch embeddings, and inference.

Suno is the concrete customer example: Bernhardsson says it runs much of its AI music inference on Modal. The common requirement is flexible compute under application code, rather than a single prescribed model API.

Slide titled “People use Modal at large scale today” lists workloads beside a green infrastructure illustration and the speaker’s video.Open full source frame
Modal workloads include generative inference, batch preprocessing, computational biology, and LLM fine-tuning.
3:403:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:40 · section reference included

A square function crosses the cloud boundary

The live demonstration starts with a deliberately small application: square returns a number’s square and writes a message to standard error. The app.function decorator makes it remotely executable; a local entrypoint triggers it from the laptop. The essential structure is:

python

import sys
import modal

app = modal.App("square-demo")

@app.function()
def square(x: int) -> int:
    print(f"Squaring {x}", file=sys.stderr)
    return x * x

@app.local_entrypoint()
def main():
    print(square.remote(12))

The arithmetic stays ordinary Python. The decorator and .remote() call establish where it runs.

With that file saved as square_demo.py, the interactive command is:

bash

modal run square_demo.py

Modal packages the code into a container, executes it in the cloud, and streams output back to the terminal. Bernhardsson then changes a print statement and runs it again. The new code is picked up and the container rebuilt automatically, removing the manual cycle of rebuilding an image, pushing it, and retrieving logs. He describes the experience as nearly local in speed; the demonstration emphasizes iteration rather than presenting a timed comparison.

4:505:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:50 · section reference included

Request hardware and define its environment

The next change requests an H100 through the function decorator. A100s and T4s are other available choices. Bernhardsson reports getting access to an H100 in a couple of seconds in this demonstration. Merely attaching a GPU does not move Python arithmetic onto it: x * x still does not use the accelerator.

To inspect the device, the function needs PyTorch installed. Modal accepts a Dockerfile or an existing Docker image, but the demonstration defines the environment directly in Python: start with modal.Image.debian_slim(), install torch, and attach the image to the function. Extending the same example gives:

python

import sys
import modal

app = modal.App("square-demo")
image = modal.Image.debian_slim().pip_install("torch")

@app.function(image=image, gpu="H100")
def square(x: int) -> int:
    import torch

    print(torch.cuda.get_device_name(), file=sys.stderr)
    print(f"Squaring {x}", file=sys.stderr)
    return x * x

@app.local_entrypoint()
def main():
    print(square.remote(12))

torch.cuda.get_device_name() returns the CUDA device’s name. It establishes hardware access without pretending that squaring an integer is useful GPU work. Bernhardsson estimates roughly one second of Torch initialization overhead in this run. That remaining initialization cost becomes important later.

6:306:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:30 · section reference included

Map the function, then inspect the actual scale

Once a function can run remotely, batch parallelism becomes a change in how it is called. After considering smaller batches, Bernhardsson settles on 10,000 invocations. Replace the local entrypoint with a mapping call and consume its result iterator:

python

@app.local_entrypoint()
def main():
    for result in square.map(range(10_000)):
        print(result)

Modal fans the work out across containers. During the ramp, Bernhardsson points out five, then six, then eight running containers. He describes hundreds or thousands of GPU containers as possible with a longer run; that larger scale is not reached in this demonstration. The mechanism is the useful part: independent batch inputs become parallel function invocations without application code explicitly provisioning each worker.

Terminal displays numeric output, a running container status line, and a progress bar labeled 7239/10000, with the speaker below.Open full source frame
The mapped batch run shows progress at 7239/10000.

The console also prints a link to the application dashboard. There, the run can be inspected through app details, container metrics, and logs; the UI also includes user management. The demonstrated run peaks at 18 containers. Bernhardsson then examines CPU and GPU utilization, a GPU temperature reading of 33°C, and power consumption. These views connect the convenient mapping call to the resources it actually created.

8:178:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:17 · section reference included

Keep the function available, not the container

Interactive runs are initiated from the laptop. Deployment makes the function persistently addressable from another Python context:

bash

modal deploy square_demo.py

In the recording, Bernhardsson opens a REPL, imports modal, and uses a lookup operation to obtain a remote function handle. The important distinction is between a persistent callable endpoint and a permanently running container. The deployed function remains available even when its execution container has shut down.

The first deployed call incurs a cold start, including Torch import in an H100-backed container. Bernhardsson describes a default 60-second idle retention period before shutdown. A subsequent call can reuse the warm container and is typically faster. The tiny arithmetic workload still wastes the GPU’s compute capacity; it demonstrates deployment and lifecycle behavior, not acceleration. More invocations can create more containers, followed by scale-down.

The same platform also supports distributed file systems mounted into containers for exchanging information, web endpoints, and cron jobs. Remote function calls are therefore one interface to a broader application environment, rather than the only way to organize work.

10:4811:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:48 · section reference included

Start containers by moving less data

The infrastructure explanation begins with the speed requirement. Bernhardsson says Kubernetes and Docker could not deliver the desired startup behavior for Modal’s internal execution stack, so the team built its own scheduler and surrounding machinery. That does not conflict with accepting Docker images as inputs. The execution system uses gVisor for isolation, alongside existing Linux facilities and cloud tools.

For this explanation, a container can be reduced to two pieces: a root file system, and process isolation that prevents workloads from interfering with one another. The file-system side contains two opportunities to avoid work. First, an image may contain data the application never reads—Perl, manual pages, locale information, or Uzbekistan time-zone data. Second, different images contain many identical files. Bernhardsson’s comparison of three otherwise different images finds extensive overlap. Moving every byte of every image independently wastes both bandwidth and startup time.

Modal uses content-addressed storage to exploit that overlap. An image becomes metadata pointing to blobs; each blob is identified by a checksum or hash, allowing identical content to be deduplicated. Bernhardsson notes that AWS Lambda uses the same broad technique.

ObservationStorage responseStartup benefit
Images share identical contentAddress blobs by content hashReuse cached blobs across images
Many files are never readLoad data on accessAvoid unnecessary transfers
Images can reference shared blobsStore image metadata separatelyKeep the image description small

Caching and lazy loading solve different parts of the problem: caching avoids fetching content already nearby, while lazy loading avoids fetching content that is not needed at all.

12:3612:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:36 · section reference included

Predict reads, then bypass initialization

Lazy loading introduces another problem: Python startup can require reading several thousand modules, with many file accesses occurring sequentially. A small delay on each read accumulates. Bernhardsson explains that millisecond-scale accesses repeated a thousand times can add seconds to startup. Saving transfer volume is not enough if each required file introduces another remote round trip.

Modal therefore uses prefetching and task tracing. Historical runs reveal which files were accessed, allowing the system to anticipate those reads on a later start. Bernhardsson describes these techniques as part of the effort to get startup below a second. They preserve demand loading’s avoidance of unnecessary data while reducing the latency of predictable reads.

The next step goes beyond fetching files efficiently. Modal builds its own container images, but also snapshots CPU memory. Instead of loading data and repeating initialization, it can restore the container’s RAM state through gVisor’s checkpoint/restore support. Image caching and memory restoration reinforce one another: one makes the underlying data available cheaply; the other avoids recreating already initialized process state.

Bernhardsson reports starting Stable Diffusion in a couple of seconds despite roughly 5–10 GB of model weights. The talk does not specify the model version, hardware, cache conditions, or latency percentile for that claim. GPU snapshotting is still under investigation at this point in the recording. CPU memory restoration should therefore not be read as a claim that the demonstrated system restores GPU state.

These optimizations sit on a distributed storage system that Bernhardsson says uses R2 and a CDN across multiple regions. Together, the scheduler, file system, caching, prefetching, and snapshots bring the discussion back to the original goal: making remote execution fast enough to support an enjoyable development loop.

15:4916:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:49 · section reference included

Fast startup changes the economics of shared GPUs

Fast cold starts also make serverless resource allocation more practical. Bernhardsson frames the promise as provisioning only what is needed and paying for used capacity. With expensive GPUs, pooling many users and assigning resources dynamically can improve utilization and reduce cost. Pooling also reduces relative demand variance, making aggregate demand more predictable than individual customers’ needs.

The capacity-planning problem moves to the provider. Modal operates thousands of GPUs across cloud vendors and regions, scaling its fleet up and down. Bernhardsson says the system solves a mixed integer programming problem to minimize total capacity spend. A request for 100 GPUs may require acquiring capacity somewhere in the world and starting machines; a maintained buffer can instead satisfy the request immediately. The simple function interface rests on continuous procurement and allocation work beneath it.

18:2418:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:24 · section reference included

From an infrastructure platform to an install command

The resulting contract is to run your own code, iterate quickly, and let the provider manage capacity. Bernhardsson describes billing as covering the time containers are active, and says hundreds or thousands of GPUs can become available within seconds or minutes. Active-container billing is not the same as billing only while useful computation runs: retained idle resources can still incur charges. The developer avoids operating an internal platform built around Kubernetes, Docker, and GPU capacity planning.

The talk closes with pip install modal and describes the client as configuring its connection automatically. For a present-day setup, the installation guide also requires an account and authentication:

bash

pip install modal
modal setup

The recording’s Python-only access and REPL lookup belong to its SDK era; current documentation also supports JavaScript/TypeScript and Go clients for calling Functions and managing resources, while Python remains primary for implementing Functions. At the talk, Bernhardsson offers $30 per month in free credits and up to $50,000 in startup credits. Those are the recorded offers, rather than a statement of current startup eligibility. The long infrastructure journey ends at a deliberately small entry point: install the client, write a function, and let the platform supply its execution environment.

Slide titled “How do you try Modal?” shows “pip install modal,” $30/month free credits for all users, and $10k–50k Modal credits for startups.Open full source frame
Getting started with Modal: pip install modal.
20:2120:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:21 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…