AI Engineer World's Fair 2025
Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI
Loading slides…
Read the talk
Dream Machine: Scaling to 1m users in 4 days
Luma’s launch surge turned GPU provisioning into a scheduling problem: keeping accelerators fed, sharing scarce capacity fairly, and deploying models across a changing fleet.
From a talk by Keegan McCallum
Before you start: Familiarity with worker queues, GPU inference, and basic Python will help with the architecture and scheduling examples.
When ten times the capacity is still not enough
At 9 AM on the launch morning—June 11, 2024, in Keegan McCallum’s account—Luma announced Dream Machine, its first video model, and waited for sign-ups. The team had allocated about 500 H100 GPUs. Within an hour, requests were accumulating faster than the fleet could process them. The contingency plan was direct: take every available GPU from every provider, SSH into the machines, and start workers pulling from a global queue.
McCallum reports expanding to about 5,000 H100s over six hours, with a backlog approaching 100,000 requests beginning to drain around 2 PM. The launch-day chat shown in the recording captures two queue counters at 14,247 and 70,600.
Then Amit, Luma’s CEO and self-appointed chaos monkey, tweeted that capacity had increased tenfold and invited people back. The queue had fallen to roughly 300; about ten minutes after the tweet, it climbed through 350, 400, and 1,400. Luma commandeered the remaining training cluster—another roughly 4,000 H100s—but McCallum says it barely dented the growing queues. The KEKW laughing emote became a recurring expression of the team’s predicament: even a dramatic capacity increase could immediately attract more demand.
McCallum reports one million users in four days and about half a million videos processed in twelve hours. He compares the adoption milestone with ChatGPT reaching one million users in five days. That comparison concerns user growth; the video count describes Luma’s processing volume, not a comparable inference benchmark.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The workload includes more than a text prompt
Luma describes itself as a foundation model lab pursuing general multimodal intelligence: systems that generate, understand, and eventually operate in the physical world. Video generation is one expression of that ambition. In the Modify Video demonstration, introduced as a feature released the previous day, footage recorded on iPhones becomes the input to a transformation directed by a text prompt. The selected frame shows a basketball player leaping toward a hoop in a blue-lit arena.
The accompanying API pitch is that developers can send raw user prompts and media while Luma handles the prompt engineering needed to return images and videos. McCallum credits Kiran with managing the public API, SDK, and developer relations. The product demonstration and API availability have separate release histories: Luma’s changelog lists Modify Video on June 4, 2025, and its API release on June 24. The API invitation should therefore not be read as establishing that this particular feature was already publicly available through the API on the day of the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The launch stack was easy to start and difficult to operate
The original serving stack consisted of tightly coupled containers. That packaging had a valuable property during the emergency: the team could launch it on raw machines without additional infrastructure dependencies. Luma initially chose NVIDIA Triton Inference Server rather than build a serving system from scratch.
As the system grew, four limitations became consequential:
- Failure coordination: If Triton went down, CPU processes did not necessarily notice. They could continue pulling jobs that would fail.
- Distributed execution: Luma’s video models needed multiple GPUs, and sometimes multiple nodes, to reach acceptable latency. McCallum found the stack poorly suited to that workload. This is a claim about Luma’s video-serving experience; NVIDIA also documents a multi-node TensorRT-LLM deployment path.
- Hardware choice: McCallum cites weak support for non-NVIDIA hardware, naming AMD and Groq as examples.
- Research iteration: Researchers had to learn unfamiliar serving idioms and configuration rituals. The environment was brittle and awkward to develop against.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate media preparation from GPU execution
Luma rebuilt GPU serving around vanilla PyTorch. Chip vendors already invest in supporting the framework, so ordinary PyTorch operations provide a useful starting point for moving models between accelerators. That makes execution portable, not automatically fast: particular operations can still need hardware-specific optimization.
The replacement also decoupled CPU workers from GPU workers. CPU workers prepare jobs and bring their video, image, and other media inputs into the cluster before a GPU needs them. The accelerator should pull ready work, rather than wait for input preparation. Separating these responsibilities lets the CPU side prepare upcoming work while the GPU side runs inference.
A GPU worker can join from an arbitrary provider or VM as long as it can reach Redis and the shared SeaweedFS storage. Tailscale connects those machines to the rest of the architecture. This reduces the work required to add capacity: even on the training cluster, inference workers can start with a command, without specially provisioned machines or another round of parallel SSH orchestration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep preparation from trapping work in the wrong cluster
Decoupling creates a new backpressure problem. Multiple clusters pull from the same global queue. If one cluster’s CPU workers pull too aggressively, they can accumulate jobs waiting for that cluster’s GPUs while GPUs elsewhere could have processed them. Preparation has effectively committed the work to a congested destination.
Luma introduced a dispatch limit on a specific intermediate state: a job has entered a cluster but has not yet been picked up by a GPU. Capping the number of jobs in that state prevents one cluster from taking too much work out of global circulation. The important quantity is not simply how many CPU workers are available; it is how much prepared work the local GPU pool can absorb. Which user’s job should enter that pipeline is a separate fairness problem.
Adding GPUs also has a startup cost. McCallum describes video models containing roughly 10–20 sub-models and traditional autoscaling warmups consuming 10–20 minutes of GPU time loading weights and compiling. Frequent cold starts can therefore spend scarce accelerator time preparing to serve rather than serving.
To handle bursts, Luma built a simple scheduler on Slurm that starts PyTorch workers on the training cluster. It expands the inference GPU pool when demand rises and shrinks it when demand falls. Researchers dislike losing training capacity, but the shared architecture makes that capacity available to users without rebuilding the deployment environment each time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When capacity is finite, priority needs a clock
The resulting system is pull based, with queues submitting to other queues and workers pulling through several stages. McCallum calls queue management the team’s least enjoyable task, invoking colleagues Sourish and Vasu as witnesses. When Ray2, a much larger and more resource-intensive model, arrived, Luma could no longer solve every backlog by adding compute. Further expansion was neither economical nor feasible.
The service had API, enterprise, Unlimited, Plus, Lite, and Free tiers. Processing them strictly in priority order creates work starvation: enough API or enterprise traffic can prevent a Lite job from ever reaching a GPU. McCallum reports users waiting seven to nine hours under this priority starvation. After the team designed an SLO-based alternative, Sourish implemented it in a couple of hours over a weekend.
The usual remedy is aging: increase a job’s priority as it waits. But an aging function still needs a policy. Luma treated that policy as a product decision: how long is it acceptable, in the worst case, for each tier to wait? The product team defines a service-level objective, or SLO, for each tier. McCallum illustrates an API target of a couple of minutes and a Lite target of ten minutes. A configurable threshold—50% of the allowed wait in his example—promotes a job to the front of the queue.
Promotion alone introduced another starvation case. Among jobs approaching their objectives, sorting by absolute waiting time favored jobs with long SLOs. A Lite job could have waited longer in minutes while a newer API job was much closer to missing its own target. Luma’s final ranking compares the fraction of each job’s SLO already consumed:
A larger fraction means greater urgency. The policy compares waiting time against the promise made to that tier, rather than treating a minute as equally consequential for every user.
For a concrete calculation, use a two-minute API objective and a ten-minute Lite objective. The following Python expresses the normalized ranking among jobs that have crossed a 50% promotion threshold; it does not specify the rest of the queue policy.
python
from dataclasses import dataclass
@dataclass(frozen=True)
class Job:
id: str
wait_seconds: int
slo_seconds: int
@property
def urgency(self) -> float:
if self.slo_seconds <= 0:
raise ValueError("SLO must be positive")
return self.wait_seconds / self.slo_seconds
jobs = [
Job("api-a", wait_seconds=90, slo_seconds=120),
Job("lite-b", wait_seconds=360, slo_seconds=600),
]
promoted = sorted(
(job for job in jobs if job.urgency >= 0.5),
key=lambda job: job.urgency,
reverse=True,
)
for job in promoted:
print(f"{job.id}: {job.urgency:.0%} of SLO")
| Job | Wait | SLO | SLO consumed |
|---|---|---|---|
api-a | 90 seconds | 120 seconds | 75% |
lite-b | 360 seconds | 600 seconds | 60% |
Both jobs qualify for promotion. The ranking places api-a first even though lite-b has waited longer. These are illustrative job values; ranking does not mean either job has executed.
McCallum gives a separate example in which an API job waiting one minute receives the same treatment as a Lite job waiting ten minutes. Equal normalized waits in that example imply SLOs with a 1:10 ratio, rather than the earlier two-minute and ten-minute configuration. The mechanism is the same: urgency follows the percentage of the allowed wait. McCallum says this produced intuitive, fair scheduling behavior in practice.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Version the model and its entire Python environment
Luma retained one useful idea from Triton: the model repository. Each model lives in an object-storage folder containing version subfolders. As researchers change code, fine-tune models, or deploy checkpoints, they create immutable versions. A YAML file at the model folder’s root selects the active version. The repository concept is borrowed; the YAML selector and rollout behavior are Luma’s implementation, not Triton’s standard repository configuration.
Each version includes both its checkpoints and its full Python environment, with the dependencies needed to run the model. That makes rollback more precise than selecting an older weights file: workers recover the same environment and checkpoint across otherwise disparate execution environments. McCallum says rollback resolves most of the issues they encounter.
An automated rollout system watches the YAML selections. Updating them causes workers to switch model versions on the fly without restarting. McCallum describes using this mechanism across a fleet of thousands of H100 and AMD GPUs. The deployment operation has moved from manually reaching individual machines to changing which immutable version the fleet should run.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Portable models still need an acceleration team
The first audience question asks what PyTorch portability actually means when every chipset has its own provider and software stack. McCallum’s answer starts with the vendors: they make sure at least a particular PyTorch version works on their hardware. Luma then tries its models on the new chipset. Usually the model runs, but initially it can run more slowly.
Luma has an acceleration team of about ten people, according to McCallum, optimizing low-level PyTorch operations and working directly with chip providers. The tools include the Triton GPU language and compiler, distinct from the Triton Inference Server discussed earlier. PyTorch provides a route to functional execution; efficient execution still requires kernel work and vendor collaboration.
Asked which hardware Luma uses, McCallum identifies close relationships with NVIDIA and AMD. He distinguishes those from exploration of other providers, including Amazon’s chips and Groq. He also points to the announced HUMAIN partnership, describing HUMAIN’s relationship with Groq as another route for exploration. These remarks do not establish production inference on Groq hardware.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Providers supply the Kubernetes clusters
Another question asks whether Luma runs Kubernetes on bare metal. McCallum initially says no, then qualifies the answer: providers supply working Kubernetes clusters, and Luma does not provision the nodes itself. He would need to check individual providers to say whether their underlying machines are VMs or bare metal. He identifies the Amazon setup as not bare metal while leaving the other providers uncertain. The clear operational boundary is who supplies the functioning cluster, not a uniform claim about the physical machines beneath it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Visual understanding already sits inside the application
The final question returns to Luma’s broader multimodal ambitions: has it shipped visual question-answering models? Not yet, McCallum says, although the application already uses image and video understanding internally. Users interact with an agent, and uploaded videos or images are captioned by vision-language models, or VLMs, to enhance the overall prompt.
That internal understanding step is distinct from a publicly shipped VQA offering. At the time of the talk, McCallum says Luma has not released that offering; he hints at possible future work without making a commitment. The generation interface already depends on interpreting the user’s media, even when visual question answering is not exposed as a separate product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The June 2025 announcement explains video transformation, motion preservation, and the Adhere, Flex, and Reimagine controls.
Documents model folders, version directories, configuration files, and supported storage locations.
Distributed storage software with object and file interfaces, setup instructions, and deployment examples.
Explains how Tailscale connects machines across networks using encrypted connections.
Introduces Slurm's cluster resource allocation, job execution, and scheduling components.
Tools and tutorials for writing GPU kernels, distinct from Triton Inference Server.
Further reading
An archived NVIDIA guide to deploying large language models across GPUs and nodes with Kubernetes.
Updates since the talk
Ray2-era API examples for modifying videos with prompts, reference frames, and strength settings; the page also directs readers to newer API documentation.
Full transcript
Loading transcript…