AI Engineer World's Fair 2024

Compute & System Design for Next Generation Frontier Models

Loading slides…

Read the talk

Compute & System Design for Next Generation Frontier Models

Larger models demand more than faster GPUs: economical inference depends on batching, workload isolation, and cache reuse, while frontier training exposes failures and stragglers at enormous scale.

From a talk by Dylan Patel

Before you start: Familiarity with tokens, model weights, GPU memory, and the distinction between training and inference will help; the serving mechanisms are explained as they appear.

Why better models can still feel like stagnation

Why can models keep improving without delivering an obvious leap in capability? Dylan Patel opens with a distinction between improving an existing model generation and deploying a substantially larger one. He characterizes GPT-4 Turbo and GPT-4o as smaller models trained longer, extending the generation represented by GPT-4. He makes the same comparison between Claude 3.5 Sonnet and Claude 3 Opus: a smaller model can become better through more training. These are Patel’s explanations of proprietary model development; their parameter counts and training histories are not publicly established by the launch material.

The deployment implications become clearer with his GPT-4 estimates. Patel estimates GPT-4 at 1.8 trillion parameters and almost 600 GFLOP of computation per token. He also mentions 200 billion parameters without explaining its relationship to the larger figure. The practical question for this audience is how to run the next generation: most application builders will not train a frontier model, but they may need to serve one.

Frontier Models slide with three bullets about training requirements and GPT-4, above a scatter plot of training compute by model publication date.Open full source frame
Frontier model scale and training compute.
0:000:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

One request, two resource bottlenecks

Inference begins with prefill, which processes the prompt. Patel estimates roughly one petaFLOP of work for a 2,000-token GPT prompt and 20 petaFLOP for a 32,000-token prompt. These quantities describe computation performed, not a processing rate. Before the user receives an answer, the system may already have done substantial arithmetic.

Then comes decode: generate a token, feed it back into the model, and repeat. This sequential process has a different bottleneck.

PhaseWorkDominant pressure in Patel’s account
PrefillProcess input tokensCompute throughput
DecodeGenerate successive tokensMemory bandwidth

During decode, the accelerators must read model weights from memory. In Patel’s mixture-of-experts example, a sufficiently large batch activates all experts across its requests, requiring the full weight set to be read for each generation step. That is a claim about the batch collectively; it does not mean every individual token uses every expert.

Patel’s serving example requires approximately 60 TB/s of memory bandwidth for 64 users receiving 30 tokens/s each, compared with approximately 3 TB/s for an H100. He treats 30 tokens/s per user as a minimum practical target, while some users want hundreds. The estimate illustrates the bandwidth gap; its exact derivation depends on weight precision, expert activation, cache traffic, and the H100 variant, which he does not specify.

This helps explain why relatively little arithmetic does not make output tokens cheap. Patel describes input prices at roughly one-third or one-quarter of output prices and uses $5 per million input tokens versus $15 per million output tokens as a contemporary example. He groups GPT-4o and Claude 3.5 Sonnet together, but Sonnet’s launch pricing was $3 per million input tokens and $15 per million output tokens. These are historical prices, not current API rates. The next deployment challenge he introduces is the then-upcoming Llama 3 405B, bringing a much larger model within reach of open-model builders.

1:271:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:27 · section reference included

Admit new requests while existing requests keep running

Access to model weights does not automatically produce an economical service. Patel contrasts a straightforward llama.cpp deployment with serving engines such as TensorRT-LLM and vLLM. TensorRT-LLM targets NVIDIA GPUs; vLLM is his example of an open-source engine spanning AMD and Intel hardware as well. One correction matters here: although he calls TensorRT-LLM closed source, NVIDIA’s public-release announcement already described it as open source.

The first essential serving mechanism is continuous batching. Running one request at a time can be reasonable on a personal device, but rented cloud GPUs change the economics. Patel estimates that batch-size-one cloud inference can cost 10–100 times as much as serving at high batch sizes. He supplies no workload, hardware configuration, or latency target for that range; the useful engineering point is that multiple concurrent requests can share expensive model-weight reads.

Here, batch size means the number of concurrent users being served. Requests do not arrive together or finish together, so a useful scheduler must admit work between iterations:

  1. A first user submits a request, and the model starts generating its answer.
  2. Five seconds later, a second request arrives while the first remains unfinished.
  3. The scheduler brings the second request into ongoing inference rather than waiting for the first answer to finish.

Patel urges builders to implement this capability or contribute to an engine that provides it. His criticism of llama.cpp describes the talk’s deployment context; the current server documentation explicitly supports continuous batching, enabled by default. That documentation does not establish when the capability appeared.

3:564:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:56 · section reference included

Separate prompt processing from token generation

At higher volumes, disaggregated prefill, also called disaggregated batching in the talk, separates the two inference phases into accelerator pools. The user still experiences one operation—submit input, receive tokens—but the infrastructure can schedule compute-intensive prompt processing separately from bandwidth-intensive generation. Patel says Google had publicly discussed doing this; he describes OpenAI and Anthropic deployments as his belief, and Together and Fireworks as having hinted at the approach.

Consider his hypothetical Llama 405B worker occupying four chips. Scaling the service need not mean repeatedly adding complete four-chip replicas. Instead, one pool can process prompts and hand the resulting request state to another pool for decoding. Separation does not require different chip models: Patel describes deployments using the same accelerator type for both roles, primarily H100s, while also mentioning A100s and L40s. The distinction is the work assigned to each pool.

The immediate benefit is protection from noisy neighbors. Patel warns that certain query patterns can slow shared inference services, including deliberately disruptive traffic. Users experience the interference through time to first token and uneven generation speed. His example of output jumping from 100 to 30 and back to 100 tokens/s illustrates why average throughput alone does not describe a usable service. Predictable latency and output pacing belong in the service guarantee.

The disruptive customer need not be malicious. A legitimate customer might submit LLM queries for every row in a database, sending 10,000 rows at once. That burst can harm other users sharing the service. Disaggregated prefill is one technique for containing interference while still serving the bulk customer. Patel argues against relying on annoying rate limits as the answer: once a service is public, it must handle demand that its operator does not control. Separation is a scheduling and isolation tool, not a claim that the incoming work disappears.

5:415:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:41 · section reference included

Reuse a document instead of paying to process it repeatedly

Another way to make strong models useful is to avoid retraining them. Patel describes fine-tuning Claude 3.5 Sonnet as unavailable in the talk’s context and estimates that fine-tuning Llama 405B would require dozens of GPUs. He introduces context caching, then newly offered by Google, as a way to supply application data to a powerful model without changing its weights. He describes Google as the only provider offering it at that point.

Gemini 1.5 Pro provides his concrete example: put the relevant data into its two-million-token context rather than fine-tuning a smaller model. Google’s June 2024 announcement documents expanded access to that context window and caching for both Pro and Gemini 1.5 Flash. Patel contrasts this route with more accessible fine-tuning targets—Llama 7B, Mixtral, and Llama 70B—whose capabilities he regards as below the strongest closed models. Supplying context and changing weights are different operations, but context can meet the application’s need to make its data available to the model.

The economic obstacle is repeatedly processing the same long input. Under Patel’s illustrative $5-per-million-input and $15-per-million-output rates, a million-token document costs $5 in input processing before any answer is generated. If the answer is short, input dominates the bill. A small calculator makes that asymmetry explicit:

python

def request_cost(input_tokens: int, output_tokens: int) -> dict[str, float]:
    input_cost = input_tokens * 5 / 1_000_000
    output_cost = output_tokens * 15 / 1_000_000
    return {
        "input_usd": input_cost,
        "output_usd": output_cost,
        "total_usd": input_cost + output_cost,
    }

# A million-token document and an illustrative short answer.
print(request_cost(input_tokens=1_000_000, output_tokens=200))

Legal research, contract review, and other enterprise document queries often have exactly this shape: a large source document followed by a small answer. Reusing the source context therefore targets the dominant recurring expense. Patel expects longer-context open models and broader caching deployment to make this option more widely available.

Inference Requirements – Context Caching slide with four bullets on reusable inputs and document queries, beside a dark pricing panel.Open full source frame
Context caching reuses document inputs, with pricing shown alongside.

The mechanism is to retain the KV cache, the attention keys and values computed for the reusable context, instead of recomputing them on every request. That creates a storage tradeoff: long contexts produce large caches. Patel proposes keeping persistent caches in CPU memory or storage rather than occupying GPU memory continuously, then making the state available when needed. This describes the proposed serving mechanism, not a disclosure of Google’s physical cache placement.

Patel points to vLLM as building this capability and invites contributions or following the project. He contrasts the anticipated long-context models with then-common open-model windows of 4K, 8K, and 32K tokens. As reusable inputs grow, avoiding repeated prefill becomes more valuable—but the system must also accommodate the larger stored state.

9:069:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:06 · section reference included

Beyond the GPT-4 training generation

The talk now moves from serving techniques to forecasts about frontier infrastructure. Patel estimates GPT-4 training used 20,000 chips for 90–100 days and consumed 38 GWh. He names OpenAI, xAI, and Anthropic among the organizations building 100,000-chip clusters, and estimates that a newer cluster of that scale could train GPT-4 in three days. These are his infrastructure estimates, rather than a disclosed training-run comparison.

The change is not just the number of chips. H100 improves on A100, and subsequent NVIDIA hardware adds another generation of capacity. Patel expects those larger systems to support models with tens of trillions of parameters. He places GPT-4 training at roughly 2 × 10²⁵ FLOP, then describes future training budgets as “ten e twenty-six” and “ten e twenty-seven.” His next inference-bandwidth estimate is “two hundred gigabytes or terabytes a second,” leaving the unit unresolved. The forecast is a substantial increase in both training work and serving demands; the spoken figures do not support a precise future hardware budget.

12:0612:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:06 · section reference included

The physical footprint of frontier compute

Patel makes that scale tangible with a displayed image of Microsoft data centers in Arizona. He identifies the site as a GPT-5 training location with approximately 100,000 GPUs and 150 MW of power demand. His loose comparison to tens or hundreds of thousands of homes emphasizes that frontier training is also a power and facilities problem. The site identification and capacity figures are his account of the project.

He then contrasts Musk’s then-current 100,000-GPU cluster construction with a discussed successor containing 300,000 GPUs. Patel estimates the proposed 300,000-GPU successor’s electricity cost at approximately $500 million per year. Power becomes a major recurring expense alongside the cost of building the cluster.

13:2613:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:26 · section reference included

A large cluster must keep training while components fail

At smaller scales, it is convenient to think of a training cluster as a connected fabric in which every GPU can communicate with every other GPU at some available bandwidth. Patel argues that this picture becomes increasingly difficult to realize at extreme scale. The 100,000-GPU clusters he describes already span buildings, with plans for multiple such clusters the following year. Connecting them makes network topology and communication capacity central design constraints.

Those connections involve electrical-to-optical conversion, fiber, and transceivers. A component that lasts years can still contribute to frequent failures when the fleet contains enormous numbers of components. Patel describes optical components with a roughly five-year failure timescale and estimates a failure somewhere every five minutes in clusters containing 100,000–500,000 GPUs. He does not derive that frequency or provide a recovery architecture. Instead, he poses the systems question: how does a long-running training job continue making progress when failures are routine?

14:0814:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:08 · section reference included

A functioning GPU can still slow the whole job

Hardware does not have to fail outright to cause trouble. Nominally identical H100s can run at different speeds—the silicon lottery, familiar from gaming-GPU comparisons. In a large synchronous training job, variation matters because participants repeatedly process data, exchange gradients, and update weights together. A slower participant can leave the others waiting at synchronization points.

Patel illustrates the bottleneck by saying that one participant running 10% slower can make the synchronized workload 10% slower. He then recounts an unnamed ByteDance paper in which one GPU was associated with a 25% slowdown in a 20,000-GPU cluster. In his account, the GPU technically worked and NVIDIA regarded it as acceptable, but it was still a training straggler. The slide compares Sync SGD with Hierarchical SGD and marks stragglers with orange arrows.

Straggler slide with three bullets and diagrams comparing Sync SGD and Hierarchical SGD, using orange arrows to mark stragglers.Open full source frame
Stragglers delay synchronized training steps.

Removing the slow GPU substantially improved performance, Patel reports. The operational lesson is to measure participation in the distributed workload, not merely whether each device passes a basic health check. The exact paper behind his anecdote is not identified here; the related ByteDance and Peking University paper MegaScale does not establish the complete single-GPU, 25%, 20,000-GPU account.

15:3815:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:38 · section reference included

Build for the models these systems might enable

These infrastructure questions eventually return to application design. Much of today’s LLM scaffolding compensates for hallucinations and other capability limits. Patel describes the hope among AGI proponents that increasing compute by 100 times will reduce those problems and enable new applications. His closing scale example combines a cluster costing more than $10 billion with roughly $500 million in annual electricity. The expected improvement in hallucinations is a motivation for that spending, not a demonstrated result in the talk.

The immediate work remains concrete: make models at the Llama 405B scale economical and reliable to serve through capabilities that Patel sees at frontier labs but not yet broadly implemented in open-source systems. At the same time, those labs are building beyond that capability level. Application builders therefore face two moving targets: the serving machinery needed to use the next available model, and the product assumptions that may change when a substantially stronger model arrives.

17:1117:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:11 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…