AI Engineer World's Fair 2024
Unlocking Developer Productivity across CPU and GPU with MAX
Loading slides…
Read the talk
Unlocking Developer Productivity across CPU and GPU with MAX
Chris Lattner explains how MAX combines familiar Python interfaces, an integrated inference stack, and Mojo to reduce the work of deploying and optimizing AI.
From a talk by Chris Lattner
Before you start: Familiarity with Python and basic model inference is helpful; no GPU kernel programming experience is required.
When an endpoint stops being enough
A hosted inference endpoint makes a generative AI prototype unusually easy to build. Choose a model, send a request, and put its response into an application. The harder question comes when that application needs something the endpoint cannot provide: control over where private data goes, integration with internal security, or fine-tuning that keeps critical company information inside the organization.
Customization adds another reason to own more of the system. A proprietary model can incorporate research that better fits a particular application. At scale, inference prices can also justify running the model yourself, while alternative hardware may offer capabilities or economics unavailable through the default provider. Moving beyond the endpoint is a decision about control, customization, cost, and deployment choice. It does not negate the endpoint’s usefulness; it changes how much infrastructure the product needs to own.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Owning inference exposes a fragmented stack
The tools for taking that step have changed. Training frameworks such as PyTorch, TensorFlow, and Caffe gave way to a larger inference ecosystem, including ONNX and TensorRT, followed by a proliferation of generative AI frameworks. Some specialize in a single model. That specialization can be convenient for one deployment, but supporting several models can mean switching technologies repeatedly.
Models then sit inside an already complicated production system. Lattner’s objection is that this fragmentation slows the movement of research into products: an optimization is less useful if adopting it requires another round of integration work. Meanwhile, AI engineers face new models and optimizations every week, pressure to add generative AI across products, and little time to evaluate new hardware. Once a prototype reaches production, per-unit inference costs become a further constraint on scale. The problem is not a shortage of effort from engineers; it is how much infrastructure work surrounds each improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One stack, adopted gradually
Modular approaches this problem with experience from building TensorFlow, PyTorch, LLVM, MLIR, and XLA. Those systems reflect earlier requirements; the proposal here is to apply what their builders learned to a stack organized around generative AI. The goals are ownership of models and data, control over the product, and freedom to choose where it runs.
The architectural choice is one integrated stack rather than a separate solution for every model. Adoption should nevertheless start with familiar Python and PyTorch workflows. Developers can go deeper as needed, without first becoming experts in the infrastructure beneath their applications.
MAX has two parts: a free AI framework and commercial managed services. This talk concentrates on the framework, with a talk-time focus on inference: deploying PyTorch models and supporting generative AI workloads whose cost and complexity make deployment difficult. The managed service is a separate offering, not a prerequisite for understanding the framework’s architecture.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Familiar interfaces over a rebuilt execution layer
Python is the primary entry point, with a C++ option also mentioned. Lattner describes several paths into MAX, depending on how much of the model implementation a developer wants to control.
| Entry path | Intended use |
|---|---|
| PyTorch ecosystem | Bring models through ONNX, TorchScript, or torch.compile |
| Native APIs | Work directly with KV caches, paged attention, and other inference mechanisms |
The first path preserves existing model workflows. The second exposes more of the machinery needed to develop new LLM and generative AI techniques. These are the integration paths presented in the recording, rather than a current API recipe.
Underneath those interfaces, MAX takes a more invasive approach. Rather than building on cuDNN and the corresponding Intel libraries, it replaces that layer with a consistent execution stack. Compiler technology, runtime support, and high-performance kernels arrive together. The intended simplification is below the API as well as at it: developers should not have to assemble or reconcile those layers themselves.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Quantization makes the CPU case concrete
Developer velocity is a major reason to change frameworks, but it is difficult to quantify. Performance gives the architecture a more concrete test. Lattner introduces a recent quantization release involving INT4 and reports roughly 5× faster performance than llama.cpp on cloud CPUs. The displayed comparison names Llama 3, specifies time to first token on AWS c6i.16xlarge, and shows bars labeled 6.7×, 4.8×, and 4.9×. Those conditions matter: this is a particular latency comparison, not a universal multiplier for every model or every stage of inference.
The product implication is that faster inference can change perceived responsiveness and cost. The implementation argument goes further: compiler and kernel work should make quantization a reusable capability, rather than a one-off optimization attached to one model. Llama 3 supplies the recognizable example; the intended benefit is an approach that generalizes beyond it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A device switch does not unify two implementations
GPU programming raises the ambition. The goal is to make it as approachable as programming a CPU in Python, while improving utilization of hardware already being paid for. Lattner speculates that audience workloads may achieve only 30–50% GPU utilization, framing that as roughly 2–3× excess GPU spending. This is a motivating estimate, not a measurement of those workloads. The desired stack must combine ease of use with CUDA-level power, rather than forcing developers to choose between them.
Rebuilding that capability is difficult: NVIDIA’s software represents years of engineering. A slide showing a one-line change from CPU to GPU does not explain how the difficulty was solved. Such a switch can conceal a fork between an Intel MKL implementation and a CUDA implementation. The interface looks consistent, but the underlying systems can behave differently.
MAX’s response is to replace the underlying matrix multiplication, fused attention, and graph machinery so the components fit together across execution targets. The meaningful claim is therefore not the brevity of device selection. It is that a shared implementation approach can make behavior more predictable beneath that selection.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Custom kernels and compiler fusion
That shared stack offers two routes to GPU performance:
- Custom kernels: Advanced developers can write kernels directly against MAX. Lattner presents this as CUDA-level capability with strong developer tools and a programming experience he considers easier than Triton.
- Automatic fusion: Developers who do not want to write kernels can rely on an auto-fusing compiler, reducing the need to create hand-fused implementations for ordinary cases.
The distinction is how much optimization work the developer takes on, while both routes remain part of the same system.
Ease of programming still needs a performance test. For matrix multiplication, MAX is compared with NVIDIA’s cuBLAS library and its more programmable C++ alternative, CUTLASS. The displayed microbenchmark uses an A100, BF16 inputs and outputs, and FP32 accumulation, with throughput plotted across matrix shapes. Lattner describes MAX as meeting or beating both libraries and cites a 30% win, while emphasizing that results depend on the data. This is a kernel-level comparison, not a claim of 30% faster end-to-end inference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Mojo supplies the implementation foundation
Rebuilding the execution stack led Modular down another layer: it built Mojo to implement MAX. Using Mojo is optional for MAX users. Its intended role is to make extending Python possible without requiring a move into C, C++, or Rust, while retaining syntax and habits familiar to Python developers.
Lattner describes a downloadable toolchain, Visual Studio Code support, and Linux, Mac, and Windows availability. The historical scope needs two distinctions: the March 2024 open-source release covered the Mojo standard library, not the whole compiler; Windows use was through WSL, with native support still forthcoming in Modular’s 2026 account. He also cites 200,000 people and 20,000 in Discord, without defining the first population.
The design separates Python’s strengths from its implementation. Its developers, libraries, community, ecosystem, and familiar packaging practices are assets to preserve. Mojo combines that familiarity with MLIR and compiler technology intended for a different execution model. Python-like code consequently need not inherit the performance characteristics developers associate with interpreted Python.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What fast ordinary control flow changes
Lattner reports users finding 100–1,000× speedups over Python in unspecified cases, with even larger outliers. Without workloads or measurement protocols, those figures do not establish a general language speedup. The more useful consequence is what developers can write: when performance matters, Python programmers often avoid putting hot loops in Python itself.
The concrete example is tokenization code from Modular’s Llama 3 implementation. It uses a standard algorithm with linked-list traversal, conditionals, and loops. These are ordinary programming constructs, but they matter because token processing is not simply a large matrix operation that can always be handed off to a library. A small Mojo scan illustrates the same freedom to express work directly, here counting adjacent repeated token IDs:
mojo
fn count_adjacent_repeats(tokens: List[Int]) -> Int:
var count = 0
for i in range(1, len(tokens)):
if tokens[i] == tokens[i - 1]:
count += 1
return count
fn main():
var tokens = List[Int](17, 17, 42, 42, 42)
print(count_adjacent_repeats(tokens))
The scan has three matching adjacent pairs. The talk’s tokenizer is more involved—it follows linked structures and merges tokens—but the programming point is the same: loops and branches can remain in a language that feels familiar to Python developers.
This changes the boundary between application work and systems work. Developers who need deeper control can pursue it without immediately switching to C or C++. Lattner connects that capability to lower execution cost, researchers’ ability to modify the full stack, and teams with fewer boundaries between language specialists. He also points to a Modular blog comparison with Rust; that is a claim about particular implementations and workloads, not evidence that one language is universally faster.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From framework adoption to deployment
The adoption decision returns to the requirements that made an endpoint insufficient. Data control, internal security, customization, cost, and hardware portability determine whether owning inference is worth the additional responsibility. MAX is presented as a free, downloadable framework for that path. For production, Modular also offers deployment support on Kubernetes and SageMaker, although the recording does not walk through a deployment procedure.
GPU availability is a roadmap statement in this recording: Lattner forecasts an official September launch and invites developers to seek early access through Discord. That forecast should not be read as a completed release; September’s MAX 24.5 announcement still described a CPU-only release, and the GPU technology preview followed in December. The closing invitation is to explore Modular’s tools, the source and examples on GitHub, and its developer community—starting with the framework and going deeper into the implementation when the product requires it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current home of Modular’s open-source components, MAX and Mojo examples, and getting-started links.
Further reading
Historical release details for quantization, Llama implementations and MAX on Apple silicon.
Related AWS Summit deck covering MAX architecture, CPU and GPU comparisons, Mojo and cloud deployment.
Explains Mojo’s compiler and SIMD design and the limits of language performance comparisons.
Updates since the talk
- MAX 24.6 GPU previewArticle
The December 2024 announcement of the MAX GPU technology preview and its initial hardware support.
Later announcements covering Mojo compiler open sourcing and planned native Windows support.
Full transcript
Loading transcript…