AI Engineer World's Fair 2026
Taming the AI Hardware Cambrian Explosion — Abdul Dakkak, Modular
Slide chapters
30 slide and demo groups · Timestamps are approximate. 5 additional groups were omitted from this pass.
Read the talk
Taming the AI Hardware Cambrian Explosion
Abdul Dakkak explains how Modular connects Mojo kernels, MAX inference and distributed deployment—and why its performance comparisons need to be read alongside their workloads, hardware and costs.
From a talk by Abdul Dakkak
At a glance
Ideas worth remembering
Modular connects three levels of work: Mojo implements portable device code, MAX executes and serves models, and Cloud operates distributed endpoints.
Mojo’s performance strategy combines portable kernel abstractions with low-level access; open kernels let developers test the claims on their own workloads.
MAX keeps model authoring in Python while exposing multi-GPU, multi-node and heterogeneous execution, including separate hardware for prefill and decode.
Time to first token, throughput and baseline-relative speedups answer different questions. Hardware price must enter the comparison before drawing a cost conclusion.
The subsecond image example makes the application stake concrete: faster execution shortens the wait between a changed prompt and its visual result.
More chips leave software with more choices to manage
Compute can be booked out for years while the choice of processors keeps expanding. That is the opening tension in Abdul Dakkak’s account of Modular: NVIDIA GPU generations coexist with hardware from Intel, Arm and Apple, each covering different needs. More hardware creates more possible places to run a model. It also creates a programming problem: how much software must change to use each one?
Hardware choices travel upward through the software stack. Dakkak introduces a diagram of vLLM’s NVIDIA and AMD support to illustrate that complexity; adding another hardware target adds more implementation choices to manage. Production deployments then multiply the stacks themselves, using vLLM, SGLang, TensorRT-LLM or llama.cpp. The useful question for the diagram is where hardware-specific dependencies enter an otherwise shared inference system.
Modular’s response is to rebuild the stack around a common hardware layer. This commits the company to solving several problems together: writing fast device code, executing models and operating inference across machines. A common interface alone would leave the performance work underneath unresolved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Mojo, MAX and Cloud divide the work
The proposed stack has three layers with different jobs. Mojo supplies a Python-like systems language targeting CPUs, GPUs and other accelerators. MAX uses Mojo kernels to execute models across text, image, video and speech workloads. Modular Cloud handles deployment and distributed serving decisions: choosing hardware, routing requests, offloading the key-value cache and arranging distributed or disaggregated inference.
What depends on what? The stack below separates language and kernel implementation from model execution and endpoint operation. MAX is built on Mojo; Cloud sits above those capabilities to operate inference across machines. These are software dependencies, rather than three consecutive stages that every request must visibly traverse. Dakkak describes deployment on premises, in Modular’s cloud or in a customer’s cloud environment.
The implementation beneath a distributed endpoint
Read the diagram as text
- Mojo. Systems programming and kernels targeting CPUs, GPUs and other accelerators.
- MAX. Model execution and inference using Mojo kernels across several modalities.
- Modular Cloud. Endpoint deployment and distributed operation, with access to MAX and Mojo tuning variables.
- MAX → Mojo: Built with Mojo kernels.
- Modular Cloud → MAX: Operates and exposes tuning of the inference stack.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Portable kernels still need low-level access
Mojo occupies the systems-programming role associated with C++, Rust and CUDA C++, while presenting Python-like syntax. The ambition is familiar programming across accelerators without giving up the control needed to compete with CUDA performance. Syntax lowers the entry cost; hardware access determines how far optimization can go.
Dakkak places existing approaches along three competing dimensions—usability, performance and portability:
- CUDA and ROCm: Strong performance, with more demanding programming and hardware-specific work.
- OpenCL: Broad portability, with compromises in usability and performance in his comparison.
- Triton: A useful balance of portability and usability, though he characterizes its performance as roughly 70% without specifying a workload or denominator.
- Mojo: Portable kernel abstractions combined with low-level access, so programmers can keep tuning when a higher-level abstraction stops being sufficient.
The kernels are the concrete test of that design. Dakkak reports matrix-multiplication results ahead of cuBLAS and hipBLAS, and attention kernels ahead of FlashAttention 4 and AITER. Practically all of Modular’s kernels, he says, are open source and can be benchmarked on a reader’s own machine. The recording does not establish the shapes, precisions or hardware configurations behind these kernel comparisons, so they support workload-specific performance claims rather than a universal ranking.
The language’s release status is separate from the kernels’ availability. At the time of the recording, Mojo 1.0 and API stabilization had been announced, while the release and open sourcing of Mojo itself were promised for the following months. Open kernels therefore did not yet imply an already open-source language implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
MAX brings model execution and optimization into one framework
MAX addresses the next layer of the problem. Dakkak’s criticism of PyTorch is historical: its original design predates today’s generative-inference requirements. Systems such as vLLM and SGLang supplied serving mechanisms including KV-cache management and PagedAttention, but those additions also left teams assembling more layers around the framework.
MAX combines graph compilation with eager execution that is portable across devices. Its compiler also understands Mojo programs, allowing fusion across work expressed in those programs. Fusion brings operations together in the compiled execution; the architectural point is that optimization can reach into the kernel language rather than treating it as an unrelated implementation beneath the model framework. Eager execution remains available for development and debugging.
Development, optimization and serving are intended to live in one system. Dakkak claims support for thousands of model architectures and provides OpenAI-compatible endpoints for familiar client access. The endpoint interface and the internal execution machinery solve different adoption problems: applications can keep a familiar request format while model developers work within MAX.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Python model definitions extend across GPUs and nodes
Model development stays in Python. Developers import modules, compose them into a model and use eager execution to debug. Dakkak points to a model-definition example as evidence that this should feel familiar to a PyTorch programmer. The useful thing to inspect is how imports and composition express the model, rather than assuming that the systems language must also become the model-authoring language.
The execution target can then expand beyond one device. MAX APIs describe models running across multiple GPUs and multiple nodes. They can also express heterogeneous placement: Dakkak’s concrete example puts prefill on hardware X and decode on hardware Y. Those are distinct phases of language-model inference, so the placement choice can follow the phase rather than forcing the entire model-serving operation onto one kind of processor. The example establishes the supported placement idea without detailing its scheduling or cross-device transfer implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A cloud endpoint with the tuning controls exposed
Modular Cloud is described as a “glass box”: the endpoint exposes variables from MAX, Mojo and the layers between them. Cloud Lab lets users experiment with parameters and choose a point suited to a particular model and workload. That matters because deployment is already a collection of decisions about machines, routing and cache handling; a convenient endpoint does not remove the need to make those decisions well.
The basic usage path is familiar: select a model, obtain an API key and place it in an HTTP request. More specialized deployments can use dedicated endpoints, fine-tuned models or custom speculative decoders. Bring-your-own-cloud deployment provides another operating option. The claimed benefit is flexibility across both models and hardware, with cost reduction as a workload-dependent outcome of those choices.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read latency, throughput and GPU cost separately
The benchmark sequence turns the architecture into measurable claims. Dakkak introduces comparisons against vLLM or SGLang configurations and then moves between language models, image generation and different GPU vendors. These are reported results from the recording; the full workload settings and measurement protocols are not established here, which limits comparisons beyond the named cases.
For Kimi K2.5, Modular reports a 5.5× improvement in time to first token relative to vLLM, alongside 1.5× throughput measured in tokens per minute per GPU on B200. These metrics answer different questions: time to first token concerns how long a user waits for output to begin, while throughput concerns how much output the hardware serves. The two multipliers should remain separate. A useful inspection of the benchmark slide would look for the conditions attached to each.
Kimi-K2.5 on MAX
Modular MAX vs. vLLM
Configuration:
Kimi-K2.5
NVIDIA B200
P50 TTFT
Modular 239 ms
vLLM 1,303 ms
5.5x
Throughput (TPM/GPU)
Modular 625k
vLLM 419k
1.5x
For FLUX.2, Dakkak reports about 7× faster execution than torch.compile with Diffusers. He uses the image-generation result to emphasize that MAX covers several modalities within the same framework: text, image, audio and video generation share the stack rather than requiring a separate framework for each category.
The AMD comparison prompts an important correction. Dakkak reports a 4× improvement on MI355 systems, then warns against concluding that the 7× result is automatically the better purchase. Each speedup is relative to a baseline, and operating cost also depends on GPU price. He characterizes AMD hardware as usually about 30% cheaper than the NVIDIA equivalent. The practical comparison therefore needs delivered performance and hardware cost together; the multipliers alone do not establish total cost of ownership.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Subsecond images turn speed into an interaction
Dakkak next describes Artificial Analysis independently verifying results through a partnership with Modular. For FLUX.2 dev, he reports roughly 3× the speed of the next competitor in the comparison shown. He explicitly attributes this to execution performance rather than distillation or quantization. That is a consequential distinction: the claimed gain comes from running the model more efficiently, rather than changing it through those techniques.
What does that speed let a user do? In the demonstration Dakkak describes, someone pastes prompts and images appear in under a second—often around 0.3 seconds. The observable relationship to inspect is the prompt change, the resulting image and any displayed timing. He also describes a booth demonstration that combines a prompt with a video feed and renders the result. The application consequence is shorter waiting between an input and a visual response; the recording does not establish a sustained frame rate or a complete end-to-end latency definition.
The final model comparison is Gemma 4. Dakkak reports roughly 6× the speed of SambaNova in the presented comparison, including a hardware-accelerator provider among the competitors. This remains a separate model-serving result, rather than another measurement of the image demonstration. He closes the performance sequence by saying further optimizations are still being added.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The invitation is to inspect and benchmark the stack
The ending returns to what developers can inspect. Dakkak says Modular has open-sourced more than 700,000 lines of kernel code, its model definitions and MAX Serve. Those expose different parts of the implementation: kernels perform the device work, model definitions express the computation and the serving layer makes it available to applications.
For developers unfamiliar with Mojo or MAX, Modular also offers AI skills intended to help port code from familiar languages. The team’s hiring spans low-level kernel engineering through Kubernetes work, reflecting how much of the system sits between an individual device operation and a deployed service.
The practical starting point is pip install modular. Dakkak’s closing invitation is to try the software and validate the results. For a prospective user, the useful test follows the distinctions made throughout the talk: choose the actual model and hardware, measure the latency or throughput the application needs, and include hardware cost when deciding where to deploy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Read the complete timestamped transcript
28 slide groups · 2 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 5 additional groups were omitted from this pass.
- 0:13
Uh, so let's get started. Uh, hi, my name is Abdul. Uh, uh, I'm from Modular. Uh, today I'm gonna be talking about what Modular is and what value it brings to you. So as you probably know, today there is a huge crunch, uh, for compute, and there's, like, no alleviation in sight. A lot of these things are booked for the next few years.
The compute crunch
Dakkak says compute is in short supply and much of it is booked for the next few years.
Text in image
Today there is a compute crunch and there is no alleviation in sight M Interface text: 2 / 38
AI hardware is exploding
Dakkak calls the growing range of hardware a Cambrian explosion: vendors serve different niches, raising the question of how to program their systems.
Text in image
AI hardware is EXPLODING... NVIDIA Blackwell, GeForce intel Gaudi arm Ethos, Neoverse Qualcomm Snapdragon, AI100 AMD Instinct, Ryzen groq LPU MTIA G TPU, Axon CPU Maia, Cobalt Apple Silicon aws Inferentia, Trainium M Interface text: 4 / 38
- 0:39
But there's also a kind of... As a result, you have an explosion of hardware. We call it like a Cambrian explosion. You have NVIDIA providing, you know, a few generations of GPUs, the Intels, the ARMs, the Apples, everything, uh, i-is, is present. They all have some niches, uh, that they cover, but the problem is that you have to ask is: How do you program all of these, uh, systems?
How do we program it?
Vendor logos and chips frame Dakkak’s question: how can developers program the expanding range of AI hardware?
Text in image
...but how do we program it? NVIDIA intel G arm Qualcomm AMD aws groq Interface text: 4 / 38
AI software is not scaling
The vLLM stack branches into NVIDIA and AMD software layers, showing how hardware-specific choices add complexity higher up the stack.
Text in image
AI hardware is exploding and AI software is not scaling The problem is at the bottom. Old technologies tied to individual vendors’ hardware. This forces complexity into AI systems upward to support different hardware options. vLLM Serving API vLLM Python API vLLM Engine TorchChat ... PyTorch Python API TorchScript torch.compile NVFuser AITemplate Inductor Triton Hardware Vendor Stacks NVIDIA CUDA vLLM NVIDIA Kernels FlashInfer DeepGEMM DeepEP cuDNN cuBLAS cuTile CuTe DSL CUTLASS Driver B200 AMD ROCm AIter ATOM MIGraphX MIOpen rocBLAS FlyDSL MoRI Composable Kernels Driver MI355x M Interface text: 5/38
- 1:06
Now, if you look at the current software as it is today, um, it's kind of a mess as well, and that's because a lot of these hardware choices bubble up from the d-- you know, from down under, uh, into upwards into the stack. So this is a diagram showing how, uh, vLLM supports both NVIDIA and, uh, AMD. If you add extra hardware, the, the, the diagram becomes even more difficult. Now, today, folks actually,
- 1:36
uh, deploy not only using vLLM, but also SGLang, TRT-LLM, uh, llama.cpp, et cetera. So at Modular, we thought, "Well, this is kind of a huge mess. Let's re-architect the, the, uh, the stack from the ground up and let's build a unifying layer, uh, that unifies all the hardware, so everything runs on our stack."
Modular’s solution
Modular proposes a unified compute layer connecting its stack to NVIDIA, AMD, Apple Silicon and future xPUs.
Text in image
Modular’s solution Build a unified compute layer NVIDIA B200 NVIDIA H200 xPUs built for whatever comes next. AMD MI355X AMD MI325X Apple Silicon *progress available in nightlies M Interface text: 6/38
Platform design principles
Modular proposes building a complete software stack around portability, performance, and programmability.
Text in image
To realize this, you need a platform built from first principals with portability, performance, and programabliity in mind M Interface text: 7 / 38
- 2:00
To realize this, you have to actually go back to first principles and build a complete software stack with portability, performance, and programmability in mind. So at the bottom-most level, we've developed Mojo, a heterogeneous programming language that's targeted towards AI, uh, as well as general purpose programming, uh, as well. It's kind of Pythonic in style, so folks that actually know Python today pretty much know ninety percent of Mojo.
Mojo kernels
Mojo sits at the kernel layer beneath serving and cloud. The Pythonic language targets CPUs, GPUs, and other accelerators.
Text in image
Modular Platform Cloud Serving Kernels Cloud Serving Mojo 🔥 Write novel AI algorithms with a Pythonic systems language that runs with SOTA perf across heterogenous AI hardware COMPILER | LIBRARIES | TOOLS M Interface text: 8 / 38
- 2:31
Uh, so we've built that language, and that targets both, uh, CPUs, GPUs, as well as other accelerators. On top of that, we leverage Mojo to build Max, our, uh, inference engine. Uh, so that provides you with, uh, AI inference for both-- for a bunch of different modalities like image, video, speech, as well as text. We've built all our kernels underneath the hood t- that are built on top of Mojo. Finally, today's AI
MAX serving
MAX sits above Mojo kernels in Modular’s stack. Dakkak describes it as an inference engine for text, image, video, and speech.
Text in image
Modular Platform Cloud Serving Kernels Cloud MAX 🧑🚀 Serve text, image, video, & speech models with SOTA perf on NVIDIA & AMD GPUs with one unified framework SERVING | MODELS | 🔥 KERNELS Kernels Interface text: M 9 / 38
MCloud
MCloud adds a cloud control plane above serving and kernels, turning MAX and Mojo into production GenAI endpoints. Dakkak frames hardware selection and request routing as challenges for distributed inference.
Text in image
Modular Platform Cloud Serving Kernels MCloud ☁️ A performance-native, heterogenous control plane that turns MAX and Mojo into production GenAI endpoints. ENDPOINT | CLUSTER | LAB Serving Kernels Interface text: M 10 / 38
- 3:01
inference is distributed, so you actually need a cloud solution as well. Uh, you need to pick which hardware to deploy your model on, how to route, uh, different user requests to different, uh, machines, how to do KV cache offloading and, and, and distributed inference and disaggregated inference, uh, to be able to serve the model at the speed and the cost that the folks expect. So putting it all together, we built the modular platform that's fully programmable,
Modular Platform
Modular stacks Mojo kernels, MAX serving, and cloud into an inference platform. The slide says the same model scales across NVIDIA, AMD, CPUs, and more, with BYOC as the preferred go-to-market motion.
Text in image
Modular Platform Cloud Modular Serving MAX Kernels Mojo - A vertically integrated, fully-programmable enterprise-grade AI inference stack - Abstracts hardware complexity; the same model scales across NVIDIA, AMD, CPUs, & more - Supports SOTA perf across text, image, video models, not just text-based LLMs - BYOC preferred GTM motion dedicated compute for POCs and initial onramp M Interface text: 11 / 38
- 3:31
it's hardware abstracted, it's, it's SOTA perf, uh, and then it actually can be deployed both on-prem as well as o-on our cloud as well. So it could be... If you have a cloud, uh, uh, setup, you can actually download the modular cloud and, and deploy it there. So let me kind of dig into each one a-at a time, each part of the stack. So the first is Mojo. Uh, so Mojo, as I mentioned, is a Pythonic system programming language. It's not
Mojo
The talk turns to Mojo, the first part of the stack, which Dakkak describes as a Pythonic systems programming language.
Text in image
Mojo 🔥 M Interface text: 12 / 38
Mojo at a glance
Mojo aims to bring Python-like programming to CPUs and GPUs. The slide pairs its interoperability and accelerator support with a vectorized Mandelbrot kernel that updates iteration counts using a mask.
Text in image
Mojo🔥 at a glance Pythonic systems programming language • CLI toolchain + VS Code support • Strong Python and C interop Unlocking performance for heterogeneous compute Blazing fast Unified programming CPU + GPU Scalable GPU kernels for Nvidia, AMD, Apple ... Best way to extend Python to CPUs and GPUs def mandelbrot_kernel[ width: Int # SIMD Width ](c: ComplexSIMD[float, width]) -> SIMD[int, width]: """A vectorized implementation of the inner mandelbrot computation.""" z = ComplexSIMD[float, width](0, 0) iters = SIMD[DType.index, width](0) mask = SIMD[DType.bool, width](True) for i in range(MAX_ITERS): if not any(mask): break mask = z.squared_norm() <= 4 iters = mask.select(iters + 1, iters) z = z.squared_add(c) return iters M Interface text: 13/38- 4:01
unlike C++ Rust or CUDA C++. The thing is what it enables is, uh, programmability a-across different accelerators. It's blazing fast. It's-- Again, it actually competes with CUDA in terms of performance. Uh, and you can think of it as the best way to program CPUs and GPUs within a Python-like programming language. In terms of how we think about Mojo, uh, you know, you can
Accelerator programming tradeoffs
Dakkak places ROCm and CUDA near performance, OpenCL near portability, and Triton between portability and usability. He presents Mojo as aiming to combine all three.
Text in image
Usability Performance Portability ROCm CUDA Triton OpenCL M Interface text: 14/38
- 4:32
think about there's t-three different quadrants: usability, performance, and portability. ROCm and CUDA-- uh, and CUDA gives you massive performance, but they sacrifice on both usability and programmability. A lot of folks use agents to actually program these system nowadays. OpenCL back in, you know, a few years ago was popular. It gives you great portability, but it sacrificed on performance and usability. Triton is-- uh, has a good sweet spot that gives you seventy percent of the perf, uh,
Usability, performance and portability
Three overlapping circles frame accelerator programming around usability, performance and portability, which Dakkak presents as competing priorities.
Text in image
Usability Performance Portability Interface text: M 15 / 38
- 5:02
but it gives you portability and usability. And Mojo is, is trying to kind of fill the niche of the sweet spot, where it's the best of both worlds. It gives you the abstraction to allow you to write portable kernels and, and, uh, and usable kernels while giving you also the low-level access, uh, to get the maximum performance. We've actually open sourced, uh, practically all of our kernels, so you can actually test this stuff. You can benchmark it on your own machine. Uh, we have
Accelerator programming tradeoffs
Dakkak places ROCm and CUDA near performance, OpenCL near portability, and Triton between portability and usability. He presents Mojo as aiming to combine all three.
Text in image
Usability Performance Portability ROCm CUDA Triton OpenCL M Interface text: 14/38
Mojo and accelerator programming
Dakkak places Mojo at the intersection of usability, performance and portability, aiming to combine portable kernel abstractions with low-level access for performance.
Text in image
Mojo 🔥 The best way to program accelerators Usability Performance Portability Interface text: M 15 / 38
- 5:32
state-of-the-art kernels on NVIDIA, AMD, and other hardware. So to tell you what state-of-the-art is, it means that our matrix multiplication is faster than the cuBLAS, the hipBLAS, everything out there, and it's not by one percent, in some cases by two percent. We have the fastest attention, uh, kernel out there, faster than FlashAttention 4, uh, as well as, like, the AITER and things like that.
Mojo 1.0 beta
The Mojo 1.0 beta page describes a language for CPUs and GPUs. Dakkak says API stabilization was announced, with the release and open sourcing planned for the following few months.
Text in image
Mojo 🔥 Install Docs Packages Releases Community Mojo 1.0 🔥 [BETA] Write like Python, run like C++. Write fast code for diverse hardware, from CPUs to GPUs, without vendor lock-in, in a language that's both user friendly and memory safe. Install Now Stable: 1.0.0b1 (May 7) | Latest nightly (2 hrs ago) Built different Modern AI native Simply performant Quicksta[illegible] Releases Roadma[illegible] Contrib[illegible] Interface text: mojolang.org 1.0.0b1 17 / 38
- 5:59
Uh, one thing that we've announced, uh, a month ago or so is Mojo 1.0, the stabilization of our APIs. Uh, and we're gonna be releasing Mojo one point O in the next few months, and we're also open sourcing Mojo, uh, in the next few months as well.
MAX
MAX sits on top of Mojo and aims to address assumptions in PyTorch that predate today’s generative AI inference.
Text in image
MAX M Interface text: 18 / 38
- 6:17
Sitting on top of Mojo is Max. So Max is trying to kind of solve a few of the, uh, assumptions that were made, uh, back in the day for PyTorch. PyTorch is about ten years old. It, it was not really designed for the GenAI inference of today. As a result, you have, like, vLLM and SGLang that stepped into the void to give you things like KV cache, PagedAttention, and, and things like that.
What have we learned?
Dakkak argues that vLLM and SGLang filled gaps in PyTorch’s inference support, but the added layers complicated its ecosystem and increased dependence on one vendor’s tools.
Text in image
What have we learned? 01 PyTorch wasn’t designed for today’s inference M Interface text: 19 / 38
Show 3 builds
- 6:19
The first takeaway says PyTorch was not designed for today's inference.
Text in image
What have we learned? 01 PyTorch wasn’t designed for today’s inference M Interface text: 19 / 38
- 6:33
The highlighted second point says tools like vLLM and SGLang filled an inference gap.
Text in image
What have we learned? 01 PyTorch wasn’t designed for today’s inference 02 Tools like vLLM/SGLang grew to fill in the gap M Interface text: 20 / 38
- 6:42
The highlighted third point says this exacerbated PyTorch complexity and dependence on one vendor’s tools.
Text in image
What have we learned? 01 PyTorch wasn’t designed for today’s inference 02 Tools like vLLM/SGLang grew to fill in the gap 03 This exacerbated PyTorch complexity and dependence on one vendor’s tools M Interface text: 21 / 38
- 6:19
- 6:45
And... But all of that kind of, kind of complicated the entire PyTorch ecosystem. A lot of folks don't really, uh, use PyTorch as is. They're resorting to these layers on top.
- 6:58
So think of a new approach, and we call that the Max Framework. So again, it's built from the ground up, uh, with the best technologies, with the best state-of-the-art graph compilers. Uh, the eager mode execution is portable across devices. It understands your Mojo programs to do novel fusion, uh, and it gives you the maximum performance and deployability that you're otherwise not able to do in, in, in regular PyTorch.
MAX Framework
Dakkak presents MAX as a ground-up framework combining a graph compiler with eager execution across devices. He says it understands Mojo programs for novel fusion.
Text in image
MAX Framework is built from the ground up with the best technologies 👉 Portable to multiple devices All the performance of graphs Easy to use Python APIs Lots of example models SoTA graph compiler LLM from scratch tutorial World’s best eager mode Portable to multiple devices Mojo-native design Domain agnostic, not just GenAI Interface text: M 23/38
- 7:26
So compared to other stacks, uh, this is a single stack that allows you to serve, develop the model, as well as optimize everything in one cohesive, uh, system.
- 7:40
It-- Within Max, we support thousands and thousands of models, in fact, thousands of architectures, uh, everything from Gemma Four to Mis-- uh, MiniMax Three to GLM. Um, so any model that you think about is, is practically supported in, uh, in, in, uh, in Max, and it has the same interface that you'd expect, uh, from what is standard today. So we also provide A-- OpenAI endpoints that you can use, uh, within
- 8:10
the stack.
MAX for PyTorch developers
A Python example builds a model from layers, runs it for development and debugging, then compiles it for production. The speaker presents the interface as familiar to PyTorch developers.
Text in image
MAX a new, intuitive way to build models for AI developers familiar with PyTorch from max import nn, functional as F, tensor, random class MyModel(nn.Module): def __init__(self): self.fc = nn.Linear(10, 10) self.proj = nn.Linear(10, 1) def __call__(self, x: tensor.Tensor) -> tensor.Tensor: x = self.fc(x) x = F.gelu(x) x = self.proj(x) return x model = MyModel() input = random.normal((10, 10)) # for development and debugging max_output = model(input) # for production compiled_model = model.compile(input.type) max_output = compile_model(input) Interface text: M 26 / 38- 8:14
If you're familiar with, like, PyTorch programming, uh, the figure on the right would... or the left will not be, like, massive departure of your mental model. Uh, you can build a model as you expect. You know, you can import modules. The, uh, programming language for developing models is Python, so that's familiar to folks. Uh, and you can compose, uh, things as, as you, as you, as you, um, as you build them. We also support, like, the eager
- 8:44
execution to enable debugging as well.
- 8:49
Now, the other thing that one has to realize is modern LLMs, uh, are run on across different GPUs and in many cases, multiple nodes. So also Max, the Max APIs provide you with a way to, uh, program multi-GPU system ac-- as well as multi-node systems from within the API, so that the model is natively supported across, uh, different GPUs and different nodes as well.
- 9:19
And this can also be heterogeneous, so there's nothing to stop you from saying the prefill is on hardware X and the decode is on hardware Y.
Modular Cloud
Dakkak introduces Modular Cloud as a layer over the stack where developers can adjust variables across MAX and Mojo.
Text in image
Modular Cloud Interface text: 28 / 38
- 9:30
Finally, layered on top of all of this is the Modular Cloud. So as I mentioned, uh, today's AI inference is distributed. It runs on many, many different machines. Uh, it's heterogeneous. So we provide the Modular Cloud, which is a performance native... You know, we think of it as a glass box because you can actually tinker with all the variables, uh, that are available for you in both Max, Mojo, and everything in between, uh, within
Modular Platform: kernel to cloud
Modular Cloud sits above the serving and Mojo layers. Dakkak describes it as a glass box for adjusting inference parameters, deploying endpoints and experimenting with workloads.
Text in image
Modular Platform is full stack control - kernel to cloud. Cloud Serving Mojo Modular Cloud ☁ A performance-native, “glass-box” cloud that turns MAX and Mojo into production GenAI endpoints. ENDPOINT | CLUSTER | LAB Interface text: M 29 / 38
- 10:00
your endpoints. So it provides, it provides endpoint deployment as well as bring your own cloud deployment as well. We also have a Modular Cloud Lab that allows you to experiment with all these different parameters and pick the best point in the parameter frontier that you have for that model or based on your workload as well.
Modular Cloud offerings
Modular Cloud offers per-token shared endpoints, reserved GPU endpoints and deployment for custom models. Dedicated endpoints and custom models can run in Modular’s cloud or the customer’s.
Text in image
Cloud ☁ Shared endpoints High-performance inference. No infrastructure to manage, no long-term commitments. Test easily. Per-token pricing. Dedicated endpoints* Reserved NVIDIA and AMD GPUs. Forward-deployed engineers optimize your deployment. Per-minute pricing that’s easy. Custom models* Bring your own custom or fine-tuned models. Deploy on optimized infrastructure with per-minute pricing. *Deploy in our cloud or yours. Dashboard illustration: Modular Home BUILD Deployments Batches Performance EXPLORE Models Playground Codespaces Kimi K2.6 Running 407 0.35s 99.9% Uptime 128k Total Context [illegible] KimiK2.6 | 407 Tok/sec | 0.35s TTFT | 99.9% Uptime FLUX-dev | 512×512 Resolution | 0.93s Process time | 99.9% Uptime DeepSeek | 200+ Tok/sec | 0.95s TTFT | 99.9% Uptime Interface text: M 30 / 38
- 10:22
Uh, so in terms of, like, the UI, uh, if you're familiar with, like, any, uh, cloud provider, I mean, it's, it should be pretty familiar. You just go to the website. You click on which model t-- you wanna run. You get an AP-- uh, an, uh, API key, and then you plug that in, uh, to your HTTP request, and, uh, you can run it. You can have dedicated endpoints as well as we can serve your own custom model. Uh, you know, in many cases, folks have fine-tuned
- 10:51
models or models with their own custom speculative decoders. Uh, we could do all of that, uh, for you as well. Now, why do you care about all of this, right? It's a lot of tech. Uh, why do folks care? Well, it's because you can actually, uh, accomplish more things, uh, through the system, right? So you can actually decrease cost. You can get greater flexibility because you can actually use different hardware. You're not beholden
Performance
Dakkak moves from hardware flexibility and portability to performance benchmarks.
Text in image
Performance Interface text: 31 / 38
- 11:21
to one hardware provider. Uh, you get portability. You can use multiple different models, uh, et cetera. So how does perf look like? Well, we benchmarked a lot of these. Uh, and this is, like, against, uh, state-of-the-art, you know, the best combination of vLLM or SGLang. In a bunch of cases, these are-- these benchmarks are independently verified as well. So in terms of Kimi K point two-- uh, Kimi K two point five, we're five point five faster on
Kimi-K2.5 on MAX
On NVIDIA B200, Modular reports 5.5x lower P50 time to first token and 1.5x higher throughput in tokens per minute per GPU than vLLM for Kimi-K2.5.
Text in image
Kimi-K2.5 on MAX Modular MAX vs. vLLM 5.5x P50 TTFT Modular: 239 ms vLLM: 1,303 ms 1.5x Throughput (TPM/GPU) Modular: 625k vLLM: 419k Configuration: Kimi-K2.5 NVIDIA B200 M Interface text: 32 / 38
- 11:51
TTFT, uh, compared to vLLM. We can also get one point five X throughput improvements, so the token per minute on per GPU is one point five X faster, uh, on B200 than it is on vLLM.
FLUX.2 on MAX
At 1024×1024, MAX generates FLUX.2-dev images 6.9x faster than Diffusers on NVIDIA B200 (1.88 vs. 13.07 s) and 3.8x faster on AMD MI355x (3.40 vs. 12.92 s).
Text in image
FLUX.2 on MAX Modular MAX vs. PyTorch Diffusers (torch.compile) 6.9x 1024 x 1024 Modular: 1.88 s Diffusers: 13.07 s 3.8x 1024 x 1024 Modular: 3.40 s Diffusers: 12.92 s Configuration: FLUX.2-dev NVIDIA B200 AMD MI355x Text to Image 1024 x 1024 M Interface text: 33/38
- 12:11
In terms of FLUX.2, and I, I encourage folks to actually visit the booth. It's behind this curtain. Uh, there's a nice demo of FLUX.2. Uh, we're about seven X faster, uh, compared to torch.compile with Diffusers. So again, Max supports different modalities within the same framework, so you don't have to use an omni variant of a, of a piece of software or anything like that. We support both video generation, audio generation, image generation, as well as text generation.
- 12:43
Uh, we also support, you know, things on AMD where you can get a TCO benefit as well, where we're four X faster than the equivalent, uh, on, um, on, on the MI355 systems. Actually, you know, some of this is not a fair comparison because, you know, some folks might say, "Oh, seven X is much larger than four X," but this is... You cannot do that comparison very easily. Uh, that you can- you have to factor in the
- 13:12
cost, uh, per, uh, GPU, and AMD is usually thirty percent cheaper than, uh, the NVIDIA equivalent.
- 13:22
Uh, Artificial Analysis, actually, we've been partnering with them, uh, and we've been... And they've independently been verifying our results. Uh, this is our results, like, as of, uh, today. Uh, we're about three X faster than the next competitor, uh, on the Flux two dev model as well. And this is not playing tricks of distillation and, you know, quantization or anything like that. This is just raw speed because we're- can leverage the hardware from the ground up and make
- 13:52
sure that, uh, we're getting the max, uh, the max perf benefit from the model.
- 14:00
And you can actually go to Artificial Analysis today and see, and see these results. Now, what does that mean to folks? Again, it means you can open up applications that are otherwise kind of not possible because of slow inference. So this is actually, uh, a kind of a live demo. It's just somebody's pasting in the prompts. Uh, but the generation is, is accurate. So you can generate things in subsecon- generate images in
FLUX.2 Demo: Jellyfish
The FLUX.2 demo shows a glowing jellyfish over a city. Dakkak says prompts are pasted in and images generate in under a second, often around 0.3 seconds.
Text in image
🚀 FLUX.2 Interface text: Image Prompt Describe the image you want to generate... Generate Total generation time: 0.27sec 35 / 38
14:13Demo milestone
FLUX.2 image generation demo
The demo shows a bowl of cherries and then an iridescent beetle. Dakkak says prompts are pasted in and many images generate in around 0.3 seconds.
Text in image
🚀 FLUX.2 Interface text: Image Prompt A giant moon resting gently on the surface of a calm ocean at night, perfectly reflected in the still water below, minimal horizon line, cool blue and silver palette Generate Total generation time: 0.28sec 35 / 38
14:17Demo milestone
FLUX.2 image generation demo
The demo shows a bowl of cherries and then an iridescent beetle. Dakkak says prompts are pasted in and many images generate in around 0.3 seconds.
Text in image
🚀 FLUX.2 Interface text: Image Prompt Describe the image you want to generate... Generate Total generation time: 0.27sec 35 / 38
14:28Demo milestone
- 14:30
subsecond, uh, perf. In many cases, these are point three seconds. Again, you can go to the booth. There's a nice demo where, uh, you can give a prompt, and then it reads the video feed and then renders it as well.
Time to First Token: Gemma 4 31B Providers
With 10,000 input tokens, Modular reaches the first token in 0.53 seconds, followed by Lightning AI at 0.61 seconds. SiliconFlow (FP8) takes 3.14 seconds.
Text in image
Time to First Token: Gemma 4 31B Providers Seconds to first token received · Lower is better · 10,000 input tokens Y-axis: Seconds to first token (s) Y-axis ticks: 0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5 Modular: 0.53 Lightning AI: 0.61 CoreWeave: 1.01 DeepInfra: 1.12 Google AI Studio: 1.13 Together AI: 1.15 FriendliAI: 2.31 SambaNova: 2.78 Parasail: 2.85 SiliconFlow (FP8): 3.14 Interface text: M; 36 / 38
- 14:48
Uh, the other one data point is Gemma four. So Gemma four was a, a recent model that was just introduced about a month ago, a month and a half ago. Uh, again, we're quite faster than other-- any other competition out there. Uh, so, uh, you know, uh, m- many cases actually, some of these are, uh, hardware companies. You know, SambaNova is a hardware accelerator, uh, where, you know, let's say six X faster than that,
- 15:18
and in general, we're much, much faster than others. And by the way, we're... For, uh, all these cases, we're just starting. There's more optimizations that we're landing, uh, uh, as, as, as time goes on. So we're never, never stopping to improving these things. So we have a bunch of, uh, places where you can get involved. You can join our open source community. So a lot of our code is open source. We have over seven hundred thousand lines of kernel code that we've open source. We've over- open
Questions
Dakkak closes by inviting questions.
Text in image
Questions Interface text: M; 38 / 38
- 15:47
source all of our model definitions, all of our Max serve. Um, we have AI skills. So if you're not familiar with Mojo or Max, you can actually use the AI skills to port, uh, from languages that you're familiar with into Mojo or Max. You can also join us. We're hiring. So there's pamphlets on the booth where we're hiring for a variety of different positions. So as I-- you probably know, we, we hire from
- 16:17
low-level kernel engineers to high-level folks that work on Kubernetes and everything in between. You can actually try all of this. So this is not, uh, demo-ware or slide-ware. You can pip install modular right now if you want to and try all of this stuff, and I welcome everybody to validate these results. So with that, I kind of... That's it, and, uh... But I'm open to questions if folks have it, and I'll also be on the booth, uh, for the next few hours as
- 16:47
well.