# Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI

Simran Arora, Together AI

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/pOvWgX7IJsc-llms-write-fast-multi-gpu-kernels) · [Video](https://www.youtube.com/watch?v=pOvWgX7IJsc)

## Can LLMs Write Fast Multi-GPU Kernels?

Simran Arora explains why communication has become a limiting resource in many production distributed training and inference workloads, distills multi-GPU kernel design into transfer and scheduling choices, and tests whether models can make those choices across 87 practical problems.

### The bottleneck moves between GPUs

[0:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=12s)

Work inside one GPU has improved through kernels such as FlashAttention, memory-efficient architectures, sparse attention, and better domain-specific languages. Arora’s starting point is the consequence of that progress: as single-GPU execution gets better, communication between GPUs becomes a larger share of the remaining runtime. The new problem is to make kernels spanning several GPUs fast without requiring months of specialized engineering for each operator.

The hardware explains why data movement matters. Threads and blocks execute on processors distributed across the GPU. Those processors retrieve weights and activations through a hierarchy that includes a relatively small L2 cache and much larger high-bandwidth memory. Registers sit closest to computation and can supply data at a cited 130 terabytes per second on an H100, but their capacity is limited. Moving farther away generally provides more storage at the cost of slower access.

Multiple GPUs extend that hierarchy into a network. PCIe carries CPU–GPU traffic, while communication across nodes can use InfiniBand or TCP. Within the NVIDIA systems emphasized in the talk, NVLink connects GPUs point to point and NVSwitch joins NVLink endpoints into a nonblocking fabric. NVSwitch can also accelerate operations such as multicast and reductions in the network rather than requiring every step to run on a GPU.

Data gets larger—and farther away
Within a GPU, fast, scarce registers give way to larger memories farther from computation. Beyond the GPU, communication links provide access to remote memory; the links are not additional storage tiers.
- Registers — Closest to computation and very fast, but scarce.
- L2 cache — On-GPU memory surrounding the processors, with limited capacity.
- High-bandwidth memory — Stores large weights and activations and supplies the GPU memory capacity reported by system tools.
- NVLink and NVSwitch — Connect GPUs within the NVIDIA scale-up domain; NVSwitch can perform multicast and reductions in the fabric.
- InfiniBand or TCP — Carries communication across nodes.

### Workloads and hardware are becoming more network-shaped

[5:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=337s)

Several optimization layers have advanced together: architectures reduce compute and memory requirements, hardware-aware algorithms improve execution, programming tools make kernels easier to express, and megakernels overlap work across operators. Yet distributed training and inference still exchange data. Arora reports that communication increasingly consumes most of the runtime in many production workloads, leaving low model FLOP utilization as systems scale.

Networking is also less uniform than matrix multiplication or memory hierarchies. The talk contrasts AMD’s point-to-point XGMI links, TPU systems using a 3D torus with optical wraparound links, and NVIDIA’s NVSwitch fabric with in-network reductions. Arora cites up to 900 gigabytes per second of unidirectional NVLink bandwidth between remote GPUs’ high-bandwidth memories on a particular NVIDIA generation. That hardware qualification matters: it is an example of one system, not a universal link rate.

The workloads themselves now cross more boundaries. KV-cache storage can span GPU memory, CPU memory, disk, and remote machines. Inference systems can place speculative decoding, prefill, and decode on different hardware. Arora also describes scale-up domains reaching 72 GPUs and cites a plan for a 576-GPU system in 2027. Device-side tensor memory acceleration adds asynchronous transfers to this environment, giving kernels finer control over when communication begins and how it overlaps with other work.

Compute has nevertheless improved faster than the links carrying its data. From NVIDIA’s A100 in 2020 to B200 in 2024, the reported BF16 tensor-core gain is 7.2×, compared with 3× for intra-node communication and 2× for inter-node communication. Faster arithmetic cannot remove that mismatch. It increases the amount of useful computation that can sit idle while communication catches up.

### Convenient collectives leave a performance gap

[11:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=670s)

NCCL makes common multi-GPU operations accessible, but its bulk-transfer orientation does not fit every kernel. It works naturally with large contiguous chunks. Fine-grained communication, tightly fused collectives, and communication that must overlap with a particular computation require more control. In the team’s benchmark, most simple PyTorch-plus-NCCL baselines fall below 50% of their communication-aware roofline, leaving headroom even after accounting for the communication limit.

The available alternatives each carry a cost:

- **Higher-level frameworks:** Systems built around bulk NCCL collectives often synchronize before and after transfers, preserving the same coarse execution structure.
- **Compilers and distributed DSLs:** These can offer better abstractions, but tuning may not transfer across rapidly changing network architectures. Arora cites a distributed compiler configuration tuned around eight H800 GPUs that did not adapt efficiently to H100s in the team’s results.
- **Hand-tuned operators:** Specialized kernels can reach peak performance, but the work does not scale. Arora says adapting some implementations from one precision to another can take five or six months.

This leads to the project’s core research question: can a small set of reusable principles capture the important decisions in multi-GPU kernel design? The team first investigated those decisions manually through **ParallelKittens**, a compact set of primitives and patterns used to build high-performance kernels across several parallelism schemes. Only after developing that understanding did they ask whether language models could apply it.

### Choose the transfer mechanism to fit the message

[16:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=960s)

The first reusable decision is how data crosses GPUs. A copy engine performs host-initiated transfers and suits large messages that can approach peak communication bandwidth. Because the transfer does not occupy the GPU’s compute processors or consume many of its registers, those resources remain available for other parts of the workload.

Device-initiated communication offers two other paths. Tensor memory acceleration, or TMA, can saturate NVLink with relatively small messages while consuming few registers and few processors, making it useful for fine-grained overlap. Register-level transfer instructions also support fine-grained movement and can exploit the in-network reductions that NVSwitch provides. TMA cannot effectively use those in-network computations in the design space Arora describes.

There is no single best path. Large independent transfers favor the copy engine’s resource isolation. Small transfers interleaved with computation favor device initiation. Operations that need reduction inside the switch may justify register-level instructions. The kernel writer must trade message size against register pressure, processor use, overlap, and network functionality.

Three paths for inter-GPU data
The useful choice depends on message size, GPU resource consumption, desired overlap, and whether the operation needs computation inside NVSwitch.
- Copy engine — Host-initiated; strong for large messages; leaves processors and registers available.
- Tensor memory acceleration — Device-initiated; saturates NVLink with smaller messages; uses few registers and processors; useful for overlap.
- Register-level instructions — Device-initiated instructions that can use NVSwitch in-network reductions.

### Overlap work within an SM—or divide it across SMs

[18:18](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1098s)

After choosing a transfer mechanism, the kernel must schedule computation, memory work, and communication. An intra-SM schedule assigns different warps or threads inside one streaming multiprocessor to compute and communication concurrently. This is effective when both activities consume the same data in compatible patterns. When their data or resource needs diverge, packing them into one SM can create awkward competition for registers and shared memory.

An inter-SM schedule instead gives different streaming multiprocessors distinct responsibilities for computation, communication, or memory work. This can avoid misaligned resource sharing and can better drive NVLink when intra-SM overlap cannot. Arora’s examples are operation-specific: intra-SM overlap works well for GEMM plus reduce-scatter, while inter-SM overlap works well for GEMM plus all-reduce when it uses NVSwitch’s in-network reduction capability.

Buffering and synchronization between senders and receivers remain part of the design. ParallelKittens packages these choices into programming primitives and templates while preserving developer control over them. Arora reports that the approach typically adds roughly a dozen lines to a single-GPU kernel and is used in production at Together AI and Cursor. The displayed comparisons show state-of-the-art results against strong reference baselines across data, sequence, and expert parallelism.

Two ways to overlap communication and computation
The schedule follows data alignment and resource pressure; neither strategy is universally superior.
- Intra-SM overlap — Different warps in one SM specialize in compute or communication. Best when their data patterns align.
- GEMM + reduce-scatter — The talk’s example where intra-SM overlap is effective.
- Inter-SM overlap — Different SMs specialize in compute, communication, or memory work, avoiding misaligned resource sharing.
- GEMM + all-reduce — The talk’s example where inter-SM scheduling uses NVSwitch reductions effectively.
- Intra-SM overlap → GEMM + reduce-scatter (example)
- Inter-SM overlap → GEMM + all-reduce (example)

### ParallelKernelBench tests practical distributed design

[21:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1290s)

**ParallelKernelBench** asks whether promising single-GPU kernel-generation results carry over to multi-GPU design. Each task gives a model an unoptimized PyTorch reference using distributed NCCL operations and a topology specifying the rank count and intra-node hardware. The model must replace that reference with a performant CUDA kernel that uses unified virtual addressing.

The search space grows combinatorially because a transformer layer can be parallelized across data, sequence, tensor, context, layer, pipeline, and expert dimensions. Each composition changes which data must move, which ranks participate, and when communication occurs. The team built a taxonomy covering these patterns and selected 87 problems arising in inference, reinforcement learning, and post-training, drawing from real repositories, optimized libraries, and existing DSL implementations.

The benchmark is intended to produce useful kernels rather than isolated programming puzzles. Its references represent work that practitioners already express through PyTorch and NCCL, while the candidates must provide an optimized implementation for the supplied topology. The public [ParallelKernelBench repository](https://github.com/togethercomputer/ParallelKernelBench) contains the benchmark and evaluation harness.

What one benchmark task asks the model to do
A topology-aware rewrite must preserve per-rank behavior while replacing coarse distributed operations with a faster kernel.
- PyTorch reference — Unoptimized implementation using distributed NCCL operations.
- System topology — Specifies rank count and intra-node hardware configuration.
- Kernel-generating model — Must reason about the operation and the available multi-GPU system.
- CUDA candidate — Uses unified virtual addressing and aims to outperform the reference.
- PyTorch reference → Kernel-generating model (operation semantics)
- System topology → Kernel-generating model (hardware constraints)
- Kernel-generating model → CUDA candidate (rewrites)

### Correctness improves faster than performance

[23:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1438s)

The evaluation separates two outcomes. `pass@K` asks whether at least one correct kernel appears within K attempts. The performance-oriented `fast₁@K` also requires the candidate to beat the PyTorch-plus-NCCL reference. That distinction matters because a compiling, numerically correct kernel may still communicate inefficiently or leave hardware idle.

In the zero-shot setting, the best tested frontier model solves 28 of 87 problems, and 22 of those solutions beat the reference. Drawing additional samples raises the number of correct solutions to 36, but the correct-and-faster share plateaus at roughly 31%. More parallel generations therefore find additional valid implementations without producing a comparable rise in high-performance ones.

When a solution is correct, its speedup often comes from removing NCCL staging and replacing it with direct NVLink loads and stores. Success clusters around familiar collective primitives, tensor-parallel GEMMs, and Ulysses-style context parallelism. Arora interprets this concentration as a sign that models perform best on patterns well represented online. It suggests uneven generalization, although the distribution alone cannot reveal the reasoning process behind any individual solution.

The talk identifies GPT 5.5 as the strongest model in the displayed comparison and reports that its qualifying solution count drops quickly as the required speedup rises. Beating a coarse reference on some tasks is therefore a lower bar than consistently finding kernels with substantial gains.

### Retries fix syntax more readily than distributed design

[26:28](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1588s)

The persistent failures are deeper than CUDA syntax. Multiple samples or error feedback often get a kernel to compile, but models still mishandle collective ordering, data partitioning, intra-SM versus inter-SM scheduling, and transfer selection. Generated kernels also often omit TMA and register-level transfer instructions. Compilation proves that the program is expressible; it does not prove that the model chose the right communication design.

The team also placed Gemini 3 Pro in a multi-turn mini-SWE-agent harness with access to a local shell. This improved the result from 24 solved problems to 35 of 87, with 26 candidates beating the reference. More execution time eventually reached another plateau. The environment helps the model inspect failures and revise code, but it does not supply the missing judgment about data movement and scheduling.

Aggregate limitations do not make the benchmark fruitless. Arora reports useful new examples for NeMo vocabulary-parallel filtering, Hyena context parallelism, and intersection-over-union suppression for the SAM 3 video-segmentation model. The talk does not give individual speedups for these kernels, so they demonstrate practical directions rather than quantified wins.

The ending preserves the central tension. A small set of primitives can express many effective multi-GPU patterns, yet the tested models still fail to apply their tradeoffs reliably even when those ideas are supplied in context. Arora points toward better methods for attacking the benchmark and toward architectures designed for larger scale-up domains, less dependence on scale-out, and much larger on-chip memories. Kernel generation is only one part of that opportunity: models and systems must ultimately be designed around the changing balance among computation, memory, and networking.

## Key takeaways

- Communication now deserves explicit optimization: the reported A100-to-B200 gains are 7.2× for BF16 compute, 3× for intra-node communication, and 2× for inter-node communication.
- Fast multi-GPU kernels match the transfer path to message size, GPU resource use, desired overlap, and any need for in-network reduction.
- Scheduling is operation-specific: intra-SM overlap works when computation and communication align around the same data, while inter-SM specialization helps when their resources or communication patterns diverge.
- Correctness and speed are separate benchmark outcomes. The best zero-shot result is 28 correct kernels out of 87, with 22 faster than the reference; more samples reach 36 correct while correct-and-faster performance plateaus near 31%.
- A shell-equipped agent harness improves iteration—Gemini 3 Pro reaches 35 solved problems and 26 faster solutions—but ordering, partitioning, scheduling, and transfer selection remain difficult after compilation errors are repaired.
- Limited aggregate coverage can still yield useful kernels, including the reported NeMo, Hyena, and SAM 3 examples, but reliable reasoning about evolving multi-GPU systems remains open.

## Resources

- [ParallelKernelBench](https://github.com/togethercomputer/ParallelKernelBench) — The public benchmark and evaluation harness for turning PyTorch-plus-NCCL references into fine-grained multi-GPU kernels, including generation and local evaluation workflows.
- [Simran Arora](https://arorasimran.com/) — Arora’s personal site for her research and contact information.
- [Simran Arora on GitHub](https://github.com/simran-arora) — Arora’s GitHub profile, including pinned systems and kernel projects.
- [Official AI Engineer talk page](https://ai.engineer/talks/pOvWgX7IJsc-youtube-7ee21c4ee99ca26b43d7) — The recording’s official read-along page with video chapters and a timestamped transcript.

## Related talks

- [AI Kernel Generation: What's Working, What's Not, What's Next](https://ai.engineer/talks/6guQG_tGt0o-ai-kernel-generation-whats-working-whats-not) — A companion discussion of progress, limitations, and next steps in AI kernel generation.
- [What every AI engineer needs to know about GPUs](https://ai.engineer/talks/y-UGrYbJsJk-what-every-ai-engineer-needs-know-about) — A broader introduction to GPUs for AI engineers.
- [Homa: The End of TCP for AI Clusters — John Ousterhout, Stanford](https://ai.engineer/talks/eZ8WWZzoaR0-homa-end-tcp-ai-clusters-john-ousterhout) — Explores Homa as an alternative to TCP for communication in AI clusters.

## Slides

### Accelerator interconnect topologies

TPU chips form a wraparound torus, Nvidia compares direct GPU links with an NVSwitch fabric, and AMD XGMI connects GPUs point to point.

[7:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=426s) [Diagrams compare TPU torus interconnects, Nvidia NVLink and NVSwitch, and AMD XGMI.](https://assets.ai.engineer/sites/aiecode2025/sha256/769d24dcfa1709f182f5ab8c705d31d098e2974ceb9eab34aeaeb10b3ace460e.webp)

Networking is rapidly evolving and the frontier of new inference algorithms (disaggregation).

TPU inter-chip interconnects (torus).
Chip  Chip  Chip  Chip
Chip  Chip  Chip  Chip
Chip  Chip  Chip  Chip
Chip  Chip  Chip  Chip

Multi-GPU Configuration without NVSwitch
GPU  GPU  GPU  GPU
GPU  GPU  GPU  GPU

Multi-GPU Configuration with NVSwitch
GPU  GPU  GPU  GPU
NVSwitch
GPU  GPU  GPU  GPU

Nvidia NVLink / NVSwitch (Switched any-to-any).

AMD XGMI (point-to-point).
GPU 0
GPU 1
GPU 2
GPU 3
GPU 4
GPU 5
GPU 6
GPU 7
Link labels: 45.21 GB/s; 47.056 GB/s; 46.432 GB/s; [illegible] GB/s

Interface text:
Simran's slides - Google Slides
Ask Gemini
Work
Relaunch to update
Google Slides address: [illegible]

### Evolving hardware and communication

Inference systems can spread KV cache across GPUs, CPUs, disks and remote machines, and run stages on different hardware. Larger scale-up domains and device-initiated transfers change how kernels communicate.

[8:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=497s) [Four panels illustrate hierarchical caching, larger scale-up domains, heterogeneous hardware, and device-initiated communication.](https://assets.ai.engineer/sites/aiecode2025/sha256/69b4bf981c66c7725d4ec44e93bd3be21226e3567de83fab1fb49451a3166aa3.webp)

Networking is rapidly evolving and the frontier of new inference algorithms (disaggregation).

Hierarchical caching (GPU, CPU, Disk).
Diagram labels: Host; PCIe Gen5 128 GB/s; B200; NVLink Gen5 1,800 GB/s; NVSwitch.

Growing scale up domains (NVL72, Helios).
Diagram labels: GPU [illegible]; 18x NVSwitches.

Heterogenous hardware (Vera + LPU).
Image labels: Rubin GPU; Groq 3 LPU.

Device-initiated communication (TMA, TDM).

Interface text:
Simran's slides - Google Slide[illegible]
docs.google.com/presentation/d/[illegible]
Ask Gemini
Work
Relaunch to update

### Hardware utilization and development simplicity

The goal is to achieve peak hardware utilization while keeping multi-GPU kernels simple to develop.

[9:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=591s) [Problem statement: maximizing hardware utilization and development simplicity.](https://assets.ai.engineer/sites/aiecode2025/sha256/9136982fd045e61f98f3262cde00142c7258868f357406fb727ec0a3da9ccfff.webp)

Problem:
Maximizing hardware utilization and development simplicity.

Interface text:
Simran’s slides - Google Slides
Ask Gemini
Work
Relaunch to update
docs.google.com/presentation/d/[illegible]

### Challenges in multi-GPU kernel development

Communication can leave GPUs idle, while networking hardware has improved more slowly than compute and memory. From A100 to B200, BF16 tensor cores improved 7.2×, versus 3× for intra-node and 2× for inter-node communication.

[10:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=604s) [Text describes communication overhead and hardware improvements, alongside images referencing Comet and DeepEP.](https://assets.ai.engineer/sites/aiecode2025/sha256/fef5788acf5065abd1ab3e3f292064e07eabf901a71fa3014413ad83867cc73c.webp)

It is difficult to write high performance multi-GPU kernels.

1. Communication can occupy over 50% of execution time in large language model workloads, leaving GPU compute idle [1].
2. There have been slow improvements in communication hardware: from A100 to the B200, BF16 tensor cores improved by 7.2× HBM by 5.1×, intra-node (NVLink) by only 3× and inter-node (PCIe/InfiniBand) by just 2×.

ByteDance | Seed
Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
Shulai Zhang, Ningxin Zheng, Haibin Lin, Ziheng Jiang, Wenlei Bao, Chengquan Jiang, Qi Hou, Weihao Cui, Size Zheng, Li-Wen Chang, Quan Chen, Xin Liu [author markers illegible]
¹ByteDance Seed, ²Shanghai Jiao Tong University

deepseek-ai / DeepEP
Code  Issues 198  Pull requests 78  Agents  Actions  More
Watch 102
DeepEP: an efficient expert-parallel communication library
MIT license
9.8k stars  1.3k forks  102 watching  14 Branches  1 Tag

[1] FLUX: Fast Software-based Communication Overlap On GPUs Through Kernel Fusion

Interface text:
Simran’s slides - Google Sli[illegible]
Ask Gemini
Work
Relaunch to update
docs.google.com/presentation/d/[illegible]

### Networking overhead and hardware utilization

The GEMM + All-Gather trace shows separated communication blocks and scattered memory operations in the PyTorch + NCCL reference: 320.6 μs versus a 4.99 μs theoretical roofline.

[11:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=668s) [A profiler trace compares a PyTorch and NCCL GEMM plus all-gather reference at 320.6 microseconds with a theoretical roofline of 4.99 microseconds.](https://assets.ai.engineer/sites/aiecode2025/sha256/9161fafd3b19ff55c5d72a88de48f9ab34c30e2361f05b2c42f8f736fec70748.webp)

Networking overheads waste hardware utilization.

roofline: 4.993 μs
LLM solution: 87.9 μs
PyTorch reference: 320.6 μs

Theoretical roofline
memory
compute
comm

torch.dist reference
memory
compute
comm

Time (μs)
X-axis ticks: 0, 50, 100, 150, 200, 250, 300
Legend: Memory, Compute, Communication

Profiler trace for GEMM + All-Gather: PyTorch + NCCL reference (320.6 μs) versus the ideal theoretical roofline (4.99 μs).

Interface text:
Simran’s slides - Google Slide[illegible]
docs.google.com/presentation/d/[illegible]
Ask Gemini
Work
Relaunch to update

### Approaches to multi-GPU kernel development

Arora says NCCL-based systems synchronize around transfers, compilers struggle to track networking advances, and adapting some hand-tuned kernels to another precision can take five or six months.

[12:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=752s) [Three approaches are shown: off-the-shelf APIs, compilers, and low-level primitives.](https://assets.ai.engineer/sites/aiecode2025/sha256/a40e50ddcc5596ce22327ef84ff93b0f6848f0e2d8f55ca0cf85723f06a944b8.webp)

There are three current approaches for multi-GPU kernel development.

NVIDIA
NVIDIA COLLECTIVE COMMUNICATION LIBRARY (NCCL)
[illegible] | May 2019
Installation Guide

NVIDIA/cutlass
CUDA Templates for Linear Algebra Subroutines
NVIDIA
164 Contributors
227 Issues
235 Discussions
6k Stars
1k Forks

Off-the-shelf APIs (NCCL, RCCL).
Compilers (Triton Distributed).
Low-level primitives (Comet, CUTLASS).

Interface text:
Simran’s slides - Google Slides
Ask Gemini
Work
Relaunch to update
docs.google.com/presentation/d/[illegible]

### Research question

Hand-tuning a method for another precision can take five or six months. Arora asks whether a small set of principles and methods could simplify fast multi-GPU kernel development.

[14:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=850s) [Question asking whether a small set of principles and methods can simplify writing fast multi-GPU kernels.](https://assets.ai.engineer/sites/aiecode2025/sha256/b59703b300285c2570afa1af5a40a87414d19a1149f3e505d0b6b01f5d4b64f2.webp)

Question:
Is there a small set of principles that suffices for writing fast multi-gpu kernels and are there methods that simplify the development of the kernels?

Interface text:
Simran’s slides - Google Slide[illegible]
docs.google.com/presentation/d/[illegible]
Ask Gemini
Work
Relaunch to update

### ParallelKittens and ParallelKernelBench

ParallelKittens distills multi-GPU kernel primitives and patterns; ParallelKernelBench evaluates frontier AI systems on generating fast kernels. Arora reports that the models performed poorly.

[14:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=870s) [Two columns introduce ParallelKittens and ParallelKernelBench, with author lists, kitten illustrations, and summaries of each project.](https://assets.ai.engineer/sites/aiecode2025/sha256/0ad11bb5a5f436cced381d5844e9544df40619f05ea9140dcf0aea257ffa0707.webp)

ParallelKittens (MLSys ’26)
Stuart Sul, Simran Arora, Benjamin Spector, Christopher Ré

We 1) show that a small set of principles suffices for writing performant multi-gpu AI kernels and thus 2) create a minimal set of multi-gpu kernel programming primitives.

ParallelKernelBench
Willy Chan, Nathan Paek, Simon Guo, Simran Arora, Daniel Y. Fu

We 1) produce a benchmark that covers the space of multi-gpu AI problems and 2) measure the abilities of frontier AI systems in generating fast multi-gpu kernels.

Interface text:
Simran’s slides - Google Slide
Ask Gemini
Warik
Relaunch to update
docs.google.com/presentation/d/[illegible]/edit?slide=[illegible]#slide=[illegible]

### Fundamentals

This section introduces the hardware mechanisms and parallelism strategies behind multi-GPU kernel tradeoffs.

[15:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=946s) [Section slide introducing hardware mechanisms and parallelism strategies.](https://assets.ai.engineer/sites/aiecode2025/sha256/80a49268c909907909d0565576ecc6cf893c1a13d28721fa21f80cf8b8d9ae20.webp)

Fundamentals:
Hardware mechanisms and parallelism strategies.

Interface text:
Simran’s slides - Google Slide…
Ask Gemini
https://docs.google.com/presentation/d/[illegible]/edit?slide=id.[illegible]
Work
Relaunch to update

### Transfer mechanisms and message size

For a 1 GB NVLink transfer between two H100s, device-initiated transfers reach high throughput at small message sizes. Host-initiated throughput climbs with message size and eventually exceeds it.

[15:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=954s) [Copy engine, TMA, and register instructions are listed beside a chart comparing host-initiated and device-initiated transfer throughput by message size.](https://assets.ai.engineer/sites/aiecode2025/sha256/cfa83144704aff3e5bdfa1f051a54cac5bbdfe2f1a4f8d6ef03a87e1b580c150.webp)

Tradeoff #1: Transfer mechanism.

1. Copy engine: host-initiated bulk communication.
2. TMA: device-initiated fine-grained communication.
3. Register instructions: device-initiated fine-grained communication.

Y-axis: Throughput (GB/s)
Y-axis ticks: 0, 100, 200, 300, 400
X-axis: Message Size (Bytes)
X-axis ticks: 2¹¹, 2¹³, 2¹⁵, 2¹⁷, 2¹⁹, 2²¹, 2²³, 2²⁵, 2²⁷, 2²⁹
Legend: Host-initiated; Device-initiated
Max SMEM

Observed memory bandwidth utilization for a 1 GB peer-to-peer transfer over NVLink between 2 H100s.

Interface text:
Simran's slides - Google Slides
Ask Gemini
[illegible address]
Work
Relaunch to update

### Transfer mechanisms and GPU resource use

The copy engine delivers the highest measured bandwidth without using GPU processors. TMA approaches it with few processors and registers; register instructions use more processors and achieve less bandwidth.

[17:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1021s) [Resource-use tradeoffs for copy engine, TMA, and register instructions, with bandwidth charts and a table for H100 and B200 GPUs.](https://assets.ai.engineer/sites/aiecode2025/sha256/35e1d9359b3114e2ce8ab1b39b45498610c4ace159358cbbc6630bcd4f966f32.webp)

Tradeoff #1: Transfer mechanism.

1. Copy engine: consumes none of the GPU processors, achieves peak B/W amongst mechanisms.
2. TMA: consumes few GPU processors and registers.
3. Register instructions: consumes many GPU processors and struggles to saturate B/W.

H100s
Y-axis: Bandwidth (GB/s)
Y-axis ticks: 0, 100, 200, 300
X-axis: Number of SMs
X-axis ticks: 0, 50, 100
Legend: TMA Op, Register Op, Copy Engine

B200s
Y-axis: Bandwidth (GB/s)
Y-axis ticks: 0, 200, 400, 600
X-axis: Number of SMs
X-axis ticks: 0, 50, 100, 150
Legend: TMA Op, Register Op, Copy Engine

Method | H100 BW (Ratio) | B200 BW (Ratio)
Copy Engine | 368.82 (82%) | 726.13 (81%)
TMA Op | 350.01 (78%) | 669.12 (74%)
Register Op | 342.68 (76%) | 628.35 (70%)

Interface text:
Simran’s slides - Google Slides
Ask Gemini
Work
Relaunch to update
Address: docs.google.com/presentation/d/[illegible]

### Transfer functionality and NVSwitch

All three mechanisms support P2P transfer and in-fabric broadcast; only register instructions support in-fabric reduction and elementwise transfer. The diagrams contrast direct GPU links with an NVSwitch fabric.

[18:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1087s) [GPU connection diagrams and a table compare copy engine, TMA, and register instruction functionality, including in-fabric reduction.](https://assets.ai.engineer/sites/aiecode2025/sha256/2dfbf928a97b711cecccd2cc0fd7763f49c5a6cc0b45a75f75734948f0ccd819.webp)

Tradeoff #1: Transfer mechanism.

Register instructions uniquely enable NVSwitch in-network reduction.

Multi-GPU Configuration without NVSwitch
GPU  GPU  GPU  GPU
GPU  GPU  GPU  GPU

Multi-GPU Configuration with NVSwitch
GPU  GPU  GPU  GPU
NVSwitch
GPU  GPU  GPU  GPU

Functionality | CE | TMA | Reg
P2P Transfer | ✓ | ✓ | ✓
In-fabric Broadcast | ✓ | ✓ | ✓
P2P Reduction | × | ✓ | ✓
In-fabric Reduction | × | × | ✓
Elementwise Transfer | × | × | ✓

Interface text:
Simran’s slides - Google Slide[illegible]
docs.google.com/presentation/d/[illegible]/edit?slide=[illegible]
Ask Gemini
Work
Relaunch to update

### Foundation models and multi-GPU fundamentals

Can models reason through multi-GPU kernel tradeoffs and generalize beyond single-GPU benchmarks?

[21:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1290s) [Question asking how well foundation models understand and exploit these fundamentals.](https://assets.ai.engineer/sites/aiecode2025/sha256/30ca95e6541e35c3c64ebf683e65c7b0748c7ec711c564ee57831d30dfb4e535.webp)

How well do foundation models understand and exploit these fundamentals?

Interface text:
Simran’s slides - Google Slide[illegible]
docs.google.com/presentation/d/[illegible]
Ask Gemini
Work
Relaunch to update

### ParallelKernelBench

ParallelKernelBench gives a model a distributed PyTorch reference and GPU topology, then compares its generated CUDA kernel with the reference per rank for correctness, performance, and communication.

[21:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1317s) [Diagram showing a language model generating a multi-GPU kernel from a distributed reference and topology, then evaluating it per GPU rank.](https://assets.ai.engineer/sites/aiecode2025/sha256/a8e3161200ec95bc01f9439e8ec540b05c7a4125b67d80d02ecb4679ed7fe3ae.webp)

ParallelKernelBench

A distributed PyTorch reference and network topology go in; the model returns a kernel, evaluated per-rank.

Task instruction

Network topology & capability
4x H100
NVLink · NVSwitch · NVLS on
NVSWITCH

PyTorch (NCCL calls)
def solution():
    C_t = B.T @ A
    dist.all_reduce()
    ...

Reference torch distributed

Language model

Custom kernel
CUDA
with multi-GPU communication

Optimized CUDA kernel

Evaluate per GPU rank
Randomized inputs
Reference
Generated
Benchmark over n_trial times

Correctness
Output match?

Performance
Wall-clock speedup?

Communication
Comm. roofline?

Interface text:
Simran’s slides - Google Slides
Ask Gemini
Work
Relaunch to update
docs.google.com/presentation/d/[illegible]

### Taxonomy of multi-GPU parallelism

PKB maps parallelism choices across transformer layers and selects representative problems for each. The example combines sequence and tensor parallelism, with AllGather and ReduceScatter between operations.

[22:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1355s) [Parallelism schemes for transformer layers alongside a representative model using sequence and tensor parallelism.](https://assets.ai.engineer/sites/aiecode2025/sha256/feb798939f530d3caca38e7b056c170732f0ce4969e18fddc3de5d140d495cb7.webp)

To ensure that PKB provides high coverage over multi-GPU settings, we identify a taxonomy of parallelism schemes and propose problems for each setting.

[PP] layers
Normalization: [SP]
Attention:
[TP] Shard heads
[CP] Ring attention / Ulysses
[DP] Different sequence
Normalization: [SP]
MLP:
[TP] Dense
[EP] MoE
[TP] MoE
[EP + TP] MoE
Other: Graph, Spatial, ...

[D]ata · [S]equence · [T]ensor · [C]ontext · [E]xpert · [P]ipeline
Parallelism

[SP] RMS norm
AllGather
[TP] GQA attention
shard on heads
ReduceScatter
[SP] RMS norm
AllGather
[TP] Dense MLP
(GeGLU)
ReduceScatter
...

Expresses representative models
e.g. one Gemma3-27B layer

Interface text:
Simran's slides - Google Slides
Ask Gemini
Work
Relaunch to update
docs.google.com/presentation/d/[illegible]

### PKB problem composition

Among 100 parallelism strategies, tensor leads with 17, followed by data and context with 15 each. Among 87 problem sources, RL post-training leads with 25, followed by LLM training with 17.

[23:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1401s) [Pie charts show parallelism strategies and problem source categories in ParallelKernelBench.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ebc64586610cca97ecd5e99df808f88cf4b6e64e854b85a7f65c63fff03f8c8.webp)

PKB problem composition.

Parallelism strategies (n = 100)
Tensor (17)
Data (15)
Context (15)
Expert (13)
FSDP / ZeRO (12)
Collective (8)
Graph (6)
Geometric / Spatial (5)
Vocab (4)
Sequence (2)
Other (2)
Pipeline (1)

Problem source categories (n = 87)
RL post-training (25)
LLM training (17)
Kernel libraries (10)
NVIDIA/nccl (8)
Vision models (8)
Other (6)
GNN frameworks (6)
Scientific computing (5)
Inference engines (2)

Interface text:
Simran’s slides - Google Slide[illegible]
docs.google.com/presentation/d/[illegible]
Ask Gemini
Work
Relaunch to update

### Results

The talk turns to how models perform on ParallelKernelBench.

[23:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1430s) [Results section title on a pale blue background.](https://assets.ai.engineer/sites/aiecode2025/sha256/9e3e614618f5dade1110bd81b60ef6f9bbc19bc471245982dd131650ff30d9d9.webp)

Results

Interface text:
Simran’s slides - Google Slide[illegible]
docs.google.com/presentation/d/[illegible]
Ask Gemini
Work
Relaunch to update

### The few correct kernels are slow

As the required speedup relative to the reference rises, the fraction of problems solved with a qualifying correct kernel falls for every model. GPT-5.5 leads, dropping from about 0.30 at p=0 to 0.12 at p=3.

[25:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1549s) [Line chart comparing fast-p scores across models as the speedup threshold increases; GPT-5.5 leads throughout.](https://assets.ai.engineer/sites/aiecode2025/sha256/4da7a3d36220ec847f360f91badd1e3c12807b61d1a3005cfa4c792fced4440e.webp)

The few correct kernels are slow.

Y-axis: fastₚ score
Y-axis ticks: 0.00, 0.05, 0.10, 0.15, 0.20, 0.25, 0.30
X-axis: Speedup threshold (p)
X-axis ticks: 0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0
Legend: GPT-5.5; Gemini 3 Pro; Claude Opus 4.7; GLM 5.1; DeepSeek V4 Pro

fast-p: fraction of problems solved with a correct kernel at least p times faster than the reference; higher is better.

Interface text:
Simran’s slides - Google Slide[illegible]
Ask Gemini
Work
Relaunch to update
docs.google.com/presentation/d/[illegible]

### PKB failure modes by model

Across 87 PKB problems per model, GPT-5.5 has 28 correct outcomes; GLM-5.1 has 6. Arora says models also struggle with tradeoffs such as collective ordering and data partitioning.

[26:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1586s) [Stacked bars show counts of correctness and failure categories across 87 problems for each model.](https://assets.ai.engineer/sites/aiecode2025/sha256/832900f3266d8f9e1ff23f15285fb879812ffe37e1f40eb8093d8fc4c65daa31.webp)

PKB failure modes by model.

GPT-5.5: 16, 3, 3, 6, 26, 5, 28
Gemini 3 Pro: 10, 7, 11, 24, 10, 24
Opus 4.7: 26, 7, 12, 12, 9, 20
GLM-5.1: 31, 6, 9, 8, 17, 10, 6
DeepSeek V4 Pro: 36, 20, 8, 6, 14

X-axis: Percentage of problems in batch
X-axis ticks: 0, 20, 40, 60, 80, 100

Legend: nvcc/syntax errors; Tensor shape error; Attribute/name/API typo; Runtime error; Output mismatch; Deadlock; Correct

Outcome distribution across 87 PKB problems per model; segment labels show problem counts.

Interface text:
Simran’s slides - Google Slides
Ask Gemini
Work
Relaunch to update
[illegible]

### Agentic evaluation loops

Gemini 3 Pro used a local shell to write and test solutions, raising the number solved from 24 to 35 of 87 problems. Performance plateaued as the team gave it more time.

[27:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1625s) [Diagram shows a language model running commands and iterating until a solution passes or the budget is exhausted. The slide says loops correct syntax and shape errors but do not foster algorithmic innovation.](https://assets.ai.engineer/sites/aiecode2025/sha256/145b1085ed7a928a060c7224061d18829fd9264f1a0e0904a33bcad9bd2d6c4c.webp)

Agentic loops help correct syntax and shape errors, but does not help the model innovate algorithmically.

Evaluation instructions
System prompt
In-context examples
Language model
Write a file?
Run evaluation?
Run test?
Shell executes command
Satisfied / out of budget?
No? Then iterate!
Yes? Done!
Solution
Python file

Agentic evaluation loop: the model writes files, runs correctness tests and benchmarks, and iterates until the solution passes or the step budget is exhausted.

Interface text:
Simran’s slides - Google Slide [truncated]
Ask Gemini
docs.google.com/presentation/d/[illegible]
Work
Relaunch to update

### NeMo vocab-parallel log-prob with top-k and top-p filtering

Gemini 3 Pro-generated CUDA reaches 1.28× the CuBLAS + NCCL reference at sequence length 1,024, then 1.06–1.07× at longer lengths.

[27:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1678s) [Bar chart shows Gemini 3 Pro-generated CUDA speedups of 1.06 to 1.28 times the CuBLAS and NCCL reference across sequence lengths.](https://assets.ai.engineer/sites/aiecode2025/sha256/743c7c9e3763c971e3bfdcc92ca79b46f4b37c08a74ef89e8b0336f3cbcd3cf8.webp)

Net new! NeMo vocab-parallel log-prob with top-k/top-p filtering.

Y-axis: Kernel speedup (x)
Y-axis ticks: 0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50
X-axis: Sequence length
X-axis ticks: 1,024, 2,048, 4,096, 8,192, 16,384
Speedup labels: 1.28x, 1.06x, 1.07x, 1.07x, 1.06x
Legend: CuBLAS + NCCL (reference); Gemini 3 Pro-generated CUDA

Mean kernel speedup relative to the CuBLAS + NCCL reference.

Interface text:
Simran’s slides - Google Slid[illegible]
docs.google.com/presentation/d/[illegible]
Ask Gemini
Work
Relaunch to update

### Hyena forward context parallelism

GPT-5.5-generated CUDA reaches 3.55× and 1.82× the CuBLAS + NCCL reference at sequence lengths 1,024 and 2,048, then falls to 0.70× at 8,192 and 16,384.

[28:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1717s) [Bar chart shows GPT-5.5-generated CUDA speedups of 3.55, 1.82, 0.70 and 0.70 times the reference at increasing sequence lengths.](https://assets.ai.engineer/sites/aiecode2025/sha256/265a4f2ed4925d220e5402e3c4a38029318818e32f78a9817563486c116e4992.webp)

Net new! Hyena forward context parallelism.

Y-axis: Kernel speedup (x)
Y-axis ticks: 0.00, 0.50, 1.00, 1.50, 2.00, 2.50, 3.00, 3.50, 4.00
X-axis: Sequence length
X-axis ticks: 1,024, 2,048, 8,192, 16,384
Bar labels: 3.55x, 1.82x, 0.70x, 0.70x
Legend: CuBLAS + NCCL (reference); GPT-5.5-generated CUDA

Mean kernel speedup relative to the CuBLAS + NCCL reference.

Interface text:
Simran’s slides - Google Slides
Ask Gemini
Work
Relaunch to update
docs.google.com/presentation/d/[illegible]

### SAM3 IoU suppression

GPT-5.5-generated CUDA reaches 1.03× to 1.38× the reference speed across 16–512 tracked objects, with the largest gain at 16 objects.

[28:43](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1723s) [Bar chart shows GPT-5.5-generated CUDA speedups of 1.03 to 1.38 times the reference across numbers of tracked objects.](https://assets.ai.engineer/sites/aiecode2025/sha256/791aa14012576fcc3c0552dfd4821ee654a103ebb61888bcd48d9162a58f6d8f.webp)

Net new! SAM3 IoU suppression.

Y-axis: Kernel speedup (x)
Y-axis ticks: 0.00, 0.25, 0.50, 0.75, 1.00, 1.25, 1.50
X-axis: Number of objects tracked
X-axis ticks: 16, 32, 64, 128, 256, 512
GPT-5.5-generated CUDA values: 1.38x, 1.25x, 1.13x, 1.03x, 1.10x, 1.03x
Legend: Reference (1x); GPT-5.5-generated CUDA

Mean kernel speedup relative to the reference implementation.

Interface text:
Simran’s slides - Google Slid[illegible]
docs.google.com/presentation/d/[illegible]
Ask Gemini
Work
Relaunch to update

### In conclusion

ParallelKittens captures multi-GPU programming tradeoffs in a few primitives, but models still struggle to reason through them in context. Next steps include solving PKB and exploring evolving networking and communication.

[28:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1730s) [Conclusion slide summarizing multi-GPU kernel programming lessons, future research directions, and a hiring announcement.](https://assets.ai.engineer/sites/aiecode2025/sha256/92f297849701644620588242b0d7527f889b50e3eeb675cdb35a0c8e5df4210e.webp)

In conclusion.

1. Lessons.
• A few simple tradeoffs govern multi-gpu kernel programming, demonstrated by our ability encapsulate the principles in a few programming primitives (ParallelKittens).
• Models do not currently understand these tradeoffs or demonstrate an ability to reason about the tradeoffs in-context.

2. Next steps.
• Solving PKB to obtain net new kernels!
• How should AI architectures change as the networking stack evolves (larger scale up, disaggregated inference, heterogeneous hardware)?
• Inter-node communication, multi-silicon communication.

We’re hiring for Frontier Performance Research! Contact [redacted].

Interface text:
Simran’s slides - Google Slide
Ask Gemini
Google Slides URL: docs.google.com/presentation/d/[illegible]
Work
Relaunch to update

### AI Engineer World's Fair

[29:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1795s) [Black slide with the white AI Engineer World's Fair logo in a rectangular frame and the event website below.](https://assets.ai.engineer/sites/aiecode2025/sha256/c24648b4784e191004b72e48b2d0c5946ecec65a1333a8db47fb2edb7933fd4c.webp)

AI Engineer
World’s Fair

HTTPS://AI.ENGINEER

## Full transcript

[0:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1s) [music]

[0:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=12s) Hi everyone. Uh, sorry it's a bit loud

[0:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=15s) in here. Was not expecting this. Um, I'm

[0:18](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=18s) Siman. I'm a principal scientist at

[0:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=20s) Together AI. Um, I previously did my PhD

[0:23](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=23s) in the Hazy Research Lab with Chris Ray

[0:25](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=25s) at Stanford and I'm an incoming

[0:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=27s) professor at Caltech. Um I lead the

[0:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=30s) frontier performance research team at

[0:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=33s) together where we develop systems,

[0:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=34s) frameworks and algorithms to extract as

[0:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=37s) much performance as possible out of

[0:39](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=39s) modern um AI hardware. Today I want to

[0:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=42s) share a little bit about our

[0:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=44s) contributions towards simplifying the

[0:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=46s) development of uh multi-GPU AI kernels.

[0:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=51s) A few years ago, um, GPU utilization

[0:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=54s) used to be limited by poor intraGPU

[0:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=57s) memory access and single GPU kernels.

[1:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=61s) But with significant investment in

[1:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=63s) better kernels like flash attention, uh,

[1:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=66s) memory efficient architectures like from

[1:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=68s) deepseek, sparse attentions, mambas and

[1:11](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=71s) so on, um, and better DSLs, we've sort

[1:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=74s) of shifted the bottleneck to multi-GPU

[1:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=77s) communication.

[1:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=81s) During this talk, I'll start by telling

[1:23](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=83s) you a little bit about why now, why GPU

[1:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=86s) networking now. Then I'll tell you about

[1:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=89s) um the sort of problem space. So what

[1:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=91s) are the challenges in maximizing

[1:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=93s) hardware utilization and development

[1:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=95s) simplicity for multiGPU kernels. Um

[1:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=98s) three, we'll talk a little bit about the

[1:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=100s) fundamentals be behind designing

[1:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=102s) effective multiGGPU kernels. Four, we'll

[1:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=105s) look at whether frontier AI models can

[1:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=109s) uh leverage these fundamental

[1:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=111s) principles. Do they understand them? Can

[1:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=113s) they reason about them? Um, you know, in

[1:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=115s) theory, these models are very good at

[1:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=117s) reasoning. Um, and then five, we'll talk

[2:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=120s) through the results of these frontier

[2:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=121s) models on a benchmark that we've

[2:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=123s) developed called um parallel kernel

[2:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=126s) bench for multiGPU kernel generation

[2:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=128s) evaluation.

[2:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=132s) Okay, before we dive into those five

[2:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=134s) parts, just basic preliminaries. So,

[2:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=137s) this is an Nvidia GPU, uh, an H100 GPU

[2:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=141s) that you can see on the screen. Um, I

[2:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=144s) always like to help ground people in GPU

[2:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=147s) kernels via looking at the hardware. So,

[2:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=150s) these rainbow colored dots are

[2:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=152s) processors where actual compute is

[2:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=154s) happening. um all of the you know

[2:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=156s) parallel threads are operating within

[2:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=158s) one of those colored dots and there's

[2:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=161s) typically you know 100 200 of them on

[2:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=164s) modern AI GPUs.

[2:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=167s) Um around those processors you can see

[2:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=170s) um some of the memory that these

[2:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=172s) processors retrieve data so large

[2:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=175s) weights activations from. So these

[2:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=178s) rectangles between the colored dots are

[3:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=181s) an L2 cache slightly faster memory. um

[3:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=184s) not a ter like crazy large amount of it.

[3:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=187s) And then these black boxes are high

[3:09](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=189s) bandwidth memory. So when you Nvidia Smi

[3:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=192s) and see you know 80 gigabytes, 1008

[3:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=195s) gigabytes, whatever it is on your GPU,

[3:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=197s) that's that memory.

[3:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=199s) Um a GPU is operating a highly parallel

[3:23](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=203s) program. So multiple threads are

[3:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=206s) combined together in uh into larger

[3:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=209s) coarser units. and we schedule these

[3:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=212s) threads and and blocks onto these

[3:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=214s) processors to perform our AI compute.

[3:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=218s) Beyond the GPU, we'll have multiple GPUs

[3:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=221s) and we'll also have, you know, CPUs um

[3:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=224s) that have memory as well.

[3:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=229s) Um so to perform computation um the

[3:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=233s) memory that these threads use is going

[3:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=235s) to be stored in a really fast register

[3:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=237s) memory that's right next to the

[3:59](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=239s) computation units. um simple physics if

[4:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=242s) I am pulling data from very very close

[4:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=246s) to my proc my compute unit it's really

[4:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=248s) fast to get to it because it you know

[4:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=250s) that data is right next to me but

[4:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=252s) there's not a large radius and not a

[4:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=254s) large volume of space that's close by to

[4:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=257s) my process uh my my compute units and so

[4:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=260s) I don't have very much of it so you can

[4:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=262s) see that the fastest memory here the

[4:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=264s) registers is sup is you know 130

[4:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=267s) terabytes per second on an H100 but we

[4:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=269s) don't have very much on of it and as we

[4:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=272s) go to the further away memory we have a

[4:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=274s) lot more of it but it takes longer to

[4:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=276s) reach it. Uh again simple physics.

[4:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=280s) So in multiGPU systems in particular um

[4:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=284s) there is a hierarchy of interconnects.

[4:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=286s) So we will have something called PCIe um

[4:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=289s) as the channel for CPUGGPU

[4:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=291s) communications. We'll have multiGPU or

[4:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=294s) multi- uh node communications over

[4:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=297s) infiniband TCP and then in the we're

[5:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=301s) going to focus mostly on the Nvidia

[5:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=303s) sphere here. Um in uh the intraGPU

[5:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=306s) regime we'll have NVLink um providing

[5:09](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=309s) point-to-point connections between GPUs

[5:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=312s) and the NV switch. NV switch connects

[5:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=315s) all NVLink in endpoints into a

[5:18](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=318s) non-blocking fabric for full GPUGGPU

[5:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=321s) communication and NV switch is exciting

[5:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=324s) because it also provides support for in

[5:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=326s) network offdevice acceleration for oper

[5:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=330s) like communication primitives like

[5:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=333s) multiccast and reductions.

[5:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=337s) Okay, so diving in with the

[5:39](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=339s) preliminaries in mind. Why GPU

[5:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=342s) networking now?

[5:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=344s) So as I mentioned at the beginning,

[5:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=346s) we've really put a lot of effort into

[5:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=349s) making AIO uh more efficient over recent

[5:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=352s) years. Um again, we have uh

[5:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=355s) architectures that use less compute,

[5:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=357s) less memory like Mamba or sparse

[5:59](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=359s) attentions. We have algorithms that make

[6:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=362s) AI more hardware affair aware like flash

[6:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=365s) attention. We have tools to make it easy

[6:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=368s) to map AI algorithms to the hardware

[6:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=370s) like tileang, mojo, triton, gluon,

[6:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=373s) thunder kittens. Um, and we have new

[6:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=376s) techniques to overlap uh execution

[6:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=379s) across many AI operators very tightly

[6:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=381s) like mega kernels. And we also have

[6:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=384s) tools to make it easy to run on multiple

[6:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=387s) vendor and silicon platforms like

[6:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=389s) thunder mittens for Apple silicon or or

[6:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=392s) hipkittens for AMD and so on.

[6:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=395s) Um, at this point we really believe that

[6:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=398s) GPU networking offers many new and

[6:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=400s) exciting opportunities for AI

[6:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=402s) efficiency.

[6:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=405s) Modern AI workloads are getting very big

[6:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=407s) and require kernels that span multiple

[6:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=410s) um, you know, GPUs. So on many

[6:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=413s) production um distributed training and

[6:56](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=416s) inference workloads, communication is

[6:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=418s) increasingly consuming the majority of

[7:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=421s) the runtime and yields low model flop

[7:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=424s) utilization at scale.

[7:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=428s) So the uh pace of innovation and

[7:11](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=431s) diversity of approaches that different

[7:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=433s) hardware providers are taking in their

[7:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=435s) networking stacks is another reason why

[7:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=437s) it's an exciting time to study um you

[7:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=440s) know networking and communication. So we

[7:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=442s) can see here um AMD hardware with uh

[7:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=446s) what's called XGMI interconnects um

[7:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=449s) providing pointto-point links between

[7:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=451s) different GPUs in a scaleup domain. We

[7:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=455s) can see here TPU and uh interconnect uh

[7:39](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=459s) as well. So the TPU will use a 3D Taurus

[7:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=462s) and also have optical wraparound links.

[7:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=465s) So yet another diverse form of the

[7:48](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=468s) topology and links. Um and then again

[7:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=470s) for Nvidia we'll have the INV um switch

[7:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=474s) which integrates um compute capabilities

[7:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=477s) directly into the interconnect fabric

[8:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=480s) like for in network reductions and then

[8:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=483s) we'll have our NV link providing up to

[8:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=485s) you know 900 gigabytes of unidirectional

[8:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=487s) bandwidth between any two remote GPUs

[8:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=490s) high bandwidth memory um on you know

[8:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=494s) particular generation of Nvidia hardware

[8:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=499s) um Beyond the diversity in the

[8:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=500s) networking stacks, there's also a lot of

[8:23](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=503s) evolution in um how AI workloads are

[8:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=507s) adapting to take advantage of this

[8:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=509s) hardware and the diversity of types of

[8:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=512s) um you know hardware that we're we're

[8:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=514s) using simultaneously for one AI

[8:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=516s) workload. So um KV cache memory in

[8:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=520s) modern inference systems is going to

[8:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=521s) beworked across GPU, CPU, disk and

[8:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=524s) remote machines. um inference systems

[8:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=527s) increasingly disagregate different steps

[8:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=530s) of inference across different hardware

[8:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=532s) backends. So you could run speculative

[8:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=534s) decoding on some hardware, decode on

[8:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=537s) different hardware, prefill on different

[8:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=538s) hardware. Um hardware is also evolving

[9:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=542s) to have larger scaleup domains than ever

[9:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=544s) before with you know 72 GPUs and a

[9:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=547s) scaleup domain in in the coming um chips

[9:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=550s) and Nvidia planning on a single system

[9:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=552s) in 2027 with 576 GPUs.

[9:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=557s) Um at the same time to take advantage of

[9:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=559s) these more uh intensive scaleup domains

[9:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=562s) we're getting richer uh primitives for

[9:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=564s) fine grained control and kernel writing

[9:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=567s) over these domains. So we have something

[9:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=570s) called tensor memory acceleration where

[9:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=572s) we can provide uh perform asynchronous

[9:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=575s) network transfers from the device side

[9:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=577s) um on these GPUs.

[9:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=580s) So all of these changes are opening up

[9:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=582s) new opportunities and challenges in um

[9:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=585s) both AI you know how do we build models

[9:48](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=588s) that take advantage of these trends and

[9:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=590s) in systems.

[9:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=593s) Okay, so the problems that we're going

[9:56](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=596s) to go after um how do we get peak

[9:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=598s) hardware utilization and also

[10:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=600s) development simplicity um for these

[10:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=603s) multiGPU kernels.

[10:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=606s) So um it's been very difficult to write

[10:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=610s) multiGGPU kernels and there's a lot of

[10:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=612s) there are a lot of papers a lot of

[10:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=614s) systems reports that document you know

[10:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=616s) challenges here. Um, one of the things

[10:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=619s) here is it's compounded by the fact that

[10:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=621s) communication hardware around GPUs has

[10:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=624s) progressed a lot more slowly relative to

[10:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=627s) compute and memory. Um, so comparing

[10:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=630s) NVIDIA A100's in 2020 to B200s in 2024,

[10:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=636s) u BF16 tensor core speeds improved by

[10:39](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=639s) 7.2x.

[10:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=640s) um while intra node communication by

[10:43](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=643s) just 3x and inter node communication by

[10:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=646s) just 2x. Um and coming back to my points

[10:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=649s) about how diverse networking is right

[10:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=651s) now things like tensor cores that run

[10:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=654s) map moles and our memory hierarchies are

[10:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=657s) pretty consistent and resemble one

[10:59](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=659s) another across diverse AI vendors and

[11:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=661s) multisilicon. Um but again as I I

[11:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=664s) mentioned the networking stack is

[11:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=665s) something that is really different

[11:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=666s) across vendors still

[11:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=670s) um you know a first step as we went

[11:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=672s) about all this work is to just study the

[11:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=674s) baselines um that are out there. So um

[11:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=677s) one of the popular tools for um

[11:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=681s) communications is this nickel library or

[11:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=684s) Rickle on AMD um that both you know

[11:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=687s) companies respectively spend a lot of

[11:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=689s) you know engineering investment into

[11:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=691s) releasing to make it easy for people to

[11:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=693s) do multiGPU work. Um but they're not

[11:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=696s) very flexible. So they're tuned for bulk

[11:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=698s) transfers for large contiguous chunks of

[11:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=701s) data transfers. And the design really

[11:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=704s) breaks down when you care about peak

[11:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=706s) performance, fine grain communication,

[11:48](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=708s) um, and sort of non-trivial collectives

[11:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=711s) that you want to fuse together.

[11:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=714s) So as a result, you can achieve much

[11:56](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=716s) higher performance by writing custom

[11:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=718s) communication kernels that directly

[12:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=721s) address these needs.

[12:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=723s) If we look at a naive baseline that's

[12:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=725s) representative of very popular libraries

[12:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=728s) in machine learning stacking pietorch

[12:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=730s) with nickel um we can we find that

[12:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=733s) across um you know the many uh problems

[12:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=737s) in our parallel kernel bench benchmark

[12:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=739s) that the majority of these um simple

[12:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=742s) baselines will fall below 50% of their

[12:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=746s) communication aware roof line bound. So

[12:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=750s) there's a lot of room for improvement

[12:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=752s) here.

[12:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=753s) Um the current frameworks beyond um

[12:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=756s) nickel which is popular in like Megatron

[12:39](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=759s) LM, Flex Flow, Nanoflow um you know all

[12:43](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=763s) again these systems are primarily

[12:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=765s) orchestrating bulk collectives via

[12:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=767s) nickel um and require synchronization

[12:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=770s) before and after data transfers. So

[12:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=772s) beyond th those off-the-shelf libraries,

[12:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=774s) we have compilers and DSLs that exist.

[12:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=778s) So there's Triton distributed is one of

[13:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=781s) them. Um and uh you know tile link is

[13:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=784s) another one. Um we have found that it's

[13:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=787s) very difficult to support the rapid pace

[13:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=790s) of networking improvements within these

[13:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=792s) frameworks. So our benchmarks and our

[13:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=795s) papers highlight results where Triton

[13:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=797s) distributed originally tuned around 8 uh

[13:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=800s) H800 GPUs fails to adapt efficiently to

[13:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=804s) other architectures like H100s.

[13:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=807s) And then the third category of how

[13:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=810s) people can proceed here is to really

[13:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=812s) handtune specific AI operators one by

[13:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=816s) one. So there's a lot of popular work um

[13:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=818s) DPP um comet ring attention um flux

[13:43](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=823s) flashdoe

[13:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=825s) um and then several distributed gem

[13:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=827s) kernels from cutless and these methods

[13:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=830s) achieve peak performance but often um

[13:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=834s) they do not you know some of these

[13:56](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=836s) methods have been designed in one

[13:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=838s) precision and it takes five or six

[14:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=840s) months to scale it to another precision

[14:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=842s) and just the scalability of this hand

[14:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=844s) tuning and um fine grain kernel writing

[14:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=847s) is not very um effective.

[14:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=850s) So um with this landscape in mind, our

[14:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=853s) research question was really about

[14:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=855s) whether there is a small set of

[14:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=857s) principles and fundamentals that really

[14:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=860s) governs multiGPU kernel writing and

[14:23](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=863s) whether there are methods that can

[14:25](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=865s) leverage those principles if they exist

[14:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=867s) to simplify the development of these

[14:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=869s) kernels.

[14:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=872s) Um I'll briefly highlight two works here

[14:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=874s) that um govern like that that represent

[14:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=877s) our approach. So first we think it's

[14:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=880s) important to build our own fundamental

[14:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=882s) understanding and to manually do the

[14:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=884s) work to understand it rather than just

[14:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=886s) throwing say an LLM at the problem. So

[14:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=890s) we spent the time to build out parallel

[14:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=892s) kittens which is a small set of minimal

[14:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=895s) primitives and patterns for multiGPU

[14:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=897s) kernels. Um we use this to understand

[15:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=900s) the trade-offs of multiGPU kernels and

[15:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=903s) to one write a large collection of um

[15:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=906s) peak performance kernels for a variety

[15:09](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=909s) of parallelism schemes. Um and this I

[15:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=912s) will use to hopefully you know educate

[15:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=915s) and bring us all on the same page on

[15:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=916s) what patterns we figured out. Um and

[15:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=919s) then once we found that there is indeed

[15:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=922s) a small set of trade-offs governing this

[15:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=924s) landscape, we were curious whether

[15:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=926s) models, especially these models right

[15:28](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=928s) now that claim to be very good at kernel

[15:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=931s) writing and also reasoning um could

[15:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=934s) reason about these trade-offs when we

[15:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=936s) provide them in context to actually

[15:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=938s) generate a bunch of net new multiGPU

[15:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=940s) kernels for us. Unfortunately, we found

[15:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=944s) they were not very good, but we'll dive

[15:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=945s) into more of that at the end.

[15:48](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=948s) Um so just the fundamental section this

[15:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=950s) is going to be more you know educational

[15:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=952s) what are the trade-offs that go into

[15:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=954s) these kernels.

[15:56](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=956s) So there are three main ways to do um

[15:59](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=959s) intraGPU data transfers. Um there's the

[16:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=963s) per GPU what's called copy engine and

[16:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=966s) this is host or CPU initiated work. It's

[16:09](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=969s) really good for large message transfers.

[16:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=972s) So when your message size, the amount of

[16:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=975s) data being transferred is really big um

[16:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=977s) and it can get to sort of like peak

[16:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=979s) bandwidth um on on the communication

[16:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=982s) side. In contrast, you can use device

[16:25](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=985s) initiated or GPU initiated transfers via

[16:28](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=988s) that tensor memory accelerator that I

[16:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=990s) mentioned or via register level

[16:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=992s) instructions called uh in sort of their

[16:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=995s) PTX lingo like LDST red multime. Um, and

[16:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1000s) the uh T TMA is really nice. These

[16:43](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1003s) device initiated ones are really nice

[16:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1005s) because they can saturate our NVLink

[16:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1007s) bandwidth using relatively small message

[16:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1011s) sizes. And this means that they can be

[16:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1013s) really nice when we're trying to do fine

[16:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1015s) grain communication rather than sending

[16:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1017s) bulk amounts of data over the links all

[16:59](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1019s) at once coarsely.

[17:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1022s) Um, there are some trade-offs here. So

[17:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1025s) the copy engine is really nice because

[17:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1027s) it doesn't take away or you know waste a

[17:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1030s) lot of our precious registers that I

[17:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1032s) mentioned are important for compute on

[17:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1034s) the GPU. Um and it doesn't also use any

[17:18](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1038s) of those rainbow colored dots the

[17:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1040s) processors on our GPU allowing us to

[17:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1042s) repurpose those for memory or um

[17:25](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1045s) computation on our uh you know other

[17:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1047s) parts of the AI pipeline. Um, TMA, the

[17:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1051s) second option here, consumes very few

[17:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1053s) registers, which is why it's nice. Um,

[17:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1056s) and it also can achieve high utilization

[17:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1058s) using very few of our processors. So,

[17:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1061s) it's a nice useful tool for fine grain

[17:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1064s) overlapping. Um, TMA does have

[17:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1067s) limitations. It can't effectively take

[17:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1069s) advantage of these in network um,

[17:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1071s) computations that I mentioned are

[17:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1073s) feasible with technologies like NV

[17:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1075s) switch. And the register level

[17:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1078s) instructions are really nice for being

[18:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1080s) able to take advantage of um you know

[18:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1083s) those those sort of in network

[18:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1085s) reductions that NV switch offers.

[18:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1088s) So again there are different tradeoffs

[18:11](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1091s) different functionalities that these

[18:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1093s) transfer mechanisms offer and they face

[18:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1095s) different trade-offs.

[18:18](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1098s) Second um beyond transfer mechanism the

[18:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1101s) trade-off is around how to overlap

[18:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1104s) compute memory and uh communication in

[18:28](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1108s) GPU kernels. So there's two main

[18:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1110s) categories of schedules. The first is

[18:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1114s) intraSM within one of those rainbow dots

[18:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1118s) um where we'll we'll have different

[18:39](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1119s) warps or threads within that processor

[18:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1122s) specialized to handle either compute or

[18:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1125s) one specialized for communication

[18:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1127s) concurrently. Um we can dedicate you

[18:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1129s) know different warps to each of these.

[18:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1132s) The challenge with this intm overlapping

[18:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1135s) is that the communication and

[18:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1137s) computation pattern really need to like

[18:59](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1139s) align and jive with one another. they

[19:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1141s) need to use the same data as inputs for

[19:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1144s) the computation and communication. When

[19:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1147s) they don't align, you could use

[19:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1148s) something like interm um schedules that

[19:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1152s) are shown on your right here where we'll

[19:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1155s) now have each of the different rainbow

[19:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1157s) colored dots on our GPUs, those

[19:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1159s) different processors specialized to

[19:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1161s) compute communication and memory. Um and

[19:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1166s) so this this is nice when the colonel

[19:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1169s) would others wise need to split across

[19:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1171s) resources like the register file or

[19:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1173s) shared memory um across these different

[19:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1176s) steps in misaligned ways.

[19:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1181s) Um this is also really nice when it's

[19:43](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1183s) hard to maximize NVLink traversal with

[19:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1185s) intram

[19:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1187s) overlapping.

[19:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1190s) So I just wanted to highlight one quick

[19:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1192s) example here where each of the patterns

[19:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1194s) excels in popular you know AI uh uh like

[19:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1197s) kind of patterns that you'll see. So on

[20:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1200s) a gem plus uh reduced uh scatter here we

[20:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1204s) can see that the um intraSM overlapping

[20:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1208s) scheduleuler schedule is very effective

[20:11](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1211s) in the gem plus all reduce we can see

[20:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1214s) that the interSM which again leverages

[20:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1217s) the in network reductions of envy switch

[20:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1219s) is very effective. So we face these

[20:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1222s) trade-offs and you can read more about

[20:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1224s) um the design decisions that go into

[20:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1226s) them in our parallel kittens paper.

[20:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1230s) Um, and then finally, ideally,

[20:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1231s) abstraction should allow the developer

[20:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1233s) flexibility to control how they're

[20:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1235s) buffering and synchronizing between data

[20:39](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1239s) senders and receivers.

[20:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1241s) So, we encapsulated these ideas into

[20:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1244s) parallel kittens. Again, a simple set of

[20:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1246s) programming primitives and templates for

[20:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1249s) these multiGPU kernels. Parallel Kittens

[20:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1252s) is used in production at Together AI as

[20:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1254s) well as our partner um you know Cursor

[20:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1257s) and other uh companies in the AI space.

[21:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1261s) Um here's some sample code. I won't

[21:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1263s) spend too much time here, but we usually

[21:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1265s) add roughly a dozen lines of code over a

[21:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1267s) single GPU kernel to insert these

[21:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1270s) multiGPU primitives.

[21:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1273s) Um, and you can see here across data

[21:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1276s) sequence and expert parallelism how our

[21:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1279s) parallel kittens kernels are achieving

[21:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1282s) state-of-the-art results um, compared to

[21:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1284s) strong reference baselines. And you can

[21:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1286s) check out our repo to learn more.

[21:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1290s) Okay, so we understand a little bit

[21:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1293s) about um the trade-offs that underly

[21:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1295s) these multiGPU kernels and there's just

[21:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1297s) a couple main, you know, ones that

[21:39](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1299s) exist. So can models reason through them

[21:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1302s) um and give us these uh you know kernels

[21:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1305s) models are getting better at reasoning

[21:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1307s) today. Um do they generalize well to

[21:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1310s) these problems or are we benchmaxed on

[21:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1313s) you know benchmarks of the past which

[21:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1315s) are more single GPU centric

[21:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1318s) uh models right now are showing really

[22:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1320s) promising results on single GPU uh

[22:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1322s) benchmarks. So it's a ripe time to to

[22:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1325s) extend it.

[22:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1327s) In our benchmark parallel kernel bench,

[22:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1330s) each task presents the model with an

[22:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1332s) unoptimized reference implementation

[22:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1334s) written in PyTorch with torch

[22:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1336s) distributed um nickel operations and

[22:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1339s) then a system topology that specifies

[22:22](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1342s) the number of ranks and intraode

[22:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1344s) hardware configuration. And the model

[22:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1346s) needs to rewrite the reference into a

[22:28](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1348s) performance CUDA kernel that uh uses

[22:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1351s) unified virtual addressing.

[22:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1357s) The um multiGPU problem space expands

[22:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1360s) combinatorally beyond single GPU um

[22:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1364s) cases. So a standard transformer layer

[22:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1366s) can be parallelized across data

[22:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1369s) sequence, tensor, context, layer,

[22:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1371s) pipeline and expert dimensions. and each

[22:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1373s) composition um induces a different

[22:56](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1376s) communication pass uh pattern. So to

[22:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1378s) make sure that our benchmark has high

[23:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1381s) coverage over the representative types

[23:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1383s) of multiGPU problems um we created this

[23:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1386s) taxonomy um that you can read more about

[23:09](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1389s) in our paper and then picked

[23:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1390s) representative problems for each part of

[23:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1392s) the taxonomy.

[23:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1395s) Um these are all patterns that arise in

[23:18](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1398s) real AI workloads from inference to RL

[23:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1400s) to post-training.

[23:23](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1403s) There are 87 problems overall um drawn

[23:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1406s) from GitHub repositories that we found

[23:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1409s) to be very informative. Um and we uh and

[23:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1414s) like optimized library implementations

[23:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1416s) and DSLs that people have written

[23:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1418s) multiGPU kernels in.

[23:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1420s) Um, we wanted to really make sure that

[23:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1422s) solving PK this parallel kernel bench

[23:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1425s) would lead to net new useful production

[23:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1427s) kernels rather than artificial or

[23:49](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1429s) useless kernels.

[23:51](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1431s) Okay, so that's parallel kernel bench.

[23:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1434s) How do models perform?

[23:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1438s) So we measured to sorry that this is a

[24:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1440s) bit small. We measured two main metrics.

[24:03](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1443s) Um pass at K which is the number of

[24:06](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1446s) correct kernels generated after K

[24:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1448s) attempts and then fast um one at K which

[24:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1453s) counts solutions that are both correct

[24:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1456s) and outperform the speed of the pi torch

[24:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1459s) plus nickel baseline.

[24:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1461s) Um so pass K just correctness fast one

[24:25](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1465s) at K is whether you're getting a 1x or

[24:28](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1468s) higher speed up over the reference. So

[24:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1470s) performanceoriented

[24:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1472s) um we found that in the zeroot setting

[24:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1475s) the best of the frontier models we tried

[24:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1477s) solves 28 out of 87 problems and 22 of

[24:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1481s) those problems are faster than the

[24:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1482s) pietorch plus nickel baseline.

[24:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1485s) If we make multiple samples, you know,

[24:47](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1487s) standard uh scaling up test time

[24:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1490s) compute, we can uh get that number from

[24:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1494s) say like to to 36 correct solutions, but

[24:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1497s) the fast uh one performance still

[25:01](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1501s) plateaus out at roughly 31%. So we don't

[25:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1505s) see much room from continuing to scale

[25:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1507s) there as we increase the number of

[25:09](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1509s) parallel generations.

[25:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1512s) Um we find that the correct once

[25:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1514s) correctness is established um speedups

[25:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1516s) naturally come from eliminating nickel

[25:18](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1518s) staging overhead in favor of direct

[25:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1520s) NVLink loads in stores.

[25:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1524s) Um the success patterns here are really

[25:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1527s) concentrated into familiar patterns. So

[25:30](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1530s) collective primitives, tensor parallel

[25:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1532s) gems and Ulyses style context

[25:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1534s) parallelism. So in other words, patterns

[25:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1537s) that we see heavily represented um on

[25:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1540s) the internet rather than necessarily

[25:43](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1543s) patterns that the model has used its

[25:45](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1545s) reasoning abilities to think through.

[25:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1550s) Okay, even the best available model that

[25:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1553s) we benchmarked here GPT 5.5 drops off

[25:56](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1556s) very quickly as the speed up threshold

[25:59](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1559s) increases. So on the x-axis here, we're

[26:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1562s) increasing the speed up threshold over

[26:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1564s) that pietor torch plus nickel baseline.

[26:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1567s) And then we're showing the number of

[26:09](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1569s) correct kernels that are faster than

[26:11](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1571s) that baseline or this much faster than

[26:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1573s) the baseline on the y-axis.

[26:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1576s) So GPT 5.5 is this orange line here and

[26:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1580s) then DeepSeek V4 Pro is the aqua line at

[26:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1584s) the bottom.

[26:28](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1588s) um we found that there's deeper issues

[26:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1591s) than CUDA syntax. So we found that if

[26:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1594s) you do multiple sampling or have the

[26:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1596s) model kind of look at its errors and

[26:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1598s) correct them, it can often compile the

[26:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1601s) kernels. But the models really struggle

[26:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1604s) to reason through the tradeoffs that we

[26:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1606s) talked about in the prior section. um

[26:48](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1608s) collective ordering, data partitioning,

[26:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1610s) thinking about intra versus interm

[26:53](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1613s) scheduling or deciding between the

[26:55](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1615s) different transfer mechanisms. Um we

[26:58](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1618s) find that they often do not use things

[27:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1620s) like the register transfer instructions

[27:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1622s) or tensor memory acceleration when

[27:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1624s) writing the kernels.

[27:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1627s) Um we wanted to try a pretty simple

[27:10](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1630s) instantiation of something like a clawed

[27:12](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1632s) code um coding agent. So, we took uh the

[27:17](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1637s) mini sui agent multi-turn harness and

[27:20](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1640s) one of the best performing models,

[27:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1641s) Gemini 3 Pro, and gave it access to a

[27:24](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1644s) local bash environment to sort of mimic

[27:26](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1646s) the standard claude code setup. We found

[27:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1649s) that this could help the agent um go

[27:33](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1653s) from solving 24 problems to 35 of 87

[27:36](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1656s) problems um with um 26 achieving over a

[27:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1661s) 1x speed up over the reference. But we

[27:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1664s) found that as we scaled the amount of

[27:46](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1666s) time um the performance plateaued as uh

[27:50](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1670s) and we find that additional techniques

[27:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1672s) would be required to continue seeing the

[27:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1674s) scaling there. Again these results are

[27:57](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1677s) discussed in more detail in our paper.

[28:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1680s) Um we think this is a really exciting

[28:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1682s) you know just to wrap up here we hope

[28:04](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1684s) that people out here can use both

[28:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1685s) parallel kittens and parallel kernel

[28:07](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1687s) bench. We think that the kernels

[28:09](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1689s) generated from solving parallel kernel

[28:11](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1691s) bench will lead to net new production

[28:14](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1694s) kernels that are important bottlenecks

[28:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1696s) for inference in RL right now. um

[28:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1699s) they're you know we tried our best to

[28:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1701s) make them you know non-artificial and we

[28:23](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1703s) can already see signs of life and

[28:25](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1705s) exciting results where people have not

[28:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1707s) invested a bunch of time to handw write

[28:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1709s) a multiGPU kernel and we've gotten some

[28:31](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1711s) net new interesting ones like this Nemo

[28:34](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1714s) vocab parallel um you know filtering

[28:37](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1717s) kernel um a hyena architecture context

[28:41](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1721s) parallelism kernel

[28:44](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1724s) and uh the SAM 3 video segmentation

[28:48](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1728s) model um IOU suppression kernel.

[28:52](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1732s) Just to conclude here um we're really

[28:54](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1734s) excited about um how uh well just

[28:59](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1739s) talking about the lessons. First off, we

[29:00](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1740s) think there aren't that many patterns

[29:02](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1742s) that are involved in writing intragpu

[29:05](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1745s) effective kernels again encapsulated by

[29:08](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1748s) our small set of programming primitives.

[29:11](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1751s) Um, but unfortunately models do not

[29:13](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1753s) currently understand how to reason

[29:15](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1755s) through these trade-offs even when we

[29:16](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1756s) provide them in context. Um, we're

[29:19](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1759s) really excited about methods that can

[29:21](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1761s) help attack this benchmark. We're

[29:23](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1763s) excited about architectures that can

[29:25](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1765s) grow with the trends of how networking

[29:27](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1767s) stacks are evolving. Um, you know,

[29:29](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1769s) larger scale up domains shift away from

[29:32](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1772s) scale out um and massive onchip memory

[29:35](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1775s) structures. And we hope that uh you know

[29:38](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1778s) we can also extend and you you can feel

[29:40](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1780s) free to reach out to me at my email.

[29:42](https://www.youtube.com/watch?v=pOvWgX7IJsc&t=1782s) Thanks.
