AI Engineer World's Fair 2026

Can LLMs Write Fast Multi-GPU Kernels? — Simran Arora, Together AI

Loading slides…

Read the talk

Can LLMs Write Fast Multi-GPU Kernels?

Simran Arora explains why communication has become a limiting resource in many production distributed training and inference workloads, distills multi-GPU kernel design into transfer and scheduling choices, and tests whether models can make those choices across 87 practical problems.

From a talk by Simran Arora

At a glance

Ideas worth remembering

  • Communication now deserves explicit optimization: the reported A100-to-B200 gains are 7.2× for BF16 compute, 3× for intra-node communication, and 2× for inter-node communication.

  • Fast multi-GPU kernels match the transfer path to message size, GPU resource use, desired overlap, and any need for in-network reduction.

  • Scheduling is operation-specific: intra-SM overlap works when computation and communication align around the same data, while inter-SM specialization helps when their resources or communication patterns diverge.

  • Correctness and speed are separate benchmark outcomes. The best zero-shot result is 28 correct kernels out of 87, with 22 faster than the reference; more samples reach 36 correct while correct-and-faster performance plateaus near 31%.

  • A shell-equipped agent harness improves iteration—Gemini 3 Pro reaches 35 solved problems and 26 faster solutions—but ordering, partitioning, scheduling, and transfer selection remain difficult after compilation errors are repaired.

  • Limited aggregate coverage can still yield useful kernels, including the reported NeMo, Hyena, and SAM 3 examples, but reliable reasoning about evolving multi-GPU systems remains open.

The bottleneck moves between GPUs

Work inside one GPU has improved through kernels such as FlashAttention, memory-efficient architectures, sparse attention, and better domain-specific languages. Arora’s starting point is the consequence of that progress: as single-GPU execution gets better, communication between GPUs becomes a larger share of the remaining runtime. The new problem is to make kernels spanning several GPUs fast without requiring months of specialized engineering for each operator.

The hardware explains why data movement matters. Threads and blocks execute on processors distributed across the GPU. Those processors retrieve weights and activations through a hierarchy that includes a relatively small L2 cache and much larger high-bandwidth memory. Registers sit closest to computation and can supply data at a cited 130 terabytes per second on an H100, but their capacity is limited. Moving farther away generally provides more storage at the cost of slower access.

Multiple GPUs extend that hierarchy into a network. PCIe carries CPU–GPU traffic, while communication across nodes can use InfiniBand or TCP. Within the NVIDIA systems emphasized in the talk, NVLink connects GPUs point to point and NVSwitch joins NVLink endpoints into a nonblocking fabric. NVSwitch can also accelerate operations such as multicast and reductions in the network rather than requiring every step to run on a GPU.

Data gets larger—and farther away

Registerson-gpu — Closest to computation and very fast, but scarce.
L2 cacheon-gpu — On-GPU memory surrounding the processors, with limited capacity.
High-bandwidth memoryon-gpu — Stores large weights and activations and supplies the GPU memory capacity reported by system tools.
NVLink and NVSwitchintra-node — Connect GPUs within the NVIDIA scale-up domain; NVSwitch can perform multicast and reductions in the fabric.
InfiniBand or TCPinter-node — Carries communication across nodes.
Within a GPU, fast, scarce registers give way to larger memories farther from computation. Beyond the GPU, communication links provide access to remote memory; the links are not additional storage tiers.
Read the diagram as text
  • Registers. on-gpu — Closest to computation and very fast, but scarce.
  • L2 cache. on-gpu — On-GPU memory surrounding the processors, with limited capacity.
  • High-bandwidth memory. on-gpu — Stores large weights and activations and supplies the GPU memory capacity reported by system tools.
  • NVLink and NVSwitch. intra-node — Connect GPUs within the NVIDIA scale-up domain; NVSwitch can perform multicast and reductions in the fabric.
  • InfiniBand or TCP. inter-node — Carries communication across nodes.
0:421:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Workloads and hardware are becoming more network-shaped

Several optimization layers have advanced together: architectures reduce compute and memory requirements, hardware-aware algorithms improve execution, programming tools make kernels easier to express, and megakernels overlap work across operators. Yet distributed training and inference still exchange data. Arora reports that communication increasingly consumes most of the runtime in many production workloads, leaving low model FLOP utilization as systems scale.

Networking is also less uniform than matrix multiplication or memory hierarchies. The talk contrasts AMD’s point-to-point XGMI links, TPU systems using a 3D torus with optical wraparound links, and NVIDIA’s NVSwitch fabric with in-network reductions. Arora cites up to 900 gigabytes per second of unidirectional NVLink bandwidth between remote GPUs’ high-bandwidth memories on a particular NVIDIA generation. That hardware qualification matters: it is an example of one system, not a universal link rate.

The workloads themselves now cross more boundaries. KV-cache storage can span GPU memory, CPU memory, disk, and remote machines. Inference systems can place speculative decoding, prefill, and decode on different hardware. Arora also describes scale-up domains reaching 72 GPUs and cites a plan for a 576-GPU system in 2027. Device-side tensor memory acceleration adds asynchronous transfers to this environment, giving kernels finer control over when communication begins and how it overlaps with other work.

Compute has nevertheless improved faster than the links carrying its data. From NVIDIA’s A100 in 2020 to B200 in 2024, the reported BF16 tensor-core gain is 7.2×, compared with 3× for intra-node communication and 2× for inter-node communication. Faster arithmetic cannot remove that mismatch. It increases the amount of useful computation that can sit idle while communication catches up.

5:376:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:37 · section reference included

Convenient collectives leave a performance gap

NCCL makes common multi-GPU operations accessible, but its bulk-transfer orientation does not fit every kernel. It works naturally with large contiguous chunks. Fine-grained communication, tightly fused collectives, and communication that must overlap with a particular computation require more control. In the team’s benchmark, most simple PyTorch-plus-NCCL baselines fall below 50% of their communication-aware roofline, leaving headroom even after accounting for the communication limit.

The available alternatives each carry a cost:

  • Higher-level frameworks: Systems built around bulk NCCL collectives often synchronize before and after transfers, preserving the same coarse execution structure.
  • Compilers and distributed DSLs: These can offer better abstractions, but tuning may not transfer across rapidly changing network architectures. Arora cites a distributed compiler configuration tuned around eight H800 GPUs that did not adapt efficiently to H100s in the team’s results.
  • Hand-tuned operators: Specialized kernels can reach peak performance, but the work does not scale. Arora says adapting some implementations from one precision to another can take five or six months.

This leads to the project’s core research question: can a small set of reusable principles capture the important decisions in multi-GPU kernel design? The team first investigated those decisions manually through ParallelKittens, a compact set of primitives and patterns used to build high-performance kernels across several parallelism schemes. Only after developing that understanding did they ask whether language models could apply it.

11:1011:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:10 · section reference included

Choose the transfer mechanism to fit the message

The first reusable decision is how data crosses GPUs. A copy engine performs host-initiated transfers and suits large messages that can approach peak communication bandwidth. Because the transfer does not occupy the GPU’s compute processors or consume many of its registers, those resources remain available for other parts of the workload.

Device-initiated communication offers two other paths. Tensor memory acceleration, or TMA, can saturate NVLink with relatively small messages while consuming few registers and few processors, making it useful for fine-grained overlap. Register-level transfer instructions also support fine-grained movement and can exploit the in-network reductions that NVSwitch provides. TMA cannot effectively use those in-network computations in the design space Arora describes.

There is no single best path. Large independent transfers favor the copy engine’s resource isolation. Small transfers interleaved with computation favor device initiation. Operations that need reduction inside the switch may justify register-level instructions. The kernel writer must trade message size against register pressure, processor use, overlap, and network functionality.

Three paths for inter-GPU data

Copy enginebulk transfer — Host-initiated; strong for large messages; leaves processors and registers available.
Tensor memory accelerationfine-grained transfer — Device-initiated; saturates NVLink with smaller messages; uses few registers and processors; useful for overlap.
Register-level instructionsfine-grained transfer and reduction — Device-initiated instructions that can use NVSwitch in-network reductions.
The useful choice depends on message size, GPU resource consumption, desired overlap, and whether the operation needs computation inside NVSwitch.
Read the diagram as text
  • Copy engine. bulk transfer — Host-initiated; strong for large messages; leaves processors and registers available.
  • Tensor memory acceleration. fine-grained transfer — Device-initiated; saturates NVLink with smaller messages; uses few registers and processors; useful for overlap.
  • Register-level instructions. fine-grained transfer and reduction — Device-initiated instructions that can use NVSwitch in-network reductions.
16:0017:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:00 · section reference included

Overlap work within an SM—or divide it across SMs

After choosing a transfer mechanism, the kernel must schedule computation, memory work, and communication. An intra-SM schedule assigns different warps or threads inside one streaming multiprocessor to compute and communication concurrently. This is effective when both activities consume the same data in compatible patterns. When their data or resource needs diverge, packing them into one SM can create awkward competition for registers and shared memory.

An inter-SM schedule instead gives different streaming multiprocessors distinct responsibilities for computation, communication, or memory work. This can avoid misaligned resource sharing and can better drive NVLink when intra-SM overlap cannot. Arora’s examples are operation-specific: intra-SM overlap works well for GEMM plus reduce-scatter, while inter-SM overlap works well for GEMM plus all-reduce when it uses NVSwitch’s in-network reduction capability.

Buffering and synchronization between senders and receivers remain part of the design. ParallelKittens packages these choices into programming primitives and templates while preserving developer control over them. Arora reports that the approach typically adds roughly a dozen lines to a single-GPU kernel and is used in production at Together AI and Cursor. The displayed comparisons show state-of-the-art results against strong reference baselines across data, sequence, and expert parallelism.

Two ways to overlap communication and computation

Intra-SM overlapwithin one processor — Different warps in one SM specialize in compute or communication. Best when their data patterns align.
GEMM + reduce-scatterwithin one processor — The talk’s example where intra-SM overlap is effective.
Inter-SM overlapacross processors — Different SMs specialize in compute, communication, or memory work, avoiding misaligned resource sharing.
GEMM + all-reduceacross processors — The talk’s example where inter-SM scheduling uses NVSwitch reductions effectively.
The schedule follows data alignment and resource pressure; neither strategy is universally superior.
Read the diagram as text
  • Intra-SM overlap. within one processor — Different warps in one SM specialize in compute or communication. Best when their data patterns align.
  • GEMM + reduce-scatter. within one processor — The talk’s example where intra-SM overlap is effective.
  • Inter-SM overlap. across processors — Different SMs specialize in compute, communication, or memory work, avoiding misaligned resource sharing.
  • GEMM + all-reduce. across processors — The talk’s example where inter-SM scheduling uses NVSwitch reductions effectively.
  • Intra-SM overlap → GEMM + reduce-scatter: example.
  • Inter-SM overlap → GEMM + all-reduce: example.
18:1818:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:18 · section reference included

ParallelKernelBench tests practical distributed design

ParallelKernelBench asks whether promising single-GPU kernel-generation results carry over to multi-GPU design. Each task gives a model an unoptimized PyTorch reference using distributed NCCL operations and a topology specifying the rank count and intra-node hardware. The model must replace that reference with a performant CUDA kernel that uses unified virtual addressing.

The search space grows combinatorially because a transformer layer can be parallelized across data, sequence, tensor, context, layer, pipeline, and expert dimensions. Each composition changes which data must move, which ranks participate, and when communication occurs. The team built a taxonomy covering these patterns and selected 87 problems arising in inference, reinforcement learning, and post-training, drawing from real repositories, optimized libraries, and existing DSL implementations.

The benchmark is intended to produce useful kernels rather than isolated programming puzzles. Its references represent work that practitioners already express through PyTorch and NCCL, while the candidates must provide an optimized implementation for the supplied topology. The public ParallelKernelBench repository contains the benchmark and evaluation harness.

What one benchmark task asks the model to do

PyTorch referenceinput — Unoptimized implementation using distributed NCCL operations.
System topologyinput — Specifies rank count and intra-node hardware configuration.
Kernel-generating modelgeneration — Must reason about the operation and the available multi-GPU system.
CUDA candidateoutput — Uses unified virtual addressing and aims to outperform the reference.
A topology-aware rewrite must preserve per-rank behavior while replacing coarse distributed operations with a faster kernel.
Read the diagram as text
  • PyTorch reference. input — Unoptimized implementation using distributed NCCL operations.
  • System topology. input — Specifies rank count and intra-node hardware configuration.
  • Kernel-generating model. generation — Must reason about the operation and the available multi-GPU system.
  • CUDA candidate. output — Uses unified virtual addressing and aims to outperform the reference.
  • PyTorch reference → Kernel-generating model: operation semantics.
  • System topology → Kernel-generating model: hardware constraints.
  • Kernel-generating model → CUDA candidate: rewrites.
21:3022:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:30 · section reference included

Correctness improves faster than performance

The evaluation separates two outcomes. pass@K asks whether at least one correct kernel appears within K attempts. The performance-oriented fast₁@K also requires the candidate to beat the PyTorch-plus-NCCL reference. That distinction matters because a compiling, numerically correct kernel may still communicate inefficiently or leave hardware idle.

In the zero-shot setting, the best tested frontier model solves 28 of 87 problems, and 22 of those solutions beat the reference. Drawing additional samples raises the number of correct solutions to 36, but the correct-and-faster share plateaus at roughly 31%. More parallel generations therefore find additional valid implementations without producing a comparable rise in high-performance ones.

When a solution is correct, its speedup often comes from removing NCCL staging and replacing it with direct NVLink loads and stores. Success clusters around familiar collective primitives, tensor-parallel GEMMs, and Ulysses-style context parallelism. Arora interprets this concentration as a sign that models perform best on patterns well represented online. It suggests uneven generalization, although the distribution alone cannot reveal the reasoning process behind any individual solution.

The talk identifies GPT 5.5 as the strongest model in the displayed comparison and reports that its qualifying solution count drops quickly as the required speedup rises. Beating a coarse reference on some tasks is therefore a lower bar than consistently finding kernels with substantial gains.

23:5824:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:58 · section reference included

Retries fix syntax more readily than distributed design

The persistent failures are deeper than CUDA syntax. Multiple samples or error feedback often get a kernel to compile, but models still mishandle collective ordering, data partitioning, intra-SM versus inter-SM scheduling, and transfer selection. Generated kernels also often omit TMA and register-level transfer instructions. Compilation proves that the program is expressible; it does not prove that the model chose the right communication design.

The team also placed Gemini 3 Pro in a multi-turn mini-SWE-agent harness with access to a local shell. This improved the result from 24 solved problems to 35 of 87, with 26 candidates beating the reference. More execution time eventually reached another plateau. The environment helps the model inspect failures and revise code, but it does not supply the missing judgment about data movement and scheduling.

Aggregate limitations do not make the benchmark fruitless. Arora reports useful new examples for NeMo vocabulary-parallel filtering, Hyena context parallelism, and intersection-over-union suppression for the SAM 3 video-segmentation model. The talk does not give individual speedups for these kernels, so they demonstrate practical directions rather than quantified wins.

The ending preserves the central tension. A small set of primitives can express many effective multi-GPU patterns, yet the tested models still fail to apply their tradeoffs reliably even when those ideas are supplied in context. Arora points toward better methods for attacking the benchmark and toward architectures designed for larger scale-up domains, less dependence on scale-out, and much larger on-chip memories. Kernel generation is only one part of that opportunity: models and systems must ultimately be designed around the changing balance among computation, memory, and networking.

26:2826:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:28 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…