Contents
  1. Work and state
    1. What additional devices buy
    2. The numerical state that must fit
    3. Workers and ownership
    4. Turning points in distributed execution
  2. Communication
    1. Collectives preserve numerical meaning
    2. Communication on the critical path
  3. Training
    1. Different examples, matching updates
    2. Preserve the intended update
    3. Shard state, materialize for use
    4. Partition a layer's arithmetic
    5. Pipeline layers and microbatches
  4. Placement and composition
    1. Physical links change the partition
    2. Compose an executable training job
    3. Partition sequence-dependent work
    4. Place experts under uneven demand
  5. Serving
    1. Replicas and cooperating serving groups
    2. Move cache between serving phases
  6. Failure and recovery
    1. Detect missing execution progress
    2. Restore a coherent training state
    3. Restore capacity without inventing continuity
  7. Measurement
    1. Find the useful scaling limit
  8. Check understanding
  9. Open questions
  10. Selected talks
  11. References
  12. Talk library
← All topics

Distributed training & inference

Distributed execution lets several devices train or serve a model together. The central design problem is deciding what each device owns, what it computes locally, and what it must obtain from others. A useful partition can make an otherwise impossible workload fit in memory, but every new dependency can also introduce communication, waiting, and coordinated recovery. Understanding those dependencies is how you distinguish a larger system from a faster or more dependable one.

I — Work and state

What additional devices buy

Start with the outcome you need: enough memory to execute the workload, less time to complete training, or more serving requests completed within their latency requirements. These goals are related but not interchangeable. Replicating a server can increase aggregate throughput without reducing the execution time of one request; partitioning a model can make it fit while adding communication to that request.

A device here is a numerical accelerator, usually a graphics processing unit, or GPU. A node is a machine containing one or more devices. Their combined memory is not automatically one equally accessible address space: workers must arrange access to remotely held values. Scale-up usually means enlarging resources within a machine or tightly connected system; scale-out adds machines. State the boundary explicitly, because those labels alone say little about communication cost.

A November 2025 multi-node inference study illustrates the distinction. For its fixed Llama 3.1 70B workloads, vLLM latency improved from four to eight GPUs, then generally flattened or increased. The system had four A100 GPUs per node, NVLink within nodes, and Slingshot-11 between nodes. This measured batch completion, not online tail latency. More arithmetic capacity did not reliably shorten the complete execution once communication became consequential.

The numerical state that must fit

A tensor is an indexed numerical array with a shape and element type. Model parameters are learned arrays. A forward pass applies them to inputs and produces intermediate activations. During training, a loss assigns a numerical penalty to the predictions. Backward computation produces gradients, derivatives describing how parameter changes affect that loss. An optimizer uses them to update parameters and may retain optimizer state, such as moving averages of earlier gradients. These arrays have different lifetimes and need not share the same device. From recorded text to parameter updates explains the learning process.

Serving keeps parameters fixed but introduces another growing allocation. Prefill processes the known input positions; decode advances the generated continuation. A key-value cache, or KV cache, retains attention representations derived from processed positions so they need not be recomputed. It is numerical request state, not the token text itself. See Prefill and decode and The KV cache for their mechanisms.

Ownership and lifetime identify which allocation a partition can reduce.
StateLogical ownerLifetime and growth
ParametersModelPersistent; grows with model size and representation width.
Optimizer stateTraining update processPersists across updates; depends on optimizer and parameter count.
GradientsCurrent updateRetained through aggregation or accumulation; storage depends on the execution policy.
ActivationsCurrent examples and operationsIntermediate results; retained for backward use or recomputed.
KV cacheRequests and compatible shared prefixesDepends on active sequences, retained positions, architecture, and cache format.
Workspaces and communication buffersRuntime operationsPersistent or temporary; can create peaks beyond the model's stored arrays.

Memory feasibility concerns the peak simultaneous allocation, not the sum of parameter shards alone. For example, vLLM profiles non-KV consumption before assigning remaining capacity to caches. Allocator-reserved memory also differs from live tensor storage; adding both can double-count bytes. Keep a ledger of actual allocations and transient peaks, using the distinctions in Where operands live.

Workers and ownership

A worker executes an assigned portion of the job, commonly as a process controlling a device. A process group identifies participants that communicate together. A worker's rank identifies it within that group; the group's size is its world size. A communicator supplies the communication context for those participants. Rank is not a permanent machine identity: membership changes can assign new ranks.

Replication keeps equivalent copies; sharding assigns portions to owners. Sharded storage need not imply partitioned arithmetic: execution may gather the parameter shards.

The main parallelism axes divide different things.
AxisWhat is dividedWhat connects workers
Data parallelismExamples processed by model replicasGradient contributions for corresponding parameters.
Tensor parallelismArithmetic inside a layerOperands, output pieces, or partial sums.
Pipeline parallelismThe ordered sequence of layersForward activations and backward gradients at stage boundaries.

These axes can coexist. A group may cooperate on each example through tensor and pipeline partitions while another group processes different examples. Before choosing a library setting, identify the owner of every required input and result. Missing local values explain the communication that the partition introduces.

Turning points in distributed execution

Distributed model execution combines two continuing lines of work: coordinating processors and arranging numerical work so that coordination is worthwhile. Earlier parallel-computing ideas supplied portable communication and whole-application reasoning. Deep-learning systems then developed partitions suited to large parameter arrays, repeated training updates, and stateful generation.

These developments addressed complementary constraints, not successive replacements for one universal design.

Complementary answers to distributed execution

  1. 1967AmdahlWhole-workload scaling limits: faster parallel arithmetic leaves other costs exposed.Sources & context

    Contributors: Gene Amdahl, IBM

    What changed: The paper includes sequential work, data management, irregular computation, and memory contention in the scaling question.

  2. June 1994MPI 1.0Portable communication contracts that implementations can optimize.Sources & context

    Contributors: Collaborating vendors, universities, laboratories, and industry

    What changed: Applications gained a common message-passing interface. The linked MPI 1.1 specification, dated June 1995, documents and clarifies the earlier standard.

  3. 2012DistBeliefPartitioned models and independently progressing replicas.Sources & context

    Contributors: Google

    What changed: Downpour SGD traded strict coordination for tolerance of variable worker progress. Gradients could use older parameters, and parameter-server shards could advance differently.

  4. November 2018 preprint; 2019 paperGPipeMicrobatch pipelines across layer partitions expand executable model capacity.Sources & context

    Contributors: Yanping Huang and colleagues, Google

    What changed: The NeurIPS paper combines computationally balanced stages, accumulated gradients, and rematerialization, with one update after the minibatch.

  5. September 2019 preprintMegatron-LMPaired within-layer partitions avoid unnecessary intermediate exchanges.Sources & context

    Contributors: Mohammad Shoeybi and colleagues, NVIDIA

    What changed: The linked March 2020 revision reports converged 8.3-billion-parameter models on 512 GPUs. Its scaling experiment enlarges the model rather than holding total work fixed.

  6. October 2019 preprintZeRORemove duplicated training state while retaining substantial local computation.Sources & context

    Contributors: Samyam Rajbhandari and colleagues

    What changed: Progressive state partitioning reduces persistent storage. Adding parameter sharding also requires parameter collection for forward and backward use.

  7. 2024Splitwise and DistServeSeparate serving phases, making KV movement and phase allocation explicit.Sources & context

    Contributors: Pratyush Patel and colleagues (Splitwise); Yinmin Zhong and colleagues (DistServe)

    What changed: Splitwise overlaps layerwise cache transfer with prompt computation. DistServe evaluates serving capacity against first-token and subsequent-token latency objectives.

Notice the shift from whole-workload limits and communication contracts to partitions of computation, training state, and serving phases. These approaches coexist; spacing is not to scale.

II — Communication

Collectives preserve numerical meaning

Point-to-point communication connects two participants. A collective operates across a group; a reduction combines corresponding values, for example by summing. The NVIDIA Collective Communications Library (NCCL) implements GPU collectives.

OperationResult ownership
All-reduceEvery rank receives the reduced array.
Reduce-scatterEach rank receives a portion of the reduced array.
All-gatherEvery rank receives the concatenated rank contributions.
BroadcastEvery rank receives the designated root's values.

Reduce-scatter then all-gather produces all-reduce. Gathering concatenates in rank order; it does not sum. NCCL requires matching counts and datatypes.

Synchronization enforces a progress or data-readiness dependency; it need not be a global barrier. The MPI nonblocking-collective contract separates initiation from local completion. Initiation returns a handle; completion establishes safe local buffer access. It generally does not establish that every other participant has finished. Participants must initiate collectives in the same order on a communicator. GPU frameworks have their own stream and completion rules, so apply their documented boundary before consuming or overwriting buffers. This extends the producer/consumer reasoning in Safe sharing and reuse.

Communication on the critical path

The interconnect carries values between devices or nodes. The critical path is the dependency chain that determines completion: shortening work outside it may leave elapsed time unchanged. Communication becomes exposed when a dependent operation is ready to run except for missing remote data. Issuing a transfer asynchronously creates an opportunity for overlap, not proof that useful work overlaps it.

Tmessage≈α+SBT_{\text{message}} \approx \alpha + \frac{S}{B} Here α\alpha is startup latency in seconds, SS is payload size in bytes, and BB is effective bandwidth in bytes per second. This simple uncontended model explains why many small transfers repeatedly pay startup cost, while large transfers emphasize bandwidth. It excludes producer delays and competing traffic.

Consider an example with a 2 ms producer operation, a 6 ms transfer, and 4 ms of independent computation. Serial execution takes 12 ms. Starting the transfer when its producer finishes, alongside the independent computation, takes 8 ms; 2 ms of transfer remains exposed after that computation ends. The transfer still consumes bandwidth for 6 ms. Overlap changes elapsed time without eliminating data movement.

The same transfer, a shorter execution

Example timings

Independent computation hides only part of the transfer.

Serialized execution0–12 msDuration 12 ms
Produce payload0–2 msDuration 2 msWithin Serialized execution
Transfer payload2–8 msDuration 6 msWithin Serialized execution
Independent computation8–12 msDuration 4 msWithin Serialized execution
Overlapped execution0–8 msDuration 8 ms
Produce payload0–2 msDuration 2 msWithin Overlapped execution
Transfer payload2–8 msDuration 6 msWithin Overlapped execution
Independent computation2–6 msDuration 4 msWithin Overlapped execution
Both schedules start transfer after its producer finishes at 2 ms. Overlap hides four of six transfer milliseconds, reducing completion from 12 to 8 ms. Parent spans measure elapsed execution; overlapping child spans must not be summed as elapsed time.
Read the diagram as text
  • Serialized execution. 0 to 12 ms; duration 12 ms.
  • Produce payload. 0 to 2 ms; duration 2 ms. Parent: Serialized execution.
  • Transfer payload. 2 to 8 ms; duration 6 ms. Parent: Serialized execution.
  • Independent computation. 8 to 12 ms; duration 4 ms. Parent: Serialized execution.
  • Overlapped execution. 0 to 8 ms; duration 8 ms.
  • Produce payload. 0 to 2 ms; duration 2 ms. Parent: Overlapped execution.
  • Transfer payload. Only its final two milliseconds remain uncovered. 2 to 8 ms; duration 6 ms. Parent: Overlapped execution.
  • Independent computation. 2 to 6 ms; duration 4 ms. Parent: Overlapped execution.

Strong scaling holds total work fixed while adding processors; weak scaling holds work per processor fixed. Amdahl's fixed-work argument exposes the work that remains unaccelerated. John Gustafson's 1988 Reevaluating Amdahl's Law instead considered a larger problem completed in approximately the same time. Its scaled-speedup expression, s+N(1−s)s+N(1-s), uses processor count NN and serial fraction ss measured on the parallel machine. It changes the workload and denominator; it does not overturn the fixed-work limit.

Also distinguish bytes sent by one worker, aggregate traffic, and traffic crossing a limiting shared link. NCCL Tests reports algorithm bandwidth as operation data size divided by time. Its bus bandwidth applies a collective-specific normalization; for all-reduce on nn ranks, the factor is 2(n−1)/n2(n-1)/n. That derived metric is not a counter of physical-link traffic.

III — Training

Different examples, matching updates

In synchronized data-parallel training, replicas process different examples from matching starting parameters. Workers average gradients for corresponding parameters, then apply matching optimizer updates. With matching optimizer state and settings, the replicas remain one logical model. PyTorch's DistributedDataParallel, or DDP, performs this gradient synchronization through all-reduce; it does not redistribute newly updated weights after every step.

Ordinary replication still stores complete parameters, gradients, and optimizer state on each worker. It distributes example processing, not that storage burden. A local batch contains the examples assigned to one replica; the global batch contains those contributing across replicas. Two workers processing four distinct examples each contribute eight examples to an update. Doubling workers while keeping the local batch fixed therefore changes the global batch.

Input coverage requires its own policy. PyTorch's DistributedSampler partitions indices by rank. For uneven dataset lengths, it can drop the tail or add indices to equalize partition lengths. Equal-length partitions thus need not mean exactly-once coverage. Shared shuffle seeds and calling set_epoch before each epoch's iterator are also part of its contract.

Gradient communication need not wait for every layer. DDP groups gradients into buckets and starts a reduction when a bucket is ready during backward computation. Larger buckets amortize startup; waiting for their last gradient can delay overlap. Reductions follow consistent bucket order across ranks, and required reductions finish before synchronized gradients reach the optimizer.

Horovod made communication and integration a joint concern. Alexander Sergeev and Mike Del Balso's February 2018 Uber paper wrapped gradient averaging in an optimizer interface and introduced Tensor Fusion to pack ready, same-type tensors into larger reductions. It built on existing ring-all-reduce work, rather than inventing the algorithm. Cross-worker timelines revealed why reducing many small arrays separately could squander otherwise parallel computation.

Preserve the intended update

A microbatch is a smaller scheduled portion of a batch. Gradient accumulation combines several microbatch contributions before an optimizer update. It can reduce activation residency and, when intermediate synchronization is suppressed, communication frequency. Preserving the intended update requires fixed parameters during accumulation, correct normalization, and accounting for batch-dependent operations. It is not enough merely to process the same number of examples.

L=∑r∑j=1nrℓr,j∑rnrL=\frac{\sum_r\sum_{j=1}^{n_r}\ell_{r,j}}{\sum_r n_r} For a token-mean objective, nrn_r counts scored, non-padding tokens on rank rr across the accumulation window, and ℓr,j\ell_{r,j} is one token's loss. Framework gradient averaging and accumulation scaling must produce this denominator.

For example, two tokens with mean loss 1 and six with mean loss 3 produce a global mean of (2×1+6×3)/8=2.5(2\times1+6\times3)/8=2.5, not the unweighted mean 2. The shorter contribution must not receive equal weight merely because it occupied one worker or microbatch.

Bulk-synchronous parallel computation, proposed by Leslie Valiant in 1990, separates local computation, communication, and progress boundaries into supersteps. It provides a useful way to reason about waiting, without requiring every collective to be a barrier. Synchronous training pays for slow participants because the next update needs their contributions.

DistBelief's Downpour stochastic gradient descent used parameter servers: workers that held parameter shards and applied incoming gradient updates. Replicas independently fetched parameters and returned gradients, so others could continue when one failed. But those gradients could use older parameters, and server shards could advance differently. This staleness changes which contributions reach each update. It is a change to the learning process, not merely faster synchronized accumulation. Training schedules covers changes to the update policy.

Shard state, materialize for use

Replication wastes capacity when every worker holds state that only needs one owner during an update. Zero Redundancy Optimizer, or ZeRO, progressively removes that duplication. Workers still cooperate on the same logical parameters, but each retains and updates an assigned portion of training state.

ArrangementPersistently partitioned state
Replicated data parallelismNone of the principal model states.
ZeRO stage 1Optimizer state.
ZeRO stage 2Optimizer state and reduced gradients.
ZeRO stage 3Optimizer state, gradients, and parameters.

The original ZeRO analysis used Adam, an optimizer retaining two arrays of gradient statistics, with different numerical precisions for different state. Parameters and gradients used 2 bytes each per parameter; a higher-precision master parameter copy and the two optimizer arrays used 4 bytes each. That totals 16 bytes per parameter before activations and workspaces. Sharding removes redundant copies, but parameter shards must be collected for forward and again for backward computation. Adding parameter sharding increased the modeled communication from 2Ψ2\Psi to 3Ψ3\Psi elements per step, where Ψ\Psi is parameter count. More transferred elements do not imply the same proportional increase in latency.

Fully Sharded Data Parallel (FSDP) keeps parameters sharded between uses. The FSDP2 configuration gathers each layer for forward and backward, then reduce-scatters gradients for shard-local updates. Other layers remain sharded; prefetching the next layer can overlap communication but increases simultaneous residency.

Both panels retain the same sharded ownership. Prefetch overlaps full next-layer residency with the current layer on both ranks. Ownership maps are not additional counted buffers; the schematic does not measure an allocation ratio.

Yanli Zhao and colleagues' 2023 PyTorch FSDP paper explains why framework integration matters: initialization can fail before sharding is established, and allocator behavior controls transient peaks. That paper describes an earlier implementation, not every FSDP2 detail. Its broader contribution is showing that a memory ownership plan must survive initialization and scheduling as well as steady-state storage.

Other allocations need other remedies. Activation recomputation, often called activation checkpointing, discards selected intermediates and regenerates them during backward computation. Offloading moves retained values to host memory and brings them back before use. These exchange storage for computation or transfers; neither is a durable restart checkpoint. Long-context training often needs them even after parameter sharding has succeeded.

Partition a layer's arithmetic

Tensor parallelism keeps the arithmetic itself divided. For a linear layer, rows of XX represent input vectors and columns of WW select output features. Splitting output features creates complete, independent output pieces. Splitting the dimension summed over creates incomplete contributions that must be added.

X∈Rb×h,W∈Rh×f,Y=XWX\in\mathbb{R}^{b\times h},\quad W\in\mathbb{R}^{h\times f},\quad Y=XW Output-column split: W=[W0 W1]W=[W_0\ W_1], so Y=[XW0 XW1]Y=[XW_0\ XW_1]. Contracted-dimension split: X=[X0 X1]X=[X_0\ X_1], W=[W0;W1]W=[W_0;W_1], so Y=X0W0+X1W1Y=X_0W_0+X_1W_1. Here bb counts input vectors, hh input features, and ff output features.

An elementwise nonlinearity transforms each value separately and can process complete output pieces locally. It cannot generally precede an unfinished sum: ReLU replaces negative values with zero, so ReLU(−1+2)=1 but ReLU(−1)+ReLU(2)=2. Megatron's multilayer perceptron, or MLP, applies a first matrix, an elementwise function φ, then a second matrix. Pairing a column-split first matrix with a row-split second matrix keeps the intermediate features local; only the final partial outputs need summing.

Keep complete features local; sum partial outputs

Example

The nonlinearity precedes the second product, not its final reduction.

X: b×h
H₀ = φ(XW₀): b×(f/2)
H₁ = φ(XW₁): b×(f/2)
P₀ = H₀V₀: b×h
P₁ = H₁V₁: b×h
Y = P₀ + P₁
A two-way MLP partition with even f. Wᵢ has shape h×(f/2); Vᵢ is the second matrix’s row block, shape (f/2)×h. Each local feature is complete before φ, so no intermediate gather is needed. Sum the b×h partial outputs; the final node denotes a mathematical result, not a coordinator.
Read the diagram as text
  • X: b×h.
  • H₀ = φ(XW₀): b×(f/2).
  • H₁ = φ(XW₁): b×(f/2).
  • P₀ = H₀V₀: b×h.
  • P₁ = H₁V₁: b×h.
  • Y = P₀ + P₁.
  • X: b×h → H₀ = φ(XW₀): b×(f/2): column block W₀.
  • X: b×h → H₁ = φ(XW₁): b×(f/2): column block W₁.
  • H₀ = φ(XW₀): b×(f/2) → P₀ = H₀V₀: b×h: row block V₀.
  • H₁ = φ(XW₁): b×(f/2) → P₁ = H₁V₁: b×h: row block V₁.
  • P₀ = H₀V₀: b×h → Y = P₀ + P₁: partial sum.
  • P₁ = H₁V₁: b×h → Y = P₀ + P₁: partial sum.

Distributed linear algebra already treated ownership and operand movement together. Robert van de Geijn and Jerrell Watts' SUMMA, published in journal form in April 1997, arranged processors in a logical grid: workers retained output blocks while receiving input panels through row and column broadcasts. Blocking amortized repeated message startup. The same engineering tension remains: finer partitions reduce local arithmetic but can increase communication overhead. Equal partitions also require compatible dimensions; the chosen layout must specify how any remainder is handled.

Decode can make that tension acute because frequent exchanges accompany relatively small local operations. The multi-node study introduced earlier found that communication growth could offset reduced computation. Separately, changing reduction order can change floating-point results even when the algebra is equivalent. Compare against a defined numerical tolerance, following Specify the numerical result, rather than assuming either bitwise identity or that any discrepancy is acceptable.

Pipeline layers and microbatches

A pipeline stage owns consecutive layers. Forward activations move downstream; backward gradients return upstream. Microbatches overlap across stages. Pipeline bubbles are idle slots caused by dependencies, filling, draining, or imbalance.

GPipe updates once after all microbatches run forward then backward. Rematerialization rebuilds intermediates from boundary activations. Stages balance computation, not parameter counts. Batch normalization computes statistics per microbatch, so division can change the forward result.

Synchronous one-forward-one-backward (1F1B) alternates passes after warmup, retiring activations sooner while keeping parameters fixed until batch completion. In the Megatron paper's balanced-stage model, its fill/drain bubble matches GPipe's: (p−1)/m(p-1)/m relative to ideal computation, not total elapsed time, for pp stages and mm microbatches. The schedule comparison show shorter activation lifetimes without earlier completion.

Earlier activation release, same completion time

F / B = forward / backward; subscript = microbatch. Each operation takes one slot, transfers are negligible, and parameters stay fixed until the update.

GPipe fill–drain0123456Stage AidleidleidleidleidleidleF₁F₂B₁B₂Stage BidleidleidleidleidleidleF₁F₂B₁B₂A retained · peak 2012221B retained · peak 2001210Updateafter slot 6Synchronous 1F1B0123456Stage AidleidleidleidleidleidleF₁F₂B₁B₂Stage BidleidleidleidleidleidleF₁B₁F₂B₂A retained · peak 2012211B retained · peak 1001010Updateafter slot 6

Counts are outstanding microbatches, not bytes: include an activation after its forward completes until its backward completes. All counts return to zero at slot 6.

Both schedules execute the same eight operations and update after slot 6. Stage B’s peak outstanding activation count falls from two to one; Stage A’s peak stays two. This unit-duration example illustrates lifetime, not measured speed or memory bytes.

PipeDream, published at SOSP in October 2019, pursued continuous overlap across minibatches. Its weight stashing retained versions so a stage's backward pass used its forward-pass weights. That did not ensure a common version across all stages, and even optional cross-stage version alignment left stale gradients. Original PipeDream therefore had different update semantics from synchronous 1F1B. Schedule names alone do not specify consistency.

IV — Placement and composition

Physical links change the partition

Topology is physical connectivity, including shared links; placement assigns logical workers to it. Multiple transfers may contend for one uplink even when every device has a fast local connection. Oversubscription means aggregate downstream demand can exceed upstream capacity. Bandwidth across a cut dividing the network constrains exchanges between its sides. Counting devices or summing their link ratings does not reveal that constraint.

Remote direct memory access, or RDMA, supports transfers into registered remote memory without ordinary application-level copying at the receiving endpoint. GPUDirect RDMA lets a network adapter exchange payloads directly with GPU memory over PCI Express, or PCIe. Software still registers memory, initiates work, and manages lifetimes. GPU–network-interface affinity means choosing devices with suitable connecting paths: traffic through PCIe switches, CPU I/O infrastructure, or links between CPU sockets has different costs and support constraints. This illustrates nonuniform memory access, or NUMA: equally sized resources need not be equally close.

Collective algorithms choose routes for the same numerical contract. NVIDIA's NCCL 2.4 account describes bandwidth-efficient rings whose latency grows with participant count, complementary trees with logarithmic latency, and hierarchical arrangements combining within-node and between-node operations. Message size and topology determine the useful choice; no route is universally fastest.

NCCL PXN supplies a concrete topology-aware example. NVIDIA's February 2022 account describes moving payloads over NVLink to an intermediate GPU, then through that GPU's network interface controller, or NIC. In a rail-optimized arrangement, corresponding NICs across servers share a leaf switch. Choosing the destination's rail can avoid higher-level spine switches. An extra local hop can therefore simplify the external path. The Crusoe networking talk applies this reasoning to multi-GPU training; the internal hop must not be omitted when describing its external one-leaf path.

Highlighted routes carry the same payload between the same endpoints. PXN uses an intermediate local GPU and its NIC to avoid the spine in this rail-optimized arrangement; unhighlighted connections are schematic.

Close placement improves communication opportunities but can concentrate failure exposure. A failure domain groups resources affected by a shared failure, such as one node or zone. Spread complete independent replicas across appropriate domains when availability requires it; spreading individual shards is not equivalent to preserving a complete surviving replica. Hard placement constraints can also leave work unscheduled.

Compose an executable training job

A device mesh is a logical arrangement whose dimensions identify cooperating groups. Fix all coordinates except one to identify a group along that dimension. PyTorch's DeviceMesh manages these memberships and communicators; the logical arrangement does not establish physical connectivity.

For an example mesh with two data replicas, two pipeline stages, and two tensor workers per stage, write coordinates as (data, pipeline, tensor). Each coordinate is 0 or 1.

With pp pipeline stages, tt tensor workers, dd data replicas, mm microbatches and bb examples per microbatch, device count is ptdptd, global batch bmdbmd. Only data replicas multiply examples. State sharding can reuse an existing mesh dimension.

Eight workers, two example subsets

Coordinates: (data, pipeline, tensor). Each outlined row is a tensor group working on the same examples.

Data replica 0 · example subset A

Pipeline stage 0
(0,0,0)(0,0,1)

↓ Forward activations · ↑ Backward gradients
Corresponding tensor partitions across stages

Pipeline stage 1
(0,1,0)(0,1,1)

Data replica 1 · example subset B

Pipeline stage 0
(1,0,0)(1,0,1)

↓ Forward activations · ↑ Backward gradients
Corresponding tensor partitions across stages

Pipeline stage 1
(1,1,0)(1,1,1)
Data groups combine gradients for matching parameter portions
Pipeline, tensorData replica 0Data replica 1
0, 0(0,0,0)(1,0,0)
0, 1(0,0,1)(1,0,1)
1, 0(0,1,0)(1,1,0)
1, 1(0,1,1)(1,1,1)

2 × 2 × 2 = 8 devices. Global batch = b × m × d = 2bm examples. Pipeline and tensor workers cooperate on each subset; they do not multiply it.

Logical group membership does not specify physical links. Each data group combines gradients for corresponding parameter portions; state sharding may reuse a mesh dimension.

One update follows those memberships: assign different example subsets to data replicas; carry each subset through tensor computations and pipeline stages; return backward contributions; combine corresponding gradients across data replicas; then update the owned parameters. Group initialization, matching model state, and compatible collective order are prerequisites, not consequences of having the right total device count.

Gang scheduling coordinates admission for processes that need to run together. Kueue distinguishes aggregate quota from physical fit: eight available GPUs across two four-GPU nodes cannot satisfy one Pod requiring eight GPUs on one node. A readiness timeout can requeue partially ready work, but does not prove that its model collectives execute. Admission, placement, initialization, and execution readiness are separate gates.

Operational inputs matter too. Krea described queue resource counts becoming stale as nodes entered maintenance; Snorkel described training waiting on shared-filesystem reads. These are different ways to starve an otherwise valid partition. Establish memory fit first, place frequent exchanges on suitable links, verify group execution, then measure whether input delivery sustains it.

Partition sequence-dependent work

Parameter sharding can leave long-context training limited by activations. The long-context training talk illustrates this shift: reducing model-state memory did not remove attention's allocations. The next partition must target sequence-dependent work rather than repeat the parameter-storage optimization.

Attention uses queries to match keys, then combines associated values; an attention head performs one such learned interaction. Matching and transmitted content supplies the model explanation. Dividing query positions does not make the required remote keys and values disappear, and their availability does not remove causal masks.

Some operations can run separately at each token position; attention also needs information from other positions. Megatron's terminology reflects that distinction. It calls partitioning selected activations in token-local operations, specifically Dropout and LayerNorm, sequence parallelism. Its context parallelism partitions inputs and activations across positions while exchanging the remote keys and values needed for attention. Other systems use sequence parallelism more broadly, so name the tensor axis and operation as well as the label.

DeepSpeed Ulysses (2023) uses all-to-all exchanges: each worker sends different slices to different peers. Position-partitioned queries, keys, and values become head-partitioned for full-sequence attention. A second exchange returns outputs to position ownership.

Ring Attention, introduced by Hao Liu, Matei Zaharia, and Pieter Abbeel in October 2023, instead retains local query blocks while circulating key/value blocks and accumulating attention contributions. Transfer can be hidden when block computation lasts at least as long as the corresponding communication. Faster arithmetic relative to the interconnect can therefore require larger blocks to maintain overlap. Increased executable sequence capacity still does not establish useful long-context behavior in a model.

Move head slices or circulate context

Ulysses · exchange position/head slices, then restore position ownership

SliceInput position ownerAfter all-to-all: attention ownerAfter all-to-all: output owner
A/H0U0U0 · A+B, H0U0
A/H1U0U1 · A+B, H1U0
B/H0U1U0 · A+B, H0U1
B/H1U1U1 · A+B, H1U1

Each attention owner has all positions for its head subset. It applies the mask locally; the second exchange returns outputs to their original position owner.

Ring Attention · one rotation of paired K and V blocks

Queries stay local as paired key/value blocks circulateR0 sends KV A to R1, R1 sends KV B to R2, and R2 sends KV C to R0. Queries A, B and C remain on R0, R1 and R2 respectively.R0Queries QA stay hereKVA before → KVC afterApply mask; accumulate attentionR1Queries QB stay hereKVB before → KVA afterApply mask; accumulate attentionR2Queries QC stay hereKVC before → KVB afterApply mask; accumulate attentionKV A →KV B →← KV C

Further rotations bring the remaining context. Partial attention results require normalization-aware accumulation, not an unweighted sum. Queries stay local; all context is not permanently gathered on every worker.

A/B/C are position blocks; H0/H1 are head subsets. Both methods preserve attention masks. Their different worker counts explain ownership, not relative performance.

Place experts under uneven demand

A mixture-of-experts, or MoE, layer contains several learned subnetworks called experts. A learned router selects a subset for each token position. Total parameter capacity can therefore exceed the parameters activated for one token. Experts need not have clean human-readable specialties, and sparse arithmetic does not guarantee low distributed latency.

With expert parallelism, experts live on different workers. Dispatch sends token representations to selected destinations; return traffic brings expert outputs back for weighted combination at their original positions. All-to-all is useful because each destination needs a different slice. Equal expert counts per worker do not imply equal load: many tokens may select the same few experts.

GShard, published as a June 2020 preprint, bounded expert buffers by limiting the tokens each expert could accept. Its top-two router selected at most two experts per token. If both destinations overflowed, the token followed the residual connection, the path carrying input around the expert computation, without receiving an expert contribution. This capacity policy changed which computation occurred; it is not a universal property of MoE.

FasterMoE addressed uneven demand differently. Jiaao He, Tiago Antunes, and Jidong Zhai's 2022 account describes expert shadowing: broadcast selected popular expert weights so other workers can execute that expert, trading extra weight movement and storage for less concentration of token work. A performance model guides the choice. Its topology-aware expert-selection proposal is a separate intervention because it changes which expert is chosen, not merely how unchanged results travel.

Bring expert weights to some of the work

Weight copy: W0’s E0 → W1’s shadow of E0
The original remains on W0; the shadow consumes additional weight storage on W1.

Token; fixed weightOriginal token → compute → outputWith shadow: token → compute → output
a; αaW1 → W0: E0(a) → W1:aW1 → W0: E0(a) → W1:a
b; αbW1 → W0: E0(b) → W1:bW1: E0(b) → W1:b (local)
c; αcW1 → W0: E0(c) → W1:cW1: E0(c) → W1:c (local)

Table arrows carry tokens and expert outputs; the separate arrow above copies weights. Every selected E0 contribution is computed once and combined at its original position with the same α. No router decision or overflow policy changes.

Three token positions select E0. Shadowing changes execution placement while preserving selection and combination weights. Added weight movement and storage do not imply a speedup or a complete training synchronization protocol.

Evaluate dispatch, expert computation, and return together under the actual token assignment. Faster transfers cannot compensate for arbitrarily concentrated expert work; replication cannot help without paying to make weights available. Token-to-expert routing is internal model execution, distinct from choosing a model endpoint or orchestrating several independent agents.

V — Serving

Replicas and cooperating serving groups

A serving replica is a complete executable model instance, potentially spanning several workers. Independent dense-model replicas can process different request batches without synchronizing training gradients. Within a tensor- or pipeline-parallel replica, however, workers cooperate on the same requests and must agree on the request positions and computation they are executing. The complete dependency group determines usable capacity.

For eight devices, these are alternative allocations only if both fit the model and leave sufficient request-state headroom.
AllocationRequest ownershipTradeoff
Four replicas of two devicesFour independently scheduled request groups and caches.More independent serving groups; each group has less aggregate memory.
Two replicas of four devicesTwo independently scheduled request groups and caches.Larger cooperating groups; different per-request arithmetic and communication costs.

Do not infer throughput from this count. Continuous batching determines how requests share execution, while KV capacity bounds their retained state. Decode repeatedly pays the communication pattern for successive tokens; pipeline concurrency across requests does not remove one sequence's dependencies. In some vLLM MoE arrangements, shared expert layers even couple otherwise separate data-parallel engines, requiring aligned forward passes and dummy work from idle participants.

Dispatch among equivalent replicas can consider both load and prefix locality: compatible cached state for the request's exact preceding tokens. NVIDIA Dynamo, a distributed-serving system, combines reusable prefix overlap with projected work. A less busy worker without that cached prefix can win over a cache-rich congested worker. Workers report cache creation and release to update the routing index; separate transfer mechanisms move state when needed. This chooses where an equivalent model executes, unlike model or provider selection.

The same boundary explains cross-datacenter reinforcement-learning rollouts. A rollout is a generated trajectory used for learning; a policy version identifies the generating model state, as introduced in Post-training. Nan Jiang's cross-datacenter design keeps tightly coupled training local while serving islands exchange versions and completed trajectories externally. Each island may still need fast internal links. Requests and responses name policy versions so independently advancing islands do not hide staleness.

Move cache between serving phases

Prefill–decode disaggregation assigns prompt processing and continuing generation to separately scheduled pools. The phase distinction matters because their resource demands differ: prompt work can delay ongoing decoding, while one allocation need not suit both phases. Separation removes some interference but introduces another queue and a transfer of the KV state needed to continue.

Splitwise's 2024 implementation overlaps layerwise cache transfers with later prompt computation; the destination waits on a completion semaphore. The study used BLOOM-176B and Llama2-70B with eight-way tensor parallelism. Figure 14 covered batched prompt sizes through 2048 tokens and reported roughly 8 ms of exposed transfer on A100 with 200-Gbps InfiniBand, and 5 ms on H100 with 400-Gbps InfiniBand. These are remaining delays after overlap, not total transfer durations or an isolated GPU-generation comparison; the figure does not separate model curves. DistServe additionally searched phase-specific parallelism and placement under memory, workload, and network constraints.

A handoff must preserve more than a model name. The documented NixlConnector compatibility checks include model dimensions, numerical types, attention backend, cache layout, software version, and protocol. Different tensor-parallel degrees or cache block sizes work only under supported conversion rules. Configuration compatibility is not proof that both sides loaded identical weight artifacts. The destination must establish that its required cache is complete and interpretable before dependent decode consumes it.

Source release is a separate obligation. vLLM 0.15.0 documents failure or recomputation after KV-load failure; recomputing introduces prompt work into the decode pool. It also documents temporary source-cache retention after some aborts until a timeout. Successful dispatch or receipt of cancellation therefore does not establish that both endpoints have released their allocations.

This snapshot separates numerical cache transfer from metadata and request scheduling. Decode waits for the required destination state; source buffers follow a backend-specific release condition. Layerwise overlap does not imply immediate cleanup or a universal speedup.

Disaggregation helps only when removed interference and improved allocation exceed transfer, conversion, buffering, and queueing costs. Short prompts can leave little interference to remove; an unsuitable pool ratio starves one phase or overloads the other. In the local heterogeneous-serving demonstration, unintended Wi-Fi routing made cache movement a bottleneck. The general lesson is to verify the path actually carrying the bytes, not merely the interfaces installed on the machines.

VI — Failure and recovery

Detect missing execution progress

A live process can be unable to complete useful model work. A straggler progresses unusually slowly; a watchdog monitors expected progress; a timeout reports that a completion expectation was missed. None alone identifies the cause. A participant may be delayed before entering communication, stuck on the GPU, executing an incompatible collective, or unable to reach peers.

PyTorch's March 2026 Flight Recorder account explains cross-rank diagnosis using group identity, collective sequence, type, shapes, datatypes, stacks, and scheduled/started/completed states. Align the same logical operation before comparing participants. The reporting rank can merely be the first to notice another participant's problem. Missing trace output remains unknown, not proof of a hardware fault.

These invented record patterns illustrate different next checks; neither is a complete incident diagnosis.
Aligned observationsDiscriminating next check
Rank 0 entered sequence 42; rank 1's latest available record is 41.Inspect rank 1's intervening execution and telemetry coverage.
Both entered sequence 42, but their tensor counts differ.Find the divergent shape or call path before blaming link speed.

Correlate logical records with physical placement. Krea reported cross-node InfiniBand and within-node NVLink telemetry revealing failures that ordinary GPU health views missed. MegaScale described stragglers that appeared normal in isolated matrix tests but slowed distributed execution. Shared node, link, and storage observations help explain why several ranks stall together. Use execution reconstruction to keep observations separate from inferred causes.

When communication cannot complete, NCCL supports detecting asynchronous errors and aborting the affected communicator. Recreating that communication context does not restore lost parameters, optimizer state, or request caches. Determine the recovery unit from the dependencies that can no longer proceed, then restore the corresponding application state.

Restore a coherent training state

A resumable checkpoint contains more than weights: optimizer, scheduler, random-generator, input-progress, and other state can determine the next update. A checkpoint is more than weights defines that inventory. Distribution adds a consistency requirement: all saved portions must describe corresponding progress. A completed-update boundary is a straightforward choice; saving mid-accumulation additionally requires preserving or deliberately discarding the unfinished contributions.

Asynchronous checkpointing separates copying a stable snapshot from writing it to storage. PyTorch's asynchronous example exposes staging and upload completion separately. It waits for staging before the next parameter mutation and for upload before another checkpoint or final cleanup. This permits overlap while protecting the copied state. Custom mutable buffers need equivalent protection, and staging consumes host memory and CPU resources.

In PyTorch Distributed Checkpoint, ranks write their assigned data and the coordinator finishes metadata after collecting write results. DCP’s coordinated exception handling prevents that finish when a participating operation reports failure. This establishes a reported-success gate, not independent shard-integrity verification or bounded detection of a process that disappears.

Publishing a checkpoint tells loaders that a save is available; crash durability concerns whether it survives a system failure. The DCP local filesystem writer, with its default file synchronization enabled, synchronizes data files, writes and synchronizes temporary metadata, then renames it to the final filename loaders open. A fresh incomplete directory without final metadata cannot be loaded through that path. However, replacing existing metadata does not guarantee that a crash leaves either the old or the new checkpoint available. Nor does the sequence establish a crash-durable transaction across the directory. Storage-backend guarantees and recovery tests must address those failures separately.

Protect the snapshot, then publish the save

Release mutable training state before storage publicationAt a coherent update boundary, create a protected snapshot. Staging completes before the next parameter mutation, which can overlap background writes. All rank writes must report success before the coordinator finishes. The default local filesystem path syncs temporary metadata and renames it to the final name.Event order → · schematic, not elapsed durationTrainingSnapshotRank writesPublicationCoherentupdate boundaryProtected stagingStaging completeNext parameter mutations may proceedRank 0 write + data syncRank 1 write + data syncAll writes reportedsuccessfulCoordinator finishWrite + sync temporary metadata → rename final metadata
  • Every participating write reports success: coordinator finish can proceed.
  • A participating operation reports failure: coordinated exception handling blocks finish.
  • A write is unfinished or a process disappears: completion remains unknown; this gate does not promise bounded detection.
The asynchronous staging contract and DCP’s reported-write gate are distinct from filesystem durability. This local-filesystem sequence assumes default synchronization. A fresh directory without final metadata is not loadable; replacing an existing save is not a crash-atomic transaction.

Restart policy is another layer. Under documented torchrun behavior, worker failure or membership change stops and reforms the worker group. Loading can reshard tensors into a different topology, but the application must revisit global batch size, random state, and input assignment. TorchData's StatefulDataLoader coordinates local loading subprocesses, not distributed ranks. Successfully loading every tensor therefore does not establish unchanged data coverage or the same update sequence after a world-size change.

Checkpoint frequency balances saving cost against repeated work after failure. John Daly's 2006 analysis treats useful computation, checkpoint writing, lost work, and restart separately. Its stochastic assumptions do not automatically fit correlated failures or asynchronous staging. Measure your saving and recovery costs rather than copying an interval formula without its failure model.

A bounded verification method is to compare an uninterrupted run with one stopped during staging or writing and restored from the last completed checkpoint. Check the selected update boundary, tensor and optimizer state, next input identities, and subsequent updates under declared numerical tolerances. Test changed membership separately: it changes more than the location of tensor shards.

Restore capacity without inventing continuity

Serving recovery has two separate goals: restore executable capacity and determine what happens to interrupted requests. Discovery tells dispatchers which endpoints exist; readiness concerns eligibility for work; draining stops new admission while allowing existing work to finish. Replacing failed workers can restore future service without recovering any particular stream.

A frontend can answer health checks while no model worker is available. Dynamo's development health reference allows this and offers active canaries: small inference requests that exercise the engine. They test more than process reachability, but not performance under representative load. Routing eligibility is another boundary. The operator implementation described here registers workers separately from Kubernetes readiness probes and leaves active health checks disabled by default. A readiness probe therefore does not establish that inference was tested or that routing will exclude an unusable worker; verify both mechanisms in the deployed version and configuration.

A recovery contract should assign these responsibilities separately.
BoundaryRequired result
Discovery and group managementRemove unusable endpoints; restore the configured cooperating group.
Execution readinessCompatible model state and communication can actually execute inference.
Request replayRetained input and continuation state are sufficient for the supported request type.
Client deliveryKnow which output can be repeated or continued without contradicting committed output.
Cancellation and cleanupConfirm backend work has stopped and transfer-owned resources are eventually released.

Replay uses retained request information to rebuild execution elsewhere. Dynamo's documented migration retains prompt and generated tokens, with attempt and sequence-length limits. It excludes multi-choice requests and structured-output continuation. In the latter, replaying tokens as context does not advance the replacement engine's constraint state machine—the state that restricts legal next tokens. Constrained decoding explains that additional state. A retained prefix is therefore not a universal continuation checkpoint.

The described Dynamo migrator records returned token IDs before forwarding them upstream, appends them to the replay request, and subtracts them from the remaining generation budget. Its boundary is tokens observed by the migrator, not client-acknowledged delivery. Rebuilding KV state may permit further generation, but this path neither transfers backend random-generator state nor establishes reconnect deduplication. Apply safe repetition contracts before promising an uninterrupted, duplicate-free stream.

Observed tokens are not acknowledged delivery

Record observed token IDs before forwardingOriginal engine sends token IDs to the migrator. The migrator appends observed IDs and accounts for their count before forwarding to the client. Client receipt remains unknown at interruption. On a later migratable error, the migrator sends retained prompt, continuation and remaining budget to a replacement engine, which rebuilds KV.Original engineMigratorClient boundaryReplacement engineReturned token IDsRetain k observed tokensAppend to replay contextRemaining budget: n − kForward upstreamInterruption:receipt unknownMigratable errorPrompt + observed continuation + budget n − kRebuild KV from contextGenerate continuation

Observed tokens condition the replacement; they are not automatically re-emitted as a prefix. This path carries neither backend random-generator state nor a client acknowledgment cursor. It does not establish identical samples or reconnect deduplication, and structured-output continuation needs additional constraint state.

The described Dynamo migrator records returned IDs before forwarding upstream. On a migratable error, replay uses its observed continuation and remaining generation budget; client receipt at interruption is a separate, unknown boundary.

Cancellation has a similar boundary. Linked request contexts can propagate a stop signal, but the engine must observe it and abort computation. Dynamo's cancellation metric counts received signals, not confirmed engine aborts. During an interrupted cache transfer, retained source blocks can outlive the signal. Recovery tests must observe execution and resource release as well as control messages.

VII — Measurement

Find the useful scaling limit

Choose a feasible baseline before calculating speedup. If one device cannot execute the model, compare against a credible multi-device configuration rather than an imaginary single-device time. Freeze the model, precision, input distribution, output policy, and semantic requirements. Then change the smallest partition or placement decision predicted to shorten the critical path. Measure completed work, not submission, and benchmark the workload, not one convenient tensor shape.

Each proposed optimization needs a mechanism-level prediction and an end-to-end acceptance measure.
ChangeInspectAccept only when
Larger gradient bucketsReadiness delay, message count, exposed reductions.Completed updates improve with intended normalization unchanged.
Earlier parameter prefetchGather overlap and peak simultaneous residency.Useful execution improves without exceeding memory.
Higher tensor-parallel degreeLocal operation sizes and repeated communication.The fixed workload completes sooner, not merely on more devices.
Separate serving phasesInterference removed, transfer tail, and both queues.More requests meet the declared latency requirements.
Different checkpoint intervalSaving cost, lost work, and recovery duration.Useful training completion improves under the relevant failures.

For training, separate step throughput from completion at a quality target. MLPerf Training measures wall-clock time to a specified quality on a specified dataset and repeats runs to account for variability. Increasing the global batch changes the learning experiment unless deliberately controlled. Production completion also includes checkpointing and recovery; MegaScale's coordinated diagnostics, node replacement, and checkpoint restore illustrate work omitted by a healthy-step benchmark.

For serving, define goodput here as completed requests satisfying configured latency objectives per benchmark second. State offered load, prompt/output distributions, errors, and completion accounting. Time to first token, or TTFT, needs a start and first-content boundary. Average time per output token, or TPOT, describes pace after the first token; it is not the distribution of individual gaps. AIPerf calls its per-request average ITL and separately reports inter-chunk gaps. Its request-latency boundary ends at the last content response, which need not be a later terminal event. Latency compliance also does not establish answer correctness.

Use per-rank progress, peak memory, transfer measurements, imbalance, and failure behavior to explain the result. Network microbenchmarks help isolate a mechanism, but the final question remains whether the model workload finishes usefully sooner. More parallelism stops helping when its communication, coordination, or recovery costs outweigh the declared gain. Full cost attribution belongs to AI Cost and Performance Engineering; the execution decision begins with ownership and ends with completed outcomes.

Open questions

  1. Elastic recovery needs a contract for more than tensor resharding. Changing membership also changes input assignment and potentially batch composition. Progress would mean tested preservation of declared data coverage and update semantics, including unfinished accumulation and application state.

  2. Portable stream recovery requires agreement on committed output and continuation state across the client, migrator, and engine. Token replay alone misses constraint machines and delivery acknowledgements. Progress would include explicit replay boundaries and failure tests covering disconnects, partial transfers, and cancellation.

  3. Automatic partition selection must predict transient memory and exposed communication under changing workloads, not just static parameter fit. Useful progress would be a placement policy whose predictions remain calibrated across prompt lengths, contention, and hardware changes, with safe responses when estimates fail.

  4. Sparse expert placement must respond to demand without making weight movement more expensive than the congestion it removes. The hard boundary is between relocating unchanged computation and changing expert selection. Progress would compare both under controlled routing distributions and report end-to-end completion alongside memory and traffic.

Follow the curated reading path through the speakers and demonstrations behind this entry.

20 min

AI Engineer World's Fair 2025 · 2025

Hacking the Inference Pareto Frontier

Kyle Kranen

Cited in this entry

Develops the interaction among phase separation, worker ratios, cache locality, and queueing, with workload-dependent tradeoffs rather than a universal disaggregation prescription.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

20 matching talks

TalkSpeakerEventYear
Alex CheemaAI Engineer Europe 20262026
Mark MoyouAI Engineer World's Fair 20242024
Rhythm Garg, Linden LiAI Engineer Code 20252025
Walden, Carter, Tanay, Alex Atallah, NavAI Engineer World's Fair 20262026
Philip Kiely, Yineng ZhangAI Engineer World's Fair 20252025
Daniel Kim, Daria SobolevaAI Engineer World's Fair 20252025
Dylan PatelAI Engineer World's Fair 20242024
Lachlan Ainley, Humza IqbalAI Engineer World's Fair 20242024
Nader Khalil, Alex Cheema, Matthew Berman, Ahmad Osman, Joseph NelsonAI Engineer World's Fair 20262026
Joe FiotiAI Engineer World's Fair 20252025
Sander DielemanAI Engineer Europe 20262026
Yuchen Fama, Ashish KamraAI Engineer World's Fair 20262026
Abdul DakkakAI Engineer World's Fair 20262026
Bogdan GazaAI Engineer Code 2025Unknown
Nishant Gupta, Naman AhujaAI Engineer World's Fair 20262026
Dylan Bristot, Sujee ManiyamAI Engineer Code 2025Unknown
Harshul Jain, Tanmay SahEvent unknownUnknown
John OusterhoutEvent unknownUnknown
Daniel KimAI Engineer World's Fair 20262026
Tarun SunkaraneniAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
16 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
9 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. vLLM 0.22.0: Parallelism and Scaling

    https://docs.vllm.ai/en/v0.22.0/serving/parallelism_scaling/

  2. vLLM: Data Parallel Deployment

    https://docs.vllm.ai/en/latest/serving/data_parallel_deployment/

  3. LLNL: Introduction to Parallel Computing Tutorial

    https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial

  4. LLM Inference Beyond a Single Node: From Bottlenecks to Mitigations with Fast All-Reduce Communication

    https://arxiv.org/html/2511.09557v1

  5. PyTorch: Training a Classifier

    https://docs.pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html

  6. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=378

  7. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=814

  8. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

    https://arxiv.org/html/1910.02054v3

  9. vLLM memory profiling: allocations and transient peaks

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/utils/mem_utils.py

  10. Transformers: cache allocation versus retained history

    https://huggingface.co/docs/transformers/en/kv_cache

  11. vLLM GPU worker: profiling available KV capacity

    https://raw.githubusercontent.com/vllm-project/vllm/main/vllm/v1/worker/gpu_worker.py

  12. NCCL: Creating a Communicator

    https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/communicators.html

  13. PyTorch: torchrun Elastic Launch

    https://docs.pytorch.org/docs/main/elastic/run.html

  14. PyTorch: Getting Started with DeviceMesh

    https://docs.pytorch.org/tutorials/recipes/distributed_device_mesh.html

  15. PyTorch: Getting Started with Fully Sharded Data Parallel (FSDP2)

    https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html

  16. Building Generative Image & Video Models at Scale

    https://www.youtube.com/watch?v=xOP1PM8fwnk&t=1374

  17. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    https://arxiv.org/html/1909.08053v4

  18. Mastering LLM Inference Optimization: From Theory to Cost-Effective Deployment

    https://www.youtube.com/watch?v=9tvJ_GYJA-o&t=1530

  19. GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism

    https://papers.neurips.cc/paper/8305-gpipe-efficient-training-of-giant-neural-networks-using-pipeline-parallelism.pdf

  20. NCCL: Collective Operations

    https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html

  21. MPI 4.1: Nonblocking Collective Operations

    https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report/node145.htm

  22. SUMMA: Scalable Universal Matrix Multiplication Algorithm

    https://www.netlib.org/lapack/lawnspdf/lawn96.pdf

  23. Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

    https://www.youtube.com/watch?v=tQTB4MU_z8w&t=642

  24. Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities

    https://www.cs.cmu.edu/~18742/papers/Amdahl1967.pdf

  25. Reevaluating Amdahl's Law

    https://course.ece.cmu.edu/~ece600/fall16/references/gustafson.pdf

  26. NCCL Tests: Performance

    https://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md?plain=1

  27. PyTorch: Distributed Data Parallel

    https://docs.pytorch.org/docs/main/notes/ddp.html

  28. DeepSpeed: Zero Redundancy Optimizer

    https://www.deepspeed.ai/tutorials/zero/

  29. Accelerate: Performing Gradient Accumulation

    https://huggingface.co/docs/accelerate/main/usage_guides/gradient_accumulation

  30. PyTorch 2.9: DistributedSampler

    https://docs.pytorch.org/docs/2.9/data.html

  31. Horovod: Fast and Easy Distributed Deep Learning in TensorFlow

    https://arxiv.org/html/1802.05799v3

  32. A Bridging Model for Parallel Computation

    https://web.mit.edu/6.976/www/handout/valiant2.pdf

  33. Large Scale Distributed Deep Networks

    https://proceedings.neurips.cc/paper/2012/file/6aca97005c68f1206823815f66102863-Paper.pdf

  34. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

    https://arxiv.org/pdf/2304.11277

  35. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    https://www.youtube.com/watch?v=TUnPNY4E2fw&t=468

  36. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    https://www.youtube.com/watch?v=TUnPNY4E2fw&t=518

  37. PyTorch: Numerical Accuracy

    https://docs.pytorch.org/docs/main/notes/numerical_accuracy.html

  38. Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

    https://arxiv.org/html/2104.04473v5

  39. PipeDream: Generalized Pipeline Parallelism for DNN Training

    https://www.microsoft.com/en-us/research/uploads/prod/2019/08/pipedream.pdf

  40. NVIDIA: Massively Scale Your Deep Learning Training with NCCL 2.4

    https://developer.nvidia.com/blog/massively-scale-deep-learning-training-nccl-2-4/

  41. NVIDIA GPUDirect RDMA: Overview and Design Considerations

    https://docs.nvidia.com/cuda/gpudirect-rdma/index.html

  42. Doubling All2All Performance with NVIDIA Collective Communication Library 2.12

    https://developer.nvidia.com/blog/doubling-all2all-performance-with-nvidia-collective-communication-library-2-12/

  43. Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

    https://www.youtube.com/watch?v=tQTB4MU_z8w&t=841

  44. Kubernetes: Pod Topology Spread Constraints

    https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/

  45. Kueue: All-or-nothing Scheduling

    https://kueue.sigs.k8s.io/docs/concepts/all_or_nothing/

  46. Infra behind Krea 2 - How to train and serve at scale

    https://www.youtube.com/watch?v=byn9PURoBNY&t=543

  47. Insights from Snorkel AI running Azure AI Infrastructure

    https://www.youtube.com/watch?v=LJa1SjCkYas&t=743

  48. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    https://www.youtube.com/watch?v=TUnPNY4E2fw&t=343

  49. Megatron Core: Context Parallel Package

    https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/features/context_parallel.html

  50. DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models

    https://github.com/deepspeedai/DeepSpeed/blob/master/blogs/deepspeed-ulysses/README.md

  51. Ring Attention with Blockwise Transformers for Near-Infinite Context

    https://arxiv.org/html/2310.01889v3

  52. From Mixture of Experts to Mixture of Agents … with Super Fast Inference

    https://www.youtube.com/watch?v=tzRvcTEapzo&t=514

  53. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

    https://arxiv.org/html/2006.16668v1

  54. FasterMoE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models

    https://pacman.cs.tsinghua.edu.cn/~zjd/projects/fastermoe/

  55. From Mixture of Experts to Mixture of Agents … with Super Fast Inference

    https://www.youtube.com/watch?v=tzRvcTEapzo&t=85

  56. NVIDIA Dynamo: KV-Aware Routing

    https://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/kv-aware-routing

  57. Automatic prefix caching — vLLM

    https://docs.vllm.ai/en/stable/features/automatic_prefix_caching/

  58. Taking Reinforcement Learning Cross Datacenter

    https://www.youtube.com/watch?v=maRzp4kImJ4&t=126

  59. Taking Reinforcement Learning Cross Datacenter

    https://www.youtube.com/watch?v=maRzp4kImJ4&t=916

  60. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=400

  61. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    https://www.usenix.org/system/files/osdi24-zhong-yinmin.pdf

  62. Splitwise: Efficient Generative LLM Inference Using Phase Splitting

    https://www.cs.cmu.edu/~18742/papers/Patel2024.pdf

  63. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving

    https://arxiv.org/html/2401.09670v2

  64. vLLM: NixlConnector Compatibility Matrix

    https://docs.vllm.ai/en/latest/features/nixl_connector_compatibility/

  65. vLLM 0.15.0: NixlConnector Usage Guide

    https://docs.vllm.ai/en/v0.15.0/features/nixl_connector_usage/

  66. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=556

  67. Frontier AI at Home (literally)

    https://www.youtube.com/watch?v=ESbWpPT_9-o&t=5491

  68. Flight Recorder: A New Lens for Understanding NCCL Watchdog Timeouts

    https://pytorch.org/blog/flight-recorder-a-new-lens-for-understanding-nccl-watchdog-timeouts/

  69. Infra behind Krea 2 - How to train and serve at scale

    https://www.youtube.com/watch?v=byn9PURoBNY&t=400

  70. MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs

    https://arxiv.org/html/2402.15627v1

  71. TorchSnapshot: Getting Started

    https://meta-pytorch.org/torchsnapshot/stable/getting_started.html

  72. PyTorch: Asynchronous Saving with Distributed Checkpoint

    https://docs.pytorch.org/tutorials/recipes/distributed_async_checkpoint_recipe.html

  73. PyTorch: Distributed Checkpoint

    https://docs.pytorch.org/docs/main/distributed.checkpoint.html

  74. PyTorch DCP utils.py: coordinated exception handling

    https://raw.githubusercontent.com/pytorch/pytorch/main/torch/distributed/checkpoint/utils.py

  75. PyTorch DCP filesystem.py: FileSystemWriter and FileSystemReader

    https://raw.githubusercontent.com/pytorch/pytorch/main/torch/distributed/checkpoint/filesystem.py

  76. TorchData: Stateful DataLoader

    https://meta-pytorch.org/data/main/torchdata.stateful_dataloader.html

  77. A Higher Order Estimate of the Optimum Checkpoint Interval for Restart Dumps

    https://graal.ens-lyon.fr/~abenoit/CR02/papers/daly.pdf

  78. NVIDIA Dynamo 0.9.1: Fault Tolerance

    https://docs.nvidia.com/dynamo/v-0-9-1/user-guides/fault-tolerance

  79. NVIDIA Dynamo: Overall Architecture

    https://docs.nvidia.com/dynamo/dev/knowledge-base/overview

  80. NVIDIA Dynamo: Health Check Reference

    https://docs.nvidia.com/dynamo/dev/observability/signals/health-checks

  81. Dynamo Operator: worker container defaults

    https://raw.githubusercontent.com/ai-dynamo/dynamo/main/deploy/operator/internal/dynamo/component_worker.go

  82. NVIDIA Dynamo: Request Migration

    https://docs.nvidia.com/dynamo/dev/kubernetes/fault-tolerance/request-migration

  83. Dynamo migration.rs: response tracking and stream recreation

    https://raw.githubusercontent.com/ai-dynamo/dynamo/main/lib/llm/src/migration.rs

  84. NVIDIA Dynamo: Request Cancellation Architecture

    https://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/fault-tolerance/request-cancellation-architecture

  85. CUDA C++ Best Practices Guide

    https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html

  86. MLCommons: MLPerf Training Benchmark

    https://mlcommons.org/benchmarks/training/

  87. AIPerf Metrics Reference

    https://docs.nvidia.com/aiperf/reference/ai-perf-metrics-reference

  88. Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

    https://www.youtube.com/watch?v=tQTB4MU_z8w&t=921

  89. Compute & System Design for Next Generation Frontier Models

    https://www.youtube.com/watch?v=gFyBdBm0AGo&t=938

  90. Frontier AI at Home (literally)

    https://www.youtube.com/watch?v=ESbWpPT_9-o&t=4210

  91. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    https://www.youtube.com/watch?v=TUnPNY4E2fw&t=556

  92. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=668

  93. MPI: A Message-Passing Interface Standard, Version 1.1

    https://www.mpi-forum.org/docs/mpi-1.1/mpi1-report.pdf

  94. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

    https://arxiv.org/html/1909.08053v4