AI Engineer World's Fair 2026

Taking Reinforcement Learning Cross Datacenter

Nan Jiang19:50

Loading slides…

Read the talk

Taking Reinforcement Learning Across Datacenters

Keep training on fast fabric, move rollout engines to available GPUs, and synchronize exact served weights through sparse patches and explicit version contracts.

From a talk by Nan Jiang

Before you start: Familiarity with neural-network training, inference, and floating-point precision is helpful.

When the training cluster limits rollout capacity

Where are the GPUs, and can an RL run get them now? At scale, post-training adds a physical capacity problem to the familiar questions about algorithms, environments, and tool calls. The available machines may not share a region or a fast interconnect. Some of the work normally confined to one cluster might instead run on scattered, autoscaled capacity—provided the RL loop can be separated enough to use it.

The standard loop starts with a trainer updating the policy. Rollout workers, also called samplers, use that policy to generate trajectories; environments return observations and rewards, and the trajectories go back to the trainer for subsequent updates. The critical connection is weight synchronization. With the trainer and samplers in the same cluster, RDMA—remote direct memory access—makes that synchronization fast. But it also couples rollout capacity to the training cluster: when sampling needs more workers, the cluster may have no more nodes available.

Diagram connecting Trainer, Sampler, and RL Env, with arrows for new weights, trajectories, actions, and rewards.Open full source frame
The default RL loop places the trainer and sampler in one cluster.

Nan Jiang calls the two available compute shapes the cathedral and the sprawl.

Compute shapeWhat it providesWhat it cannot promise
CathedralMany GPUs in one region, joined by a fast interconnectElastic access to additional nodes
SprawlCapacity across providers and regions, with different prices and availabilityOne coherent RDMA fabric

The mismatch is that usable compute is distributed, while the default RL loop asks for one tightly coupled cluster.

That cluster must satisfy four requirements together: enough GPUs, the same region, fast fabric, and immediate availability. Each requirement is manageable in isolation; their intersection is restrictive. Inference capacity can be elastic without belonging to an expandable RDMA cluster. Keeping the whole RL loop inside the training cluster makes rollout inherit the hardest capacity constraint, even if rollout does not need the same communication pattern.

0:280:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:28 · section reference included

Move serving islands, keep backpropagation local

Training is a tightly coupled job. Its steps involve collectives, all-reduce, and communication between model-parallel workers, so it belongs on fast fabric. Rollout is a fleet of serving jobs: workers generate trajectories, call environments or tools, and send data back. There is no global all-reduce across those rollout jobs. Backpropagation stays in the cluster; the rollout fleet can leave.

The movable unit is a rollout serving island: one coherent endpoint, or a local group of endpoints, serving one policy version. An island can retain the local parallelism a large model needs, including prefill/decode disaggregation and other serving constraints. Its external dependency is much lighter: a policy version comes in; trajectories and metadata go out. The resulting architecture keeps the trainer’s collectives inside the RDMA cluster while rollout islands fan out across available capacity, without adding a global collective between them.

3:253:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:25 · section reference included

A full checkpoint is the wrong synchronization unit

Separating rollout from training leaves one expensive link: publishing new weights. The naive approach moves a full checkpoint through storage and over the network whenever rollout needs another policy version. Jiang uses a roughly 500 GB checkpoint at Kimi scale in NVFP4 as his size illustration. He describes full weight synchronization at that scale as taking minutes to hours, depending on the setup. Even asynchronous RL benefits from fresh weights arriving within seconds, so moving rollout elsewhere is useful only if the synchronization object becomes much smaller.

The proposed shortcut starts with a hypothesis: fewer than 1% of rollout-visible weights change between versions. Rollout-visible means the representation the serving engine actually consumes, such as FP8 or NVFP4—not FP32 master weights or Adam’s optimizer moments. Transfer the changes to that served representation, while requiring bitwise reconstruction of the version a full checkpoint would have produced. With sufficient sparsity, the proposed transfer shrinks from hundreds of gigabytes to hundreds of megabytes. The next question is why a broadly trained model would have so few visible changes.

5:015:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:01 · section reference included

Small Adam steps meet finite precision

The first ingredient is finite precision. An optimizer can maintain high-precision master weights while the forward pass reads a BF16 view. That view has discrete representable values. For a weight magnitude Θ, Jiang uses the following approximate spacing and rounding scale:

ULPBF16(Θ)≈Θ128,half-ULP scale≈Θ256.\mathrm{ULP}_{\mathrm{BF16}}(\Theta) \approx \frac{\Theta}{128}, \qquad \text{half-ULP scale} \approx \frac{\Theta}{256}.

ULP means unit in the last place: the spacing between adjacent representable values. Around a weight value of one, the illustrated BF16 spacing is about 0.0078, with a half-ULP scale of about 0.0039. An update that leaves the master weight inside the same rounding bin produces no change in the BF16 view.

Precision qualification: Half an ULP is a rounding scale, not every master weight’s distance to its nearest boundary. A master weight already near a rounding boundary can cross it with a smaller update.

The second ingredient is Adam’s update magnitude. Ignoring weight decay, an Adam or AdamW update is the learning rate multiplied by a normalized direction. Raw gradients can be dense and vary widely in magnitude, but Adam normalizes them using running gradient statistics. The PULSE study bounds the per-parameter step by a constant b times the learning rate under its assumptions:

Δθi=−ηui,∣Δθi∣≤bη.\Delta\theta_i = -\eta u_i, \qquad |\Delta\theta_i| \le b\eta.

Here, η is the learning rate and uᵢ is the normalized direction. The useful point is the controlled scale of the update: at the RL post-training learning rates under discussion, the per-weight push is tiny.

Combining the two ingredients gives Jiang’s push versus floor picture. A served value changes only when the optimizer’s push crosses a rounding boundary. In the illustrative comparison, an Adam step of 3e-6 is more than a thousand times smaller than the approximately 0.0039 half-ULP scale around one. That disparity explains why many updates can be absorbed by rounding; it does not, by itself, prove that a particular cast stays unchanged. The master weights still change, and accumulated updates can eventually become visible.

The chart expresses the mechanism with weight magnitude on the horizontal axis and update magnitude on the vertical axis. A red line represents the approximate BF16 visibility scale, increasing with weight magnitude. A green band represents the Adam push and its conservative bound. Where the red line lies above the green band, small master-weight changes are often invisible in the served view. Small-magnitude weights have finer spacing and can change more readily; larger weights more readily absorb the same small push. This is Adam absorption, the mechanism behind a sparse served-weight update.

7:107:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:10 · section reference included

Construct a lossless patch

The synchronization object is a difference between served versions, not a difference between optimizer states. Constructing it follows a specific order:

  1. Project into the serving representation. Cast or otherwise convert the weights into the dtype the rollout engine consumes.
  2. Compare versions. Compare the served representations of t−1 and t bitwise.
  3. Encode the changes. Record changed positions, replacement bits, and the metadata needed to apply the patch.
  4. Reconstruct the version. Apply selective overwrites, or an equivalent lossless encoding such as XOR, to recover the target served bits.

This is not repeated floating-point addition of numerical deltas. Replacement and XOR encodings avoid additive reconstruction drift: correctly applying the patch recovers exactly the served version that full synchronization would have delivered.

That small transfer has a different explanation from LoRA’s small transfer.

Training approachWhat the optimizer changesWhy synchronization can be small
Full-parameter RLThe model’s parameters broadlyAbsorption can leave most served weight bits unchanged
LoRASmall adapters while the base model stays frozenThe trainable object is small by construction

Full-parameter RL needs the precision argument to explain its sparse served-view patch. LoRA does not: its small adapter already limits the synchronization object.

10:4110:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:41 · section reference included

What the sparsity measurements count

The measurement is deliberately about weight bits, not gradients or optimizer state. Cast the weights to BF16, compare consecutive versions bitwise, and count the entries that did not change. Jiang reports that roughly 99% of BF16 weights were bit-identical per step across the model families examined in the cited study. He also reports that differences remain sparse when rollout lags behind the trainer. These are observations from the evaluated runs, not a universal sparsity guarantee. In either case, changed indices plus replacement values reconstruct the selected target version exactly.

Slide showing a three-step measurement procedure, green bars across model families, a staleness line chart, and a lossless-patch note.Open full source frame
PULSE measures unchanged weights by casting to BF16 and comparing bitwise.

Sparse served-weight changes do not imply sparse gradients. Jiang reports that about 99% of parameters had nonzero gradients in the cited study. The FP32 master-weight updates were also dense, but small. The apparent contrast disappears once the representations are separated: many parameters receive updates, while only a small fraction cross a boundary in the rollout engine’s lower-precision view. Nothing in the patch construction requires pruning those gradients.

12:0312:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:03 · section reference included

Lower serving precision introduces shared-scale constraints

Rollout engines may serve in FP8 or NVFP4 even when training uses BF16; serving precision and training precision are separate choices. Quantization-aware training is another possibility, but it is not required for that distinction. For a floating-point format with fixed scales and m mantissa bits, the illustrated visibility scale becomes:

visibility scale≈Θ2m+1.\text{visibility scale} \approx \frac{\Theta}{2^{m+1}}.

Fewer mantissa bits mean coarser spacing. In this simplified picture, FP8 has a higher visibility threshold than BF16, and FP4 has a higher threshold still, so fewer small updates may alter the served representation.

Shared-scale formats require more care. With independent rounding, one value crossing a boundary can change one encoded value. INT4 instead quantizes weights relative to a shared group scale, and NVFP4 uses hierarchical scales with its own encoding rules. The exact patch must preserve the relevant scale bits as well as the encoded weight values. Counting unchanged element codes alone does not establish that the complete served checkpoint is equally sparse.

Jiang then reports an internal run serving GLM-4.7-Air in FP8:

Point in the internal runReported changed weightsStated condition
First stepApproximately 0.15%Learning rate described as high
Later stepsApproximately 0.5% per stepAdam behavior described as stabilizing

These figures describe one internal run, not a guarantee across models or training schedules. Jiang also points to related RL synchronization research and cites Cursor, Composer-2, and MAI when discussing Adam’s use in post-training. With small, exact served-version updates as the working premise, the next problem is making remote rollout engines participate reliably.

13:0613:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:06 · section reference included

Publish immutable versions and make staleness explicit

The deployment architecture leaves the trainer in its RDMA cluster and places rollout engines outside it, potentially across regions and providers. After an update, the trainer publishes an immutable rollout weight version to a shared bulletin board. The request path carries a version contract: a request identifies the desired policy version and which versions it will accept. The response reports the version actually used, alongside generated tokens, log probabilities, router replay information, and other rollout metadata. Version identity becomes part of the interaction rather than an assumption about colocated processes.

After each optimizer step, the trainer publishes the version; engines pull it and materialize it locally in a checkpoint layout they can serve. An engine can choose how to load and shard that version, but it cannot substitute different served weights. Jiang describes a Hugging Face-style safetensors layout for compatible engines such as vLLM. Compatibility still depends on the attention and mixture-of-experts backends, parallelism, serving dtype, and GPUs. A common file format does not remove those execution constraints.

15:0615:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:06 · section reference included

A sidecar gates requests on committed versions

A sidecar makes an ordinary rollout engine aware of the version contract. It handles three cases:

  • An acceptable version is already committed: Proxy the request to the engine.
  • The engine is behind but can catch up: Apply the missing transactions so it can serve an acceptable version.
  • The required version cannot be reached: Return not ready rather than serving an unacceptable version.

This gate lets additional compatible GPU capacity join the rollout fleet without assuming that every engine is current at every moment. Each engine must either satisfy the request’s version requirements or decline it.

16:4116:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:41 · section reference included

From smaller transfers to an elastic rollout fleet

In-cluster synchronization is fast because RDMA provides the appropriate fabric. Moving full checkpoints across regions replaces that advantage with a large network transfer. Sparse patches change the amount of data that must cross the link. Jiang illustrates reducing a transfer from roughly 500 GB to 500 MB, with latency falling into seconds. This is a size-and-latency illustration, not a benchmark for a specified network or region pair. The actual payload also includes indices, replacement bits, scales where applicable, and metadata; fewer than 1% changed weights alone does not establish the illustrated byte count.

Modal’s Stitch is the concrete implementation Jiang describes for this protocol. On the trainer side, it publishes what defines a rollout weight version through the bulletin-board contract. On the rollout side, engines pull weights and synchronize across regions and providers. Jiang describes Stitch as agnostic to the trainer framework, serving engine, and transport, with an emphasis on asynchronous RL and agentic workloads.

Each rollout engine synchronizes its own weights, serves accepted versions, and returns rollout metadata. That gives the fleet a unit of independent growth: another compatible serving island can join without expanding the training cluster itself. Scattered inference capacity can become RL rollout capacity, while the trainer retains the tightly connected infrastructure its computation needs.

17:1417:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:14 · section reference included

What remains open beyond the Adam-based design

The architecture leaves three distinct research questions unresolved:

  • Other optimizers: Does served-weight sparsity persist under Muon? Jiang cites Moonshot and DeepSeek-V4 when discussing Muon adoption in post-training, but the preceding absorption argument specifically concerns Adam. It does not automatically establish the same behavior for another optimizer.
  • Fully asynchronous RL at scale: Access to global compute expands the possible rollout fleet. How far fully asynchronous RL can scale with that capacity remains an open question; distributing the workers does not itself answer it.
  • Training beyond RL: Could the same synchronization pattern help pre-training, mid-training, or supervised fine-tuning? Those extensions are proposed directions, not results established by this design.
18:4418:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:44 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…