# Taking Reinforcement Learning Cross Datacenter

Nan Jiang

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/maRzp4kImJ4-taking-reinforcement-learning-cross-datacenter) · [Video](https://www.youtube.com/watch?v=maRzp4kImJ4)

## Taking Reinforcement Learning Across Datacenters

Keep training on fast fabric, move rollout engines to available GPUs, and synchronize exact served weights through sparse patches and explicit version contracts.

Familiarity with neural-network training, inference, and floating-point precision is helpful.

### When the training cluster limits rollout capacity

[0:28](https://www.youtube.com/watch?v=maRzp4kImJ4&t=28s)

Where are the GPUs, and can an RL run get them now? At scale, post-training adds a physical capacity problem to the familiar questions about algorithms, environments, and tool calls. The available machines may not share a region or a fast interconnect. Some of the work normally confined to one cluster might instead run on scattered, autoscaled capacity—provided the RL loop can be separated enough to use it.

The standard loop starts with a trainer updating the policy. Rollout workers, also called samplers, use that policy to generate trajectories; environments return observations and rewards, and the trajectories go back to the trainer for subsequent updates. The critical connection is **weight synchronization**. With the trainer and samplers in the same cluster, RDMA—remote direct memory access—makes that synchronization fast. But it also couples rollout capacity to the training cluster: when sampling needs more workers, the cluster may have no more nodes available.

![Diagram connecting Trainer, Sampler, and RL Env, with arrows for new weights, trajectories, actions, and rewards.](https://ai.engineer/api/talk-essay-media/24e12cfbe63155b515bdaf3cd834b99a681ce5a1cb2f16400de11aeebf1c197c)

The default RL loop places the trainer and sampler in one cluster.

Nan Jiang calls the two available compute shapes the *cathedral* and the *sprawl*.

| Compute shape | What it provides | What it cannot promise |
|---|---|---|
| Cathedral | Many GPUs in one region, joined by a fast interconnect | Elastic access to additional nodes |
| Sprawl | Capacity across providers and regions, with different prices and availability | One coherent RDMA fabric |

The mismatch is that usable compute is distributed, while the default RL loop asks for one tightly coupled cluster.

That cluster must satisfy four requirements together: enough GPUs, the same region, fast fabric, and immediate availability. Each requirement is manageable in isolation; their intersection is restrictive. Inference capacity can be elastic without belonging to an expandable RDMA cluster. Keeping the whole RL loop inside the training cluster makes rollout inherit the hardest capacity constraint, even if rollout does not need the same communication pattern.

### Move serving islands, keep backpropagation local

[3:25](https://www.youtube.com/watch?v=maRzp4kImJ4&t=205s)

Training is a tightly coupled job. Its steps involve collectives, all-reduce, and communication between model-parallel workers, so it belongs on fast fabric. Rollout is a fleet of serving jobs: workers generate trajectories, call environments or tools, and send data back. There is no global all-reduce across those rollout jobs. **Backpropagation stays in the cluster; the rollout fleet can leave.**

The movable unit is a **rollout serving island**: one coherent endpoint, or a local group of endpoints, serving one policy version. An island can retain the local parallelism a large model needs, including prefill/decode disaggregation and other serving constraints. Its external dependency is much lighter: a policy version comes in; trajectories and metadata go out. The resulting architecture keeps the trainer’s collectives inside the RDMA cluster while rollout islands fan out across available capacity, without adding a global collective between them.

### A full checkpoint is the wrong synchronization unit

[5:01](https://www.youtube.com/watch?v=maRzp4kImJ4&t=301s)

Separating rollout from training leaves one expensive link: publishing new weights. The naive approach moves a full checkpoint through storage and over the network whenever rollout needs another policy version. Jiang uses a roughly 500 GB checkpoint at **Kimi** scale in [NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/) as his size illustration. He describes full weight synchronization at that scale as taking minutes to hours, depending on the setup. Even asynchronous RL benefits from fresh weights arriving within seconds, so moving rollout elsewhere is useful only if the synchronization object becomes much smaller.

The proposed shortcut starts with a hypothesis: fewer than 1% of rollout-visible weights change between versions. *Rollout-visible* means the representation the serving engine actually consumes, such as FP8 or NVFP4—not FP32 master weights or Adam’s optimizer moments. Transfer the changes to that served representation, while requiring **bitwise reconstruction** of the version a full checkpoint would have produced. With sufficient sparsity, the proposed transfer shrinks from hundreds of gigabytes to hundreds of megabytes. The next question is why a broadly trained model would have so few visible changes.

### Small Adam steps meet finite precision

[7:10](https://www.youtube.com/watch?v=maRzp4kImJ4&t=430s)

The first ingredient is finite precision. An optimizer can maintain high-precision master weights while the forward pass reads a BF16 view. That view has discrete representable values. For a weight magnitude Θ, Jiang uses the following approximate spacing and rounding scale:

```math
\mathrm{ULP}_{\mathrm{BF16}}(\Theta) \approx \frac{\Theta}{128},
\qquad
\text{half-ULP scale} \approx \frac{\Theta}{256}.
```

ULP means *unit in the last place*: the spacing between adjacent representable values. Around a weight value of one, the illustrated BF16 spacing is about 0.0078, with a half-ULP scale of about 0.0039. An update that leaves the master weight inside the same rounding bin produces no change in the BF16 view.

**Precision qualification:** Half an ULP is a rounding scale, not every master weight’s distance to its nearest boundary. A master weight [already near a rounding boundary](https://huggingface.co/blog/delta-weight-sync) can cross it with a smaller update.

The second ingredient is Adam’s update magnitude. Ignoring weight decay, an Adam or AdamW update is the learning rate multiplied by a normalized direction. Raw gradients can be dense and vary widely in magnitude, but Adam normalizes them using running gradient statistics. The [PULSE study](https://arxiv.org/abs/2602.03839) bounds the per-parameter step by a constant `b` times the learning rate under its assumptions:

```math
\Delta\theta_i = -\eta u_i,
\qquad
|\Delta\theta_i| \le b\eta.
```

Here, η is the learning rate and `uᵢ` is the normalized direction. The useful point is the controlled scale of the update: at the RL post-training learning rates under discussion, the per-weight push is tiny.

Combining the two ingredients gives Jiang’s *push versus floor* picture. A served value changes only when the optimizer’s push crosses a rounding boundary. In the illustrative comparison, an Adam step of 3e-6 is more than a thousand times smaller than the approximately 0.0039 half-ULP scale around one. That disparity explains why many updates can be absorbed by rounding; it does not, by itself, prove that a particular cast stays unchanged. The master weights still change, and accumulated updates can eventually become visible.

The chart expresses the mechanism with weight magnitude on the horizontal axis and update magnitude on the vertical axis. A red line represents the approximate BF16 visibility scale, increasing with weight magnitude. A green band represents the Adam push and its conservative bound. Where the red line lies above the green band, small master-weight changes are often invisible in the served view. Small-magnitude weights have finer spacing and can change more readily; larger weights more readily absorb the same small push. This is **Adam absorption**, the mechanism behind a sparse served-weight update.

### Construct a lossless patch

[10:41](https://www.youtube.com/watch?v=maRzp4kImJ4&t=641s)

The synchronization object is a difference between served versions, not a difference between optimizer states. Constructing it follows a specific order:

1. **Project into the serving representation.** Cast or otherwise convert the weights into the dtype the rollout engine consumes.
2. **Compare versions.** Compare the served representations of `t−1` and `t` bitwise.
3. **Encode the changes.** Record changed positions, replacement bits, and the metadata needed to apply the patch.
4. **Reconstruct the version.** Apply selective overwrites, or an equivalent lossless encoding such as XOR, to recover the target served bits.

This is not repeated floating-point addition of numerical deltas. Replacement and XOR encodings avoid additive reconstruction drift: correctly applying the patch recovers exactly the served version that full synchronization would have delivered.

That small transfer has a different explanation from LoRA’s small transfer.

| Training approach | What the optimizer changes | Why synchronization can be small |
|---|---|---|
| Full-parameter RL | The model’s parameters broadly | Absorption can leave most served weight bits unchanged |
| LoRA | Small adapters while the base model stays frozen | The trainable object is small by construction |

Full-parameter RL needs the precision argument to explain its sparse served-view patch. LoRA does not: its small adapter already limits the synchronization object.

### What the sparsity measurements count

[12:03](https://www.youtube.com/watch?v=maRzp4kImJ4&t=723s)

The measurement is deliberately about weight bits, not gradients or optimizer state. Cast the weights to BF16, compare consecutive versions bitwise, and count the entries that did not change. Jiang reports that roughly 99% of BF16 weights were bit-identical per step across the model families examined in the cited study. He also reports that differences remain sparse when rollout lags behind the trainer. These are observations from the evaluated runs, not a universal sparsity guarantee. In either case, changed indices plus replacement values reconstruct the selected target version exactly.

![Slide showing a three-step measurement procedure, green bars across model families, a staleness line chart, and a lossless-patch note.](https://ai.engineer/api/talk-essay-media/d984939eb61d511ad7ffb45ebd0c7081063b3767ef3eaa79864a664b072883a6)

PULSE measures unchanged weights by casting to BF16 and comparing bitwise.

**Sparse served-weight changes do not imply sparse gradients.** Jiang reports that about 99% of parameters had nonzero gradients in the cited study. The FP32 master-weight updates were also dense, but small. The apparent contrast disappears once the representations are separated: many parameters receive updates, while only a small fraction cross a boundary in the rollout engine’s lower-precision view. Nothing in the patch construction requires pruning those gradients.

### Lower serving precision introduces shared-scale constraints

[13:06](https://www.youtube.com/watch?v=maRzp4kImJ4&t=786s)

Rollout engines may serve in FP8 or NVFP4 even when training uses BF16; serving precision and training precision are separate choices. Quantization-aware training is another possibility, but it is not required for that distinction. For a floating-point format with fixed scales and `m` mantissa bits, the illustrated visibility scale becomes:

```math
\text{visibility scale} \approx \frac{\Theta}{2^{m+1}}.
```

Fewer mantissa bits mean coarser spacing. In this simplified picture, FP8 has a higher visibility threshold than BF16, and FP4 has a higher threshold still, so fewer small updates may alter the served representation.

Shared-scale formats require more care. With independent rounding, one value crossing a boundary can change one encoded value. INT4 instead quantizes weights relative to a shared group scale, and NVFP4 uses hierarchical scales with its own encoding rules. **The exact patch must preserve the relevant scale bits as well as the encoded weight values.** Counting unchanged element codes alone does not establish that the complete served checkpoint is equally sparse.

Jiang then reports an internal run serving **GLM-4.7-Air** in FP8:

| Point in the internal run | Reported changed weights | Stated condition |
|---|---:|---|
| First step | Approximately 0.15% | Learning rate described as high |
| Later steps | Approximately 0.5% per step | Adam behavior described as stabilizing |

These figures describe one internal run, not a guarantee across models or training schedules. Jiang also points to related RL synchronization research and cites Cursor, **Composer-2**, and MAI when discussing Adam’s use in post-training. With small, exact served-version updates as the working premise, the next problem is making remote rollout engines participate reliably.

### Publish immutable versions and make staleness explicit

[15:06](https://www.youtube.com/watch?v=maRzp4kImJ4&t=906s)

The deployment architecture leaves the trainer in its RDMA cluster and places rollout engines outside it, potentially across regions and providers. After an update, the trainer publishes an **immutable rollout weight version** to a shared bulletin board. The request path carries a version contract: a request identifies the desired policy version and which versions it will accept. The response reports the version actually used, alongside generated tokens, log probabilities, router replay information, and other rollout metadata. Version identity becomes part of the interaction rather than an assumption about colocated processes.

After each optimizer step, the trainer publishes the version; engines pull it and materialize it locally in a checkpoint layout they can serve. An engine can choose how to load and shard that version, but it cannot substitute different served weights. Jiang describes a Hugging Face-style `safetensors` layout for compatible engines such as vLLM. Compatibility still depends on the attention and mixture-of-experts backends, parallelism, serving dtype, and GPUs. A common file format does not remove those execution constraints.

### A sidecar gates requests on committed versions

[16:41](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1001s)

A **sidecar** makes an ordinary rollout engine aware of the version contract. It handles three cases:

- **An acceptable version is already committed:** Proxy the request to the engine.
- **The engine is behind but can catch up:** Apply the missing transactions so it can serve an acceptable version.
- **The required version cannot be reached:** Return `not ready` rather than serving an unacceptable version.

This gate lets additional compatible GPU capacity join the rollout fleet without assuming that every engine is current at every moment. Each engine must either satisfy the request’s version requirements or decline it.

### From smaller transfers to an elastic rollout fleet

[17:14](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1034s)

In-cluster synchronization is fast because RDMA provides the appropriate fabric. Moving full checkpoints across regions replaces that advantage with a large network transfer. Sparse patches change the amount of data that must cross the link. Jiang illustrates reducing a transfer from roughly 500 GB to 500 MB, with latency falling into seconds. This is a size-and-latency illustration, not a benchmark for a specified network or region pair. The actual payload also includes indices, replacement bits, scales where applicable, and metadata; fewer than 1% changed weights alone does not establish the illustrated byte count.

Modal’s **Stitch** is the concrete implementation Jiang describes for this protocol. On the trainer side, it publishes what defines a rollout weight version through the bulletin-board contract. On the rollout side, engines pull weights and synchronize across regions and providers. Jiang describes Stitch as agnostic to the trainer framework, serving engine, and transport, with an emphasis on asynchronous RL and agentic workloads.

Each rollout engine synchronizes its own weights, serves accepted versions, and returns rollout metadata. That gives the fleet a unit of independent growth: another compatible serving island can join without expanding the training cluster itself. Scattered inference capacity can become RL rollout capacity, while the trainer retains the tightly connected infrastructure its computation needs.

### What remains open beyond the Adam-based design

[18:44](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1124s)

The architecture leaves three distinct research questions unresolved:

- **Other optimizers:** Does served-weight sparsity persist under Muon? Jiang cites Moonshot and **DeepSeek-V4** when discussing Muon adoption in post-training, but the preceding absorption argument specifically concerns Adam. It does not automatically establish the same behavior for another optimizer.
- **Fully asynchronous RL at scale:** Access to global compute expands the possible rollout fleet. How far fully asynchronous RL can scale with that capacity remains an open question; distributing the workers does not itself answer it.
- **Training beyond RL:** Could the same synchronization pattern help pre-training, mid-training, or supervised fine-tuning? Those extensions are proposed directions, not results established by this design.

## Resources

- [PULSE: original sparse synchronization study](https://arxiv.org/html/2602.03839v1) — The February 2026 study explains measured served-weight sparsity and lossless replacement patches.
- [Delta weight synchronization in TRL](https://huggingface.co/blog/delta-weight-sync) — Implementation walkthrough using Hub storage and vLLM. Experimental branch instructions provide an actionable example, but were not executed or checked against current releases.
- [NVIDIA's explanation of NVFP4](https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/) — Explains the shared scales and storage overhead that qualify the talk's lower-precision rounding argument.

## Slides

### RL across the globe

Nan Jiang explores whether RL post-training can use scattered, autoscaled GPU capacity instead of relying on one cluster.

[0:12](https://www.youtube.com/watch?v=maRzp4kImJ4&t=12s) [Title slide introducing RL post-training across scattered, autoscaled GPU capacity.](https://assets.ai.engineer/sites/aiecode2025/sha256/2c3a50b95c892e2f8de382ad14b99ed8c860a373b597df88b0fb75bbe11dee68.webp)

Modal

AI ENGINEER WORLD'S FAIR 2026
RL across the globe
RL post-training across scattered, autoscaled GPU capacity.

Presented by Nan Jiang · Modal

1 / 31

Interface text:
claude.ai

### The RL loop in one cluster

The trainer sends new weights to the sampler, which interacts with the RL environment and returns trajectories and reward. RDMA makes weight sync fast, but ties rollout capacity to the trainer cluster.

[1:12](https://www.youtube.com/watch?v=maRzp4kImJ4&t=72s) [A trainer exchanges weights and trajectories with a sampler, which interacts with an RL environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/aad50d9099f15cb3a8bac11e2c39fcee2c1c8349c459e10e4eb2d857234d98d6.webp)

THE RL LOOP
By default, the whole loop lives in one cluster

one cluster · one region · RDMA

Trainer
weight update
full-param
LoRA

new weights
trajectories + reward

Sampler
rollout worker · policy

action / generation
reward / feedback

RL Env
tasks · rewards

RL across the globe
2 / 31

Interface text:
claude.ai

### One cluster and distributed compute

A tightly connected cluster suits the trainer, but usable compute is scattered across providers and regions, with different prices and availability. The default RL loop must fit into the harder-to-find cluster.

[2:04](https://www.youtube.com/watch?v=maRzp4kImJ4&t=124s) [A GPU cluster is contrasted with capacity distributed across providers, regions and GPU types.](https://assets.ai.engineer/sites/aiecode2025/sha256/52c456fa16f1890e9875980e14fa1e01b99ce746653fabb7a8c18d0228c8a0b4.webp)

THE SETUP
One cluster is powerful. Compute is everywhere else.

THE CATHEDRAL
one cluster

one cluster · one region · RDMA

Many GPUs, one fast interconnect - but you get exactly one.

THE SPRAWL
everywhere else

provider: A B C D
H100 H200 B200 GB300
same provider · 2 regions
cheaper                         pricier →

Same chip, different region - different price and availability.

RL across the globe
3 / 31

Interface text:
claude.ai

### The hard part is getting all four

RL needs enough GPUs in one region, fast fabric, and immediate availability at the same time. RDMA capacity is provisioned, so the trainer cluster cannot be assumed to grow midway through a run.

[2:41](https://www.youtube.com/watch?v=maRzp4kImJ4&t=161s) [RL requires enough GPUs in the same region, a fast fabric and immediate availability.](https://assets.ai.engineer/sites/aiecode2025/sha256/444541f5664f92517222026ffd57361b656c014a908b428623f745ede7c57e8b.webp)

THE CATCH
The hard part is getting all four

Enough GPUs
Same region
Fast fabric
Available now

RL wants all four at once.
Any one is easy. All four together are scarce.
RDMA capacity is provisioned, not elastic.

RL across the globe
4 / 31

Interface text:
claude.ai

### Rollout is the movable half

Backprop stays in the cluster, while rollout serving islands can run across regions and clouds. Each island receives a policy version and returns trajectories and metadata; local parallelism can still happen within an island.

[3:23](https://www.youtube.com/watch?v=maRzp4kImJ4&t=203s) [Rollout is parallel, can run across regions and clouds, and needs fresh weights from the trainer.](https://assets.ai.engineer/sites/aiecode2025/sha256/6f352b81e1b27ba58849a4748f2eb93e0e5167d4c67e965bb44d3b4caa20827a.webp)

THE KEY IDEA
Rollout is the movable half

Training is tightly coupled - all-reduce every step, one fast fabric. Rollout is the opposite.

01 Parallel by construction — independent trajectories - no cross-worker sync
02 Runs anywhere — no all-reduce, no trainer co-location - any region, any cloud
03 Only needs fresh weights — the single link back to the trainer

Backprop stays in the cluster. Rollout can leave.

RL across the globe
5 / 31

Interface text:
claude.ai

### Trainer stays, rollout fans out

The trainer stays in one RDMA cluster and sends weight updates to rollout workers across clouds and regions. Trajectories and rewards return to the trainer.

[4:27](https://www.youtube.com/watch?v=maRzp4kImJ4&t=267s) [A trainer in one cluster sends weight updates to rollout workers across providers and regions; trajectories and rewards return to the trainer.](https://assets.ai.engineer/sites/aiecode2025/sha256/b6002db04e8c7b9ff414f0fd465eaa25597e60819210bd2fbea358ad9e83ef05.webp)

DISAGGREGATE
Trainer stays. Rollout fans out.

rollout-weight updates

one cluster · RDMA
Trainer
gradient updates

rollout
aws · us-east

rollout
gcp · eu-central

rollout
azure · ap-se

rollout
nebius · eu

trajectories + reward → back to trainer · cheap

RL across the globe
6 / 31

Interface text:
claude.ai

### Weight sync is the remaining link

Shipping a full checkpoint for each fresh weight version can take minutes to hours, leaving rollout engines waiting. The transferred data must reconstruct the same served version bit-for-bit.

[5:14](https://www.youtube.com/watch?v=maRzp4kImJ4&t=314s) [A trainer sends a full checkpoint to rollout engines, which wait for weights.](https://assets.ai.engineer/sites/aiecode2025/sha256/50499e88a5e683dfb51b394c96864424930746c085bf401f7d4ca55359c23d6c.webp)

THE REMAINING LINK
Weight sync is the remaining link

Rollout needs fresh weight versions. Naively, that means shipping a full checkpoint.

one cluster · RDMA
Trainer
weight update

full checkpoint
minutes → hours, every step

rollout engine
aws · us-east
rollout engine
gcp · eu-central
rollout engine
azure · ap-se

idle - waiting for weights

… and whatever crosses it must reconstruct the same served version, bit-for-bit.

RL across the globe
7 / 31

Interface text:
claude.ai

### What if only less than 1% of rollout-visible weights change?

The proposal is to send only changed served weights if fewer than 1% change between versions, then reconstruct the same served version bitwise. The slide compares that delta with a 500 GB checkpoint.

[6:07](https://www.youtube.com/watch?v=maRzp4kImJ4&t=367s) [A small green delta is compared with a 500 GB full checkpoint; the slide proposes bitwise reconstruction.](https://assets.ai.engineer/sites/aiecode2025/sha256/22d0063352bf5cd40dd5c67599703ba2af5778e4250ef514e16d72e404eb4a93.webp)

THE BET

What if only <1% of rollout-visible weights change?

Ship the changed served view, not the checkpoint. Reconstruct the same version bitwise.

the delta · <1% rollout-visible
500 GB · full checkpoint

RL across the globe
8 / 31

Interface text:
claude.ai

### Why the update is sparse

The next section examines how small Adam steps and finite precision make updates sparse.

[7:08](https://www.youtube.com/watch?v=maRzp4kImJ4&t=428s) [Section slide introducing small Adam steps and finite precision.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c0f309aa5fa77d6feb21bd2b022d78635935eec09ab9985bde75bfaaf31562a.webp)

Why the update is sparse

Small Adam steps meet finite precision.

RL across the globe
9 / 31

Interface text:
claude.ai

### The BF16 view has a floor

At θ = 1.0, a +0.0009 nudge stays below the +0.0039 rounding boundary, so the BF16 view rounds back. Small FP32 updates can accumulate until they cross the boundary.

[7:17](https://www.youtube.com/watch?v=maRzp4kImJ4&t=437s) [Formulas and a number line illustrate BF16 spacing, a rounding boundary, and a small update that rounds back.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d9b273f3ffe403097ecccdf05c4e0ef07a818b835a02a995244940ca3dd1339.webp)

THE FLOOR
The BF16 view has a floor

The next forward pass reads the BF16 view of the weights - finite resolution. Around a magnitude |θ|, the nearest representable value is a fixed step away.

ULP · spacing between BF16 values
ULP(|θ|) ≈ |θ|/128

the floor · nearest rounding boundary
τ(|θ|) ≈ |θ|/256

Below the boundary, the BF16-visible value is unchanged this step - tiny FP32 updates can still accumulate and cross it later.

θ = 1.0 · inside one BF16 cell
boundary +0.0039
+0.0009 nudge → stays in the cell, rounds back
1.0000
1.0078 · next cell

RL across the globe
10 / 31

Interface text:
claude.ai

### Adam makes small steps

Adam divides gradients by running statistics, making the typical per-weight update roughly the size of the learning rate. A conservative bound puts the step at no more than B times that rate.

[8:20](https://www.youtube.com/watch?v=maRzp4kImJ4&t=500s) [Adam update equations show typical learning-rate-sized steps and a conservative bound.](https://assets.ai.engineer/sites/aiecode2025/sha256/35d3210112bb8c25fa226bb83046e6950db6b2dce6de0f23f9f5a59b8819b327.webp)

THE PUSH
Adam makes small steps
Bound · PULSE, arXiv:2602.03839

Δθᵢ = −η m̂ᵢ / (√v̂ᵢ + ε)
Adam / AdamW - the case PULSE explains
Δθᵢ per-parameter update · η learning rate · the fraction m̂ᵢ/(√v̂ᵢ + ε) is Adam's per-weight normalized direction, order 1. AdamW decay term omitted.

typical
|Δθ| ≈ η

conservative
|Δθ| ≤ B η
B = √((1 − β₁)/(1 − β₂))
β₂=0.95 → B=1.4
β₂=0.999 → B=10

Dense gradients. Learning-rate-sized updates.

RL across the globe
11 / 31

Interface text:
claude.ai

### A weight is visible only if the push clears the floor

At θ = 1, an RL Adam step of approximately 0.000003 falls below the ≈0.0039 BF16 boundary, so the served value rounds back to 1.0. The master weight is not necessarily frozen forever.

[9:09](https://www.youtube.com/watch?v=maRzp4kImJ4&t=549s) [The slide compares an Adam step with a BF16 rounding boundary and gives an example that rounds back to 1.0.](https://assets.ai.engineer/sites/aiecode2025/sha256/cc9ebc0a387b68dec000df25cdb957f2380a25007bbda7200d132c2f7c0e7784.webp)

THE KEY IDEA
A weight is visible only if the push clears the floor

push = η (Adam step)
floor = τ(|θ|) ≈ |θ|/256
visible ⟺ |Δθ| ≥ τ

η learning rate · τ nearest BF16 rounding step · Δθ the push

θ = 1.0
bf16 boundary ≈ 0.0039
RL Adam step ≈ 0.000003
→ rounds back to 1.0 · absorbed

RL across the globe
12 / 31

Interface text:
claude.ai

### Adam absorption, visualized

The BF16 boundary rises with weight magnitude while the Adam push stays roughly level. Updates below the boundary remain invisible in served weights; about 99% of weights are absorbed this step.

[9:54](https://www.youtube.com/watch?v=maRzp4kImJ4&t=594s) [Graph compares the BF16 visibility boundary with the Adam update and marks most weights as absorbed.](https://assets.ai.engineer/sites/aiecode2025/sha256/0573eb45037661f2a5866534c5f14736830c725a72ddf25bd8e20f6f7abaa466.webp)

THE CENTERPIECE
Adam absorption, visualized

x = weight magnitude |θ| · y = update magnitude
both log scale

BF16 boundary ≈ |θ| / 256
push = Adam step ≈ η (up to Bη)
can move
Absorbed this step
~99% of weights

Y-axis: update magnitude →
X-axis: weight magnitude |θ| →

RL across the globe
13 / 31

Interface text:
claude.ai

### The patch is the changed rollout weight

The patch compares consecutive rollout views and ships changed positions, replacement bits, and metadata. New values or XOR bits can reconstruct the served version bitwise.

[10:39](https://www.youtube.com/watch?v=maRzp4kImJ4&t=639s) [A formula and grid show changed rollout weights encoded as indices and replacement bits.](https://assets.ai.engineer/sites/aiecode2025/sha256/245ac6eaa198abc422d13c806d72630715bcaf16402d75d13677e25ef0822dc3.webp)

THE OBJECT WE SHIP
The patch is the changed rollout weight

Ωₜ = { i : qₜ[i] ≠ qₜ₋₁[i] },    qₜ = quantize_D(θₜ)

qₜ = the weights cast to the rollout dtype D · Ωₜ = the positions that changed · we diff the rollout view, not the FP32 optimizer state

rollout vₜ₋₁
rollout vₜ

patch
indices: 2, 5, 9, 13
+ new bits for those
+ metadata

ENCODING CHOICE  store new values · or store XOR bits qₜ ⊕ qₜ₋₁
both reconstruct the served version bitwise

RL across the globe
14 / 31

Interface text:
claude.ai

### Two update shapes, one protocol

Full-parameter training ships a sparse rollout-view patch, small by absorption. LoRA ships adapter weights or a delta while the base stays frozen; the adapter is small by construction.

[11:29](https://www.youtube.com/watch?v=maRzp4kImJ4&t=689s) [Full-parameter patches are small by absorption; LoRA updates are small by construction.](https://assets.ai.engineer/sites/aiecode2025/sha256/91b2f8a7accda86bb81b06a0c918a1adccb9b9accd6aa628b3521182314133bd.webp)

THE UPDATE OBJECT
Two update shapes, one protocol

Full-parameter
sparse rollout-view patch - indices + new values
small by absorption
the whole push-vs-floor argument

LoRA
adapter weights / adapter delta - base frozen
small by construction
no precision argument needed

Same protocol object. Different reason it is small.

RL across the globe
15 / 31

Interface text:
claude.ai

### PULSE measured the same view we ship

Around 99% of BF16 weights remain bit-identical per step across the models shown; over 98.5% remain unchanged even at S=32. A lossless patch sends changed indices and replacement values.

[12:01](https://www.youtube.com/watch?v=maRzp4kImJ4&t=721s) [Measurement steps, model-family bars and a staleness chart show high proportions of unchanged BF16 weights.](https://assets.ai.engineer/sites/aiecode2025/sha256/b14a8e22207352c04b7d03855c1df3c648f9becd42ee5edbb04e8d9be393be1e.webp)

MEASURED · PULSE
PULSE measured the same view we ship
The metric we ship, measured directly - cast to the rollout BF16 dtype, compare bitwise, count what didn't change.

HOW IT'S MEASURED
1 cast to BF16
2 compare bitwise
3 count unchanged

HOLDS ACROSS MODEL FAMILIES
Qwen 0.5B  99.3
Qwen 1.5B  99.4
Qwen 7B  99.1
Llama 3B  99.1
Gemma 4B  98.9
bit-identical per step

SURVIVES STALENESS
Y-axis ticks: 99.4, 98.5
sync interval →: S=1, S=32
>98.5% even at S=32 · >97.5% across k-step gaps

Lossless patch: changed indices + replacement values.
PULSE · arXiv:2602.03839
RL across the globe
16 / 31

Interface text:
claude.ai
Slide thumbnails: 1–19

### Gradients are dense. The update is sparse.

About 99% of gradients are nonzero, and the FP32 master update is small but dense. Only about 1% of rollout weights visibly change, producing a similarly sparse network delta.

[12:42](https://www.youtube.com/watch?v=maRzp4kImJ4&t=762s) [Bars compare dense gradients and FP32 updates with approximately one percent visible rollout changes and network delta.](https://assets.ai.engineer/sites/aiecode2025/sha256/fbbb9fae0025428f9d6a97dd01fbfbc438b22a6baa820010b7542d1606a1bbf0.webp)

THE MECHANISM, PRECISELY
Gradients are dense. The update is sparse.

Source · PULSE, arXiv:2602.03839

gradient: ~99% nonzero
FP32 master update: small, but dense
rollout-weight change: ~1% visible
network delta: ~1% - shipped

RL across the globe
17 / 31

Interface text:
claude.ai

### Project first. Diff second.

For fixed-scale floats, coarser precision raises the visibility floor: FP4 has the highest threshold, FP8 falls in between, and BF16 has the lowest. Fewer weight changes survive projection into a coarser served format.

[13:04](https://www.youtube.com/watch?v=maRzp4kImJ4&t=784s) [Graph compares FP4, FP8 and BF16 visibility boundaries against an update band.](https://assets.ai.engineer/sites/aiecode2025/sha256/c8124bb69dfdf77d3043f37d8415c3a9caa2ac4d7fd3a89b2478b89a6444585b.webp)

ROLLOUT PRECISION
Project first. Diff second.
Project into the served format, then diff - coarser mantissa raises the floor, so fewer weights move.

Y-axis: update magnitude →
X-axis: weight magnitude |θ| →
Bη
fp4 · |θ|/4
fp8 · |θ|/16
bf16 · |θ|/256
coarser → more frozen

Fixed-scale intuition. Real payload depends on the target format.
RL across the globe
18 / 31

Interface text: claude.ai; slide thumbnails 1–19

### Shared scales couple the diff

Int4 weights share a quantization step, while NVFP4 uses hierarchical scales: unchanged codes can decode differently when a scale changes. Both changed codes and changed scales contribute to wire size.

[13:40](https://www.youtube.com/watch?v=maRzp4kImJ4&t=820s) [Comparison of plain floats, int4 groups, and NVFP4 shared scales.](https://assets.ai.engineer/sites/aiecode2025/sha256/1e42cd5747e12c216b00b381382f1e970be5b7b2fbda0845e44a98aa0d5a4366.webp)

BLOCK-SCALED FORMATS
Shared scales couple the diff
Plain floats have per-value floors. Group-scaled formats share scales.

Plain float · Per-element floor
BF16 / FP8 - each value has its own τ
before
after
one value crosses its floor → only that code changes

int4 group · One shared step
qᵢ = round(wᵢ / s_g)
before  s_g 0.10
after   s_g 0.10
codes re-round against the shared step; step moves only if group amax does

NVFP4 · Hierarchical scales
x = x_e2m1 · s_block · s_global
before  s_blk 0.10
after   s_blk 0.11
same codes · new scale → decodes differently

Wire size = changed codes + changed scales. Shared scales couple the diff.

RL across the globe
19 / 31

Interface text:
claude.ai

### Observed in one FP8 run: density settles low

In one GLM 4.7 Air FP8 run, changed-weight density falls from about 0.15% early to about 0.05% per step. The slide cautions that this is not a universal monotonic law.

[14:14](https://www.youtube.com/watch?v=maRzp4kImJ4&t=854s) [A chart shows changed-weight density falling from about 0.15% early to about 0.05% in one FP8 run.](https://assets.ai.engineer/sites/aiecode2025/sha256/75001519cb9483510b3862715ef11fea8926e66fda4ecb2c1048a68f4a9fa3b8.webp)

OVER TRAINING
Observed in one FP8 run: density settles low

INTERNAL MEASUREMENT
GLM 4.7 Air · FP8 · changed-weight density per step

~0.15% early
~0.05% settled
Y-axis ticks: 0.15%, 0.10%, 0.05%
X-axis ticks: 1, 20, 40, 60, 80, 100
X-axis label: step

The early, high-LR phase is the densest - in this run the delta keeps thinning; not a universal monotonic law.

RL across the globe
20 / 31

Interface text: claude.ai

### The pattern is showing up elsewhere

PULSE and SparseRL-Sync report ~99% BF16 sparsity and 99%+ element sparsity, respectively, with large reductions in synchronization or communication. Other projects illustrate related rollout practices.

[14:41](https://www.youtube.com/watch?v=maRzp4kImJ4&t=881s) [A table summarizes related research and projects reporting sparse synchronization or related rollout practices.](https://assets.ai.engineer/sites/aiecode2025/sha256/b00e4b576351203ba2744be94bf2c06965dcfa2d00b89dd187813c0d000d15e4.webp)

NOT ONE PAPER
The pattern is showing up elsewhere

SOURCE | WHAT THEY’RE SEEING
PULSE | The absorption mechanism + a lossless sparse patch - ~99% BF16 sparsity, >100× less sync.
SparseRL-Sync | Separate team, same result - 99%+ element sparsity, ~100× less comms, at MoE / TB scale.
Cursor Composer 2 · MAI-Thinking-1 | Per-step S3 delta chain across US / EU rollout · public AdamW RL recipe (β₁=β₂=0.95) - the optimizer the argument relies on.
slime · miles | BF16 training → FP8 rollout · miles adds NVFP4 rollout - the block-scaled regime we handle.

Independent teams are landing on the same idea.

Sources · PULSE (2602.03839) · SparseRL-Sync · Cursor · slime · miles · MAI-Thinking-1
RL across the globe
21 / 31

Interface text:
claude.ai

### The disaggregated rollout protocol

The trainer publishes immutable weight versions to a bulletin board, which rollout engines across regions and providers pull. Requests specify accepted versions; responses return the served version, tokens and metadata.

[15:14](https://www.youtube.com/watch?v=maRzp4kImJ4&t=914s) [A trainer publishes versions to a bulletin board, which rollout engines pull and apply; responses return to the trainer.](https://assets.ai.engineer/sites/aiecode2025/sha256/79502156dc24643a4e31ea07de5ebedd1186cf1d54e8c25bad4ef586da15ae60.webp)

THE PROTOCOL
The disaggregated rollout protocol

one cluster · RDMA
Trainer
publishes versions

bulletin board
latest → version 42
version 40/ anchor
version 41/transition
version 42/transition

versions out - pull + apply

rollout engine
aws · us-east · sidecar

rollout engine
gcp · eu-central · sidecar

rollout engine
azure · ap-se · sidecar

rollout engine
nebius · eu · sidecar

response → trainer · cheap

REQUEST →
accepted_versions = 42

← RESPONSE · RETURNED TO THE TRAINER, SAME AS IN-CLUSTER
served_version = 42
tokens
logprobs
routing metadata. router-replay
tool traces

RL across the globe
23 / 31

Interface text:
claude.ai

### Write a version. Materialize it locally.

After an optimizer step, the trainer publishes an immutable version. Engines pull and materialize it locally, choosing a compatible backend and parallelism while preserving the served version.

[15:58](https://www.youtube.com/watch?v=maRzp4kImJ4&t=958s) [A trainer writes an immutable version to a bulletin board for compatible engines to materialize locally.](https://assets.ai.engineer/sites/aiecode2025/sha256/75acae688f16adddac774b649a485737bd0ad4c40fe81d763fbf9022ee324558.webp)

PUBLISH → APPLY
Write a version. Materialize it locally.

Trainer
optimizer step
export changed weights
diff vs previous version

write version 42 →

bulletin board
latest → version 42
version 40/ anchor
version 41/ transition
version 42/ transition · manifest

pull + apply →

HF / safetensors checkpoint
version 42 · tensor names · shapes · dtype

Any compatible backend
SGLang · vLLM

Any parallelism
TP · PP · EP

Declared serving contract
dtype · quant layout

Any compatible GPU
H100 · B200 · GB300

Immutable versions · transition = new-value or XOR bits, no float accumulation. The artifact defines the version - engines choose how to materialize it, not a different served version.

RL across the globe
24 / 31

Interface text:
claude.ai

### The sidecar makes engines version-aware

The sidecar checks an engine’s committed version before serving a request. It proxies an acceptable version, applies missing transitions if it can catch up, or returns NotReady so the request can retry elsewhere.

[16:38](https://www.youtube.com/watch?v=maRzp4kImJ4&t=998s) [A sidecar proxies requests at an acceptable committed version, catches up when possible, or returns NotReady.](https://assets.ai.engineer/sites/aiecode2025/sha256/02665e87b864b51078e4f092c927ad94080d021a7da1a55d6bd8a6112d547c85.webp)

THE SIDECAR
The sidecar makes engines version-aware

bulletin board
latest                         version 42
version 40                     anchor
version 41                     transition
version 42                     transition

request asks for version 42 → sidecar checks committed version
already acceptable → proxy, serve now
behind, catch-up possible → apply transitions, commit version 42, then proxy
not ready → return NotReady, retry elsewhere

Engines only serve committed versions - no request runs on half-applied weights.
accepted_versions = 42 → served_version = 42 - staleness is request-level policy

RL across the globe
25 / 31

Interface text:
claude.ai

### The link is thin enough to leave the cluster

Cross-region transition broadcasts cut the transfer from 500 GB to 500 MB and take ~seconds, versus minutes–hours for a full checkpoint. Serving continues on an accepted version while the next catches up.

[17:12](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1032s) [Latency comparison showing cross-region transition broadcasts taking seconds and reducing transfer size from 500 GB to 500 MB.](https://assets.ai.engineer/sites/aiecode2025/sha256/267af5551a85782823142b01cf2793222aa996a93c4f4aa6ad842970cad9bc25.webp)

LATENCY
The link is thin enough to leave the cluster

in-cluster · RDMA: ~seconds
full checkpoint · cross-region: minutes–hours
transition broadcast · cross-region: ~seconds

500 GB → 500 MB
~1000× less on the wire · Kimi K2.7 NVFP4 ·
<1% conservative, ~0.1% this run

Serve version 42 → local weights are that version, bit-for-bit - same policy for the same committed version. Serving continues on an accepted version while the next catches up.

RL across the globe
26 / 31

Interface text:
claude.ai

### stitch: versioned rollout serving

Modal’s stitch connects trainer-published weight versions to rollout engines that sync weights and serve the requested version. The contract includes transitions, anchors and a prepare-to-commit step.

[17:39](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1059s) [Diagram of a version contract between a trainer and rollout engines, including transitions, anchors and commit steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/d634fa2c5b076785ad9a3215112223f9f2c2c28d6706a69ca801b96865c24166.webp)

MODAL · CONCRETE IMPLEMENTATION
stitch: versioned rollout serving
One concrete implementation of the protocol.

Trainer
publishes versions
ships base checkpoint · transitions · anchors · LoRA

THE CONTRACT
weight_version → served_version
transition · anchor
prepare → commit
WeightVersionNotReady

Rollout engines
serve the exact version
backend owns loading · sharding · GPU · vLLM / SGLang

Framework-agnostic · workload-opinionated: async-first, agentic-first, elastic rollout.

RL across the globe
27 / 31

Interface text:
claude.ai

## Full transcript

[0:00](https://www.youtube.com/watch?v=maRzp4kImJ4&t=0s) [upbeat music] All right. Cool. Hi, everyone. Uh, hope you all have a good time at the conference.

[0:17](https://www.youtube.com/watch?v=maRzp4kImJ4&t=17s) Uh, I'm Nan from Modal. Uh, at Modal, we spend a lot of time thinking about GPU capacity, like where it exists, how do we make it elastic, and what kind of workload can we actually use it.

[0:28](https://www.youtube.com/watch?v=maRzp4kImJ4&t=28s) Today, I want to talk about one place where everything became, like, gets really interesting, the RL post-training. A lot of our discussion right now is about algorithm and the environments.

[0:37](https://www.youtube.com/watch?v=maRzp4kImJ4&t=37s) Sandbox, PPO, GRPO, like tool call, maybe low-precision training, maybe deterministic kernels. Um, but when you run those experiments at a scale, the problem became more physical. Where are the GPUs?

[0:50](https://www.youtube.com/watch?v=maRzp4kImJ4&t=50s) Are they in the same region? Uh, do they have fast fabric? Uh, can we get them right now? Maybe the default shape of RL compute is too restrictive. Maybe some of the work we usually force in one cluster, uh, like can actually run on scattered auto-scaled capacity.

[1:06](https://www.youtube.com/watch?v=maRzp4kImJ4&t=66s) So can we do RL across the globe? So this is my talk about, mainly about. So to make this more concrete, let's start with the RL loop itself. So in the standard, uh, RL post-training loop, we can see there's one trainer, and the trainer updates the policy.

[1:21](https://www.youtube.com/watch?v=maRzp4kImJ4&t=81s) Rollout worker, or maybe people call sampler, use those policy to generate tra- tra- trajectories. The environment will be returning the reward and observations. Those trajectory will go back to trainer for the next updates.

[1:34](https://www.youtube.com/watch?v=maRzp4kImJ4&t=94s) The important arrow here is the weight sync. In the default setup, trainer and the rollout will be living in the same cluster, and the, the weight sync will be super fast with RDMA.

[1:44](https://www.youtube.com/watch?v=maRzp4kImJ4&t=104s) But that also coupled the rollout fleet to the, to the trainer cluster. If the rollout needs to more ca- to have more capacity, maybe no, no more nodes during runs, um, you are normally limited by the fixed size during your trainer, maybe your trainer cluster.

[2:00](https://www.youtube.com/watch?v=maRzp4kImJ4&t=120s) So the ne- next question is, what kind of compute shape did we actually force everything into?

[2:06](https://www.youtube.com/watch?v=maRzp4kImJ4&t=126s) On the left side is the cathedral. Uh, one region, one fast interconnect. Uh, many GPUs wired to- together. This is the right shape for the trainer. On the right is the sprawl.

[2:17](https://www.youtube.com/watch?v=maRzp4kImJ4&t=137s) This is where a lot of like usable, useful, usable compute actually lives. Different providers, different regions, different price, and different availabilities. There's still a lot of capacity out there, but it's not one perfect RDMA island.

[2:32](https://www.youtube.com/watch?v=maRzp4kImJ4&t=152s) This is the mismatch. Available compute is distributed, but the default RL loop asks one tightly coupled cluster, and that cluster is exactly the hard part to get.

[2:43](https://www.youtube.com/watch?v=maRzp4kImJ4&t=163s) RL wants all four of these at the same time, enough GPU, same region, fast fabric, and available now. Any of these, like, is manageable, but all four of them, they are pretty hard to get at the same time.

[2:56](https://www.youtube.com/watch?v=maRzp4kImJ4&t=176s) RDMA capacity is not as elastic in the way that inference capacity is elastic. You cannot assume you can grow the trainer cluster, uh, halfway through a run just because rollout wants more nodes for a trainer.

[3:08](https://www.youtube.com/watch?v=maRzp4kImJ4&t=188s) So if the whole RL loop has no live, like, has to live inside the, the one cluster, rollout inherits the hardest part, capacity constraint in here. So that leads to the key question: Does the whole RL loop actually need that, like this kind of shape?

[3:25](https://www.youtube.com/watch?v=maRzp4kImJ4&t=205s) So let's dive into this. Training is one tightly coupled job. Every step has collectives, all-reduce, and the model parallel communication. That part, it actually wants one fast fabric RDMA-connected.

[3:37](https://www.youtube.com/watch?v=maRzp4kImJ4&t=217s) Rollout is a fleet of serving jobs. It generates trajectories, call environments, or maybe tools, and they will be sending back data back to the trainer. So across, uh, rollout jobs, there's no global all-reduce.

[3:49](https://www.youtube.com/watch?v=maRzp4kImJ4&t=229s) So the thing I want to move here is not backpropagation. Backpropagation should stay in the cluster. The run- the rollout fleet is the one that can leave. More precisely, the movable unit is the rollout serving island, a coherent endpoint, or maybe a local group of endpoint, or that they can be serving one policy version.

[4:09](https://www.youtube.com/watch?v=maRzp4kImJ4&t=249s) Inside the island, a large model may still be having, like local parallelism. They can do PD disaggregation. They can have, like local serving constraints. So across islands, the dependence is much lighter right now.

[4:20](https://www.youtube.com/watch?v=maRzp4kImJ4&t=260s) Policy version in, and the trajectory and the metadata out. So once we define the unit that way, the architecture is much, much more natural.

[4:29](https://www.youtube.com/watch?v=maRzp4kImJ4&t=269s) Once we define the movable unit, the architecture is very straightforward in this case. We just have trainer staying in RDMA cluster, and that's where the backprop and the collective go.

[4:38](https://www.youtube.com/watch?v=maRzp4kImJ4&t=278s) The rollout side will be fanning out across the sprawl. Each rollout island will be, can be single engine or maybe a local serving group, depending on the model and the serving topology there.

[4:48](https://www.youtube.com/watch?v=maRzp4kImJ4&t=288s) Across island, there is no global all-reduce. That's the most important thing there. The global interface is very simple. The trainer send policy weight version out, and the rollout sending trajectory and the metadata back.

[5:01](https://www.youtube.com/watch?v=maRzp4kImJ4&t=301s) At this point, the architecture depends on one remaining link, the weight update. So if we want to s- if we want to send the full parameter, like full checkpoint from disk, or maybe through the network, then everything is, like meaningless and, like it will be breaking immediately.

[5:16](https://www.youtube.com/watch?v=maRzp4kImJ4&t=316s) So, uh, after disaggregation, the things that we will be discussing about, like the size of like go th- the size of the full parameters go through the disk.

[5:24](https://www.youtube.com/watch?v=maRzp4kImJ4&t=324s) Naively, that means shipping all f- full checkpoints every time rollout needs a new weight version. At this scale, the checkpoint is very huge. So a Kimi-scale NVFP4 checkpoint, you have like five hundred gigabytes.

[5:36](https://www.youtube.com/watch?v=maRzp4kImJ4&t=336s) Normally, it take a minute or maybe i- normally it take mult- multiple minutes to hours to just do the weight sync. So moving that over commodity links might not be the smartest choice 'cause like, uh, when you're doing async, maybe even, even fully async training, you still want the weight updates latency to be as low as possible,

[5:53](https://www.youtube.com/watch?v=maRzp4kImJ4&t=353s) like within seconds. So the problem here is not whether rollout can leave the cluster. The problem is like the full checkpoint is the wrong unit of synchronization. So the next question is, can we keep the exact same serve version there but send a much smaller object?

[6:09](https://www.youtube.com/watch?v=maRzp4kImJ4&t=369s) So this is the bet. What if less than 1% of the rollout visible weights got changed from one version to another one? By rollout visible weights, I mean the weights in the served rollout ver- uh, checkpoint, uh, not the F- FP32 optimizer state, not the Adam, like, moments.

[6:25](https://www.youtube.com/watch?v=maRzp4kImJ4&t=385s) Like, the weights are the rollout's engine, would ac- which we'll actually use to serve, maybe let's say, the FP8 or maybe NVFP4 format. If that's true, we do not need to ship the entire full parameter over the network.

[6:37](https://www.youtube.com/watch?v=maRzp4kImJ4&t=397s) We just need to ship the change to serv- served view, the precision delta, got, got difference. The important part here is like still bitwise re- reconstruction. The rollout engine gets the same served, ser- uh, served version you would have gotten to as synced to the full checkpoint there.

[6:53](https://www.youtube.com/watch?v=maRzp4kImJ4&t=413s) So if this works, then the link shrinks from hundreds of gigabytes to maybe hundreds of megabytes, and this is something small enough we can just send it across, like, the network.

[7:03](https://www.youtube.com/watch?v=maRzp4kImJ4&t=423s) So right now we need to justify the less than 1%, like, claim. Why would this rollout visible, like, weights barely change?

[7:10](https://www.youtube.com/watch?v=maRzp4kImJ4&t=430s) Uh, now we get, we will get into this mechanism. So it's kind of small Adam, like, step meets finite precision. We need, we need a two,

[7:19](https://www.youtube.com/watch?v=maRzp4kImJ4&t=439s) uh, prerequisite. The first one... Two gra- two ingredients. The ingredient one is the precision. The optimizer may keep very high precision ma- uh, master weights, but the next forward pass read the BF16 visible view.

[7:31](https://www.youtube.com/watch?v=maRzp4kImJ4&t=451s) That view has finite resolution. Around a value of magnitude Theta, uh, BF16 spacing is roughly, uh, Theta over 128. Uh, that spacing is, people call it ULP, basically the unit in the last place.

[7:45](https://www.youtube.com/watch?v=maRzp4kImJ4&t=465s) Basically, it's the distance between the adjacent representable BF16 value. But an update only needs to cross the nearest rounding boundary to be v- viable, for it to be visible.

[7:53](https://www.youtube.com/watch?v=maRzp4kImJ4&t=473s) That boundary is about half of the ULP, so roughly, like, this Theta over 256. For weight around one, the BF16 ULP is around, uh, .0078, and the nearest rounding boundary is about .0039.

[8:08](https://www.youtube.com/watch?v=maRzp4kImJ4&t=488s) If the optimizer nudged the master weight by something, like, smaller than that, uh, the BF16's visible value will round back. So you will not see any change from the weight perspective, rollout weight perspective.

[8:20](https://www.youtube.com/watch?v=maRzp4kImJ4&t=500s) So that's the floor. The second primitive there is, we call push. So for Adam, or maybe AdamW here, we just, like, we ignore the weight decay term. The per, per...

[8:30](https://www.youtube.com/watch?v=maRzp4kImJ4&t=510s) Sorry, the, the per- parameter update is the learning rate times the normalized direction. The raw gradient can be dense and can have very different magnitudes across parameters. An Adam, like, divides by running, uh, gradients statistics.

[8:45](https://www.youtube.com/watch?v=maRzp4kImJ4&t=525s) So the per-weight push is usually on the order of learning rates. The paper Paused, I said there, uh, they have, they prove a bound. The Adam step is at most b times the learning rates.

[8:56](https://www.youtube.com/watch?v=maRzp4kImJ4&t=536s) So you do not need to actually remember the exact, like, bound there. Like, the important notes here is the Adam makes the push small, and they're very controlled. So at RL post-training learning rates, the push is very, very tiny.

[9:09](https://www.youtube.com/watch?v=maRzp4kImJ4&t=549s) So that's the push. Combining these two pr- primitives, now we have to, we have a whole, like, better picture. A served va- value changed only if the push cleared the floor.

[9:20](https://www.youtube.com/watch?v=maRzp4kImJ4&t=560s) The push is the Adam step, roughly the learning rates. The floor is the nearest BF16 rounding boundary, uh, roughly Theta over 256. Take a Theta equal one, the BF16 boundary is about .0039.

[9:32](https://www.youtube.com/watch?v=maRzp4kImJ4&t=572s) A typical, uh, Adam step i- here is around 3e-6. So the update is more than a thousand smaller than the boundary, so the BF16 visible value will not change.

[9:42](https://www.youtube.com/watch?v=maRzp4kImJ4&t=582s) This is not saying the master weights are in, is frozen forever. It is saying the value that rollout engine would serve does not change on this part. So the whole mechanism is pushing, uh, is, is push versus floor.

[9:56](https://www.youtube.com/watch?v=maRzp4kImJ4&t=596s) Let's visualize this to have better understanding. The x-axis is the weight magnitude, and the y-axis is the u- uh, update magnitude. First, we look at the red line. The red line now is the BF16 visible boundary.

[10:07](https://www.youtube.com/watch?v=maRzp4kImJ4&t=607s) It's like Theta over 256. And now we look at the green bound. This is the Adam push. It sits roughly around the learning rate and, uh, with a conservative upper bound.

[10:16](https://www.youtube.com/watch?v=maRzp4kImJ4&t=616s) So now we ask, like, where the most point fits. For most of the weights, the red floor is above the green push. Those updates exist in the master weights, but they are not visible in the served BF16 view this step.

[10:30](https://www.youtube.com/watch?v=maRzp4kImJ4&t=630s) Small weights on the left can move. Large weight on the right, they will just stay the same. They will be absorbed it. This is the Adam absorption. This is why the served update become very sparse.

[10:41](https://www.youtube.com/watch?v=maRzp4kImJ4&t=641s) In this case, the object will be shipped as just a diff, uh, it's just a diff, not the entire FP32 optimizer state. We first look at the rollout view.

[10:51](https://www.youtube.com/watch?v=maRzp4kImJ4&t=651s) The weight cast, like, all projected to the dtype that the rollout engine will be serving. Then we will be comparing the version T minus one and the version T, uh, in this view.

[11:01](https://www.youtube.com/watch?v=maRzp4kImJ4&t=661s) The patch is, the patch is the changed positions, plus, like, re- uh, replacement bits and also, like, some metadata. There are multiple lossless encoding. People can do selective overwrites.

[11:12](https://www.youtube.com/watch?v=maRzp4kImJ4&t=672s) They can... People can also do XOR. The important part is, like, they are bit equivalent, bit-level equivalent, so it's not a floating point addition, so there's no additive, uh, delta drift.

[11:24](https://www.youtube.com/watch?v=maRzp4kImJ4&t=684s) If a rollout engine apply the patch correctly, it reconstruct the same served version bitwise.

[11:30](https://www.youtube.com/watch?v=maRzp4kImJ4&t=690s) So everything so far we explained is about the full parameter, which is hub, hub RL. So in full parameter reinforcement learning, the optimizer update the whole model. But the rollout view patch is sparse, as the thing we just explained.

[11:43](https://www.youtube.com/watch?v=maRzp4kImJ4&t=703s) LoRA is a small for a different purpose, uh, for, for a different reason. The base model is frozen, and the adapter is small enough by construction, so do not need to put, so do not need to have the push versus the floor argument here.

[11:56](https://www.youtube.com/watch?v=maRzp4kImJ4&t=716s) So full parameter delta is small by absorption, and the LoRA updates are small by construction.

[12:03](https://www.youtube.com/watch?v=maRzp4kImJ4&t=723s) Let's dive into deeper about the paper itself. So the paper, they mentioned more stats I will be showing here. The measurement is not gradient sparsity. It's not optimizer state sparsity.

[12:12](https://www.youtube.com/watch?v=maRzp4kImJ4&t=732s) They cast weights to BF16, compare consecutive versions delta, and they compare the, the version bitwise, and they count what did not change over time. Across model family, the result around 99% of them is bit identical per step.

[12:27](https://www.youtube.com/watch?v=maRzp4kImJ4&t=747s) It's also s- it also survives staleness. Even when the rollout lags, the changes that remain are very small. The important part is not only the number, it is the patch is lossless.

[12:38](https://www.youtube.com/watch?v=maRzp4kImJ4&t=758s) Change index plus replacement value reconstructs the exact same version.

[12:44](https://www.youtube.com/watch?v=maRzp4kImJ4&t=764s) So a common misconception there is, like, the work- it works because all gradient are sparse. They are not. The paper reports the gradients are dense. About 99% of parameter gives non-zero gradients.

[12:56](https://www.youtube.com/watch?v=maRzp4kImJ4&t=776s) The FP32 master update is also dense, it's just small. The main thing is, like, the rollout weight change is just 1% from the perspective of rollout engine.

[13:06](https://www.youtube.com/watch?v=maRzp4kImJ4&t=786s) So far, we mostly talk about BF16, but the rollout often serving even lower precision, such as, like, MF- FP4- FP8 and NVFP4, and, uh, we can see many, many model providers doing this in their rollout.

[13:18](https://www.youtube.com/watch?v=maRzp4kImJ4&t=798s) Uh, this is not a training precision. The training is ju- just, like, in the normal BF16, although people can do QAT on that. So f- for fixed scale float format, the visibility floor is roughly theta over two to the mantissa plus one.

[13:32](https://www.youtube.com/watch?v=maRzp4kImJ4&t=812s) So as you can see, the FP4 will be higher, and FP8 also will be between BF16 and FP4, which means in even lower precision, there will be less weight change.

[13:42](https://www.youtube.com/watch?v=maRzp4kImJ4&t=822s) So plain floats are easy to reason about, so each element has its own rounding ceiling. One value cross the floor, one value they just flip and it changed. Group scale such Int4, they are qui- they are a bit different.

[13:54](https://www.youtube.com/watch?v=maRzp4kImJ4&t=834s) This is the regime where many low precision serving system are moving towards right now. For Int4, each weight is quantized against a shared group scale, and we can apply the same rationale, and also we can observe similar thing for NVFP4.

[14:07](https://www.youtube.com/watch?v=maRzp4kImJ4&t=847s) It's hierarchical scales, and, uh, we can see there are different encoding and scaling mechanism for NVFP4.

[14:16](https://www.youtube.com/watch?v=maRzp4kImJ4&t=856s) So this is the... from one internal run. So here's the model we serve, like, GLM-4.7-Air in FP8, and we can see in the beginning, there are only, like, 0.15% of weights got changed in the first step where the learning rate is high.

[14:29](https://www.youtube.com/watch?v=maRzp4kImJ4&t=869s) And after we have more training step, like, when the Adam is going r- r- relatively stable, and you can see the entire curve goes stable. We got only 1...

[14:39](https://www.youtube.com/watch?v=maRzp4kImJ4&t=879s) 0.5%, uh, weight change during each step. So we can see this pattern showing more generally. We have a different research. Uh, we, we saw different research across RL sync have a similar conclusion, and we saw a different model provider such as Cursor, Composer-2, MAI, they all using Adam in their post-training.

[14:58](https://www.youtube.com/watch?v=maRzp4kImJ4&t=898s) At, at this point, assume we can produce exact rollout weights version cheaply. The next question is, how do we do this in practice?

[15:06](https://www.youtube.com/watch?v=maRzp4kImJ4&t=906s) So from sparse delta to, to RL across globe, how do we do this with elastic rollout engines and also explicit staleness?

[15:16](https://www.youtube.com/watch?v=maRzp4kImJ4&t=916s) This is the whole shape. The trainer stay in the RDMA cluster. After it got updated, it published immutable rollout weights version to a shared bulletin board. Rollout engine live outside of the training cluster, which means they don't need to be RDMA connected with the trainer.

[15:31](https://www.youtube.com/watch?v=maRzp4kImJ4&t=931s) They can be in different region or different providers. This is also request lane. You can see a request does not just say, "Give me the completion." You will also say which version you will be sending request to and then which version you will be accepting.

[15:44](https://www.youtube.com/watch?v=maRzp4kImJ4&t=944s) And the response will come back with the version and also, like, exact same information as if, as if they are in the same cluster as the trainer. They will be returning tokens, log prob, like, router replay information, and many more me- metadata.

[16:00](https://www.youtube.com/watch?v=maRzp4kImJ4&t=960s) The trainer writes immutable version to the board aft- after optimizer state. Uh, af- after optimizer step. Engine pull version and materialize it locally in the checkpoint layout, so they can just, like, serve directly.

[16:13](https://www.youtube.com/watch?v=maRzp4kImJ4&t=973s) The auto-defined version, the engine choose how to, how to load and shard it. It does not change... It does, it does not choose the different served version.

[16:22](https://www.youtube.com/watch?v=maRzp4kImJ4&t=982s) Since it will be displaying in a HuggingFa- HF, it will be safetensors format, which is accepted widely by many rollout engines such as AstroNet and vLLM. So we can support any compatible backend.

[16:33](https://www.youtube.com/watch?v=maRzp4kImJ4&t=993s) Attention backend, MOE backend, different parallelism, compatible serving dtype, and any compatible GPUs there.

[16:41](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1001s) We can talk more about the sidecar itself. The sidecar is basically what makes a normal rollout engine version-aware. If the version is already at acceptable committed version, the sidecar just proxy the re- request.

[16:52](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1012s) If the engine is behind, but it can, they can catch up, the sidecar just apply the missing transaction. If they cannot get there, the, the sidecar just sim- simply return not ready.

[17:04](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1024s) So this will be supporting elastic rollouts, and id- any idle GPU can just be used with design, with this design to support disaggregated rollout.

[17:14](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1034s) So this is more like a system latency analysis. In-cluster weight sync is fast because they have RDMA. Uh, a full checkpoint across regions through network is pretty slow.

[17:25](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1045s) And but if we use the delta c- If, if we use exactly what we described previously, we can decrease the number of, like, transfer, t- transfer size from, like, five hundred gigabytes to fi- five hundred megabytes, so you will be, like, extremely fast, in seconds.

[17:42](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1062s) So everything above was very general protocol. Stitch is one of very concrete implementation from Modal that we imple- im- implementing everything above. So on the, on the trainer side, Stitch publish, uh, what defines a rollout weights version, and on the contract side, you'll be pulling out and it require everything on the bulletin board.

[18:00](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1080s) And on the rollout side, you'll be pulling the latest weights, and it start doing a weight sync across different region and different providers. So Stitch itself is a very framework-agnostic about trainer and engine and also transport.

[18:13](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1093s) It's very a- async first and also agent first, agentic first.

[18:17](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1097s) By doing this, we can have rollouts, uh, engines like auto-scale globally. Each one self sync its weights, serve accepted version, and return rollout metadata. That means scattered inference capab- uh, capacity became one, uh, elastic rollout fleet.

[18:33](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1113s) Instead of being limited by the training cluster, rollout can be the global pool. So inference capacity can now become RL capacity.

[18:44](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1124s) Last section, we have some, uh, ongoing explorations.

[18:48](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1128s) So w- we can see a lot of model providers such as Moonshot and also DeepSeek-V4, they have, like... they are adopting Muon in their post-training. Um, does the sparsity still hold for Muon?

[18:59](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1139s) 'Cause a lot of thing we discussed previously only for Adam. Second question is async RL at scale. Right now, we can use the compute across the globe. Then how, like, how scalable is the fully async RL?

[19:12](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1152s) This is a very open-ended question there. And, uh, third question is, like, does it generalize past RL, 'cause, like, we have pre-training, mid-training, and SFT. Like, do we have-- can we apply same paradigm there?

[19:25](https://www.youtube.com/watch?v=maRzp4kImJ4&t=1165s) Last but not least, we are working on some very hard problem, and come work with us. You can check the link there, modal.jobs. Thank you.
