AI Engineer Code 2025
Efficient Reinforcement Learning
Loading slides…
Read the talk
Efficient Reinforcement Learning: Keeping GPUs Busy Without Letting Policies Drift
Enterprise RL needs predictable training time as well as speed. A model of sampling, training and policy staleness helps find an efficient GPU allocation before an expensive run.
From a talk by Rhythm Garg and Linden Li
Before you start: Familiarity with autoregressive token generation, GPU batching and gradient-based model training will help; reinforcement-learning staleness is explained from first principles.
Training an expert for a company’s own work
How do you turn a general model into an in-house expert that performs a company’s work—and improves as people use it? For Rhythm Garg and Linden Li, co-founders of Applied Compute, the goal is automation with quantitative business returns. Along with their co-founder Yash, they previously worked on reinforcement learning at OpenAI. Their enterprise approach specializes a system for a particular use case, then deploys it with a data flywheel so that usage supports further improvement.
Reinforcement learning supplies the adaptation mechanism. Garg describes it as bringing previously out-of-distribution enterprise tasks into the model’s distribution: the same broad approach that improves public benchmark performance can target a company’s private benchmarks. The target is an expert in how that company operates, rather than simply a stronger general-purpose assistant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From graded attempts to a delivery constraint
Start with the simplified math-training loop. Select four problems for a training step, and ask an open model such as GPT-OSS or Llama to attempt each problem 100 times. Each attempt contains a reasoning trajectory followed by a final answer. Grade the answers, then update the weights to reinforce behavior associated with correct attempts and discourage behavior associated with incorrect ones. Repeating this process teaches the model to solve the task more reliably. Math makes the loop easy to explain; enterprise workflows supply the actual tasks Applied Compute wants to improve.
That loop must fit a different operating model from a frontier lab’s large training run. Garg contrasts runs lasting weeks with specialized customer delivery on the order of days. Three requirements follow:
- Fast delivery: train and hand over a useful model quickly.
- Low cost: keep the unit economics sustainable as the business scales.
- Predictable duration: keep variance in training-time estimates low enough to make dependable customer commitments.
Being reliably fast matters as much as being fast on average. The systems problem is to make RL efficient enough that use-case-specific training, together with an agent-building platform, can become a repeatable delivery process.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The last sample sets the pace
In synchronous RL, sampling and training happen in lockstep. For a simplified batch of eight samples, all eight must finish before training can begin. If seven finish quickly and the third request straggles, the completed requests do not free the system to start the next training step. Sampling duration is dictated by the slowest response.
Garg reports that, when sampling 32 responses for each of 40 arithmetic problems with Qwen-30B, about 99% of samples completed in 40 seconds, while the final 1% took another 80 seconds. The talk does not specify the exact checkpoint, hardware or decoding configuration. The throughput curve explains the waste: GPUs initially process a large batch, but the active batch shrinks as responses finish. By the end, substantial capacity is waiting on a handful of long responses—the slide’s “GPUs Slackin’” annotation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Overlap the work, then account for stale tokens
Removing the lockstep barrier allows training to proceed while sampling continues. In PipelineRL, attributed here to Piché and colleagues, separate GPU pools handle sampling and training. Sampling workers run continuous inference at high batch sizes. Completed responses enter a queue, and training workers pull batches from it. After each training step, the new weights propagate back to the samplers.
The distinctive mechanism is the in-flight weight update: a sampler can receive new weights halfway through generating a response. Continuous generation does not mean weight transfer is free; PipelineRL briefly pauses inference to receive weights, without waiting for whole responses to finish. A single completed response can therefore contain tokens produced by several policy versions.
Follow the response in Garg’s example. It starts under policy t, then continues under t + 1 and t + 2 after two training steps trigger in-flight updates. When the response is consumed in the training batch from t + 3 to t + 4, its token groups have different ages:
| Token group | Generating policy | Staleness at training |
|---|---|---|
| Initial tokens | t | 3 policy steps |
| Middle tokens | t + 1 | 2 policy steps |
| Final two tokens | t + 2 | 1 policy step |
Staleness belongs to the tokens’ generating policies, not just to the completed response’s arrival time. Updating the sampler cannot retroactively refresh tokens it has already produced.
Now impose a staleness limit of two. In the illustrated schedule, the update following the t + 1 to t + 2 training batch must wait until the outstanding response finishes. The training workers cannot propagate that update and move on to the next batch without allowing the response to reach staleness three. They become idle. A limit of one makes the wait longer still. Relaxing the limit generally reduces idle time, but changes the learning problem.
Data generated under policy t is being used to train policy t + k. Garg introduces an importance ratio to correct for this mismatch, describing it as making the policy-gradient estimate unbiased. As staleness increases, the ratio’s variance can increase, making learning unstable or even divergent. The talk does not develop the algorithmic innovations needed to tolerate that mismatch. Instead, it fixes an acceptable staleness threshold and a compute budget, then asks: what configuration achieves the highest RL throughput within those limits?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four inputs to the systems model
Li approaches the question through a first-principles model of the end-to-end system. Its first input is the GPU budget. Synchronous execution gives the entire fleet to sampling, then to training. Asynchronous execution introduces an allocation decision: how much of the fixed fleet should serve each phase? The second input is the training batch size. A batch contains a subset of dataset problems, with multiple parallel attempts per problem. Harder problems may warrant more attempts so the model explores a wider range of solution strategies.
The third input is a model of sampling latency. GPU memory contains model weights, activations and the KV cache. Each autoregressive forward pass produces the next token for active requests and writes additional state to that cache. Measuring forward-pass latency as a function of the active sampling batch captures a central determinant of inference throughput. A large batch puts more work into each pass, but it must fit within the available KV-cache memory.
The fitted latency curve follows roofline-style behavior:
| Sampling regime | Main bottleneck | Effect of adding requests |
|---|---|---|
| Small batch | Loading model parameters | Little additional latency |
| Large batch | Processor computation | Longer forward pass |
At small batch sizes, the processor can perform the arithmetic faster than memory can supply the parameters. Extra requests reuse that parameter-loading work. As the batch grows, computation becomes the limiting resource. Li’s fit uses a sigmoid to smooth the transition between these regimes. This is why doubling the batch does not necessarily double latency—and why batching is such a powerful utilization tool.
The fourth input is training throughput, measured in tokens per second per training GPU. Fit it using representative workloads and the chosen training batch size. Unlike the sampling latency model, this measurement includes the forward pass, backward pass and optimizer work. Together, the four inputs describe the available compute, the amount of work and the service rate of each side of the pipeline.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Simulating a batch that drains
A synchronous baseline remains useful: it avoids stale training data and assigns the whole GPU fleet to whichever phase is active. To estimate its duration, the model also needs the generation batch size and the response-length distribution. In Li’s animation, squares represent requests and darken as generation progresses. Completed requests enter a queue, while a Grafana-like time series shows the active sampling batch falling toward zero. Only then does optimization run and the cycle restart.
The simulation follows the decoding loop directly:
- Advance through token positions up to the longest response.
- At each position, determine how many requests remain active.
- Apply the fitted forward-pass latency for that active batch.
- Remove requests that finish, using the response-length distribution.
- Sum the sampled tokens and divide by aggregate training throughput to estimate training duration.
For an illustrative Python implementation, engine_lengths holds response lengths assigned to each inference engine. decode_seconds supplies the fitted latency for an active batch; engines sample concurrently, and training starts after the slowest engine finishes.
python
from collections.abc import Callable, Sequence
def synchronous_step_seconds(
engine_lengths: Sequence[Sequence[int]],
decode_seconds: Callable[[int], float],
training_gpus: int,
train_tokens_per_second_per_gpu: float,
) -> float:
if training_gpus <= 0 or train_tokens_per_second_per_gpu <= 0:
raise ValueError("Training capacity must be positive")
engine_seconds = []
total_tokens = 0
for lengths in engine_lengths:
if any(length < 0 for length in lengths):
raise ValueError("Response lengths must be nonnegative")
total_tokens += sum(lengths)
elapsed = 0.0
for position in range(max(lengths, default=0)):
active = sum(length > position for length in lengths)
elapsed += decode_seconds(active)
engine_seconds.append(elapsed)
sampling_seconds = max(engine_seconds, default=0.0)
training_seconds = total_tokens / (
training_gpus * train_tokens_per_second_per_gpu
)
return sampling_seconds + training_seconds
The response-length CDF expresses the same shrinking-batch calculation statistically: at each token position, it tells the simulator what fraction of responses have already finished.
The completed simulation shows the response-length CDF alongside predicted step latency as GPU count increases. More GPUs reduce the predicted latency, as expected, but the simulator retains the inefficient tail: near the end of sampling, few requests remain active even though the fleet is still allocated to generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Asynchronous execution still needs balance
Splitting the fleet does not automatically solve utilization. If training consumes samples faster than sampling produces them, the queue empties and training waits. In the first allocation animation, the red training square turns gray when idle. Removing the synchronous barrier has merely moved the waiting to a different place.
The opposite allocation produces another failure mode. Li doubles the sampling GPU count and halves the training GPU count. Samples now arrive faster than training can consume them, so they accumulate in the queue. The indices on the yellow sample squares show increasing staleness; their growing transparency represents diminishing learning value. High sampling throughput alone is therefore an inadequate optimization target. The system needs enough training capacity to use what sampling produces before it becomes too stale.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match rates and bound policy age
With continuous sampling, every completed response is immediately replaced by a new request. The active batch can remain approximately constant instead of draining to zero. That steady state simplifies modeling, although it still depends on response lengths: longer responses occupy more KV cache and can force a smaller feasible batch. Li incorporates the response-length distribution into the model to account for this pressure.
The first constraint is approximate equality between sample production and training consumption. Let Gₜ and Gₛ be the training and sampling GPU counts, uₜ the per-GPU training throughput, b the active sampling batch per GPU, and ℓ(b) the forward-pass latency in seconds. In token-per-second units, the balance is:
Each decoding pass produces one token per active request, so batch size is divided by pass latency to obtain a rate. This notation makes the units explicit for the talk’s simplified per-GPU model.
The second constraint is a bound on maximum staleness. Compare the longest response’s generation time with the duration of a training step: how many policy advances can occur while that response is being generated? Let Lₘₐₓ be the maximum response length, L̄ the mean response length, and B the number of responses in a training batch. Expressing both quantities as durations gives:
The training-token workload needs its throughput divisor to become seconds. This is the talk’s longest-request approximation, rather than a complete accounting of queue residence and update boundaries; the end-to-end simulation checks the resulting schedule against the tolerated staleness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Search layouts before spending the compute
With those constraints, layout selection becomes a bounded search:
- Sweep candidate training GPU counts. The fixed fleet size determines the remaining sampling GPU count.
- Select a steady-state generation batch that respects KV-cache capacity and supports the desired sampling throughput.
- Reject configurations that exceed the permitted staleness.
- Simulate the remaining layouts end to end using the response-length distribution.
The useful comparison is between feasible complete systems, not between isolated sampling or training peaks.
Li reports roughly 60% simulated speed-up relative to their synchronous baseline, assuming optimal GPU allocation between training and sampling. The talk does not specify the speed-up definition or the hardware, response distribution and staleness setting behind that estimate, so it should not be read as a measured 60% reduction in production elapsed time.
The simulator’s practical value extends beyond that comparison. It lets the team explore configurations before paying for an expensive training run. What allocation should they use if responses become much longer? That question matters because models often learn to produce longer reasoning during RL, changing the workload as training progresses. What empirical throughput should performance engineering target? A system model identifies the rates that would improve the whole pipeline, rather than merely making one component faster. Li says these simulations inform both systems and research decisions: choose the layout for the workload the model is developing, while keeping policy age within what learning can tolerate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains in-flight weight updates, the inference/trainer architecture, and early reasoning-model experiments.
Piché and colleagues' paper on overlapping generation and training while keeping rollout policies fresh.
Open-source asynchronous RL implementation with installation instructions and examples for defining agent-training tasks.
Updates since the talk
Later Applied Compute analysis of response-length tails, queue capacity, rollout concurrency and GPU allocation, with formulas and validation.
Linden Li and Agastya Goel explain how selecting informative rollouts can reduce training work and free compute for sampling.
Full transcript
Loading transcript…