AI Engineer Code 2025

Efficient Reinforcement Learning

Rhythm Garg· Co-founder, Applied ComputeLinden Li· Co-founder, Applied Compute20:19

Loading slides…

Read the talk

Efficient Reinforcement Learning: Keeping GPUs Busy Without Letting Policies Drift

Enterprise RL needs predictable training time as well as speed. A model of sampling, training and policy staleness helps find an efficient GPU allocation before an expensive run.

From a talk by Rhythm Garg and Linden Li

Before you start: Familiarity with autoregressive token generation, GPU batching and gradient-based model training will help; reinforcement-learning staleness is explained from first principles.

Training an expert for a company’s own work

How do you turn a general model into an in-house expert that performs a company’s work—and improves as people use it? For Rhythm Garg and Linden Li, co-founders of Applied Compute, the goal is automation with quantitative business returns. Along with their co-founder Yash, they previously worked on reinforcement learning at OpenAI. Their enterprise approach specializes a system for a particular use case, then deploys it with a data flywheel so that usage supports further improvement.

Reinforcement learning supplies the adaptation mechanism. Garg describes it as bringing previously out-of-distribution enterprise tasks into the model’s distribution: the same broad approach that improves public benchmark performance can target a company’s private benchmarks. The target is an expert in how that company operates, rather than simply a stronger general-purpose assistant.

Three bar charts compare model performance on competition math, competition code, and PhD-level science questions beneath a statement about reinforcement learning enabling customization.Open full source frame
Benchmark comparisons motivate reinforcement learning for customization.
0:220:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:22 · section reference included

From graded attempts to a delivery constraint

Start with the simplified math-training loop. Select four problems for a training step, and ask an open model such as GPT-OSS or Llama to attempt each problem 100 times. Each attempt contains a reasoning trajectory followed by a final answer. Grade the answers, then update the weights to reinforce behavior associated with correct attempts and discourage behavior associated with incorrect ones. Repeating this process teaches the model to solve the task more reliably. Math makes the loop easy to explain; enterprise workflows supply the actual tasks Applied Compute wants to improve.

That loop must fit a different operating model from a frontier lab’s large training run. Garg contrasts runs lasting weeks with specialized customer delivery on the order of days. Three requirements follow:

  • Fast delivery: train and hand over a useful model quickly.
  • Low cost: keep the unit economics sustainable as the business scales.
  • Predictable duration: keep variance in training-time estimates low enough to make dependable customer commitments.

Being reliably fast matters as much as being fast on average. The systems problem is to make RL efficient enough that use-case-specific training, together with an agent-building platform, can become a repeatable delivery process.

1:441:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:44 · section reference included

The last sample sets the pace

In synchronous RL, sampling and training happen in lockstep. For a simplified batch of eight samples, all eight must finish before training can begin. If seven finish quickly and the third request straggles, the completed requests do not free the system to start the next training step. Sampling duration is dictated by the slowest response.

Garg reports that, when sampling 32 responses for each of 40 arithmetic problems with Qwen-30B, about 99% of samples completed in 40 seconds, while the final 1% took another 80 seconds. The talk does not specify the exact checkpoint, hardware or decoding configuration. The throughput curve explains the waste: GPUs initially process a large batch, but the active batch shrinks as responses finish. By the end, substantial capacity is waiting on a handful of long responses—the slide’s “GPUs Slackin’” annotation.

Slide with a request-latency histogram, a percentile table, and a declining generation-throughput chart annotated “GPUs Slackin’.”Open full source frame
Synchronous steps wait for the slowest sample as generation throughput declines.
4:064:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:06 · section reference included

Overlap the work, then account for stale tokens

Removing the lockstep barrier allows training to proceed while sampling continues. In PipelineRL, attributed here to Piché and colleagues, separate GPU pools handle sampling and training. Sampling workers run continuous inference at high batch sizes. Completed responses enter a queue, and training workers pull batches from it. After each training step, the new weights propagate back to the samplers.

The distinctive mechanism is the in-flight weight update: a sampler can receive new weights halfway through generating a response. Continuous generation does not mean weight transfer is free; PipelineRL briefly pauses inference to receive weights, without waiting for whole responses to finish. A single completed response can therefore contain tokens produced by several policy versions.

Follow the response in Garg’s example. It starts under policy t, then continues under t + 1 and t + 2 after two training steps trigger in-flight updates. When the response is consumed in the training batch from t + 3 to t + 4, its token groups have different ages:

Token groupGenerating policyStaleness at training
Initial tokenst3 policy steps
Middle tokenst + 12 policy steps
Final two tokenst + 21 policy step

Staleness belongs to the tokens’ generating policies, not just to the completed response’s arrival time. Updating the sampler cannot retroactively refresh tokens it has already produced.

Now impose a staleness limit of two. In the illustrated schedule, the update following the t + 1 to t + 2 training batch must wait until the outstanding response finishes. The training workers cannot propagate that update and move on to the next batch without allowing the response to reach staleness three. They become idle. A limit of one makes the wait longer still. Relaxing the limit generally reduces idle time, but changes the learning problem.

Data generated under policy t is being used to train policy t + k. Garg introduces an importance ratio to correct for this mismatch, describing it as making the policy-gradient estimate unbiased. As staleness increases, the ratio’s variance can increase, making learning unstable or even divergent. The talk does not develop the algorithmic innovations needed to tolerate that mismatch. Instead, it fixes an acceptable staleness threshold and a compute budget, then asks: what configuration achieves the highest RL throughput within those limits?

5:275:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:27 · section reference included

Four inputs to the systems model

Li approaches the question through a first-principles model of the end-to-end system. Its first input is the GPU budget. Synchronous execution gives the entire fleet to sampling, then to training. Asynchronous execution introduces an allocation decision: how much of the fixed fleet should serve each phase? The second input is the training batch size. A batch contains a subset of dataset problems, with multiple parallel attempts per problem. Harder problems may warrant more attempts so the model explores a wider range of solution strategies.

The third input is a model of sampling latency. GPU memory contains model weights, activations and the KV cache. Each autoregressive forward pass produces the next token for active requests and writes additional state to that cache. Measuring forward-pass latency as a function of the active sampling batch captures a central determinant of inference throughput. A large batch puts more work into each pass, but it must fit within the available KV-cache memory.

The fitted latency curve follows roofline-style behavior:

Sampling regimeMain bottleneckEffect of adding requests
Small batchLoading model parametersLittle additional latency
Large batchProcessor computationLonger forward pass

At small batch sizes, the processor can perform the arithmetic faster than memory can supply the parameters. Extra requests reuse that parameter-loading work. As the batch grows, computation becomes the limiting resource. Li’s fit uses a sigmoid to smooth the transition between these regimes. This is why doubling the batch does not necessarily double latency—and why batching is such a powerful utilization tool.

The fourth input is training throughput, measured in tokens per second per training GPU. Fit it using representative workloads and the chosen training batch size. Unlike the sampling latency model, this measurement includes the forward pass, backward pass and optimizer work. Together, the four inputs describe the available compute, the amount of work and the service rate of each side of the pipeline.

8:579:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:57 · section reference included

Simulating a batch that drains

A synchronous baseline remains useful: it avoids stale training data and assigns the whole GPU fleet to whichever phase is active. To estimate its duration, the model also needs the generation batch size and the response-length distribution. In Li’s animation, squares represent requests and darken as generation progresses. Completed requests enter a queue, while a Grafana-like time series shows the active sampling batch falling toward zero. Only then does optimization run and the cycle restart.

The simulation follows the decoding loop directly:

  1. Advance through token positions up to the longest response.
  2. At each position, determine how many requests remain active.
  3. Apply the fitted forward-pass latency for that active batch.
  4. Remove requests that finish, using the response-length distribution.
  5. Sum the sampled tokens and divide by aggregate training throughput to estimate training duration.

For an illustrative Python implementation, engine_lengths holds response lengths assigned to each inference engine. decode_seconds supplies the fitted latency for an active batch; engines sample concurrently, and training starts after the slowest engine finishes.

python

from collections.abc import Callable, Sequence


def synchronous_step_seconds(
    engine_lengths: Sequence[Sequence[int]],
    decode_seconds: Callable[[int], float],
    training_gpus: int,
    train_tokens_per_second_per_gpu: float,
) -> float:
    if training_gpus <= 0 or train_tokens_per_second_per_gpu <= 0:
        raise ValueError("Training capacity must be positive")

    engine_seconds = []
    total_tokens = 0
    for lengths in engine_lengths:
        if any(length < 0 for length in lengths):
            raise ValueError("Response lengths must be nonnegative")
        total_tokens += sum(lengths)
        elapsed = 0.0
        for position in range(max(lengths, default=0)):
            active = sum(length > position for length in lengths)
            elapsed += decode_seconds(active)
        engine_seconds.append(elapsed)

    sampling_seconds = max(engine_seconds, default=0.0)
    training_seconds = total_tokens / (
        training_gpus * train_tokens_per_second_per_gpu
    )
    return sampling_seconds + training_seconds

The response-length CDF expresses the same shrinking-batch calculation statistically: at each token position, it tells the simulator what fraction of responses have already finished.

The completed simulation shows the response-length CDF alongside predicted step latency as GPU count increases. More GPUs reduce the predicted latency, as expected, but the simulator retains the inefficient tail: near the end of sampling, few requests remain active even though the fleet is still allocated to generation.

Two plots show an increasing curve labeled F(L) against L and a decreasing latency curve against N, beside the presenter.Open full source frame
Simulation curves show response-length distribution and latency versus N.
12:5313:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:53 · section reference included

Asynchronous execution still needs balance

Splitting the fleet does not automatically solve utilization. If training consumes samples faster than sampling produces them, the queue empties and training waits. In the first allocation animation, the red training square turns gray when idle. Removing the synchronous barrier has merely moved the waiting to a different place.

The opposite allocation produces another failure mode. Li doubles the sampling GPU count and halves the training GPU count. Samples now arrive faster than training can consume them, so they accumulate in the queue. The indices on the yellow sample squares show increasing staleness; their growing transparency represents diminishing learning value. High sampling throughput alone is therefore an inadequate optimization target. The system needs enough training capacity to use what sampling produces before it becomes too stale.

15:0015:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:00 · section reference included

Match rates and bound policy age

With continuous sampling, every completed response is immediately replaced by a new request. The active batch can remain approximately constant instead of draining to zero. That steady state simplifies modeling, although it still depends on response lengths: longer responses occupy more KV cache and can force a smaller feasible batch. Li incorporates the response-length distribution into the model to account for this pressure.

The first constraint is approximate equality between sample production and training consumption. Let Gₜ and Gₛ be the training and sampling GPU counts, uₜ the per-GPU training throughput, b the active sampling batch per GPU, and ℓ(b) the forward-pass latency in seconds. In token-per-second units, the balance is:

Rtrain=Gtut,Rsample=Gsbℓ(b),Rtrain≈Rsample.\begin{aligned} R_{\mathrm{train}} &= G_t u_t, \\ R_{\mathrm{sample}} &= G_s\frac{b}{\ell(b)}, \\ R_{\mathrm{train}} &\approx R_{\mathrm{sample}}. \end{aligned}

Each decoding pass produces one token per active request, so batch size is divided by pass latency to obtain a rate. This notation makes the units explicit for the talk’s simplified per-GPU model.

The second constraint is a bound on maximum staleness. Compare the longest response’s generation time with the duration of a training step: how many policy advances can occur while that response is being generated? Let Lₘₐₓ be the maximum response length, L̄ the mean response length, and B the number of responses in a training batch. Expressing both quantities as durations gives:

Tlongest≈Lmax⁡ℓ(b),Ttrain≈BLˉGtut,Kestimate≈TlongestTtrain≤Kallowed.\begin{aligned} T_{\mathrm{longest}} &\approx L_{\max}\ell(b), \\ T_{\mathrm{train}} &\approx \frac{B\bar{L}}{G_tu_t}, \\ K_{\mathrm{estimate}} &\approx \frac{T_{\mathrm{longest}}}{T_{\mathrm{train}}} \leq K_{\mathrm{allowed}}. \end{aligned}

The training-token workload needs its throughput divisor to become seconds. This is the talk’s longest-request approximation, rather than a complete accounting of queue residence and update boundaries; the end-to-end simulation checks the resulting schedule against the tolerated staleness.

16:2816:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:28 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…