AI Engineer World's Fair 2026

Bringing Continual Learning into Enterprises

Samuel Denton· Platform Research Lead, Applied Compute19:03

Loading slides…

Read the talk

Bringing Continual Learning into Enterprises

Production traces can teach an agent better behavior before an enterprise has a replayable environment. The next step is to make hints adapt to what the current model actually does.

From a talk by Samuel Denton

Before you start: Familiarity with language-model distillation, agent rollouts, and supervised fine-tuning will help; the article explains how trace provenance and hint provenance affect learning.

What can an agent learn from yesterday’s traces?

You have a batch of production traces from an agent that works, but makes mistakes. How do you turn those traces into a better model? That is the starting point for Samuel Denton, who leads platform research at Applied Compute: learn through hindsight from the requests an enterprise has already served.

The first choice is how tightly learning connects to production. A one-time batch provides a fixed record to learn from. A daily batch creates a recurring cycle: deploy a model, collect its traces, and use them to improve the next version. At the fully online end, inference and training become a unified engine:

  1. Serve a production request and generate a rollout.
  2. Derive a learning signal from that trace.
  3. Update the model.
  4. Serve the next request with the updated model.

These are points along a spectrum, rather than entirely separate training methods.

Offline/Online Distillation slide with three labeled points along an axis from Offline to Online.Open full source frame
The distillation spectrum spans One-Time Batch, Daily Batch, and Unified Engine.

Many enterprises are still at the first point: they can supply historical traces and identify shortcomings, but do not yet have a live learning loop. Others are beginning to update models while serving production traffic. Supporting continual learning therefore means providing a useful entry point at both ends, rather than making a fully integrated serving-and-training system a prerequisite.

0:251:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:25 · section reference included

Where the teacher’s extra information comes from

The timing of trace collection is only one axis. The other is hint provenance: what information makes the teacher better informed than the student, or the policy currently doing the work? A hint supplies privileged information that the teacher can use to guide learning.

An offline hint comes from static information: a task rubric, a behavioral prior, or a recurring failure observed in production. For example, a customer-support agent may issue refunds too readily. That known problem can motivate a hint applied across many traces. The distinguishing feature is that the hint does not depend on the current model’s rollout.

An online hint is constructed from what the current policy actually did. It may incorporate the same behavioral priors, but its corrective information depends on the particular rollout. Crossing hint provenance with trace provenance produces four useful combinations. The boundaries simplify two continuous spectra; intermediate cases still matter.

CombinationTrace used for learningBasis of the hint
Offline trace, offline hintRecorded production traceShared behavioral prior
Online trace, offline hintCurrent-policy rolloutShared behavioral prior
Offline trace, online hintHistorical prefix plus one current-policy stepThat newly generated step
Online trace, online hintFull current-policy rolloutWhat happened in that rollout

In the first combination, the refund-related hint is applied to historical traces, and distillation targets the known behavior. In the second, the agent generates a fresh rollout, receives the shared hint, updates its policy, and continues serving traffic.

The third combination separates the history from the next decision. A trace recorded days earlier supplies an off-policy prefix: it was not generated by the policy now being trained. The current model then generates one step from that prefix without interacting with the environment. A hint is constructed from this new step. The fourth combination extends that dependence to the full rollout: let the current model finish, inspect what it did, construct a tailored hint, and distill against that online trace.

The Training Taxonomy slide shows a filled two-by-two grid, with Hint labeling the rows and Rollout labeling the columns.Open full source frame
The training taxonomy crosses offline and online hints with offline and online production traces.
3:403:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:40 · section reference included

Learning without replay or a golden answer

Applied Compute concentrates on the first and fourth combinations. Offline traces plus offline hints do not require a replayable production environment. The team can inspect what happened and target a specific change: fewer unnecessary refunds, a formatting convention, or a different amount of reasoning. The enterprise can start with the data it already has.

Online traces plus online hints provide a more adaptive loop. They can come from a replayable environment or actual production traffic. Because each hint responds to the current rollout, the learning process can address different behaviors as they appear, then feed the resulting updates back into the model. Denton describes this as the more scalable route to improving overall evaluations.

Denton frames the progression as “improve for free today” and raise ceilings tomorrow. Here, the immediate opportunity is reuse of existing production traces, not a demonstration of zero training cost. As an online policy becomes available, its new traces can support repeated updates.

A central constraint is that enterprise tasks often lack a golden answer or a comprehensive rubric for every request. The learning method needs to extract useful behavioral guidance without assuming that a perfect target response is waiting in the dataset. Bringing serving and training infrastructure closer together makes continual updates possible; continuous ceiling-raising remains Denton’s outlook for that system.

7:447:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:44 · section reference included

Teaching a coding agent to submit before its deadline

The first experiment uses offline traces and offline hints to change a specific behavior on SWE-bench. Denton identifies the model as Qwen3.5-Thinking. In this setup, it could take up to eighty turns to submit an answer; the target was to make it call a submission tool before turn forty. The experiment asks whether the model can learn to wrap up its work earlier instead of following its usual longer reasoning path.

Three metrics separate successful submission from successful coding:

MetricWhat counts as success
Task complete rateAgent calls the submission tool
Test pass rateEnvironment passes all task tests, whether or not the agent submits
Combined SWE-bench pass rateBoth submission and test success

The combined pass rate is the experiment’s intersection of the first two behaviors, not an interchangeable name for the underlying test pass rate. The aim is to increase combined success without degrading the code’s ability to pass tests.

The hint makes the approaching deadline concrete. It tells the model that roughly three turns remain before its forty-turn limit, points out its tendency to keep exploring rather than finish, and directs it to finalize the fix, verify it, and call the submission tool before time runs out. This supplies a behavioral correction without supplying the solution to the software issue.

Denton reports that the submission-tool call rate rose from about 22% to 60%, while the test pass rate stayed approximately constant, with a slight increase. The measured gain is therefore in the targeted submission behavior, rather than a claim that test success itself rose from 22% to 60%.

The surprising part is how that behavior emerged. The recorded traces essentially never called the desired submission tool. Distillation remained conditioned on those traces, and the teacher did not explicitly replace the tool-call tokens. Instead, it nudged the student’s reasoning toward finishing the task. The student became more likely to reach the submission decision even though the recorded trajectory did not contain the reasoning path that would have led to that call.

10:4010:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:40 · section reference included

One current-policy step changes what the teacher can teach

A small extension loosens the constraint imposed by the recorded trajectory: generate one step with the current model from the historical trace. As the student learns to wrap up its reasoning, that new step gives the teacher an opportunity to encourage the actual submission-tool token. The learning process now sees a decision made by the policy being trained, rather than only decisions preserved in old data.

Denton reports a larger improvement in the combined SWE-bench pass rate with this one-step extension than with fully offline learning, without giving a numerical value. This experiment still uses an offline hint. It adds a current-policy step but does not adopt the earlier taxonomy’s third combination in full, where the hint itself is constructed dynamically from that step. Trace freshness and hint provenance remain separate choices.

13:3913:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:39 · section reference included

Put the hint at the decision that needs to change

Hint content is only part of the method. Hint placement determines where the teacher’s extra information can affect a decision. Applied Compute uses a judge to choose the point in a rollout where a hint should be injected, rather than placing every hint at the beginning. Distillation then focuses on the next step, or a few subsequent steps, rather than the entire rollout.

This keeps the update close to the moment when the teacher has something relevant to teach. Denton’s plotted KL learning signal decreases as the model moves farther from the hint. The practical implication is local training: identify the decision that needs correction, provide the hint there, and concentrate learning on the nearby steps where its effect is strongest.

16:1316:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:13 · section reference included

Learn the relevant tokens, not every teacher preference

Relevance-masked self-distillation narrows learning further, from selected steps to selected tokens. In Denton’s description, an LLM judge chooses which teacher tokens the student should learn from. A teacher may prefer particular connector words, for example, even though those preferences have nothing to do with the behavior being taught. Copying those differences adds unrelated changes to the update.

Selecting relevant tokens aims to preserve the corrective signal while excluding incidental teacher preferences. Denton reports better acquisition of a strongly out-of-distribution behavior and better avoidance of catastrophic degradation with this approach. The two locality techniques address different scopes: per-step hinting chooses the moment to teach, while relevance masking chooses which token-level differences matter within the learning signal.

RMSD slide with explanatory bullets, two line plots below, and text comparison screenshots on the right.Open full source frame
Relevance Masked Self-Distillation uses LLM judging to choose tokens for updates.

That leaves an enterprise with a practical progression. Historical traces and static hints provide an entry point for teaching a known behavior. Online traces and dynamic hints make the loop responsive to new behavior as the model changes. Denton’s higher-ceiling vision rests on the adaptive judge: instead of repeatedly correcting a fixed list of mistakes, it can respond to what the current model actually does in production and guide improvement across multiple areas.

16:5817:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:58 · section reference included

Resources

From the talk

Updates since the talk

  • A later account of training from stored traces, resampling individual turns, and running full environments, with tools for inspecting hint-driven updates.

Transcript download
Full transcript

Loading transcript…