Contents
  1. What transfers
    1. Learning from another model
    2. The supervision interface
    3. Origins of distillation
  2. What supervision teaches
    1. Selected answers and sequences
    2. Probabilities over alternatives
    3. Matching internal representations
    4. Written reasoning as training data
  3. Where supervision comes from
    1. Select for the student's workload
    2. Supervise student-visited contexts
    3. Teacher-only information
  4. Fitting and diagnosing the student
    1. Build a coherent training objective
    2. Why transfer fails
  5. Evidence for replacement
    1. Measure capability and fidelity
    2. Validate the deployed task
  6. The replacement decision
    1. Repay the transfer investment
  7. Check understanding
  8. Open questions
  9. Selected talks
  10. References
  11. Talk library
← All topics

Distillation

Distillation trains a student model using behavior supplied by a teacher. Instead of paying for the larger model on every request, a team can use its answers, probability distributions or other training signals to develop a student suited to a narrower workload. The student is often smaller, but reduced size is not the definition of the method: the defining step is learning from another model. This chapter explains what to transfer, how to train with it and how to check that the resulting model is useful.

I — What transfers

Learning from another model

Distillation trains a student using supervision supplied by a teacher. Training changes the student’s learned parameters; subsequent predictions can use those parameters without running the teacher. Teacher and student name roles, not mandatory size differences.

The practical opportunity is specialization: learn the part of a teacher’s behavior that a deployment needs. The teacher can be an ensemble, several predictors whose outputs are combined, even when serving that ensemble would be impractical. The investment moves computation into labeling examples and training a reusable student.

Distillation differs from copying weights, which can initialize a model but does not itself teach it from teacher behavior. Caching stores answers for reuse; in-context demonstrations guide a model through its current input. Distillation instead produces a changed checkpoint, a saved model state. Quantization changes numerical precision rather than the source of supervision; a student can be both distilled and quantized.

Capability means performing a specified class of tasks. Behavioral fidelity means matching specified aspects of teacher behavior under stated conditions. These are different goals: reproducing an incorrect teacher answer improves agreement, not correctness. Before choosing a transfer method, specify student-visible inputs, allowed outputs, the intended task population, and acceptable gains and regressions.

For an extraction specialist, a preservation contract could distinguish these requirements.
PropertyRequired behaviorWhat may change
CorrectnessRecover the right field meanings and values.Correct a teacher’s mistaken extraction.
Output formatReturn the application’s required fields and types.Omit conversational explanation.
Abstention and refusalDistinguish insufficient information from prohibited requests.Change wording while preserving the required decision.
StylePreserve tone only if the application needs it.Avoid imitating incidental phrasing.

The supervision interface

The feasible method depends on what the teacher exposes. Returned text supports learning from selected outputs. Scores can support learning from alternatives. Internal access permits matching numerical states inside the model. A written explanation is still an output: receiving it does not expose the hidden computation that produced it.

Teacher artifactAvailable supervisionCompatibility requirement
Selected label or textThe particular output returned.The student must represent the target in its own output format.
Probabilities or logitsRelative preference among alternatives. A logit is an unnormalized output score.Direct comparison needs corresponding outcomes and prediction positions.
Internal activationsAn intermediate numerical representation.Internal access and a chosen correspondence between teacher and student states.

A soft target is a probability distribution used as supervision. A top-k report contains only k selected alternatives; it is not the complete distribution. The TRL distillation interface, for example, distinguishes full-vocabulary losses from top-k approximations. Check the actual scoring contract rather than assuming an API returns every probability or scores every supplied token.

Different tokenizers can change both vocabulary entries and sequence boundaries, obstructing direct coordinate-by-coordinate probability matching. Teacher-only documents or hints introduce a separate issue: different information at training and deployment. Constructing these inputs or targets produces synthetic examples; making a student learn from them is the transfer step.

Origins of distillation

Teacher–student methods developed around two recurring problems: expensive computation worth consolidating, and useful information that ordinary labels omit. Their history includes internal-state transfer, ensemble compression, and supervision for generated sequences. These approaches address different constraints and can coexist.

DevelopmentContribution
History Compression — 1992Jürgen Schmidhuber’s recurrent-network work trained one network to reproduce another’s internal state, potentially making the second network unnecessary. This was an early example of consolidating learned computation, distinct from the later soft-target formulation.
Model Compression — 2006Buciluă, Caruana, and Niculescu-Mizil trained compact networks on ensemble-labeled inputs to address storage and real-time constraints, including generated inputs when suitable unlabeled data were scarce.
Distilling the Knowledge in a Neural Network — March 2015 preprintHinton, Vinyals, and Dean extended compression through teacher probabilities. March dates the posting, not invention.
Sequence-Level Knowledge Distillation — November 2016Yoon Kim and Alexander Rush adapted transfer to complete translations, using selected teacher sequences to train students.

This development explains why distillation is broader than reducing parameter count. A selected answer, a distribution over answers, and an internal state preserve different information. The next sections examine what each teaches, then turn to where the training examples must come from.

II — What supervision teaches

Selected answers and sequences

A hard target records one selected label or output rather than probabilities over alternatives. With teacher text, supervised fine-tuning, or SFT, increases the likelihood of the recorded response. A token is a model-specific vocabulary unit; a prefix is the sequence preceding a prediction. Under teacher forcing, each prediction receives recorded predecessors, not the student’s earlier sampled choices. The term does not require a live teacher during training. Response supervision develops the underlying objective.

For an abstract recorded completion containing tokens A, B, and C, one sequence supplies three conditional targets.
PredictionSupplied contextTarget
FirstPromptA
SecondPrompt + recorded AB
ThirdPrompt + recorded A + recorded BC

The teacher’s selection procedure becomes part of the supervision. Kim and Rush selected translations using beam search, which retains several promising partial continuations. Their sequence-interpolation variant instead selected the candidate most similar to a reference translation. The teacher stayed the same, but the training target changed. Sequence training also concentrated the tested students’ output distributions, helping greedy decoding—choosing the most likely next token—work better in those translation experiments.

Retaining several responses can expose more output variation, while retaining one concentrates exposure. Neither choice recovers the teacher’s complete distribution. Fluent imitation also need not transfer broad competence: The False Promise of Imitating Proprietary LLMs found that persuasive response style could coexist with substantial factual gaps. The unit being copied is observable behavior, including its mistakes.

Probabilities over alternatives

A selected label hides preferences among other alternatives. Matching distributions preserves that guidance. Distillation temperature controls how concentrated teacher and student probabilities are during comparison; raising it can expose useful distinctions or noise.

pT(i)=exp⁡(zi/T)∑jexp⁡(zj/T),T>0.p_T(i)=\frac{\exp(z_i/T)}{\sum_j\exp(z_j/T)},\qquad T>0. Here ziz_i is a teacher logit. Softmax exponentiates scores and normalizes their sum to one; student logits similarly give qTq_T.
These illustrative teachers share a winner but disagree about the alternatives. Holding logits fixed, temperature-two values normalize square roots of temperature-one probabilities (rounded).
TargetClass AClass BClass C
Teacher 1, T = 10.60000.30000.1000
Teacher 2, T = 10.60000.10000.3000
Teacher 1, T = 20.47270.33430.1930
Teacher 2, T = 20.47270.19300.3343

Kullback–Leibler divergence, or KL, measures discrepancy between distributions. Its direction matters because it determines which distribution weights the comparison.

DKL(p∥q)=∑ipilog⁡piqi,DKL(q∥p)=∑iqilog⁡qipi.D_{\mathrm{KL}}(p\|q)=\sum_i p_i\log\frac{p_i}{q_i},\qquad D_{\mathrm{KL}}(q\|p)=\sum_i q_i\log\frac{q_i}{p_i}. Here pp is the teacher distribution and qq the student distribution over the same alternatives. Forward KL weights discrepancies by teacher probability; reverse KL weights them by student probability.

Forward KL pressures the student to cover teacher-supported alternatives. Reverse KL penalizes student probability where the teacher assigns little. MiniLLM uses reverse sequence-level KL on student-generated responses. A student unable to reproduce the whole teacher distribution may concentrate on teacher-preferred continuations, but omit others; neither direction universally wins. To isolate the weighting difference, hold a three-class teacher distribution fixed and redistribute the student’s probability. This is a distribution calculation, not a trained-model result.

Which distribution weights the penalty?

A takes its share first. B receives the selected fraction of what remains; C gets the rest. Every probability stays positive and the sum stays one.

Class A
Teacher p: 0.6000
Student q: 0.5000
Class B
Teacher p: 0.3000
Student q: 0.2000
Class C
Teacher p: 0.1000
Student q: 0.3000

Forward KL, p‖q
0.121171 nats

Reverse KL, q‖p
0.157330 nats

Signed contributions using natural logarithms
ClassForward: pᵢ ln(pᵢ/qᵢ)Reverse: qᵢ ln(qᵢ/pᵢ)
A0.109393 nats-0.091161 nats
B0.121640 nats-0.081093 nats
C-0.109861 nats0.329584 nats
Sum0.121171 nats0.157330 nats
Individual contributions may be negative; their summed KL is nonnegative. Natural logs give nats. Teacher probability is not correctness, and neither direction universally wins.

Training minimizes a loss, a numerical penalty for mismatching a target. For an independently supplied label yy, cross-entropy is −log⁡q1(y)-\log q_1(y): assigning less probability to that label incurs a larger penalty. The subscript one means probabilities are computed at temperature one.

L=(1−α)(−log⁡q1(y))+αT2DKL(pT∥qT).L=(1-\alpha)(-\log q_1(y))+\alpha T^2D_{\mathrm{KL}}(p_T\|q_T). This combines the label penalty with teacher matching; 0≤α≤10\leq\alpha\leq1 sets their relative weight. For a fixed teacher, KL and teacher-target cross-entropy differ by a constant. The conventional T2T^2 approximately compensates for temperature’s effect on gradients—the signals used to update parameters—not teacher error.

Training temperature controls the comparison being learned; sampling temperature controls generated choices. A partial probability report requires an explicit approximation rather than silently treating omitted alternatives as absent. Finally, next-token likelihood is not the probability that a complete answer is true. Calibration requires comparing a specified probability forecast with observed correctness.

Matching internal representations

Outputs supervise the end of a computation. Representation matching adds guidance inside it. A layer is a stage of learned transformation; its hidden state is an intermediate numerical output. Matching selected states requires a correspondence between layers and sometimes a learned projection between dimensions. Equal-width arrays are not automatically comparable, as representation spaces explains.

FitNets, developed by Adriana Romero and colleagues, appeared as a December 2014 preprint and at ICLR 2015. It addressed training deeper, thinner students: a student layer predicts a selected teacher layer’s output through a trainable regressor, followed by output distillation. The hint stage trains the student up to that layer and the regressor. Choosing a guided layer too deep can overconstrain learning.

For coordinate matching, a trainable projection can map two student coordinates to three teacher coordinates. Squared error then compares corresponding values in equal-sized operands. The dimensions and point configurations in the comparison are illustrative; validate any learned guidance against output performance.

Relational Knowledge Distillation instead teaches the student to preserve relationships between examples. Its distance objective divides pairwise distances by their mean within a minibatch—a group of training examples processed together—then compares teacher and student distances. Its angle objective compares the angles formed by triples of examples using normalized difference vectors. These comparisons do not require identical coordinates or even equal representation dimensions. They preserve selected relationships, not every feature or a guarantee of equivalent predictions.

Match coordinates or match relationships

Coordinate targets: project into compatible dimensions
Same input → fixed teacher

Target (1, 2, 0): 3 coordinates

Same input → trainable student

State (1, 1): 2 coordinates

Current trainable projection r(u,v) = (u,v,u−v)

Projected state (1, 1, 0): 3 coordinates

Compare the two 3-coordinate operands: squared error = 1
Loss ⇢ update projection and student through the guided layer. Teacher parameters stay fixed in this hint-training example.
Relational targets: match geometry among examples
Teacher spaceABCyx0Matched coordinate scale

A=(0,0); B=(3,0); C=(0,4)

Student spaceABCyx0Matched coordinate scale

A=(10,2); B=(10,8); C=(2,2)

A, B and C identify the same three examples in both spaces. The student configuration rotates the teacher triangle 90°, doubles its scale and translates it.

Normalize each distance by that minibatch’s mean pairwise distance
PairTeacher distanceStudent distanceNormalized, both
AB360.75
AC481
BC5101.25
Mean distance481

Corresponding angles: A = 90.00°, B = 53.13°, C = 36.87° in both spaces. Matching these relationships does not guarantee equal predictions.

Coordinate matching aligns projected values. Relational matching preserves selected distances and angles while allowing different coordinates; it does not establish equivalent task behavior.

Written reasoning as training data

A reasoning trace contains emitted intermediate steps associated with a proposed solution. Those steps can provide additional learning targets, but they are not hidden states. In hint-intervention experiments, reasoning models sometimes omitted information that influenced their answers. A rationale can therefore be useful training material without being a faithful account of the teacher’s computation.

Rationale supervision creates different deployment contracts.
Training contractSupervised outputIntended deployment output
Answer onlyThe answer or label.Answer.
Trace plus answerWritten steps followed by the answer.Steps and answer, unless a separately tested policy changes this.
Separate rationale taskAnswer prediction and rationale generation are separate tasks.Answer can be requested without a rationale.

Distilling Step-by-Step, published by Cheng-Yu Hsieh and colleagues at the University of Washington and Google in July 2023, used task prefixes to select label prediction or rationale generation. Its objective, Llabel+λLrationaleL_{\text{label}}+\lambda L_{\text{rationale}}, weights two tasks; λ\lambda controls the rationale contribution. It does not require generating a rationale before every deployed answer. Its gains were task-specific, and some settings combined human labels with teacher rationales.

Final-answer checking and step checking establish different things. Consider this deliberately faulty explanation: “3×4=103\times4=10, so the answer is 12.” The answer is correct; the stated intermediate equality is not. Training on the entire completion rewards both. Longer traces also change training exposure and output length, so their value must be separated from simply spending more computation.

DeepSeek’s May 28, 2025 release used the updated R1 teacher’s emitted reasoning chains to post-train Qwen3-8B Base into DeepSeek-R1-0528-Qwen3-8B. This is a separate student lineage, not a lower-precision copy of R1. The Paper Club discussion illustrates this distinction: learning from a teacher trained with reinforcement learning does not mean the student repeats that training procedure. Spending computation on a new problem belongs to Reasoning and Test-Time Compute.

III — Where supervision comes from

Select for the student's workload

A transfer set is the collection of inputs and supervision used to teach the student. Observed requests help represent actual usage; generated inputs can expand coverage. In a function-calling example, combining schema-generated cases with filtered teacher responses to real user inputs worked better than either alone in the reported experiments. That supports testing complementary sources, not a universal mixture.

Select teachers by the students they produce. OpenThoughts reported that QwQ-32B produced stronger students than DeepSeek-R1 under shared sampling settings, despite the latter’s stronger teacher benchmark performance. Its experiments also exposed a tradeoff between more distinct questions and several responses per question. The data-recipe talk recommends testing choices cheaply, then rechecking them at scale.

Selection experiments need to distinguish improved targets from changed exposure.
ComparisonWhat it revealedInterpretation boundary
Fine-tune-CoT: Date Understanding, 6.7B studentChecked rationales beat a random subset of equal size; the larger answer-correct set performed best.The stricter check helped at matched size, but lost useful training volume.
OpenThoughts answer filteringRandom selection beat the tested targeted filters on math; an unfiltered comparison consumed more examples.Filtering outcomes cannot all be attributed to quality at equal compute.

Inspect difficult cases and teacher–student disagreements rather than assuming they should all be removed. Confidence filtering can omit precisely the region where the student needs instruction; an easy-to-hard curriculum can also fail to address the actual weakness. Keep trusted labels distinct from teacher targets, and use coverage design and filtering analysis to assess what survives.

A useful transfer record identifies the input, teacher version, teacher-visible context, generation and selection settings, retained target, and verification result. Keep rejected candidates traceable when permitted: they explain both selection effects and collection cost. Follow record-level lineage, protect independent assessment, and establish permitted reuse before collecting training material.

Supervise student-visited contexts

A student can fit recorded completions yet struggle after its own different choices. Its policy is the conditional distribution from which choices are made; a trajectory is the resulting sequence of choices and observations. Recorded demonstrations cover their own prefixes. Free-running generation visits the student’s prefixes, including mistakes. Trajectory-based learning explains this feedback between behavior and collected data.

DAgger—Dataset Aggregation—was presented at AISTATS 2011 by Stéphane Ross, Geoffrey Gordon, and Andrew Bagnell. It collects learner-visited states, obtains expert action labels, adds them to an accumulated dataset, and trains again. Collection can mix expert and learner actions. Its driving experiment showed why more expert-only laps need not teach recovery. This was supervised imitation learning, not language-model distillation or environmental reward optimization.

On-policy distillation queries the teacher at prefixes generated by the current student. Generalized Knowledge Distillation mixes these trajectories with recorded sequences and matches teacher–student token distributions. Sampled choices supply contexts, not correct answers. During each update, the sampled sequence is fixed: gradients pass through the distribution-matching loss, not the discrete sampling operation. Refreshing trajectories as the student changes keeps supervision aligned with its behavior.

Compare distributions at the same prefix

Shared initial prompt; same abstract next-token vocabulary
Recorded lane
Prompt + recorded A

Recorded predecessors A supply this lane’s context.

↓ same prefix supplied to both predictors

Teacher: p(· | Prompt, A)

Student: q(· | Prompt, A)

↓ both distributions enter the matching loss
Match at A
Loss ⇢ student parameter update
Student-sampled lane
Prompt + sampled B

Sampling chooses B before this update. Hold B fixed; it is context, not a correct-answer target.

↓ same prefix supplied to both predictors

Teacher: p(· | Prompt, B)

Student: q(· | Prompt, B)

↓ both distributions enter the matching loss
Match at B
Loss ⇢ student parameter update

Solid arrows show data dependencies; dashed update boxes mark gradient-based learning. No gradient crosses discrete sampling and no loss pairs teacher-after-A with student-after-B. This comparison keeps the teacher fixed. After an update, collect fresh student trajectories separately.

Each loss compares teacher and student predictions conditioned on the same prefix. Student sampling selects where supervision is requested, not the correct answer.
Three independent choices are often compressed into the word “online.”
ChoiceAlternativesWhat changes
Context sourceRecorded sequences or current-student trajectories.Which prefixes receive supervision.
Teacher-query timingCached scores or teacher inference during training.When teacher computation is paid.
Teacher stateFixed parameters or an explicitly updated teacher.Whether the supervisor itself changes.

LinkedIn’s training report calls teacher inference during student training “online” even though the teachers remain fixed. Separately, on-policy training may share rollout infrastructure with reinforcement learning while using teacher likelihoods instead of task rewards, as Modern Post-Training explains. More relevant contexts can therefore require more teacher scoring without changing the source of the learning signal.

Teacher-only information

Privileged information is available to the teacher during training but absent from ordinary student inputs. Generalized distillation explicitly permits different teacher and student representations of a case. A richer teacher can supply better targets without requiring the same information at deployment. Whether those targets are learnable from student-visible information remains a separate condition.

This also enables on-policy self-distillation: the same model serves as a better-informed teacher when given a useful hint, while the unhinted student learns from its guidance along student-generated trajectories. The advantage comes from information, not necessarily a larger architecture. Scaling up Continual Learning develops this setup.

The danger is hint leakage: a target relies on a fact the student could not obtain. In the talk’s login example, revealing an expired single-sign-on token can encourage an unsupported diagnosis. Guidance to inspect available logs instead teaches a discoverable procedure. This is a useful distinction, not a guarantee that filtering hints removes shortcuts. If two cases have identical student-visible inputs but require different answers because of a hidden runtime fact, imitation cannot make that fact observable.

For token-level supervision, teacher and student may receive different prompts, but the compared scores must refer to corresponding continuation positions. A hint inserted into one prompt changes offsets; blindly pairing positions can teach the wrong target. Keep the information boundary explicit and test the unhinted student. Context Engineering owns the separate task of assembling information at runtime.

Illustrative login-support case. P and H are schematic prompt and hint segments; horizontal whitespace aligns continuation positions and is not runtime padding. Matching scores updates the student. Guidance can teach an available investigation step but cannot supply a hidden fact before inspection.

IV — Fitting and diagnosing the student

Build a coherent training objective

The recipe must identify a starting checkpoint, trainable components, compatible targets, and exposure budget. Distillation can train a fresh student or adapt an already useful one. Full fine-tuning and Low-Rank Adaptation, or LoRA, are choices about parameter updates, not supervision sources; Post-training explains their difference.

DistilBERT, introduced by Hugging Face’s Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf in October 2019, illustrates initialization followed by learning. It halves BERT’s layers and initializes from alternating teacher layers. Training then combines predicting masked tokens, matching teacher output distributions, and aligning hidden-state directions through a cosine loss. These terms teach complementary aspects of prediction and representation. In the revised paper, random initialization reduced the reported General Language Understanding Evaluation (GLUE) aggregate by 3.69 points. Copying layers supplied a useful starting point, not the complete transfer.

Output, distribution, and representation losses can be combined with explicit weights, but each should serve a tested purpose. TinyBERT separately matches input representations, hidden states, attention-score matrices, and predictions. It tests alternative layer correspondences and removes individual supervision signals in ablation experiments to examine their contribution. Adding a signal adds an assumption about what ought to match; comparing trained students tests whether that assumption helps.

A loss mask selects positions that contribute directly to the penalty. Masking does not remove their text from the conditioning input. The following contract follows Tinker’s multi-turn distillation recipe.
Record regionAvailable as contextDirect distillation loss
System and user messagesYesExcluded
Tool results and assistant headersYesExcluded
Student-generated continuationEarlier generated tokens condition later ones.Included

Distinguish task weighting from span weighting. Weighting a separate rationale-generation task does not imply assigning the same weight to rationale tokens inside a trace-plus-answer response. Record exactly which terms and positions are included, which teacher targets are fixed, and whether student parameters or additional projections change. Identical visible examples can therefore produce different updates.

Count candidates generated, targets accepted, and examples or tokens actually consumed separately. Repeated exposure changes the training mixture even when the saved dataset does not change. Select loss weights, exposure, and stopping checkpoints using validation cases and retained-capability checks. Freeze the choice before the final test: repeated adjustment after seeing test results converts that set into development data.

Why transfer fails

Student capacity is the range of behavior its architecture and parameters can represent, not merely its parameter count. A weak result can also arise because suitable behavior was never supervised, inputs lack needed information, targets are misleading, or optimization failed to fit an attainable solution. These explanations call for different interventions.

Use controlled changes to test competing explanations rather than diagnosing from one final score.
ObservationDiscriminating intervention
Poor agreement even on training inputsCheck target alignment, optimization, and initialization before attributing everything to generalization.
Good fit on familiar inputs, poor fit elsewhereExpand missing coverage while holding student and objective fixed.
Different answers require an unavailable factRestore the input or narrow the task.
Suspected capacity limitCompare student sizes under controlled data, supervision, and training conditions.

Cho and Hariharan found that larger, more accurate teachers sometimes produced worse students, supporting a capacity-mismatch interpretation in their classification experiments. Conversely, Stanton and colleagues observed fitting failures even between identical architectures capable of representing the same solution. Teacher strength and theoretical representability do not establish successful learning.

Born-Again Neural Networks demonstrated improved students while retaining the teacher’s architecture in image-classification and language-modeling experiments. Same architecture did not mean copied learned weights, and individual-student results were separate from ensembles of successive generations. Teacher performance is therefore neither a universal ceiling nor a promise that a student will surpass it.

For large reductions, staged transfer is another candidate intervention. The 360Brew team reported better results from progressively smaller students than one large jump. That is a recipe to compare against direct transfer, not a universal requirement. Use failure investigation to decide which experiment addresses the observed weakness.

V — Evidence for replacement

Measure capability and fidelity

Evaluate the teacher, starting student, and distilled student on matched held-out cases. Where trusted labels exist, add a student trained on those labels without teacher supervision to isolate the teacher’s contribution. A common execution protocol helps attribute differences; a second comparison using intended deployment configurations establishes practical usefulness. Do not confuse the two.

For a defined answer-matching rule, agreement and correctness form separate axes.
Student independently correctStudent independently wrong
Agrees with teacherUseful agreement.Shared error; imitation succeeded at the wrong answer.
Disagrees with teacherPotential correction of teacher error or another valid answer.Failure requiring inspection of both outputs.

Define fidelity at the required level. Top-label agreement compares winners; predictive KL compares distributions; semantic agreement requires a task-specific interpretation of generated answers. Preserve separate checks for formatting, refusals where relevant, retained capabilities, and changed inputs. The teacher should not be the sole judge: model judges can favor verbosity or familiar output patterns, so validate them independently.

The official DeepSeek-R1-0528-Qwen3-8B model card illustrates why task slices matter.
EvaluationDistilled Qwen3-8B studentQwen3-8B comparator
AIME 202486.076.0
GPQA Diamond61.162.0

The model card permits 64K generated tokens and, where sampling applies, uses temperature 0.6, top-p 0.95, and 16 responses per query to estimate single-attempt success, or pass@1. The comparator is Qwen3-8B, not the Qwen3-8B Base initialization. This is not a clean before/after ablation, and no uncertainty interval establishes the significance of the GPQA difference.

For reasoning transfer, compare answer-only and rationale-trained students with explicit training exposure and inference budgets, recording response length, attempts, and held-out task families. A gain obtained with longer responses may be useful, but it is not evidence of improvement at equal inference work. Reasoning scores also do not establish preservation of safety-critical behavior; those requirements need their own cases and checks.

Validate the deployed task

A replacement claim concerns a complete operating configuration and a defined request population. Distillation need not replace every teacher call: cached results can handle repeated inputs while a student serves uncached requests. The surrounding software still determines what information reaches the model and how outputs are used.

Instacart’s Intent Engine report describes semantic role labeling as extracting product, brand, and attribute concepts from search queries. Offline teacher tags populate a frequent-query cache and train a Llama-3-8B student for uncached, long-tail queries—queries that are individually infrequent. Similar F1 concealed different precision and recall. Meeting latency goals also required adapter merging and upgraded hardware; quantization reduced latency but hurt recall, so the deployed model remained unquantized.

Conceptual reconstruction of the engineering report, not a system screenshot. Cache records are loaded for lookup and the trained checkpoint is deployed for prediction. Hits reuse individual outputs; misses use learned parameters. The common schema does not imply equal accuracy.

Precision measures correctness among predicted tags; recall measures recovery of required tags. F1 is their harmonic mean. Two models can therefore share an F1 while making different omissions and extra predictions. Use an explicit tag-matching policy and keep the components visible rather than treating one aggregate as interchangeability.

The conference account described replacing the extreme-tail fallback with a distilled student as work in progress; the engineering report supplies the production account. Neither supports attributing every system improvement to distillation. Keep the learning change, cache coverage, hardware, and numerical precision separate.

After offline validation, shadow evaluation copies current inputs to a candidate without applying its outputs; bounded live exposure tests consequences that shadowing cannot. Side-effecting tools still require isolation in shadow runs. Follow the live-experiment contract, then benchmark the actual workload. Only validated coverage counts toward the volume available to repay training.

VI — The replacement decision

Repay the transfer investment

Begin with alternatives that meet the same task requirements, including the untrained starting student. If it already suffices, distillation adds avoidable work. AWS’s Bedrock distillation launch account separated teacher generation, student fine-tuning, storage, and provisioned inference charges. Those historical categories illustrate lifecycle accounting, not current universal hosting requirements.

Define a useful lifetime HH: the period over which the student and its validated task remain suitable. Let NN count eligible requests during that period. Use comparable financial boundaries for the incumbent teacher path and the candidate path, including any incumbent caching. The following is an accounting model, not a published deployment result.

TermIncluded work
C_createAll teacher generations, including rejected candidates; verification; training experiments; and evaluation.
C_fixed(H)Incremental hosting, maintenance, and refresh work over H that is not allocated per request.
c_teacherAverage baseline cost per eligible request under the incumbent completion policy.
c_effectiveStudent execution, actual output lengths, repeated attempts, and expected remaining teacher work per eligible request.
Cbaseline(N)=Ncteacher,Creplacement(N,H)=Ccreate+Cfixed(H)+Nceffective.C_{\text{baseline}}(N)=Nc_{\text{teacher}},\qquad C_{\text{replacement}}(N,H)=C_{\text{create}}+C_{\text{fixed}}(H)+Nc_{\text{effective}}. With constant average rates and positive per-request savings: Nbreak-even=Ccreate+Cfixed(H)cteacher−ceffective.N_{\text{break-even}}=\frac{C_{\text{create}}+C_{\text{fixed}}(H)}{c_{\text{teacher}}-c_{\text{effective}}}. Equality repays the investment; net savings require greater volume. A nonpositive denominator provides no finite payback for a positive investment.

Assign each charge once. If hosting is already allocated into request cost, do not add it again as fixed cost. If requests outside the validated subset remain unchanged, exclude them from both incremental paths; do not count them as student-saving opportunities. A changing workload can shorten the lifetime even when total company traffic is large.

For an illustrative financial comparison, assume a $200 combined investment, a $0.02 teacher path, and 20,000 eligible requests before refresh. At $0.005 effective student cost, continuous break-even occurs at 13,333.33 requests; 13,334 is the first whole-request count with net savings. If retained teacher work raises effective cost to $0.015, equality occurs at 40,000 requests—beyond the useful horizon; net savings require still greater volume. Both candidates must first satisfy the same quality requirements.

Savings must arrive before refresh

Example

More retained teacher work moves the crossover beyond the same useful-volume horizon.

Effective student cost: $0.005

At the 20,000-request horizon, the baseline costs $400 and the replacement costs $300.

01125022500337504500002505007501000Cumulative eligible requests (requests)Cumulative financial cost (USD)Teacher baselineInvestment plus effective student costUseful-volume horizonBreak-evenBaseline after horizon (hypothetical)Replacement after horizon (hypothetical)13,333.33: equalityRefresh horizonBaselineReplacement
  • 1. Teacher baseline
  • 2. Investment plus effective student cost
  • 3. Useful-volume horizon
  • 4. Break-even
  • 5. Baseline after horizon (hypothetical)
  • 6. Replacement after horizon (hypothetical)
Read coordinates and regions as data

X: 0–45000 requests; Y: 0–1000 USD, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Teacher baseline (polyline)

(0, 0); (20000, 400)

Investment plus effective student cost (polyline)

(0, 200); (20000, 300)

Useful-volume horizon (polyline)

(20000, 0); (20000, 950)

Break-even (points)

(13333, 266.67)

Baseline after horizon (hypothetical) (polyline)

(20000, 400); (45000, 900)

Replacement after horizon (hypothetical) (polyline)

(20000, 300); (45000, 425)

13,333.33: equality: (11000, 340)

Refresh horizon: (21000, 940)

Baseline: (28000, 600)

Replacement: (28000, 300)

Effective student cost: $0.015

Only recurring cost changes. At the horizon, replacement costs $500 against the $400 baseline.

01125022500337504500002505007501000Cumulative eligible requests (requests)Cumulative financial cost (USD)Teacher baselineInvestment plus effective student costUseful-volume horizonBreak-evenBaseline after horizon (hypothetical)Replacement after horizon (hypothetical)40,000: equalityRefresh horizonBaselineReplacement
  • 1. Teacher baseline
  • 2. Investment plus effective student cost
  • 3. Useful-volume horizon
  • 4. Break-even
  • 5. Baseline after horizon (hypothetical)
  • 6. Replacement after horizon (hypothetical)
Read coordinates and regions as data

X: 0–45000 requests; Y: 0–1000 USD, increasing up. Axes scaled independently; screen angles and distances are not comparable.

Teacher baseline (polyline)

(0, 0); (20000, 400)

Investment plus effective student cost (polyline)

(0, 200); (20000, 500)

Useful-volume horizon (polyline)

(20000, 0); (20000, 950)

Break-even (points)

(40000, 800)

Baseline after horizon (hypothetical) (polyline)

(20000, 400); (45000, 900)

Replacement after horizon (hypothetical) (polyline)

(20000, 500); (45000, 875)

40,000: equality: (39000, 730)

Refresh horizon: (21000, 940)

Baseline: (28000, 500)

Replacement: (28000, 680)

These invented dollar inputs assume a $200 investment, a $0.02 baseline request, and 20,000 eligible requests before refresh. Only effective student cost changes, from $0.005 to $0.015. Equality occurs at 13,333.33 and 40,000 requests; 13,334 is the first whole-request count with savings in the first case. Dashed cost segments beyond 20,000 show hypothetical continuation. The cited carbon study does not supply these dollar amounts.

Two amortization boundaries deserve separation. LinkedIn’s multi-teacher training report caches outputs by teacher version and data fingerprint for reuse across student experiments. That saves repeated supervision work, not necessarily production request cost. Joseph Attieh and colleagues’ February 2026 translation study separately models production impact plus recurring inference impact. Its carbon accounting supports the same intercept-versus-slope reasoning, but its crossover cannot be converted into dollar payback.

The resulting decision has three legitimate outcomes: replace teacher calls across the validated workload, specialize on an identifiable subset, or retain teacher calls. Cost engineering develops the broader ledger; model routing owns selection and escalation policies. Distillation earns its place when the student preserves what matters and enough useful work remains to justify creating and maintaining it.

Open questions

  1. Predicting teaching utility remains difficult: teacher accuracy does not determine student learning. Progress would mean selecting teachers using transferable diagnostics rather than repeatedly training full students for every candidate.

  2. Rationale supervision needs stronger separation of useful decomposition from extra exposure and longer inference. Controlled comparisons matching training tokens and inference budgets across held-out task families would clarify when traces earn their cost.

  3. Privileged hints can improve supervision while encouraging inaccessible shortcuts. Progress would combine reliable leakage diagnostics with unhinted deployment tests that establish recovery procedures rather than memorized diagnoses.

  4. Targeted adaptation can change unrelated behavior. Reliable preservation of refusal decisions and other consequential capabilities requires coverage beyond reasoning benchmarks, with repeatable regression measurements across successive updates.

  5. Financial replacement claims need complete lifecycle ledgers. Collection failures, verification labor, repeated experiments, residual teacher work, and refresh can dominate savings; prospective accounting through a full useful lifetime would make payback claims more credible.

Follow the curated reading path through the speakers and demonstrations behind this entry.

23 min

AI Engineer World's Fair 2026 · 2026

Scaling up Continual Learning

Ronak Malde

Cited in this entry

Explore privileged-context self-distillation and the risk that teacher hints encourage shortcuts unavailable to the deployed student.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

23 matching talks

TalkSpeakerEventYear
Vibhu SapraAI Engineer World's Fair 20252025
Ishan AnandAI Engineer World's Fair 20242024
Low Level Technicals of LLMs

Transcript reviewed

Daniel HanAI Engineer World's Fair 20242024
Scaling Compute on Context

Transcript reviewed

Jack MorrisAI Engineer World's Fair 20262026
Vinesh GudlaAI Engineer World's Fair 20252025
Hamel Husain, Emil SedghAI Engineer World's Fair 20242024
Evaling Video Slop

Transcript reviewed

Maor BrilAI Engineer World's Fair 20262026
Mahmoud MabroukAI Engineer Europe 20262026
Kyle KranenAI Engineer World's Fair 20252025
Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Bertrand CharpentierAI Engineer Europe 20262026
Samuel DentonAI Engineer World's Fair 20262026
Fryderyk Wiatrowski, Peter AlbertAI Engineer World's Fair 20242024
Alessandro CappelliAI Engineer Europe 20262026
Kobie CrawfordAI Engineer Europe 20262026
Jesse HuAI Engineer Code 20252025
Abi AryanAI Engineer Summit 20232023
Ben KunkleAI Engineer Europe 20262026
Vivek TrivedyAI Engineer World's Fair 20262026
RL Environments at Scale

Metadata candidate

Will BrownAI Engineer Code 20252025
Ziv IlanAI Engineer Europe 20262026
Raghav SabooAI Engineer World's Fair 20262026
Alvin PaneAI Engineer NYC 2026 Online Talks2026

References

Coverage and source review
Processed transcripts
22 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
6 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Build faster, more cost-efficient, highly accurate models with Amazon Bedrock Model Distillation (preview)

    https://aws.amazon.com/blogs/aws/build-faster-more-cost-efficient-highly-accurate-models-with-amazon-bedrock-model-distillation-preview/

  2. Life Cycle-Aware Evaluation of Knowledge Distillation for Machine Translation: Environmental Impact and Translation Quality Trade-offs

    https://arxiv.org/html/2602.09691v1

  3. Distilling the Knowledge in a Neural Network

    https://www.cs.toronto.edu/~hinton/absps/distillation.pdf

  4. Model Compression

    https://www.cs.cornell.edu/~caruana/compression.kdd06.pdf

  5. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    https://arxiv.org/html/1910.01108v4

  6. Model Compression via Distillation and Quantization

    https://arxiv.org/pdf/1802.05668

  7. Does Knowledge Distillation Really Work?

    https://arxiv.org/html/2106.05945

  8. NIST AI RMF Playbook: Measure

    https://airc.nist.gov/airmf-resources/playbook/measure/

  9. Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

    https://arxiv.org/html/2402.12030

  10. Reasoning Models Don't Always Say What They Think

    https://www.anthropic.com/research/reasoning-models-dont-say-think

  11. Attention Is All You Need: token representations and next-token probabilities

    https://arxiv.org/html/1706.03762v7

  12. FitNets: Hints for Thin Deep Nets

    https://arxiv.org/pdf/1412.6550

  13. TRL v1.7.0: Distillation Trainer

    https://huggingface.co/docs/trl/v1.7.0/distillation_trainer

  14. Unifying Distillation and Privileged Information

    https://arxiv.org/pdf/1511.03643

  15. Learning Complex, Extended Sequences Using the Principle of History Compression

    https://gwern.net/doc/ai/nn/rnn/1992-schmidhuber.pdf

  16. Distilling the Knowledge in a Neural Network

    https://www.cs.toronto.edu/~hinton/absps/distillation.pdf

  17. Sequence-Level Knowledge Distillation

    https://aclanthology.org/D16-1139.pdf

  18. SFT Trainer: objective, shifting, and masking

    https://huggingface.co/docs/trl/sft_trainer

  19. Transformers T5 documentation: Training

    https://huggingface.co/transformers/v3.5.1/model_doc/t5.html

  20. Sequence-Level Knowledge Distillation

    https://aclanthology.org/D16-1139.pdf

  21. OpenThoughts: Data Recipes for Reasoning Models

    https://arxiv.org/html/2506.04178v1

  22. The False Promise of Imitating Proprietary LLMs

    https://arxiv.org/html/2305.15717

  23. MiniLLM: On-Policy Distillation of Large Language Models

    https://arxiv.org/html/2306.08543

  24. Language Models (Mostly) Know What They Know

    https://arxiv.org/html/2207.05221v4

  25. TinyBERT: Distilling BERT for Natural Language Understanding

    https://aclanthology.org/2020.findings-emnlp.372.pdf

  26. FitNets: Hints for Thin Deep Nets

    https://arxiv.org/pdf/1412.6550

  27. Relational Knowledge Distillation

    https://arxiv.org/html/1904.05068

  28. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

    https://arxiv.org/html/2305.02301

  29. Large Language Models Are Reasoning Teachers

    https://aclanthology.org/2023.acl-long.830.pdf

  30. Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

    https://aclanthology.org/2023.findings-acl.507.pdf

  31. DeepSeek-R1 更新,思考更深,推理更强

    https://api-docs.deepseek.com/zh-cn/news/news250528/

  32. Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek-R1/V3) — Vibhu Sapra

    https://www.youtube.com/watch?v=9k3xPh-40mo&t=776

  33. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

    https://arxiv.org/html/2501.12948v1

  34. Model-Maxxing: RFT, DPO, SFT (Fine-tuning with OpenAI) — Ilan Bigio, OpenAI

    https://www.youtube.com/watch?v=JfaLQqfXqPA&t=1005

  35. OpenThoughts: Data Recipes for Reasoning Models

    https://www.youtube.com/watch?v=liG97YXaTSA&t=396

  36. Statistics Canada Quality Guidelines: Coverage and frames

    https://www150.statcan.gc.ca/n1/pub/12-539-x/steps-etapes/4147788-eng.htm

  37. Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel

    https://www.youtube.com/watch?v=TNwJ1LMiENk&t=1008

  38. PROV-DM: The PROV Data Model

    https://www.w3.org/TR/prov-dm/

  39. Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant

    https://www.youtube.com/watch?v=qqXdLf3wy1E&t=523

  40. A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

    https://proceedings.mlr.press/v15/ross11a/ross11a.pdf

  41. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

    https://arxiv.org/pdf/2306.13649

  42. The Training Infrastructure Behind AI-Powered Job Search: 8X Faster Multi-Teacher Distillation

    https://www.linkedin.com/blog/engineering/infrastructure/the-training-infrastructure-behind-ai-powered-job-search-eight-x-faster-multi-teacher-distillation

  43. Scaling up Continual Learning

    https://www.youtube.com/watch?v=zL1kLftVTlo&t=473

  44. Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect

    https://www.youtube.com/watch?v=V-EDrhIhHzQ&t=727

  45. Scaling up Continual Learning

    https://www.youtube.com/watch?v=zL1kLftVTlo&t=966

  46. Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect

    https://www.youtube.com/watch?v=V-EDrhIhHzQ&t=2491

  47. Domain adaptation and fine-tuning for domain-specific LLMs

    https://www.youtube.com/watch?v=cXPYtkosXG4&t=454

  48. Tinker Documentation: Model Distillation

    https://tinker-docs.thinkingmachines.ai/cookbook/recipes/distillation/

  49. Cross-validation and held-out evaluation

    https://scikit-learn.org/stable/modules/cross_validation.html

  50. On the Efficacy of Knowledge Distillation

    https://arxiv.org/pdf/1910.01348

  51. Born-Again Neural Networks

    https://proceedings.mlr.press/v80/furlanello18a/furlanello18a.pdf

  52. 360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI

    https://www.youtube.com/watch?v=U0S6CfzAY5c&t=379

  53. Weak supervision needs evaluation beyond supervisor agreement

    https://arxiv.org/pdf/2312.09390

  54. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

    https://arxiv.org/html/2306.05685

  55. DeepSeek-R1-0528-Qwen3-8B model card

    https://huggingface.co/deepseek-ai/DeepSeek-R1-0528-Qwen3-8B

  56. Demystifying evals for AI agents

    https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents

  57. How Instacart transformed its search and discovery using an LLM-driven approach

    https://www.youtube.com/watch?v=PjaVHm_3Ljg&t=655

  58. Building the Intent Engine: How Instacart Is Revamping Query Understanding with LLMs

    https://company.instacart.com/tech-innovation/building-the-intent-engine-how-instacart-is-revamping-query-understanding-with-llms

  59. Introduction to Information Retrieval: Precision and recall

    https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-unranked-retrieval-sets-1.html

  60. Scaling up Continual Learning

    https://www.youtube.com/watch?v=zL1kLftVTlo&t=591

  61. Model-Maxxing: RFT, DPO, SFT (Fine-tuning with OpenAI) — Ilan Bigio, OpenAI

    https://www.youtube.com/watch?v=JfaLQqfXqPA&t=411

  62. Spreadsheets-are-all-you-need: Decoding the Decoder LLM without de code

    https://www.youtube.com/watch?v=NamKkerrlnQ&t=520

  63. Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect

    https://www.youtube.com/watch?v=V-EDrhIhHzQ&t=2469

  64. Hacking the Inference Pareto Frontier

    https://www.youtube.com/watch?v=Y2qc0UhDSnc&t=812

  65. OpenThoughts: Data Recipes for Reasoning Models

    https://www.youtube.com/watch?v=liG97YXaTSA&t=532

  66. Bringing Continual Learning into Enterprises

    https://www.youtube.com/watch?v=ZTA0GwpAUak&t=1018

  67. Modern Post-Training: A Deep Dive — Will Brown, Prime Intellect

    https://www.youtube.com/watch?v=V-EDrhIhHzQ&t=1328

  68. 360Brew: LLM-based Personalized Ranking and Recommendation — Hamed Firooz and Maziar Sanjabi, LinkedIn AI

    https://www.youtube.com/watch?v=U0S6CfzAY5c&t=766