Contents
  1. How training creates a checkpoint
    1. What a training stage changes
      1. Runtime input versus learned state
    2. From recorded text to parameter updates
      1. One sequence, many scored positions
    3. Objectives define the prediction problem
      1. Fit is narrower than capability
  2. How staged training developed
    1. Must-know turning points
  3. Designing model exposure
    1. Mixture weight is not corpus size
      1. Selection becomes exposure
      2. Coverage versus repetition
    2. Schedules change exposure over time
  4. Allocating finite training work
    1. Parameters, tokens, and compute form a budget
      1. A conditional allocation model
      2. Why fitted optima differ
      3. Freshness and lifetime demand
    2. Parallel execution sets feasibility
  5. Checkpoints as evidence
    1. A checkpoint is more than weights
      1. Identity and lineage
    2. Measure gains and regressions separately
      1. One scalar cannot describe a checkpoint
    3. Protect independent capability evidence
      1. Surface separation is not independence
      2. Scores can leak into selection
  6. Extending an existing model
    1. Continue rather than restart
      1. The inherited state creates choices
      2. What continuation can change
    2. Adapt to a target distribution
      1. Target gain and broad retention
    3. Classify the stage by its contract
    4. Specialization can interfere with retention
      1. Optimization and capability can disagree
      2. Mitigate, measure, and retain rollback
  7. Choosing the next stage
    1. Write the training decision before the run
  8. Check understanding
  9. Open questions
  10. Selected talks
  11. References
  12. Talk library
← All topics

Pretraining & midtraining

A model cannot use a capability merely because an application asks for it. The capability must already be supported by the model’s parameters, supplied in the current request, or produced by tools and external systems. Pretraining is the large-scale fitting process that establishes broadly reusable parameterized behavior. Later weight-updating stages can extend that behavior toward new domains, languages, sequence structures, or tasks.

The useful engineering question is therefore not simply, “Is this pretraining or midtraining?” It is: What checkpoint did the run start from? What prediction problem and data did it use? How were examples mixed and scheduled? How much training work was spent? Which capabilities should change, and which independent measurements would establish that they did? Those questions remain meaningful even when organizations use stage names differently.

I. How training creates a checkpoint

What a training stage changes

A training stage repeatedly evaluates a computable objective and changes model parameters to reduce it. The stage starts from some parameter state—perhaps a fresh initialization or a capable existing checkpoint—and produces a new one. The objective, data, schedule, and optimization budget determine the pressure applied along that path.

Runtime input versus learned state

A prompt follows a different path. It changes the current input and therefore the computation performed with fixed parameters; ordinary inference does not rewrite the checkpoint. Prompting and In-Context Learning develops that invocation-time mechanism. Weight-updating stages include broad pretraining, continued training, and the response-oriented methods introduced in Post-training and Alignment, but those stages need not share an objective or purpose.

Stage labels become useful when expanded into an operational contract.
InterventionStarting stateWhat changesTypical purpose
PromptingFixed checkpointCurrent input and activationsSpecify or demonstrate the current task
Initial pretrainingFresh or minimally initialized modelParameters through large-scale predictionEstablish broadly reusable capabilities
Continued pretrainingExisting pretrained checkpointParameters under further prediction trainingExtend exposure, context, or capability
Behavioral post-trainingPretrained or continued checkpointParameters under demonstrations, preferences, or rewardsShape responses and task behavior

From recorded text to parameter updates

In causal language-model training, a recorded sequence supplies many prediction events. For tokens build, the, index, the model predicts the from build, then predicts index from build the. Teacher forcing means that each prediction sees the recorded prefix rather than an earlier sampled model output. Causal masking prevents a position from reading future targets; the full mechanism is developed in Predicting recorded successors.

One sequence, many scored positions

For scored positions tt, a common token objective is L(θ)=−1∣S∣∑t∈Slog⁡pθ(xt∣x<t), L(\theta)=-\frac{1}{|S|}\sum_{t\in S}\log p_\theta(x_t\mid x_{<t}), where θ\theta denotes the parameters, x<tx_{<t} is the visible recorded prefix, and SS is the set of positions included by the loss mask. Gradients describe how small parameter changes affect this loss; an optimizer uses them to produce the next parameter state. Loss and the objective and Parameter updates cover those borrowed mechanics.

The model does not store the document as an ordinary database row. Each scored prediction contributes numerical pressure shared across parameters and across many examples. Consequently, a source’s influence depends on how it is sampled, tokenized, masked, weighted, repeated, and combined with other examples—not merely on whether its files appear in a corpus.

Objectives define the prediction problem

A self-supervised objective constructs targets from the observations themselves. Its design determines what information is visible and what must be recovered. Under causal prediction, a position sees only an allowed prefix and predicts a successor. BERT-style masked prediction corrupts selected positions and recovers their original tokens using surrounding context. T5-style span corruption replaces consecutive spans with sentinels and generates the removed spans as a target sequence. The source text may be identical, but these are different learning problems.

One sequence, four prediction problems

Example

The objective changes what the model can see and which recorded tokens it must predict.

Causal input: build theAt this scored position, only the recorded prefix is visible. The target token index is excluded.
Target: indexPredict the recorded successor. Other eligible positions supply their own prefix–successor prediction events.
Masked input: build [MASK] indexThe selected middle token is corrupted; both surrounding tokens remain visible.
Target: theRecover the original token at the selected position, using surrounding context.
Span input: build <s1>Replace the consecutive span the index with a sentinel marking its location.
Target: <s1> the index <s2>Generate the removed span after its corresponding sentinel, then a final sentinel. Uncorrupted input text is not repeated in the target.
FIM input: <prefix> build <suffix> index <middle>Place the prefix and suffix before the missing middle. Marker names describe their roles rather than a specific tokenizer.
Target: theGenerate the removed middle conditioned on both surrounding pieces, using a causal decoder.
Each pair reuses “build the index.” Input nodes show available information; target nodes show what is scored. Causal prediction shows one scored position. Sentinel markers identify removed spans or segment boundaries. These are target constructions, not comparisons of model quality.
Read the diagram as text
  • Causal input: build the. At this scored position, only the recorded prefix is visible. The target token index is excluded.
  • Target: index. Predict the recorded successor. Other eligible positions supply their own prefix–successor prediction events.
  • Masked input: build [MASK] index. The selected middle token is corrupted; both surrounding tokens remain visible.
  • Target: the. Recover the original token at the selected position, using surrounding context.
  • Span input: build <s1>. Replace the consecutive span the index with a sentinel marking its location.
  • Target: <s1> the index <s2>. Generate the removed span after its corresponding sentinel, then a final sentinel. Uncorrupted input text is not repeated in the target.
  • FIM input: <prefix> build <suffix> index <middle>. Place the prefix and suffix before the missing middle. Marker names describe their roles rather than a specific tokenizer.
  • Target: the. Generate the removed middle conditioned on both surrounding pieces, using a causal decoder.
  • Causal input: build the → Target: index: predict successor.
  • Masked input: build [MASK] index → Target: the: recover selected token.
  • Span input: build <s1> → Target: <s1> the index <s2>: generate removed span.
  • FIM input: <prefix> build <suffix> index <middle> → Target: the: generate missing middle.
The objective determines the information relationship presented to the model.
ObjectiveVisible informationScored targetUseful pressure
Causal predictionPermitted preceding tokensRecorded successorsContinue sequences from prefixes
Masked-token recoveryCorrupted sequence with surrounding contextOriginal selected tokensUse bidirectional context to recover missing content
Span corruptionInput with sentinel-marked gapsRemoved spans in target orderRecover longer missing regions
Fill in the middlePrefix and suffix arranged before the missing spanThe removed middleGenerate content constrained by both sides

Fit is narrower than capability

Lower loss establishes better prediction under the implemented scoring procedure. It does not by itself establish truthful answers, reliable reasoning, safe action, or product value. TruthfulQA, for example, was designed around plausible falsehoods that can be well represented in human text: accurate imitation of a distribution is not the same property as factual correctness. The general objective warning belongs to Machine Learning Fundamentals.

II. How staged training developed

Must-know turning points

Predictive training long predates the modern foundation-model pipeline. The important history is not a march through product releases; it is a sequence of bottlenecks that prompted new forms of representation sharing, transfer, scale allocation, and checkpoint reuse.

From shared representations to checkpoint reuse

  1. 2003Neural Probabilistic Language ModelJointly learns word representations and next-word probabilities so related sequences share statistical strength.Sources & context

    Contributors: Bengio and colleagues.

    What changed: Addresses sparse observations beyond short-history n-gram counts: learned representations let observed sequences inform predictions for related unseen sequences.

  2. 2006Deep Belief NetsGreedy layer-wise learning initializes a difficult multilayer generative model.Sources & context

    Contributors: Hinton, Osindero, and Teh.

    What changed: Preliminary learning makes subsequent whole-model refinement practical in the studied digit-image setting. This is initialization-oriented pretraining, not broad language pretraining.

  3. 2018ELMo, ULMFiT, and GPTELMo reuses frozen contextual features; ULMFiT adapts on target text; GPT transfers a shared transformer to supervised tasks.Sources & context

    Contributors: The ELMo and ULMFiT research teams; OpenAI’s generative-pretraining team.

    What changed: These approaches establish distinct reuse strategies rather than a single replacement sequence. ULMFiT uses an LSTM; ELMo can adapt its language model before freezing it. OpenAI’s June 11 announcement describes language-model training followed by supervised adaptation of the same transformer core.

  4. 2020Scaling Laws for Neural Language ModelsFits relationships among prediction loss, model size, data, and compute.Sources & context

    Contributors: Kaplan and colleagues.

    What changed: Makes allocation a quantitative design problem. Under the studied accounting and regimes, the compute-efficient prescription favors larger models stopped well before convergence.

  5. 2022ChinchillaEqual-compute comparisons favor scaling training tokens and parameters in roughly equal proportion.Sources & context

    Contributors: Hoffmann and colleagues.

    What changed: Compared with the earlier fitted prescription, the results favor more tokens relative to parameters. Training horizons and learning-rate decay are matched in the comparisons; the result is an empirical allocation model, not a universal constant.

  6. 2024–2026OLMo 2, GLM-4.5, and Krea 2Use midtraining for different later capability-focused stages, without a shared operational boundary.Sources & context

    Contributors: The OLMo 2, GLM-4.5, and Krea teams.

    What changed: OLMo 2 changes its curated mixture while annealing; GLM-4.5 introduces repository and reasoning structures with longer contexts; Krea 2 places capability-focused midtraining before SFT. The interval groups these recipes, not an invention date. The OLMo 2 report is a January 2025 preprint; Krea’s report is dated June 2026.

Notice how the purpose expands from sharing statistical strength and initializing networks to transferring capabilities, allocating compute, and extending existing checkpoints. Spacing is not to scale.

Autoregressive, masked, and text-to-text objectives continue to coexist because they expose different information and serve different architectures. Similarly, freezing a representation, adapting the whole model, and continuing self-supervised training remain alternative reuse strategies. Newer practice expanded the design space; it did not make every earlier approach obsolete.

III. Designing model exposure

Mixture weight is not corpus size

A data mixture is a sampling distribution over source groups. It is not merely a list of datasets. A source can contain many eligible bytes yet receive little training exposure, while a small source can be sampled repeatedly. Eligibility and lineage belong to A release is a dependency graph; this chapter begins after a source has become eligible for a particular run.

Selection becomes exposure

Consider two sources selected equally often by document. If source A’s documents contain 100 scored tokens and source B’s contain 900, consuming whole documents yields about 10% versus 90% of processed tokens. If the loss is then averaged across every valid token, B also supplies roughly nine times as many loss terms. Equal document-selection probability, equal token exposure, and equal source-level loss influence are therefore different policies.

From eligible sources to update influence

Source size and selection probability pass through document length, packing, masking, and reduction before becoming loss influence.

Eligible source AA source with its own size and document-length distribution.
Eligible source BAnother source whose size need not match its sampling weight.
Mixture samplerSelects sources and examples under declared probabilities.
Tokenize and packDocument length and packing determine realized token exposure.
Mask and reduce lossOnly scored positions enter the declared aggregation.
Parameter updateCombined loss contributions produce gradients used by the optimizer.
Sampling, document length, packing, masking, and reduction successively determine influence. Gradient magnitude is not simply proportional to token count.
Read the diagram as text
  • Eligible source A. A source with its own size and document-length distribution.
  • Eligible source B. Another source whose size need not match its sampling weight.
  • Mixture sampler. Selects sources and examples under declared probabilities.
  • Tokenize and pack. Document length and packing determine realized token exposure.
  • Mask and reduce loss. Only scored positions enter the declared aggregation.
  • Parameter update. Combined loss contributions produce gradients used by the optimizer.
  • Eligible source A → Mixture sampler: eligible examples.
  • Eligible source B → Mixture sampler: eligible examples.
  • Mixture sampler → Tokenize and pack: selected documents.
  • Tokenize and pack → Mask and reduce loss: packed token positions.
  • Mask and reduce loss → Parameter update: aggregated prediction error.

Coverage versus repetition

Upsampling increases a source’s opportunities to affect updates but also increases repetition. Downsampling preserves budget for other sources but may reduce coverage. Multilingual XLM-R experiments illustrate the tension: stronger smoothing increased exposure for low-resource languages, whereas less smoothing favored high-resource languages; fixed capacity also created dilution tradeoffs as more languages were included. No sampling exponent is universally correct.

Schedules change exposure over time

A training schedule says how update conditions vary with progress. Learning-rate schedules change update scale. Data schedules change source or task exposure. Sequence schedules change the length and structure of examples. A curriculum changes the distribution of examples—often, but not necessarily, from easier to harder cases. These are separate interventions even when one training recipe changes several together.

A phase record keeps unlike schedule dimensions separate.
Schedule dimensionWhat changesQuestion it creates
Learning rateUpdate scale over stepsWas adaptation speed or instability caused by rewarming or decay?
Source sharesSampling distribution over data groupsDid a later capability change because of composition or recency?
Sequence lengthTokens and relationships available per exampleDid longer context help, or did extra computation and changed data structure confound the result?
Final annealingLate exposure and learning rateWhich effect came from curated data, reduced learning rate, or their combination?

Order matters because neural optimization is path-dependent: each update changes the state on which later gradients operate. Equal cumulative tokens therefore do not imply equivalent checkpoints. Attribution becomes especially weak when a phase simultaneously changes its mixture, sequence construction, learning rate, and token budget. A useful comparison changes one hypothesis at a time or reports the bundle honestly as a recipe.

IV. Allocating finite training work

Parameters, tokens, and compute form a budget

Training work couples model capacity with exposure. For a simplified dense transformer, a common approximation is

A conditional allocation model

C≈6ND, C \approx 6ND, where CC is training floating-point work, NN is the parameter count, and DD is the number of processed training tokens. Holding this approximation fixed creates a tradeoff: more parameters leave room for fewer processed tokens, and more tokens require a smaller model. Architecture, sequence operations, sparsity, utilization, and accounting conventions qualify the approximation.

Why fitted optima differ

Kaplan-style fits favored relatively rapid parameter growth and shorter training under the studied setup. Chinchilla’s equal-compute comparisons and matched learning-rate horizons favored roughly proportional growth of parameters and tokens. Later reconciliation work showed that parameter accounting—especially embeddings in small models—can materially change fitted prescriptions. These are empirical models of measured regimes, not physical constants.

Freshness and lifetime demand

Processed tokens are not necessarily distinct information. Controlled repetition studies found that repeated data can remain useful but has declining marginal value; some heavily repeated runs developed rising validation loss. Nor is training loss always the final economic objective. When a model will answer many requests, a smaller model trained longer may reduce lifetime inference work, but that conclusion depends on forecast demand and on loss being an adequate quality proxy. Replit described this practical tradeoff for latency-sensitive code completion: spend more training work on a smaller model so later inference can remain fast, with training cost amortized across use.

Parallel execution sets feasibility

Large runs can exceed one accelerator because model parameters and attention activations consume memory. Distributed execution partitions work or state across devices. Its implementation belongs to Distributed Training and Inference; here, the relevant point is that parallelism defines which model, sequence length, and batch are feasible within a time budget.

Removing one bottleneck can reveal another.
ConstraintObservable consequencePlanning implication
Parameter stateModel does not fit on one devicePartitioning parameters may make placement feasible.
Attention activationsOut-of-memory remains after parameter shardingActivation or context parallelism needs a separate intervention.
Inter-node communicationAccelerators wait during synchronizationTopology and node count constrain useful scaling.
Input pipelineLow utilization while batches loadStorage and preprocessing throughput limit realized progress.

Adding devices does not guarantee proportional speedup. Global batch size can become too large for the intended optimization regime, and accelerators can be underused. Stage planning should therefore use measured tokens per second and utilization—not accelerator count alone.

V. Checkpoints as evidence

A checkpoint is more than weights

A model checkpoint identifies learned parameter values at a point in training. Those weights may be sufficient for inference when the architecture and tokenizer are separately known. A resumable training checkpoint has a larger responsibility: it may need optimizer moments, scheduler state, random-number-generator state, gradient-scaler state, progress counters, and the sampler or dataloader position that determines what examples come next.

Checkpoint contents correspond to different recovery claims.
Preserved stateWhat it supportsWhat remains separate
Model parametersLoad learned weightsArchitecture, tokenizer, and behavioral evidence
Optimizer and schedulerContinue update dynamicsExact next examples and random operations
RNG and data-iterator positionReconstruct subsequent sampling more faithfullyEnvironment-dependent numerical execution
Corpus order and configuration lineageExplain what the checkpoint encounteredA guarantee of identical rerun results

Identity and lineage

Logical restoration is not a promise of bitwise-identical continuation across releases, platforms, or CPU and GPU implementations. PyTorch explicitly limits such reproducibility guarantees. For experimental use, give each checkpoint immutable identity and parentage, then bind it to tokenizer, code revision, data release, mixture, schedule phase, cumulative exposure, and evaluation records. This is analogous to identifying what software was tested and released, but a checkpoint is not merely a source commit.

Checkpoint ancestry binds training and evidence

Example

Immutable checkpoint nodes make branches, rollback points, and exact evaluation targets visible.

Checkpoint C0Starting learned state with immutable identity.
Run A manifestMixture A, schedule A, code revision, and processed-token budget.
Checkpoint C1-ABranch produced by Run A with resumable-state inventory.
Run B manifestMatched alternative with one declared intervention changed.
Checkpoint C1-BAlternative descendant retained for comparison or rollback.
Evaluation receipt AMetrics and artifacts bound to C1-A.
Evaluation receipt BMetrics and artifacts bound to C1-B.
Schematic branches bind each checkpoint to tokenizer, code, data release, mixture, schedule, cumulative exposure, and available resume state. Evaluation evidence remains specific to its exact checkpoint and protocol.
Read the diagram as text
  • Checkpoint C0. Starting learned state with immutable identity.
  • Run A manifest. Mixture A, schedule A, code revision, and processed-token budget.
  • Checkpoint C1-A. Branch produced by Run A with resumable-state inventory.
  • Run B manifest. Matched alternative with one declared intervention changed.
  • Checkpoint C1-B. Alternative descendant retained for comparison or rollback.
  • Evaluation receipt A. Metrics and artifacts bound to C1-A.
  • Evaluation receipt B. Metrics and artifacts bound to C1-B.
  • Checkpoint C0 → Run A manifest: starts training branch.
  • Run A manifest → Checkpoint C1-A: produces.
  • Checkpoint C0 → Run B manifest: starts matched branch.
  • Run B manifest → Checkpoint C1-B: produces.
  • Checkpoint C1-A → Evaluation receipt A: assessed as exact version.
  • Checkpoint C1-B → Evaluation receipt B: assessed as exact version.

Measure gains and regressions separately

Training loss measures fit on optimization examples. Held-out prediction loss estimates the same kind of prediction on separate data. Targeted capability assessments test behavior the stage was meant to improve. Broad regression suites test capabilities the starting checkpoint already possessed. Application assessments add workload, latency, cost, and human consequences. These layers answer different questions and should not be collapsed into one unexplained score.

One scalar cannot describe a checkpoint

Prediction loss and desired behavior can move differently. Fill-in-the-middle experiments found training choices with small loss differences but substantial differences on executable code infilling. XLM-R’s downstream task performance continued improving after validation perplexity had plateaued. Conversely, a small-model workshop reported falling training loss alongside rising held-out loss later in one run. The generalization contract is developed in Generalization to new cases.

A checkpoint selection record preserves each measurement in its native unit.
Evidence layerRequired bindingSupported claim
Held-out prediction lossCheckpoint, tokenizer, corpus, scoring contextFit to the specified prediction distribution
Target capability suiteExact cases, protocol, checkpointChange on the intended capability
Retention suiteStarting and candidate checkpoints on matched workKnown capability gains or regressions within suite coverage
Application assessmentComplete deployed configuration and workloadUsefulness under the evaluated operating conditions

Protect independent capability evidence

Contamination occurs when assessment information reaches parameter fitting or model selection through a path the evaluation intended to keep independent. Exact item overlap is only one route. Paraphrases, translations, benchmark-derived synthetic examples, shared source documents, published solutions, runtime retrieval, and repeated tuning against reported scores can all provide causal access to protected information. Protect the independent assessment develops the evaluation boundary.

How protected assessments can influence a result

Contamination includes indirect paths through synthetic data, runtime tools, and adaptive checkpoint selection—not only direct training overlap.

Protected assessmentCases, source documents, solutions, and reported outcomes intended to remain independent.
Corpus constructionDirect overlap or transformed derivatives enter training data.
Runtime retrievalTools expose a solution or source during the evaluated trial.
Adaptive selectionRepeated scores guide schedules, prompts, or checkpoint choice.
Fitted parametersAssessment information can influence learned weights.
Evaluation trialRuntime access changes what information the system can use.
Selected checkpointSelection depends on the reused assessment.
Corpus inclusion, runtime access, and adaptive selection provide different information paths. A path breaches independence when the assessment contract intended to exclude it.
Read the diagram as text
  • Protected assessment. Cases, source documents, solutions, and reported outcomes intended to remain independent.
  • Corpus construction. Direct overlap or transformed derivatives enter training data.
  • Runtime retrieval. Tools expose a solution or source during the evaluated trial.
  • Adaptive selection. Repeated scores guide schedules, prompts, or checkpoint choice.
  • Fitted parameters. Assessment information can influence learned weights.
  • Evaluation trial. Runtime access changes what information the system can use.
  • Selected checkpoint. Selection depends on the reused assessment.
  • Protected assessment → Corpus construction: items or paraphrases copied.
  • Corpus construction → Fitted parameters: training exposure.
  • Protected assessment → Runtime retrieval: solution remains reachable.
  • Runtime retrieval → Evaluation trial: runtime information.
  • Protected assessment → Adaptive selection: results repeatedly inspected.
  • Adaptive selection → Selected checkpoint: model-selection influence.

Surface separation is not independence

Literal n-gram decontamination is useful but incomplete. Rephrased benchmark derivatives can evade surface-overlap detectors while still improving benchmark scores when included in training. Semantic detectors introduce their own retrieval and judgment errors. A detector can provide evidence of overlap; failure to detect overlap is not proof that none exists.

Scores can leak into selection

Selection is another access path. If engineers repeatedly choose schedules or checkpoints after inspecting one held-out suite, that suite has influenced development even when its rows never enter a gradient calculation. Preserve useful cases as regression tests, but use fresh, independently sampled cases for an ordinary final holdout claim. Corpus membership and transformations should remain traceable through data lineage.

Tool access can violate the same boundary at evaluation time. In SWE-rebench, checking out an old commit did not prevent an agent from reading future Git history; after that history was removed, the agent could retrieve the public issue and solution through other tools. The lesson is not that web access is always invalid. It is that allowed information paths must match the capability claim and be enforced across equivalent tools.

VI. Extending an existing model

Continue rather than restart

Continued pretraining performs further parameter updates from an existing pretrained checkpoint. It often retains a self-supervised objective family while changing the corpus, mixture, schedule, context regime, or token budget. The starting checkpoint supplies reusable capabilities, making continuation potentially cheaper than rebuilding them from initialization. It also supplies path dependence: the incoming data must modify a model that already represents many capabilities.

The inherited state creates choices

Continuation is not one mechanical operation. Rewarming the learning rate can speed adaptation while increasing forgetting; replaying earlier-distribution data can protect retention; resetting optimizer state changes the trajectory; tokenizer changes require compatible embeddings; and longer sequences alter both example structure and computation. Each belongs in the run contract and matched comparison.

What continuation can change

Concrete recipes illustrate the design space. Replit continued training a code model on filtered public Replit code with language emphasis matching its users. YouTube described continued training that linked text with semantic video identifiers and learned relationships from watch sequences. GLM-4.5 placed related files, issues, pull requests, and commits into longer repository-level contexts. These examples show changed exposure and sequence construction; none alone proves that continuation always dominates restarting.

Adapt to a target distribution

Domain adaptation aims to improve behavior on a target distribution. Domain-adaptive pretraining continues self-supervised training on domain-relevant material. Task-adaptive pretraining narrows the incoming material to unlabeled inputs associated with a target task; task labels are not its prediction targets. Both differ from instruction fine-tuning even when all are implemented with token-level cross-entropy, because their data and supervision contracts differ.

The target shift determines what coverage and retention evidence are needed.
ShiftNeeded exposureTarget evidenceRetention concern
DomainRepresentative documents, discourse, and workflowsTransfer across domain tasksGeneral capability and other domains
LanguageText varieties and relevant scripts or registersLanguage-specific and cross-lingual tasksCapacity dilution and high-resource languages
FormatRepositories, long records, structured events, or multimodal unitsTasks requiring those relationshipsShort-form or ordinary-text behavior
Task inputsUnlabeled inputs from the intended task distributionLater supervised task behaviorOver-specialization to one input family

Target gain and broad retention

Useful adaptation requires more than a glossary. ChipNeMo continued Llama 2 base models on chip-design code and documentation, improving domain assessments while reporting slight degradation on general benchmarks in detailed results. A Thomson Reuters example mixed legal and public data, retaining a majority representative of the earlier distribution; the reported run improved LegalBench by about five percentage points without reported general-capability loss, but the base model, mixture, and retention suite were not fully specified. These outcomes are recipe-specific evidence, not guarantees.

Retrieval, prompting, continued training, and behavioral post-training solve different problems. Frequently changing facts may belong in retrieval or current context; stable patterns repeatedly needed across many requests may justify parameter adaptation. Distribution shift provides the broader vocabulary for defining the target population.

Classify the stage by its contract

“Midtraining” is useful as a recipe label, but it has no field-wide operational definition in the inspected sources. It commonly names a weight-updating stage after broad pretraining and before response-oriented post-training. Continued pretraining names a mechanism, domain adaptation names a purpose, and midtraining usually names a stage’s role or position. A single run can satisfy all three descriptions.

Classify the contract before choosing the label.
RecipeStarting state and signalAuthors’ use of the stage
GLM-4.5Broad pretrained model; later prediction training on repository and reasoning structuresCalls several domain-focused, longer-context stages mid-training.
OLMo 2Initial checkpoint; changed curated mixture with learning rate reduced to zeroCalls the later capability-focused phase mid-training.
Krea 2Higher-resolution pretrained image model; undisclosed exact midtraining recipePlaces midtraining before SFT and uses it to move toward downstream use and add capabilities.
Llama 3Existing pretraining trajectory; long-context continuation and final annealingKeeps those stages within its pretraining recipe rather than requiring the label midtraining.
ChipNeMoLlama 2 base checkpoint; autoregressive prediction on chip-design materialDescribes domain-adaptive pretraining, followed separately by response adaptation.

Instruction formatting or synthetic provenance does not determine the category by itself. GLM-4.5 includes instruction data in stages it calls mid-training; another recipe might reserve instruction-response pairs for SFT. State the starting checkpoint, objective, loss mask, data form, breadth, sequence construction, and intended change. That contract tells readers more than the label.

Specialization can interfere with retention

Capabilities share parameters. Gradients that improve behavior on a narrow or shifted distribution can alter representations used elsewhere. Catastrophic forgetting names a severe loss of earlier capability, but retention is normally a collection of measured changes rather than one binary property.

Optimization and capability can disagree

The target gain–retention balance depends on incoming mixture, replay of earlier data, learning-rate restart, update magnitude, duration, and checkpoint selection. In controlled continuation experiments, learning-rate rewarming accelerated adaptation while increasing forgetting; replay comparisons kept continuation compute constant and improved earlier-data retention in some settings. ChipNeMo also shows that a lower training or validation loss under a higher learning rate need not translate into better task benchmarks.

Mitigate, measure, and retain rollback

Mixing broad data, limiting updates, using parameter-efficient adapters, and regression-aware selection can reduce risk, but none proves preservation. Even a small adapter constrains where updates occur rather than establishing that every unrelated behavior remains unchanged. Evaluate the target capability and retained capabilities on matched checkpoint versions, then stop or roll back according to predefined tolerances.

VII. Choosing the next stage

Write the training decision before the run

A corpus and accelerator budget do not constitute a training plan. Begin with an exact starting checkpoint and a measured capability gap. State why adding current context, retrieval, tools, prompting, or behavioral post-training is insufficient. Then freeze the capability hypothesis and the smallest training intervention capable of testing it.

A reviewable training decision record.
Decision fieldRequired statement
Starting stateImmutable checkpoint, tokenizer, architecture, prior exposure, and available resume state
Capability hypothesisSpecific behavior expected to improve and why parameter training is needed
Training contractObjective, loss masks, eligible corpus release, mixture, schedule, sequence construction, processed-token and compute budgets
Matched comparisonBaseline or branch that isolates the intended intervention as far as practical
EvidenceDevelopment metrics, protected target assessment, retention suite, and contamination controls
Decision pointsBranch checkpoints, stop criteria, acceptable regressions, and rollback target

Precommit what would cause the run to continue, branch, stop, or roll back. The latest checkpoint is not automatically the best one. A prior checkpoint may offer a better balance of target gain, retained capability, inference cost, and evidence quality. Preserve every assessment against immutable checkpoint identity so a later decision can be reconstructed.

The central discipline is simple: change a bounded part of the learning contract, measure the capability it was meant to change, protect evidence from the training and selection process, and retain a recoverable alternative. A stage name summarizes that contract only after the contract has been written.

Open questions

  1. How should a field-wide vocabulary distinguish continued pretraining, midtraining, domain adaptation, and early behavioral data when real recipes mix objectives and move the same data between stages? Progress would look like reports publishing operational stage contracts rather than labels alone.

  2. Can practitioners predict the specialization–retention frontier before running a costly adaptation? The difficulty is that shared parameters, data order, update magnitude, replay, and evaluation coverage interact. Progress would require controlled studies that vary these factors while reporting several target and retained capabilities in their native units.

  3. How can benchmark independence be established when pretraining corpora are incompletely disclosed and paraphrased or synthetic derivatives evade literal overlap checks? Progress would combine auditable lineage, protected sources, detector validation, and fresh confirmatory cases rather than claiming a perfect decontamination filter.

  4. What is the best allocation when unique high-quality data are scarce but lifetime inference demand is large? Repetition, model size, data quality, and expected serving volume change the objective in different ways. Progress would connect controlled data-constrained training curves to task quality and measured deployment economics.

Follow the curated reading path through the speakers and demonstrations behind this entry.

25 min

AI Engineer Summit 2023 · 2023

Building AI For All

Amjad Masad · Michele Catasta

Cited in this entry

Shows how one code-model program connected corpus filtering, repeated exposure, continued pretraining, and a smaller-model latency objective.

Watch talk

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

33 matching talks

TalkSpeakerEventYear
The Base Model is Dead

Cited in this entry

Varun SinghAI Engineer World's Fair 20262026
Abi AryanAI Engineer Summit 20232023
Diego CarpinteroAI Engineer Europe 20262026
Nan JiangAI Engineer World's Fair 20262026
Leo PekelisAI Engineer World's Fair 20242024
Jack MorrisAI Engineer Code 20252025
Jyh-Jing HwangAI Engineer World's Fair 20252025
2025 in LLMs so far

Transcript reviewed

Simon WillisonAI Engineer World's Fair 20252025
Ilan BigioAI Engineer Summit 20252025
Maxime LabonneAI Engineer Europe 20262026
Angelos PerivolaropoulosAI Engineer Europe 20262026
Low Level Technicals of LLMs

Transcript reviewed

Daniel HanAI Engineer World's Fair 20242024
Vikhyat KorrapatiAI Engineer World's Fair 20242024
Gus Martins, Ian BallantyneAI Engineer Europe 20262026
Devendra Chaplot, Devendra Singh ChaplotAI Engineer World's Fair 20242024
Sangwu LeeAI Engineer World's Fair 20262026
Yesu FengAI Engineer World's Fair 20252025
Aakanksha ChowdheryAI Engineer World's Fair 20252025
Richard SocherAI Engineer World's Fair 20262026
Max RyabininAI Engineer Europe 20262026
Lachlan Ainley, Humza IqbalAI Engineer World's Fair 20242024
Ievgen VakulenkoAI Engineer World's Fair 20242024
Ibragim BadertdinovAI Engineer Europe 20262026
Will BrownAI Engineer World's Fair 20262026
Samuel DentonAI Engineer World's Fair 20262026
Soheil FeiziAI Engineer World's Fair 20262026
Nathan LambertAI Engineer World's Fair 20252025
Olive SongEvent unknownUnknown
Freddy LecueEvent unknownUnknown
Deepak PathakAI Engineer World's Fair 20262026
Zixuan LiEvent unknownUnknown
Charles DickensAI Engineer World's Fair 20262026
Dhruv NathawaniAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
31 processed in full · 4 in the curated path
Automated source review
Passed
Metadata candidates
6 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. Training Compute-Optimal Large Language Models

    https://arxiv.org/html/2203.15556

  2. Reconciling Kaplan and Chinchilla Scaling Laws

    https://arxiv.org/html/2406.12907v2

  3. Language Models are Few-Shot Learners

    https://arxiv.org/html/2005.14165v4

  4. Optimizing LLMs for Speed and Memory

    https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization

  5. Krea 2 Technical Report

    https://www.krea.ai/blog/krea-2-technical-report

  6. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models

    https://arxiv.org/html/2508.06471

  7. Training language models to follow instructions with human feedback

    https://arxiv.org/pdf/2203.02155

  8. From token identifiers to next-token probabilities

    https://arxiv.org/html/1706.03762

  9. Megatron-LM GPT dataset implementation

    https://raw.githubusercontent.com/NVIDIA/Megatron-LM/main/megatron/core/datasets/gpt_dataset.py

  10. PyTorch CrossEntropyLoss

    https://docs.pytorch.org/docs/2.9/generated/torch.nn.CrossEntropyLoss.html

  11. DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining

    https://arxiv.org/html/2305.10429

  12. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    https://aclanthology.org/N19-1423.pdf

  13. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

    https://www.jmlr.org/papers/volume21/20-074/20-074.pdf

  14. Efficient Training of Language Models to Fill in the Middle

    https://arxiv.org/html/2207.14255v1

  15. TruthfulQA: Measuring How Models Mimic Human Falsehoods

    https://aclanthology.org/2022.acl-long.229.pdf

  16. A Neural Probabilistic Language Model

    https://jmlr.org/papers/volume3/bengio03a/bengio03a.pdf

  17. A Fast Learning Algorithm for Deep Belief Nets

    https://www.cs.toronto.edu/~hinton/absps/fastnc.pdf

  18. Deep Contextualized Word Representations

    https://aclanthology.org/N18-1202.pdf

  19. Universal Language Model Fine-tuning for Text Classification

    https://aclanthology.org/P18-1031.pdf

  20. Improving language understanding with unsupervised learning

    https://openai.com/index/language-unsupervised/

  21. Scaling Laws for Neural Language Models

    https://arxiv.org/html/2001.08361

  22. 2 OLMo 2 Furious

    https://arxiv.org/html/2501.00656v1

  23. Unsupervised Cross-lingual Representation Learning at Scale

    https://arxiv.org/html/1911.02116

  24. Curriculum Learning

    https://cognitivemedium.com/assets/rmnist/Bengio2009.pdf

  25. Effective Long-Context Scaling of Foundation Models

    https://aclanthology.org/2024.naacl-long.260.pdf

  26. The Llama 3 Herd of Models

    https://arxiv.org/html/2407.21783

  27. Simple and Scalable Strategies to Continually Pre-train Large Language Models

    https://arxiv.org/html/2403.08763

  28. Scaling Data-Constrained Language Models

    https://arxiv.org/html/2305.16264

  29. Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

    https://arxiv.org/html/2401.00448

  30. Building AI For All

    https://www.youtube.com/watch?v=ju73sWVtvU0&t=564

  31. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    https://www.youtube.com/watch?v=TUnPNY4E2fw&t=237

  32. Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

    https://www.youtube.com/watch?v=TUnPNY4E2fw&t=343

  33. Insights from Snorkel AI running Azure AI Infrastructure

    https://www.youtube.com/watch?v=LJa1SjCkYas&t=665

  34. Insights from Snorkel AI running Azure AI Infrastructure

    https://www.youtube.com/watch?v=LJa1SjCkYas&t=743

  35. Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

    https://www.youtube.com/watch?v=tQTB4MU_z8w&t=460

  36. Saving and Loading Models — PyTorch Tutorials

    https://docs.pytorch.org/tutorials/beginner/saving_loading_models.html

  37. Accelerate v1.7.0: Checkpointing

    https://huggingface.co/docs/accelerate/v1.7.0/en/usage_guides/checkpoint

  38. TorchData Stateful DataLoader Tutorial

    https://meta-pytorch.org/data/main/stateful_dataloader_tutorial.html

  39. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

    https://proceedings.mlr.press/v202/biderman23a/biderman23a.pdf

  40. PyTorch 2.9: Reproducibility

    https://docs.pytorch.org/docs/2.9/notes/randomness.html

  41. Training an LLM from Scratch, Locally

    https://www.youtube.com/watch?v=UsB70Tf5zcE&t=2910

  42. Data Quality is the Compute Multiplier

    https://www.youtube.com/watch?v=_PdK6x7PQNM&t=859

  43. Rethinking Benchmark and Contamination for Language Models with Rephrased Samples

    https://arxiv.org/html/2311.04850

  44. Language Models are Few-Shot Learners

    https://arxiv.org/html/2005.14165v4

  45. Generalization in Adaptive Data Analysis and Holdout Reuse

    https://arxiv.org/html/1506.02629

  46. Data Quality is the Compute Multiplier

    https://www.youtube.com/watch?v=_PdK6x7PQNM&t=265

  47. SWE-rebench: Lessons from Evaluating Coding Agents on Real Software Engineering Tasks — Ibragim Badertdinov, Nebius

    https://www.youtube.com/watch?v=wcUJWP6WpGM&t=679

  48. ChipNeMo: Domain-Adapted LLMs for Chip Design

    https://arxiv.org/html/2311.00176v5

  49. Building AI For All

    https://www.youtube.com/watch?v=ju73sWVtvU0&t=1074

  50. Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+ DAU

    https://www.youtube.com/watch?v=LxQsQ3vZDqo&t=366

  51. Domain and task adaptation change the training distribution

    https://arxiv.org/pdf/2004.10964

  52. Stuffing Context is not Memory, Updating Weights is

    https://www.youtube.com/watch?v=Jty4s9-Jb78&t=3202

  53. Decoding Mistral AI's Large Language Models

    https://www.youtube.com/watch?v=AUuktOQPWYg&t=698

  54. The Base Model is Dead

    https://www.youtube.com/watch?v=xbPriQWXtWM&t=813

  55. Stuffing Context is not Memory, Updating Weights is

    https://www.youtube.com/watch?v=Jty4s9-Jb78&t=2042

  56. Continual Learning for AI Agents: From Failures to Durable Improvements - Soheil Feizi, RELAI

    https://www.youtube.com/watch?v=2IxD9OB3XuQ&t=800

  57. Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

    https://aclanthology.org/2021.emnlp-main.98.pdf

  58. Z.ai GLM-4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

    https://www.youtube.com/watch?v=m6MF1OR_9kM&t=333

  59. The Base Model is Dead

    https://www.youtube.com/watch?v=xbPriQWXtWM&t=764

  60. Data Quality is the Compute Multiplier

    https://www.youtube.com/watch?v=_PdK6x7PQNM&t=381

  61. Building AI For All

    https://www.youtube.com/watch?v=ju73sWVtvU0&t=636