AI Engineer World's Fair 2026

The Base Model is Dead

Loading slides…

Read the talk

The Base Model is Dead

As reinforcement learning takes on reasoning and software tasks, pre-training recipes are changing to teach the representations and behaviors those tasks need.

From a talk by Varun Singh

Before you start: Familiarity with next-token prediction, supervised fine-tuning, and reinforcement learning will help; mixture-of-experts routing is explained where it matters.

What should a base model contain?

What should a base model learn before anyone teaches it to chat, reason, or use tools? The traditional answer was a broad representation of human knowledge, acquired by predicting text from a large portion of the internet. Varun Singh, introducing himself as Arcee AI’s pre-training lead, opens The Base Model Is Dead by immediately qualifying its title: the model is still needed; the familiar conception of its job is changing.

The training pipelines for Trinity Large Thinking, GLM-4.5, and GLM-5 all retain a pre-training phase. That shared starting point matters: later stages depend on representations learned before the model becomes an assistant or an agent.

Slide titled “What is a base model?” with an Arcee training flowchart on the left and GLM training diagrams on the right.Open full source frame
Training pipelines for Arcee Trinity Large Thinking, GLM-4.5, and GLM-5.

In a decoder-only transformer, next-token prediction supplies the learning objective. Given a prefix, the model learns to predict its continuation. Repeating that process across text builds representations of language and world knowledge. The GPT-3 paper illustrates a further consequence: a model trained this way can learn to use examples in its context to perform a task. This training is often called self-supervised learning because the text supplies its own targets; Singh also notes the unsupervised and supervised terminology used for it.

GPT-3’s recipe makes the original conception concrete. Common Crawl supplied a large web scrape, WebText2 supplied another, and books and Wikipedia contributed additional material. Web sources account for roughly 85% of GPT-3’s training mixture, using the paper’s sampling weights rather than raw corpus sizes. Singh then points to Llama 3, describing 50% of its training tokens as general knowledge. The categories are different, but both examples establish the prominence of broad knowledge in the starting model.

In that earlier division of labor, post-training largely taught the model how to surface what it already knew: follow a chat template, answer questions, and behave usefully in conversation. Reinforcement learning shaped the interaction more than it supplied new capabilities. Pre-training therefore consumed the bulk of the compute budget and largely determined the ceiling of the resulting assistant.

0:000:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

RL becomes a source of capability

The arrival of OpenAI o1 in 2024 and DeepSeek-R1 in January 2025 changed that allocation of responsibility. o1 demonstrated the potential of reasoning models; R1 made a route to building them more openly accessible. Reinforcement learning could now substantially improve task performance, rather than merely adjust conversational behavior. Singh uses o1’s AIME competitive-mathematics graphs to illustrate the shift.

Claude Code extended the practical setting to a terminal, where developers could ask a model to build applications. As function calling improved, the interaction became more than a question followed by an answer: a model could act within a software environment. Training that interaction end to end with RL lets the model learn how to work in the environment and construct software.

Slide juxtaposing two performance graphs with a DeepSeek-R1 paper heading and a Claude Code excerpt and partial terminal image.Open full source frame
“Everything changed”: reasoning-performance graphs, DeepSeek-R1, and Claude Code.

If RL is responsible for learning increasingly consequential behavior, pre-training should prepare the model to learn that behavior well. The question is no longer just how much knowledge a base model contains. It is whether a conventional web-trained model provides the best starting point for large-scale reasoning and agent training.

3:564:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:56 · section reference included

Two routes away from a web-heavy recipe

The emerging recipes do not agree on whether synthetic data is essential. Singh favors it, but presents the direction of the research as unsettled. His first example is MAI-Thinking-1, whose recipe he describes as excluding synthetic data and outputs from other language models, including filtering web scrapes for such material. This retains the idea of using human-produced knowledge to bootstrap representations.

Yet retaining human-produced data does not mean retaining the old mixture. Singh puts MAI-Thinking-1’s web-text share at 15%, compared with roughly 85% for GPT-3. His comparison illustrates a change in emphasis, rather than an independently established equivalence between the two recipes’ categories. Web text still matters, but code and STEM receive more attention as downstream uses demand those capabilities.

The alternative is to move data associated with post-training into pre-training itself. In the Nemotron 3 Ultra report, Singh highlights three categories prefixed with SFT in the recipe chart. These are question-and-answer or conversational examples of the kind normally associated with supervised fine-tuning. Putting them into the earlier mix exposes the model to the shape of conversations and expected downstream tasks from the beginning.

The two approaches disagree about data provenance while converging on a more task-oriented recipe:

Recipe directionData choiceIntended preparation
MAI-Thinking-1, as described by SinghHuman-produced data; less web textGreater code and STEM emphasis
Nemotron 3 Ultra, as interpreted by SinghSynthetic and SFT-style examples earlierFamiliarity with downstream conversations and tasks

Books and general web text occupy less of the mix. Singh contrasts GPT-3’s lack of a dedicated code dataset with the prominence of code in newer recipes. The shift in what a base model is for does not depend on every lab choosing synthetic data.

5:596:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:59 · section reference included

Rephrasing changes the exposure, not just the token count

Synthetic data raises a legitimate concern: indiscriminately feeding generated text back into training can degrade performance or contribute to model collapse. Singh’s concrete counterexample is Trinity Large, where he reports using synthetic data at web scale, largely through rephrasing. Start with a seed item and generate multiple expressions of the same information. Adding those variants to the mix increases exposure to the information without requiring the model to encounter exactly the same wording each time.

At the record level, the operation can be represented by keeping a seed identifier attached to every variant. In this illustrative Python example, the candidate additions express the same fact in different forms:

python

seed = {
    "id": "dict-keys",
    "text": "Python dictionary keys must be hashable.",
}
rephrases = [
    "A Python dictionary requires hashable keys.",
    "To use an object as a dictionary key in Python, it must be hashable.",
]

candidate_records = [
    {"seed_id": seed["id"], "text": text}
    for text in [seed["text"], *rephrases]
]

This constructs candidate training records; the operation itself does not establish that generated variants preserve the seed’s meaning. The mechanism Singh describes is repeated exposure to the same information through different expressions.

Singh cites Kimi K2 as a larger-scale example of rephrasing applied broadly across a pre-training dataset. He also points to SwallowCode and SwallowMath, introduced in Rewriting Pre-Training Data Boosts LLM Performance in Math and Code, as earlier examples of the approach.

Slide titled “The advent of synthetic data for pretraining,” containing highlighted Trinity Large text, research excerpts, and a colored rephrasing flowchart.Open full source frame
Synthetic pretraining data: research excerpts and a chunk-rephrasing diagram.

The intended benefit is therefore broader than acquiring more tokens. Rewriting can clean up material, improve its usefulness, and present it in instruction or agent-oriented forms. Those forms let the model begin learning downstream task representations during pre-training, rather than first encountering them after the base model is finished.

9:419:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:41 · section reference included

Earlier exposure can reduce later distribution shifts

For mixture-of-experts models, moving downstream data earlier also has a routing motivation. Experts specialize over training, while load-balancing objectives encourage broadly equal utilization across a batch or a sequence, depending on the objective. A model trained on one distribution can route a very different distribution unevenly when post-training begins.

Singh cites MAI-Thinking-1 as an example of a distribution shift producing large expert imbalances, and describes the response as increasing the balancing coefficient during SFT. The report is more specific: it gives a load-balancing coefficient of 1e-2 during self-distillation and 1e-5 during RL. Singh’s preferred direction is to expose the model to relevant data early enough that it learns stable representations, reducing the need to force a major adjustment to expert balance late in training.

Mid-training addresses a related mismatch. It exposes the model to distributions it will encounter during post-training and RL, often at longer context lengths. Longer contexts admit agentic traces into the training mix, preparing the model for interactions that extend beyond a short question and answer. As pre-training itself adopts longer contexts, Singh argues that these datasets can also move earlier, allowing the corresponding representations to develop from the start.

11:2011:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:20 · section reference included

Separate the learning objectives from the stage names

Once conversational data, agentic traces, and longer contexts can appear at multiple stages, the labels pre-training, mid-training, and post-training become less informative. Singh proposes a simpler distinction between two learning paradigms:

  • Supervised next-token learning: learn from supplied sequences and their continuations.
  • Reinforcement learning: improve behavior through the training environment and its feedback.

The important relationship is how the first prepares the model for the second, wherever a dataset happens to enter the schedule.

Compute allocation makes that relationship consequential. Citing an interview with the head of Xiaomi’s MiMo Labs, Singh reports roughly equal pre-training and post-training compute allocations for the final model. He presents Composer 2.5 as a more extreme case, claiming that its RL compute exceeded its supervised-learning compute. Cursor’s public description groups additional training and RL together, so it does not isolate the RL-only comparison.

As RL takes a larger role in the budget, supervised learning acquires a different optimization target: build representations that make subsequent RL effective. The base model still matters, but its value increasingly includes what it enables the next learning process to discover.

13:2013:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:20 · section reference included

Give RL skills it can compose

What must the supervised phase supply? Singh highlights research suggesting that the base model needs exposure to atomic skills that RL can later compose. With an environment of sufficient difficulty, RL can learn to extrapolate from those ingredients. This makes the training distribution important at two levels: the model needs useful component skills, and the RL environment needs to demand something beyond merely reproducing them.

The familiar AlphaGo comparison illustrates the possibility of reinforcement learning eventually overtaking supervised learning. But Singh leaves its applicability to language models unresolved. Human language is an exceptionally broad distribution to acquire through RL alone. Less supervised learning and more RL remain a possible direction, not a demonstrated endpoint for LLM training. Even without reaching that endpoint, thinking of a base model as a collection of skills available for RL becomes increasingly useful.

14:4415:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:44 · section reference included

Prepare the model for the behavior you want next

Some of the useful ingredients may have forms that are uncommon in ordinary human writing. Reasoning traces are one example: their structure differs from much of the text that a broad web scrape would supply. Supervised learning can introduce these unfamiliar forms before RL asks the model to use them effectively.

The same idea extends to test-time compute schemes. Singh describes warming a model up to such schemes during SFT, or even during pre-training, using excerpts he attributes to a Zyphra report. The underlying purpose is to choose data that prepares the model to explore effectively during RL. Supervised data can teach not only information, but also forms of behavior that later training will develop.

Slide titled ““Novel” data during supervised learning” with two paper excerpts and blue highlighting in the test-time compute passage.Open full source frame
Reasoning-aware training and test-time compute in supervised-learning data.

That changes the meaning of the base model: from a general prior over human knowledge toward a prior for reasoning and agentic behavior. Singh acknowledges that this framing reflects the current prominence of reasoners and agents. If a different way of interacting with models becomes important, its demands should shape the prior instead. The durable design question is what the model’s eventual use requires it to be ready to learn—not how closely its training corpus resembles a massive scrape of the web.

15:4615:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:46 · section reference included

Resources

From the talk

  • Zyphra's report on a reasoning-focused mixture-of-experts model with 700M active and 8B total parameters.

Transcript download
Full transcript

Loading transcript…