AI Engineer World's Fair 2026
Scaling Compute on Context
Loading slides…
Read the talk
Scaling Compute on Context
A model can know the public literature yet lack depth in your domain. Learning from private context requires more than fitting documents: it requires training tasks that keep improving with the model.
From a talk by Jack Morris
Before you start: Familiarity with language-model pretraining, context windows, and gradient-based training will help with the approach survey.
Where does depth come from?
How can AI connect distant mathematical literatures yet lack the intuition of someone who has spent years working in one specialty? Jack Morris opens with this contrast, drawing on his account of Terence Tao’s experience using AI for mathematics. A model’s broad reading lets it bridge topics that no individual could know equally well. But breadth does not necessarily produce the practiced judgment of a graduate student who has spent five years in one area.
Scaling compute on context is the pursuit of that depth. For Morris and Engram, training personal or organizational data into models means more than adding facts. The goal is to acquire skills and a deeper understanding of a particular domain—the kind of understanding that makes those facts useful in unfamiliar situations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What public training leaves out
Three different gaps motivate continued learning:
- Recent information. A model cannot learn during pretraining about events that happen afterward. Morris uses the question of whether Mexico won its game the previous night; he does not supply the result.
- Rare skills. Morris points to writing AMD GPU kernels as a difficult specialty with relatively few good public examples. A capability can remain weak when the material needed to learn it appears rarely in training.
- Private context. Your emails, writing preferences, annual family vacations, and a company partnership with SunTrust Bank are not facts a general model can be expected to acquire from public sources.
Morris treats this last gap as a central limitation of the prevailing training paradigm: better general models do not automatically acquire personalized knowledge after training. The question is therefore not only how to give a model another fact, but how to teach it enough about a person or organization to develop expertise in that context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Applying scale to your data
This research territory has many overlapping names: sleep-time compute, continual learning, neural memory, write-time compute, note-taking, dreaming, studying, machine studying, and amortized inference. Morris groups them around a shared ambition rather than treating them as identical technical definitions. The unsettled vocabulary reflects a field whose methods are less established than pretraining and post-training.
Conventional model scaling has three main axes:
| Axis | What increases |
|---|---|
| Data | The amount of training material |
| Compute | Training time or computational work |
| Model size | Capacity to acquire information |
These axes have supported learning from Wikipedia, Reddit, arXiv, GitHub, and expert-created post-training data from companies such as Scale AI, Surge AI, and Mercor. Morris uses public broadly here to mean knowledge that a general model can share across users, not that every training dataset is publicly downloadable.
The METR study of long-task capability provides a motivating picture: newer models can complete tasks that take humans longer. The relevant duration is the human time required for a task, evaluated at a specified model success rate—not how long the model itself runs. Morris attributes the capability trend to scaling; the trend alone does not establish that causal explanation.
The missing connection is to your own data. More training, more capacity, and more general data can improve public-code programming and textbook mathematics without teaching a model anything new about your company, work, or life. General capability growth does not automatically create private domain knowledge.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A fixed corpus and a pretrained model
Suppose Engram wants a model to understand its own company. The existing company corpus gives it a limited data budget: it cannot simply manufacture more historical company information. Training a model from scratch would also discard an enormous amount of useful outside-world knowledge. Starting from pretrained weights is the practical choice because understanding internal work depends on understanding the world around it. With the corpus and starting model given, compute becomes the central variable to scale.
The fixed-data assumption is an idealization. Someone studying a textbook can find another textbook, search the web, or talk to an expert. Someone learning a language can speak with people who know it. A learning system could likewise seek additional evidence. Morris keeps that possibility open while isolating a narrower question: how much more can a model learn by spending additional compute on the same context?
The abstract setup has two inputs: a pretrained model with parameters θ and a large unstructured corpus D. That corpus might contain every email a person has written or every meeting transcript a company has accumulated. The desired output is a new model, θ★, that knows D.
The word knows carries much of the research problem. Memorizing a document, answering questions about it, and using it to perform new work are different achievements. Morris leaves the criterion open rather than specifying an evaluation, and uses the setup to organize several possible approaches.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Fitting documents is not the same as using them
The simplest attempt is to continue next-token training directly on D. Morris argues that this works poorly for the narrow corpora under discussion, while allowing that a sufficiently broad dataset could begin to reproduce the effects of pretraining. Consider his financial-report example: the goal is to encode the reports in the model’s weights so it can answer questions about them and create new reports. Repeatedly predicting the documents’ next tokens does not directly train all of those behaviors.
In Morris’s illustrative financial-report example, training loss can reach about 0.0001 while generation collapses. This is not a reported benchmark with a specified model, loss definition, or evaluation protocol; it illustrates the gap between fitting the training text and using its contents.
There are two separate problems. First, repeated training on a finite corpus eventually exhausts its learning signal, so it cannot supply an indefinite scaling axis. Second, the resulting fit may not generalize to the interactions the user wants. Morris describes a model that struggles to answer a question unless the question and answer already occur together in the training material. The desired question-answering behavior is broader than reproducing the document.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keeping the effect of context in a smaller representation
A different starting point is the effectiveness of in-context learning. If a model answers well when the relevant documents are in its prompt, perhaps a smaller representation can preserve that behavior. Morris compares the idea to compaction in coding agents such as Claude Code, Codex, and OpenCode. Here, however, the proposed representation is a compact set of attention keys and values, rather than necessarily a textual summary; the analogy does not establish how those coding tools implement compaction.
The slide points to Fast KV Compaction via Attention Matching. Morris mentions both learned compression and a greedy approximation approach. The aim is to represent the source context succinctly enough that the model can behave as though it still has access to the larger body of text.
For the direct compaction approach he is considering, Morris identifies two limitations: the source data must fit into context, and compression may miss benefits obtainable through gradient-based learning. The context-size restriction belongs to this approach, not to every learned context representation. That distinction matters as the survey moves from compressing an existing context to training on interactions derived from it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Training the behavior that context enables
Context distillation moves the target from reproducing documents to reproducing behavior enabled by those documents. Show a model the text, then train a system to answer as though that text were still available. Morris connects this to on-policy distillation and to updating the model during training. More precisely, on-policy refers to learning from trajectories sampled by the current student policy; context conditioning and on-policy sampling are separate choices.
The next question is what interactions to train on. Raw documents do not directly provide the question-answer exchanges that users will need. Cartridges addresses this through self-study: generate question-answer interactions conditioned on D, then train the system to behave as if it were seeing D while answering. In Cartridges, the trained representation is a compact KV cache; this does not necessarily write the corpus into the base model’s weights.
This gets closer to useful behavior than simply repeating document text. But the selected training interactions still determine what the system practices. Producing a useful set of questions is not yet a method for continually discovering everything else worth learning from the corpus.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Simulating pretraining, then constructing RL tasks
Pretraining remains a compelling model of knowledge acquisition. Morris points to the abilities of GPT-5 and to asking Claude about results in his own published papers: the model can recall knowledge acquired from the literature. That suggests another route—generate synthetic material conditioned on D, then continue training θ on that material as though extending pretraining. The synthetic corpus supplies additional ways to encounter and use the original information.
Morris sees promise here, but also practical costs. Continued pretraining can overwrite earlier learning and can be difficult to scale. It may require another post-training stage afterward to recover the desired assistant behavior. Starting with an already post-trained model, rather than a suitable base model, complicates that sequence.
A further alternative is to construct unsupervised reinforcement-learning environments. Instead of training with a distillation objective, the model learns through an RL objective such as GRPO. Morris introduces this as a related direction, without specifying an environment, reward design, or concrete result. Changing the objective creates another way to learn, but does not by itself resolve the common limit of these approaches.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What happens after the model fits the training data?
The desired property is stronger than successful initial adaptation: additional compute should keep producing additional domain depth. Morris argues that the surveyed approaches do not yet provide this property. Each defines training data or tasks, then optimizes against them. Unless the model is underparameterized, it can eventually fit that material, and improvement saturates.
Synthetic data creates its own version of a data wall. A model may learn all the generated examples without learning everything useful about the source corpus D. More optimization against those same examples then stops helping. Generating a dataset once has changed the available training material, but has not created a process that keeps renewing it.
The crucial question becomes what the second stage of training should look like. Attention matching, self-study, or continued pretraining may each teach something about D. After that first fit, however, the system needs a way to create another productive learning task. Otherwise, more compute no longer buys more depth.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Making the learning tasks improve with the model
AlphaGo supplies an analogy for an adaptive training process. As the system improves, self-play provides stronger challenges. Its training against earlier policy versions illustrates how a learning system can help create its own next tasks; it does not establish that an equivalent process will improve indefinitely on private documents.
The proposed loop is:
- Generate training data from the available context.
- Train the model so it becomes more capable.
- Use the improved model to generate better or harder training data.
- Repeat with a learning task that has advanced along with the model.
Morris identifies this recursive process as a major focus at Engram. The important change is that the model’s improvement should also improve the material it trains on next.
Early Engram experiments, Morris reports, produced plateauing curves: generating more data or training longer eventually stopped helping. He then describes more sophisticated ways to make training gradually harder. The slide’s blue curve represents the plateau; its rising red curve represents the desired relationship between compute and depth. It is a schematic research target, not a quantitative demonstration of sustained scaling or a disclosed complete algorithm.
For an email corpus, the intended result is a progressively richer internal understanding of the person behind the messages: what they do, what they are trying to achieve, and what they value. That is the depth this research seeks—learning from the same context in ways that make the model increasingly useful as it studies further.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
METR’s original study defines task-completion horizons using human task durations and model success rates.
Trains compact KV caches through synthetic conversations and context distillation.
The original AlphaGo paper combines neural networks, search, and reinforcement learning through self-play.
Further reading
- Introducing EngramArticle
Engram’s launch essay explains its goal of training models on personal and organizational context.
Code for training and evaluating reusable context representations with self-study.
Constructs compact keys and values by matching attention outputs and attention mass.
- On-Policy DistillationArticle
Explains distillation using student-generated trajectories and teacher feedback.
Full transcript
Loading transcript…