AI Engineer World's Fair 2026

Scaling to Long Horizons

Loading slides…

Read the talk

Scaling to Long Horizons

From Galactica’s base-model demo to agents that learn across extended tasks, long-horizon reasoning depends on memory, learning signals, realistic environments, and compute working together.

From a talk by Ross Taylor and Chengxi Taylor

Before you start: Basic familiarity with language-model pre-training and reinforcement learning will help; compaction, critics, policy lag, and bootstrapping are explained as they arise.

What turns a capable model into a usable product?

What turns a capable language model into something people can reliably use? Ross Taylor’s route into that problem began with Papers With Code. He describes a London startup acquired by Meta in 2019, followed by a transition into research on Galactica, then post-training for Llama 2 and Llama 3 after ChatGPT appeared. By the time of this talk, he and Chengxi Taylor were leading General Reasoning, pursuing the algorithms, environments, and compute needed for longer agent tasks. That history also shaped Ross’s conviction that small, focused teams can make substantial contributions even when model development demands enormous scale.

Slide titled “My AI journey started here” with a group dinner photo and a list including Papers with Code, Galactica, Llama 2, Llama 3, and ReasoningLlama.Open full source frame
The team and projects behind the speaker’s AI journey.

The product lesson became painfully concrete in 2022. Galactica appeared roughly two weeks before ChatGPT, whose initial system was based on GPT-3.5. Both had capable underlying models, but Galactica shipped a base-model demo. Its willingness to continue a prompt included hallucinations and compliance with unsuitable requests. ChatGPT added a crucial reinforcement learning from human feedback pipeline that shaped how the model responded to users.

A good base model is not enough. Ross treats the two launches as a personal natural-experiment analogy: similar underlying promise, very different public outcomes. It is not a controlled comparison isolating RLHF, but it explains why his attention shifted from base-model capability toward the training that makes that capability useful.

0:150:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

Strong results did not solve the behavior problem

Evidence for that distinction was already available in InstructGPT. On the study’s prompt distribution, human evaluators preferred outputs from 1.3B-parameter InstructGPT over those from 175B-parameter GPT-3. Ross rounds the smaller model to one billion parameters in the talk. The result concerns human preference after instruction tuning and human-feedback training, rather than superiority across all tasks or an isolated measurement of RL alone. A much smaller model could better satisfy the people using it.

Line chart plotting win rate against SFT 175B by model size, with series for PPO-ptx, PPO, SFT, GPT prompted, and GPT; an InstructGPT reference appears below.Open full source frame
InstructGPT results shown on the slide “RLHF made LLMs products.”

Galactica’s public demo exposed the opposite problem. Users could request a research paper about Dyson spheres or a report claiming benefits from eating crushed glass, and the model would produce text in the requested form. Ross recalls the backlash—and the Meta association—as overshadowing the technical work. The failure was consequential precisely because a scientific-looking answer could be generated without the behavioral judgment a public product needed.

Meanwhile, Ross reports strong scientific-domain results against PaLM, Chinchilla, and GPT-3.5 with less compute. The equation result has a more precise scope: on manually evaluated zero-shot LaTeX-equation generation, Galactica 120B scored 68.2%, versus 49.0% for text-davinci-002. That exact comparator matters; this was not an evaluation against the ChatGPT product. Ross also recalls a chain-of-thought comparison of roughly 36% for Galactica 30B versus 19% for PaLM 540B, without naming the benchmark in the spoken passage. These domain comparisons explain why the demo’s reception could not be reduced to a weak underlying model.

3:273:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:27 · section reference included

Curated data and computation before the answer

Galactica also supplied lessons about how to obtain those capabilities. Ross contrasts its curated 105B-token corpus with roughly one trillion tokens for Chinchilla. His emphasis is on the quality of the source data, not simply accumulating more tokens. Repeated training over that data challenged the contemporary convention of stopping after one epoch. He connects this experience to the later rule of thumb about four epochs of repeated data; that is a historical connection, not a claim that four epochs is always optimal. Corpus size and total exposure during repeated training are different quantities.

The next contribution concerned what happens between the question and the answer. Several approaches were emerging at the time:

ApproachRole of intermediate generation
Chain-of-thought promptingAsk for the steps leading to an answer
Numerical scratchpadsRecord intermediate calculations
Galactica’s thinking tokensDelimit an internal working-memory process

Ross presents Galactica’s tagged working space as an early way to make the model spend inference compute before answering. The practical distinction is that intermediate tokens become a place to work through the problem, rather than merely a stylistic request to explain the final response. He regards the data, repeated-training, and thinking-token ideas as pioneering contributions; their technical usefulness does not depend on resolving historical priority.

5:125:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:12 · section reference included

Optimizing the work between thinking tags

After Galactica, the team took on Llama post-training. Ross kept pursuing a narrower question: what if reinforcement learning optimized the intermediate work inside the thinking tags? He connects that intuition to later DeepSeek reasoning models and OpenAI o1, but the starting conditions were very different. Llama 2 had weak mathematical training data and a context window of roughly 4,000 tokens, leaving limited space for extended reasoning.

The unpublished training recipe addressed both the starting model and the learning objective:

  1. Continue pre-training on mathematics and science. Improve the base model’s ability to represent and solve the relevant problems.
  2. Apply PPO with verifiable rewards. Optimize against outcomes that can be checked.
  3. Initialize the value model from a strong outcome reward model. Give the critic a useful starting point rather than learning it entirely from scratch.

Ross explicitly distinguishes this recipe from GRPO: the value model was a central part of their PPO setup. He reports internal state-of-the-art math and reasoning results, but those experiments were unpublished.

Better scores did not produce everything the team hoped to see. They did not observe inference-time scaling or the characteristic reflection and backtracking later associated with DeepSeek-R1 and o1. An objective that improved answers had not yet produced a model that reliably used additional thinking to reconsider its approach.

When later reasoning models appeared, Ross revisited that gap. His explanation combines better base models, more RL compute, and larger context windows. An apparently similar research idea can behave differently once those prerequisites change. This is also a point about access: having a GPT-4-level model earlier can expose behaviors and research directions that remain invisible with weaker models. He presents this as a retrospective scaling lesson, not a controlled demonstration of which ingredient caused reflection.

That experience did not convince him reasoning was solved. It motivated the formation of General Reasoning to pursue larger tasks, and it sets up Chengxi’s next question: what changes when useful reasoning must continue far beyond one response or one context window?

6:236:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:23 · section reference included

Continuing beyond the context window

Cancer research, Millennium Prize problems, and reaching Mars are Chengxi’s motivating ambitions, not capabilities demonstrated here. Their common requirement is sustained work. A system meant to contribute to such problems must preserve progress through reading, experimentation, failed attempts, and new ideas rather than treating each response as a self-contained task.

She uses Fermat’s Last Theorem as an analogy: years of reading papers, writing scratch work, and stepping away to develop ideas. Her illustrative conversion puts such intellectual activity at tens or hundreds of billions of tokens, against the million-token context windows she uses as a contemporary reference point. This is an intuition for the scale mismatch, not a measured token count or a universal ceiling on context length.

Compaction lets work continue despite that mismatch:

  1. Generate within the available context.
  2. Summarize the accumulated context as the window fills.
  3. Carry that summary into fresh context and continue generating.

The summary becomes part of the state on which later decisions depend. Chengxi proposes applying reinforcement learning to both the task policy and the compaction behavior, so the system learns not only how to act but also what to preserve for subsequent work.

Slide titled “One solution is compaction” shows existing context summarized into a green box, then carried above a new context box. Text describes learning compaction and task policies using reinforcement learning.Open full source frame
Compaction carries a summary into new context.
9:329:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:32 · section reference included

Learning from long trajectories and retaining their history

Continuing generation does not by itself make a long trajectory easy to train on. Chengxi identifies three optimization problems:

  • Gradient variance: longer trajectories increase the variability of the learning signal.
  • Sparse rewards and credit assignment: a result arriving much later gives limited direct information about which earlier decisions helped.
  • Variable trajectory lengths: episodes of different lengths complicate optimization.

These problems concern learning from an extended sequence of actions, even if memory is sufficient to keep acting.

A critic, or value model, estimates how promising a state is. Chengxi proposes it as a way to reduce variance, work naturally with compacted trajectories, and encourage batch diversity. It also supports bootstrapping: using an estimate to obtain a learning signal before the episode ends. The cost is an additional model to train alongside the policy, making the setup more complicated than GRPO. The critic’s role in keeping training moving becomes especially important when episodes are too long to wait for their final rewards.

External memory provides complementary ways to retain information:

  • Filesystem tools act as a scratchpad for reasoning that need not remain in the current context.
  • Self-search tools retrieve material from the agent’s previous trajectory.
  • Archive tools preserve results that later work can build on.

Archives require a task-specific boundary. Reusing an earlier result can be the point of automated research, where knowledge should accumulate. In another evaluation, retrieving an existing answer could bypass the reasoning the environment is supposed to test. Memory access therefore changes the definition of the task, not just the amount of information available to the model.

10:5611:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:56 · section reference included

A season of decisions, not a single correct answer

How well do frontier models handle this kind of extended task? General Reasoning built KellyBench, which Chengxi says attracted Financial Times front-page coverage. Agents build machine-learning models for trading on Premier League football matches over a one-year horizon. The appeal is an objective with financial meaning: sports betting connects forecasts and decisions to returns rather than stopping at a competition score. The suggested commercial upside is motivation, not a demonstrated outcome.

In the presented KellyBench evaluation, all evaluated frontier models had negative average returns from a normalized £100,000 starting bankroll in a historical-market simulation. The distinction matters: this was not live money lost by deployed agents, and an average loss does not mean every run lost money. The result remains a demanding test of whether model-building and repeated decisions add up to a successful season; the current public leaderboard has subsequent updates and should not be substituted for the talk’s results.

Chengxi’s diagnosis begins with the industry’s emphasis on coding and procedural work. A narrowly specified repair or implementation task often leaves relatively few acceptable solutions. Success there need not establish the ability to search creatively through a much larger space of possible strategies.

Open-ended environments add two further requirements. First, the agent must handle uncertainty and complexity that a tightly specified task may remove. Second, it must operate among other actors with different beliefs and objectives. Chengxi argues that current benchmarks and simulations do not capture enough of these interactions. A longer sequence of procedural instructions alone would not supply the real-world fidelity she wants.

Slide titled “Why are models bad at long-horizons?” lists industry bias toward coding and procedural tasks, insufficient focus on open-ended tasks, and insufficient long-horizon tasks with real-world fidelity.Open full source frame
Three proposed reasons models struggle with long horizons.
12:2412:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:24 · section reference included

Keeping GPUs busy while episodes continue

Long horizons also change how scarce GPUs should be divided between inference and training. In a sequential RL cycle, inference produces trajectories, then training updates the policy. As trajectories lengthen, waiting for generation to finish can leave training resources idle. Pipelined RL overlaps those activities: training starts on available sequences while additional sequences are still being generated.

The overlap creates off-policy data. A trajectory can have been generated by an older policy than the one currently being trained. Chengxi reports that, in their experience, up to eight steps of policy lag is normally acceptable. That is an empirical tolerance for their setup, not a general bound. If inference were to take weeks, ongoing updates could exceed that tolerance before a trajectory completed, forcing training to wait again.

Bootstrapping offers another route. Before the episode finishes, the value model estimates the expected outcome, allowing training to use a signal without waiting for the terminal result. Chengxi likens this anticipatory signal to dopamine; the operational mechanism is an estimate of future value. Better utilization now depends on the accuracy of a learned prediction, introducing value-model bias.

Training approachWhat enables an updateMain cost
Wait for completed rolloutsFinished trajectoriesIdle time during long generation
Pipeline generation and trainingAvailable sequences while others continuePolicy lag
Bootstrap with a criticEstimated value before completionValue-model bias

These choices connect systems scheduling to the learning algorithm. Keeping hardware occupied is useful only if the resulting training signal remains good enough to improve the policy.

14:3414:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:34 · section reference included

Environments for sustained progress

The environment infrastructure must support the same ambition. Chengxi introduces OpenReward as General Reasoning’s shared platform for RL environments. At the time of the talk, she reported more than 350 environments accessible through a single API endpoint. She says the company uses it for internal RL and that frontier and newer labs also use it. The infrastructure proposition is straightforward: a common access layer makes many environments available to the training effort.

The progression from pre-training to reasoning and agents leaves a connected set of problems. Longer tasks need memory that preserves useful work, algorithms that can learn before distant rewards arrive, environments that expose uncertainty and competing actors, and compute schedules that do not spend the horizon waiting. None of the proposed techniques removes every trade-off: compaction changes what is retained, critics add learned estimates, and pipelining permits stale-policy data.

Chengxi closes by returning to patience. If the goal is to help solve problems that already demand years of human effort, sustained work must be part of both the agent’s design and the research program building it. That ambition is the mission she attaches to General Reasoning: advancing algorithms, environments, and compute together so that progress can continue across much longer horizons.

16:1816:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:18 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…