AI Engineer World's Fair 2025

Thinking Deeper in Gemini

Jack Rae18:13

Loading slides…

Read the talk

Thinking Deeper in Gemini

Gemini’s thinking stage adds an adjustable computation loop before the final answer, opening a new path to better reasoning, finer cost control, and longer investigations.

From a talk by Jack Rae

Before you start: Basic familiarity with language models and tokens is helpful; recurrent state, attention, and test-time compute are explained as they arise.

Each solution exposes the next bottleneck

How do you improve a language model built from hand-calculated word statistics? In Jack Rae’s historical account, Claude Shannon faced that question in 1948. His word bigram model sampled text using statistics from a textbook. The samples improved on simpler character models, suggesting an obvious direction: scale up the statistics. But that required resources Shannon did not have—digitized human knowledge and modern computing. The example establishes the research approach behind Thinking in Gemini: identify the resource limiting intelligence, then change the system to overcome it.

By the 2000s, Google could aggregate language statistics at enormous scale. Rae describes n-gram models trained over trillions of tokens, powering speech recognition and translation. Their next limitation was context: storing statistics for increasingly long word sequences incurred exponential storage growth. More data and computing had made short-context models powerful, but extending the same representation became impractical.

Recurrent neural language models changed what the system stored. Instead of enumerating longer word sequences, an RNN compressed the past into a neural state. That representation could carry information beyond a five-word context, across sentences and even paragraphs. The improvement introduced another constraint, however: a fixed-size state has limited capacity. As more context passes through it, some information is lost.

Attention addressed that information bottleneck by retaining past neural representations and aggregating them on demand. The model no longer had to squeeze everything it might need into one recurrent state. In Rae’s progression, this led from attention to transformers: another architectural change that made a previously limiting resource less restrictive.

Slide titled “2014” lists the LSTM context bottleneck and attention solution beside a diagram connecting past inputs to a context vector.Open full source frame
Attention preserves past context beyond a fixed-size recurrent state.
2:322:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:32 · section reference included

Give the model more computation before it commits

By 2024, conversational models were useful across a wide range of tasks, yet they were still trained to begin answering immediately. This exposed a different bottleneck: the computation available before committing to an answer. Test-time compute matters because it is the work the model performs on your particular problem, rather than the work already spent training it.

Mechanically, a request becomes tokens that pass through the model. Computation happens in parallel within a layer and sequentially across layers. For an immediate answer, the architecture provides a bounded amount of processing at that request-to-response transition. This does not mean every request has identical total inference cost; it identifies the limited opportunity to work on a problem before starting the answer.

Making the model larger increases that computation, but it does not provide the full flexibility a user might want. A difficult or valuable problem may justify vastly more effort than a routine request. Rae motivates a possible thousandfold or millionfold range of effort as a user need, not a demonstrated improvement. The desired allocation is also adaptive: spend less on easy questions and more on hard ones, with the model helping decide which is which.

Thinking inserts a text-generation stage before the final answer. The model emits intermediate thinking tokens, repeatedly passing through another generation step before deciding to commit. This creates a computation loop whose length can vary by task. Rae describes potentially thousands or tens of thousands of iterations; those are opportunities for additional processing, not a universal multiplier for end-to-end compute. Because the loop has a stopping point, the model can learn how much thinking to perform before answering.

5:235:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:23 · section reference included

Outcome rewards teach the model how to think

Extra tokens are useful only if the model learns to do useful work with them. After pretraining, Gemini undergoes reinforcement learning on many tasks, receiving positive or negative rewards according to whether it solves them correctly. That outcome feedback shapes how it uses the thinking stage. The training recipe does not need to prescribe every intermediate reasoning step, although Rae recalls that the team initially did not know how much explicit structure would be necessary.

An early integer-prediction example made the mechanism tangible. The model used its thinking tokens to:

  1. Propose a hypothesis.
  2. Test whether the hypothesis worked.
  3. Recognize that its formula failed.
  4. Reject that idea and try an alternative approach.

The notable behavior was self-correction before the final response. A general reward signal had encouraged the model to spend computation testing and revising its own proposal, rather than simply extending the first idea it generated.

The learned repertoire extends beyond that example. Models can break a problem into components, explore several possible solutions, draft code fragments and assemble them modularly, perform intermediate calculations, and use tools. These are different ways to use the same added resource: computation devoted to the current problem before delivering an answer.

8:068:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:06 · section reference included

A new scaling path and a finer cost control

For developers, the immediate benefit is more capable models. Thinking also adds an improvement path alongside the existing ones:

Improvement pathWhat can grow or improve
PretrainingTraining data and model size
Post-trainingHuman-feedback quality and task diversity
ThinkingCompute spent on an individual problem

Rae describes investment across these paths as producing a multiplicative effect. The practical claim is that progress in thinking can compound with improvements to the underlying model, rather than requiring the other approaches to stop paying off.

The launch-history chart illustrates that relationship empirically. It places test-time compute on a logarithmic horizontal axis and reasoning performance across mathematics, code, and science on the vertical dimension. Rae reports that performance rises alongside compute, from the non-thinking Gemini 2.0 Flash Experimental at the low-compute end to the first launched Gemini 2.5 Pro at the other. This is a comparison across model launches, not a controlled isolation of compute from every other model improvement.

Thinking also changes how an application chooses its cost-performance tradeoff. Selecting a model size provides a discrete choice; setting a thinking budget provides finer control over effort within a model.

ControlDeveloper choice
Model selectionChoose among model sizes and capabilities
Thinking budgetAllow more or less reasoning effort

An application can permit greater expenditure when a stronger answer is worth the cost. In the talk’s 2025 product context, Rae describes thinking budgets as launched for Gemini 2.5 Flash and Pro; this is a historical description of token-budget controls, not a specification of today’s API.

10:1610:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:16 · section reference included

Avoid wasted thought, then extend useful thought

The next research goals run in two complementary directions:

  • More efficient thinking: Make effort adaptive enough that developers need less manual tuning. Rae acknowledges that models sometimes overthink, creating opportunities to reduce latency and cost.
  • Deeper thinking: Increase the maximum useful inference budget for problems that justify a longer investigation.

Efficiency does not require every answer to arrive immediately. It requires matching the work to the task, while retaining the ability to spend much more when necessary.

Gemini Deep Research provides a familiar interaction pattern: submit a query, let a longer process run, and return for the result. Gemini Deep Think applies that willingness to wait to difficult reasoning problems. At the time of the talk, it was an announced experimental mode built on 2.5 Pro, being introduced to trusted testers. Its mechanism combines deeper chains of thought with parallel chains that can integrate their results. Deep Research is an analogy for the interaction, not a claim that the two systems use identical mechanisms.

Rae reports benefits on difficult multimodal, coding, and mathematics problems. For the USA Math Olympiad, he describes January’s state-of-the-art performance as negligible, then gives the following participant-percentile comparison:

Model or modeRae’s reported participant percentile
Gemini 2.5 ProApproximately 50th
Gemini 2.5 Pro with Deep ThinkApproximately 65th

These are participant percentiles, not percentages of questions answered correctly. The talk does not specify the grading, sampling, or compute conditions behind the comparison. Rae also suggests that the newly updated Pro might do better, but supplies no measurement for that update.

Deep Think slide describes a high-budget thinking mode built on 2.5 Pro and multiple deeper chains of thought, beside a podium comparing o3 (high), 2.5 Pro, and 2.5 Pro Deep Think.Open full source frame
Deep Think overview with a USA Math Olympiad 2025 comparison.

Improving the base model and improving Deep Think’s algorithms can compound, just as thinking can complement pretraining and post-training. The accompanying proof animation illustrates the intended interaction between reasoning paths: begin with a proof-by-contradiction idea, explore approaches involving Rolle’s theorem and Newton’s inequalities, then integrate them into a correct proof. It conveys how multiple ideas might contribute to a solution; it does not supply enough detail to reconstruct the algebra problem or verify the proof.

13:2313:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:23 · section reference included

Longer reasoning for substantial coding tasks

Open-ended coding offers a different reason to allow longer thinking. The goal is to produce substantial software in one or a few interactions, giving the model time to work through the interconnected pieces. Rae’s example comes from colleagues who used Gemini to build from DeepMind’s original DQN research. The reported output included the training setup, the algorithm, and even an Atari emulator that could play some games.

Rae contrasts work that once took him and his colleagues months with examples now beginning to happen in minutes. That is a development anecdote, not a controlled productivity benchmark or a verified reproduction of the original research. Its significance is the scope of the generated artifact: several coordinated parts of a research system, rather than an isolated function.

16:0316:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:03 · section reference included

Turn limited starting knowledge into new work

The longer-term reference point is human discovery. Rae wants models to contemplate a small body of knowledge deeply enough to extend it. His motivating analogy is Ramanujan: in Rae’s telling, a mathematician with limited access to the mathematical community and one foundational textbook generated an extraordinary quantity of mathematics through sustained independent thought. The closing slide expresses that idea as a textbook plus Ramanujan yielding volumes of mathematical work.

Full-screen slide shows a mathematics textbook, a plus sign, Ramanujan’s portrait, an equals sign, and a set of yellow books.Open full source frame
“Towards Deeper Thinking”: a mathematics textbook plus Ramanujan leads to volumes of mathematical work.

The corresponding ambition for models is data-efficient discovery through extended inference. Rae envisions millions or more inference tokens used to build up knowledge and artifacts, allowing a model to derive more from its starting material. This is a research aspiration, not a demonstrated result: move beyond answering from existing knowledge toward sustained work that can push the frontier of human understanding.

16:5217:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:52 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…