AI Engineer World's Fair 2024

Going beyond RAG: Extended Mind Transformers

Phoebe Klett· Machine Learning Engineer, Normal Computing16:04

Loading slides…

Read the talk

Going Beyond RAG: Extended Mind Transformers

Extended Mind Transformers retrieve cached token representations inside attention, separating memory from the prompt and enabling retrieval traces and uncertainty-driven regeneration.

From a talk by Phoebe Klett

Before you start: Familiarity with transformer attention, token generation, and retrieval-augmented generation will help; the memory mechanism is explained without requiring its mathematical derivation.

Why distinguish memory from the prompt?

How much of a codebase does a model need to answer one question? Pretraining supplies general knowledge, but an application also needs specific, current information. Long context makes room for that information by extending the sequences a transformer can accept. Phoebe Klett points to two costs: extending a model beyond its training context can require expensive fine-tuning, and filling the prompt can introduce irrelevant information. A question about a repository usually needs only a subset of its function definitions.

Conventional retrieval-augmented generation, or RAG, selects that subset before generation. In the setup discussed here, an external retriever decides what matters once, upfront, using representations that are coarser than the token representations the transformer uses internally. The selection cannot respond to what the model discovers it needs while writing its answer.

Separating memory from the inference prompt makes retrieval available during generation. That separation enables two further capabilities: tracing which memory tokens were retrieved for an answer, and retrieving more information when the model becomes uncertain.

Introduction slide with red and black token bars, curved retrieval arrows, and a comparison table using green checks and red crosses.Open full source frame
Long context, RAG, and extended memory compared alongside a token-retrieval diagram.
1:041:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:04 · section reference included

Retrieve memory inside attention

Extended Mind Transformers, or EMTs, modify the transformer's existing attention mechanism. Within each decoder layer, tokens already have key-value representations: keys participate in selecting information, and values carry the information attention combines. These internal representations provide the basis for memory retrieval.

The process has two phases:

  1. Construct memory. Pass the memory tokens through the model and save their key-value representations.
  2. Retrieve during generation. Let each query token use cosine similarity to select a configured number of cached memory tokens, then attend to their representations.

Retrieval therefore uses representations formed inside the model, and its selections can change as generation proceeds. In the diagram, the red-highlighted entries are the retrieved tokens.

Extended Mind Attention slide showing input tokens, a black model block, and rows of cached tokens with selected entries outlined in red.Open full source frame
Extended Mind Attention retrieves from an external cache of key-value memories.
3:323:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:32 · section reference included

Making retrieved tokens fit the position scheme

The difficult integration question is how to assign position information to retrieved tokens. Attention needs a way to distinguish token order. Klett attributes part of earlier approaches' need for fine-tuning to their use of absolute position embeddings: the model had to learn how to use memory tokens introduced into that positional scheme. The relative-position approaches tested for EMT allow the modified models to use retrieved representations without additional fine-tuning.

Tested model familyPosition mechanismHow position enters attention
LlamaRotary position embeddings, or RoPERotates pairs of embedding coordinates to encode relative position
MPTALiBiApplies linear attention biases that penalize greater distance

ALiBi is not a position embedding: it changes attention scores directly. The distinction matters because accommodating memory requires working with the model's particular position mechanism, even when no weight updates are needed.

4:374:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:37 · section reference included

Testing retrieval rather than memorization

A model can answer a factual question correctly without reading the supplied document. To distinguish retrieval from memorization, the counterfactual WikiQA benchmark pairs questions with supporting contexts containing deliberately altered facts. Supporting contexts span approximately 2,000–16,000 tokens.

The songwriting example asks who wrote These Boots Are Made for Walkin'. The original answer is Lee Hazlewood. The benchmark replaces every occurrence of his name in the supplied context with Terry Allen, a plausible but deliberately incorrect alternative. The expected benchmark answer then becomes Terry Allen. This tests whether the model prioritizes inference-time evidence over facts memorized during pretraining or fine-tuning; it does not make the substituted answer historically true.

6:166:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:16 · section reference included

Query-only retrieval, then retrieval with RAG

The first comparison includes models fine-tuned for longer context and a base Llama model using interpolated position embeddings. Klett describes the base model as trained on 2,048-token sequences; that is the talk's characterization, rather than a general specification for the Llama family. In the reported counterfactual retrieval comparison, the base model remains reasonably effective around 8K tokens but declines sharply at 16K.

EMT performs better than the fine-tuned comparators on shorter inputs and remains competitive through 16K without additional fine-tuning. Klett interprets the shorter-input results as evidence that long-context fine-tuning can degrade attention on shorter sequences, not as a universal finding about every long-context model.

For EMT in this first experiment, the supporting document stays in memory: the prompt contains the question rather than the document. The paper specifies one-shot prompting, so “query only” describes where the supporting evidence goes, not the absence of all other prompt scaffolding. Answering the counterfactual songwriting question requires the internal retriever to find the substituted name.

The second experiment adds RAG context to the prompt while retaining the entire supporting document in memory. The paper describes retrieving five nonoverlapping 500-token chunks using Ada embeddings. In the June 2024 counterfactual fact-retrieval experiment, EMT combined with RAG outperforms the tested GPT-4 baseline. This is a result for that benchmark and setup, not a claim of general superiority; the inspected evaluation does not identify the exact GPT-4 snapshot. The combination also shows that prompt retrieval and memory attention can complement each other.

7:277:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:27 · section reference included

Connect an answer token to retrieved evidence

Citations help users assess an answer, but matching an output to a supplied passage leaves a gap. If a date appears in both the prompt and the completion, that agreement supports the answer without establishing which information the model used. EMT exposes the memory tokens retrieved and attended to during generation. Klett calls these causal citations; more precisely, the paper reports retrieval provenance, including tokens most frequently retrieved across layers, rather than an intervention-based proof of causality.

The demonstration stores a Wikipedia passage about Alexander Grothendieck and asks when he received French citizenship. Its supplied answer is 1971. When the model generates that date, the display highlights retrieved tokens containing 1971 and parts of Grothendieck's name. The trace connects the generated date to specific memory content, giving the reader a more granular account of retrieval than a document-level citation alone.

8:589:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:58 · section reference included

Retrieve more when generation becomes uncertain

Memory access can also respond to uncertainty. The simplest signal discussed is token-level entropy over the output distribution. Other uncertainty measures can serve the same purpose. Klett mentions Bayesian fine-tuning work at Normal Computing as another way to estimate uncertainty, but it is not a requirement for this loop. When uncertainty is detected, the model can regenerate the affected step with more information from memory.

The citizenship example makes the sequence concrete:

  1. Start with a default retrieval budget. Each query token can retrieve a baseline number of memory entries.
  2. Observe an incorrect answer. Under that budget, the demonstration produces 1993 instead of its target, 1971.
  3. Expand retrieval and regenerate. For uncertain tokens, retrieve more cached information and regenerate a subset of the output. The revised example produces 1971.

This is an active-learning-inspired demonstration of hallucination reduction; the paper defers full-scale experiments. Efficiency gains are also conditional: avoiding a large retrieval budget everywhere must save more computation than the extra retrieval and regeneration consume.

Active learning generation slide with two memory diagrams and stacked answer boxes about Alexander Grothendieck's French citizenship; the lower answer has a green check.Open full source frame
The citizenship example contrasts a 1993 answer with a 1971 answer marked by a green check.
10:2710:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:27 · section reference included

Tune memory construction and retrieval separately

Stride length controls how memory representations are constructed. The source text must pass through a model with a fixed context length, so processing advances through it in windows. Overlapping windows give tokens preceding context when the model forms their cached key-value representations. Smaller strides improve representation quality in the described setup, but require more computation because the model processes more overlapping text.

top-k controls generation-time access: it is the number of memory key-value pairs each query token may retrieve and attend to. Klett identifies it as the most important retrieval parameter and recommends adapting it to memory length. Longer memories generally benefit from retrieving more entries. Stride determines the quality and construction cost of the stored representations; top-k determines how much of that memory a generation step can consult.

11:5312:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:53 · section reference included

Keep irrelevant matches out of attention

Increasing retrieval indiscriminately can recreate the problem of an overloaded prompt. The implementation therefore includes similarity masking: retrieve candidate entries, then mask those whose query-key similarity falls below a threshold. Klett gives 0.25 as an illustrative threshold, not a universal setting. This permits a broad initial selection while withholding weak matches from attention. The paper did not use similarity masking in its reported retrieval experiments.

A second filter removes unknown-token entries, particularly for models using RoPE. Messy Wikipedia-derived text can contain unknown tokens whose representations match many queries without carrying useful information. Those entries are removed from the constructed memory before retrieval begins. The paper's pruning also includes beginning- and end-of-sequence special tokens. Unlike similarity masking, this changes the pool of available entries rather than filtering the results of an individual query.

13:1513:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:15 · section reference included

Use memory as a model input

The released implementation, dataset, paper, and Hugging Face models provide the starting points for using EMT. One released model is Extended Mind Llama-2-7B Chat. These are the research implementation and interfaces presented in 2024, rather than a verified current serving recipe.

The usage pattern is to tokenize the memory, supply those tokens when instantiating the model, and set the model configuration for memory retrieval. Memories can also be changed dynamically after instantiation. This makes the memory a separately managed input rather than information that must always accompany the question in the prompt.

That separation carries through the practical benefits demonstrated here: retrieval can supply facts without additional fine-tuning, its traces can expose the memory tokens associated with an answer, and uncertainty can trigger another retrieval attempt during generation. The open-source models and code make those capabilities available together as changes to attention and memory handling.

Conclusion slide with five bullets about Extended Mind Transformers and Hugging Face and GitHub icons.Open full source frame
Closing claims cover retrieval performance, citations, hallucination reduction, and open-source availability without fine-tuning.
14:4214:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:42 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…