AI Engineer World's Fair 2026

How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI

Loading slides…

Read the talk

Teaching agents to retrieve the evidence they need

Better reasoning cannot compensate for missing evidence. Mixedbread’s search harness combines semantic queries, specialized tools and training rewards to improve what agents retrieve.

From a talk by Hanna Lichtenberg and Aamir Shakir

Before you start: Familiarity with semantic search, retrieval chunks and LLM tool calls will help; no reinforcement-learning background is required.

What if the model can reason, but cannot find the evidence?

A model may be capable of answering a difficult legal or financial question when given the relevant documents, yet struggle to find those documents itself. As reasoning improves, that access problem becomes more consequential: retrieval is how the model reaches knowledge beyond its immediate context. Mixedbread calls the widening distance between these capabilities the knowledge gap.

Hanna Lichtenberg leads agentic search at Mixedbread; co-founder Aamir Shakir opens with the contrast between rapidly improving language models and slowly improving search. He characterizes the progression from GPT-3.5 to GPT-5.5 as exponential, while describing search’s progress over two decades as much slower. The practical concern is knowledge work beyond code: a stronger reasoning model still needs the right evidence to do useful legal or financial work.

A chart plots capabilities over time, with a steep yellow Reasoning curve above a slowly rising gray Search line and the shaded difference labeled Knowledge gap.Open full source frame
The knowledge gap between reasoning and search.
0:000:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Measure the difference between finding and using evidence

Two benchmarks make that distinction concrete. BrowseComp-Plus asks complex browsing questions over a fixed corpus of approximately 100,000 documents, unlike OpenAI’s original BrowseComp, which uses the open web. Databricks’ OfficeQA Pro asks difficult questions over historical U.S. Treasury material. Shakir describes its coverage as roughly a century; the published corpus spans 1939–2025.

The comparison starts with an oracle reference: give the model the correct documents alongside the question and measure its performance. This establishes supplied-evidence performance for that setup, rather than a universal ceiling on what the model could ever achieve. Then require an agent to locate the evidence in the full corpus. The difference exposes the cost of searching through noise.

Shakir reports the following rounded figures. The browsing discussion begins with BrowseComp-Plus, although he subsequently shortens its name to BrowseComp.

TaskReported oracle performanceReported drop with Codex’s default tools
Browsing benchmark93%Nine points
OfficeQA Pro64%Eight points

The point figures are described as drops; they do not establish separate absolute Codex scores. The model’s ability to use supplied evidence is stronger than its ability to obtain that evidence through the default tools.

Replacing the search tool recovers much of that difference. Mixedbread uses late interaction for retrieval. Shakir reports that GPT-5 with Mixedbread comes within three points of the browsing oracle, while the OfficeQA Pro gap almost completely closes. Mixedbread’s published companion experiments use different precise figures, so these rounded talk results should be kept separate from those published runs. Better retrieval helps immediately—but the queries agents send to it reveal another limitation.

1:281:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:28 · section reference included

A semantic search tool receives a pile of keywords

One query found during benchmarking reads: “Senator woman questions billionaires, not a company. Then okay, thank you staff. Will check hearing.” It contains fragments that might occur in a hearing transcript, but it does not clearly describe the evidence being sought. A retrieval system designed for natural questions receives guesses about word overlap instead.

Shakir attributes this behavior to three influences:

  • Codebase exploration. Coding agents learn to use tools such as grep, where matching expressions in a file is useful. They carry that habit into document search by guessing what strings the document might contain.
  • Web search habits. Human-oriented web tools encourage agents to imitate familiar keyword queries.
  • Benchmark incentives. Shakir argues that entity-heavy queries in BEIR and NanoBEIR structurally favor BM25, reinforcing lexical matching rather than a description of semantic intent.

These are the speakers’ explanations for the observed behavior. Their design target is clear: teach the agent to describe the evidence it needs, while retaining exact matching for tasks that require it.

Slide quotes “Senator woman questions billionaires not at company then ok thank you staff will check hearing,” followed by bullets about codebase exploration, web-search query distributions, and benchmark bias.Open full source frame
A keyword-heavy query illustrates the mismatch with semantic search.
3:554:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:55 · section reference included

Four tools inside a bounded search loop

Lichtenberg’s agent is designed for knowledge work beyond code search, with precision, speed and cost as explicit goals. Its harness runs on Mixedbread and separates four retrieval needs instead of asking one tool to serve every purpose.

ToolWhat the agent receives or does
Overview searchSummaries of up to 50 retrieved chunks
Semantic searchFull payloads of the top 10 chunks
Filter chunksFinds and sorts chunks using metadata facets
grepMatches keywords or exact patterns

Overview search exposes breadth without loading every full chunk into context. Semantic search supplies detail once the agent has a more focused intent. Metadata filtering and grep preserve structured and lexical access where those are the appropriate operations.

The harness permits at most four search rounds, with parallel searches inside each round. Its sequence is:

  1. Start with the user’s query, results from an initial semantic search on that query, and hints about available metadata facets.
  2. Use that corpus preview to split the search intent into at most four queries covering separate aspects.
  3. Choose the appropriate tool for each intent and run the searches in parallel.
  4. Deduplicate returned chunks before adding them to the agent’s context.
  5. Continue searching if necessary; once the evidence is sufficient, submit the relevant chunks in a plausible ranking.

The output is a ranked evidence list. The search agent’s responsibility is to find and order useful material, rather than necessarily compose the final answer itself.

Deduplication makes parallel exploration cheaper in context space: two searches can discover the same chunk without making the agent read it twice. A compact TypeScript implementation of that operation keeps a set of chunk IDs across rounds:

typescript

type Chunk = { id: string; text: string };

function collectUnseen(
  batches: readonly (readonly Chunk[])[],
  seen: Set<string>,
): Chunk[] {
  const fresh: Chunk[] = [];
  for (const batch of batches) {
    for (const chunk of batch) {
      if (seen.has(chunk.id)) continue;
      seen.add(chunk.id);
      fresh.push(chunk);
    }
  }
  return fresh;
}

const seen = new Set<string>(["hearing-17"]);
const fresh = collectUnseen([
  [
    { id: "hearing-17", text: "The senator questions the witness." },
    { id: "hearing-18", text: "The chair thanks the staff." },
  ],
  [{ id: "hearing-18", text: "The chair thanks the staff." }],
], seen);
// fresh contains hearing-18 once.

In this hearing-themed example, hearing-17 is already in context and hearing-18 arrives through two searches. Only one new chunk is retained; this collection step does not replace the agent’s final relevance ranking.

5:235:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:23 · section reference included

Ask for evidence before asking for a query

The tool menu alone does not ensure good queries. Lichtenberg describes five harness choices that shape how the agent uses it:

  • Goal. Require the agent to articulate the evidence it needs before writing a query.
  • Tool. Distinguish a semantic aspect search from an exact keyword match. Use semantic retrieval for the former and grep for the latter.
  • Prompt. Ask for “one concise sentence describing what it wants to find,” rather than telling the model to write a search query. The wording is intended to avoid triggering its familiar BM25-style habits.
  • Examples. Demonstrate good queries and how to divide an input into separate aspects worth investigating.
  • Seed. Supply initial semantic results so the agent can learn the corpus’s language and decide where to dig deeper.

Together, these choices move query writing after an explicit decision about what evidence would help. The agent is no longer asked to guess a bag of terms before it has seen the corpus.

Slide titled “Our harness encourages better semantic queries” shows five guidance rows labeled Goal, Tool, Prompt, Examples, and Seed, alongside the presenter.Open full source frame
Five harness choices encourage better semantic queries.
8:258:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:25 · section reference included

Reward both the retrieved list and the search that produced it

The next layer is training. Mixedbread chooses a small language model for speed, then trains its search strategy: tool selection, semantic-query quality, exploration, ranking and efficiency. Training begins with supervised fine-tuning using a larger teacher model, followed by on-policy reinforcement learning with a custom search reward.

That reward combines two views of success:

  • Retrieval reward: evaluate the final ranked list using recall and nDCG, alongside an LLM judge. Its rubrics assess relevance of the returned chunks and whether their ranking is plausible.
  • Trajectory reward: use an LLM judge to assess the search process, including tool choice, efficiency, natural-sentence query formulation and the amount of exploration. Both insufficient and excessive exploration matter.

The distinction lets training address the result and the behavior separately. A useful final list does not, by itself, show that the agent chose efficient tools or wrote good semantic queries. Conversely, a well-formed search trajectory still needs to retrieve relevant evidence.

10:0110:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:01 · section reference included

From parallel exploration to reported benchmark results

The example trajectory starts with the initial search and metadata hints, launches four parallel searches in its first round, then performs a single grep search in its second. Its long, rambling input comes from the Congress Hearings task in OBLIQ-Bench, called Oblique Congress in the presentation. The semantic queries are sentences describing what the agent wants to find; the grep queries remain keyword patterns. That difference is intentional: the agent changes query style with the tool rather than enforcing natural language everywhere.

At the time of the presentation, the trained agent is unreleased. Lichtenberg reports an intermediate Oblique Congress nDCG@10 of 0.4, compared with 0.18 for the benchmark paper’s GPT multi-hop agent. The paper identifies that baseline as GPT-5.2 Multi-Hop Agent and reports 0.183 in its first version. The Mixedbread figure remains a preliminary reported result; the presentation does not provide enough experiment detail to establish identical evaluation conditions.

A separate beta, Mixedbread agentic search, is already running in production. Lichtenberg reports first place on Snowflake’s MADQA benchmark with 93.4% accuracy when Gemini 3.5 Flash, as named in the presentation, uses Mixedbread agentic search as its search tool. She also reports lower effort than comparable systems using other search agents, without giving a numerical effort measure, and points to the benchmark leaderboard for comparison. These are the team’s reported results, rather than an independently established historical leaderboard snapshot.

An Evals slide contains side-by-side obliq-congress and MADQA tables, with Mixedbread search agent entries highlighted in yellow and the presenter on the right.Open full source frame
Evaluation tables compare retrieval results and answer accuracy with effort.

The ending returns to the original constraint: capable models still have room to improve through their access to knowledge. Better retrieval tools recover missing performance; a dedicated search agent adds tool selection, query formulation and controlled exploration around those tools. The production beta and the unreleased trained agent represent different stages of that work, with evidence quality and the effort needed to obtain it both part of the objective.

11:5912:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:59 · section reference included

Resources

From the talk

  • Code and evaluation materials for deep research over a fixed corpus of approximately 100,000 documents.

  • Databricks' document-reasoning benchmark, with scoring code and instructions for accessing Treasury Bulletin data.

  • OBLIQ-BenchPaper12:17

    Five retrieval tasks involving implicit relevance, including recovering congressional-hearing passages from imperfect recollections.

  • An introduction to multimodal document question answering and evaluation of answer accuracy alongside search effort.

Updates since the talk

Transcript download
Full transcript

Loading transcript…