AI Engineer World's Fair 2026
How we taught agents to use good retrieval - Hanna Lichtenberg, Mixedbread AI
Loading slides…
Read the talk
Teaching agents to retrieve the evidence they need
Better reasoning cannot compensate for missing evidence. Mixedbread’s search harness combines semantic queries, specialized tools and training rewards to improve what agents retrieve.
From a talk by Hanna Lichtenberg and Aamir Shakir
Before you start: Familiarity with semantic search, retrieval chunks and LLM tool calls will help; no reinforcement-learning background is required.
What if the model can reason, but cannot find the evidence?
A model may be capable of answering a difficult legal or financial question when given the relevant documents, yet struggle to find those documents itself. As reasoning improves, that access problem becomes more consequential: retrieval is how the model reaches knowledge beyond its immediate context. Mixedbread calls the widening distance between these capabilities the knowledge gap.
Hanna Lichtenberg leads agentic search at Mixedbread; co-founder Aamir Shakir opens with the contrast between rapidly improving language models and slowly improving search. He characterizes the progression from GPT-3.5 to GPT-5.5 as exponential, while describing search’s progress over two decades as much slower. The practical concern is knowledge work beyond code: a stronger reasoning model still needs the right evidence to do useful legal or financial work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure the difference between finding and using evidence
Two benchmarks make that distinction concrete. BrowseComp-Plus asks complex browsing questions over a fixed corpus of approximately 100,000 documents, unlike OpenAI’s original BrowseComp, which uses the open web. Databricks’ OfficeQA Pro asks difficult questions over historical U.S. Treasury material. Shakir describes its coverage as roughly a century; the published corpus spans 1939–2025.
The comparison starts with an oracle reference: give the model the correct documents alongside the question and measure its performance. This establishes supplied-evidence performance for that setup, rather than a universal ceiling on what the model could ever achieve. Then require an agent to locate the evidence in the full corpus. The difference exposes the cost of searching through noise.
Shakir reports the following rounded figures. The browsing discussion begins with BrowseComp-Plus, although he subsequently shortens its name to BrowseComp.
| Task | Reported oracle performance | Reported drop with Codex’s default tools |
|---|---|---|
| Browsing benchmark | 93% | Nine points |
| OfficeQA Pro | 64% | Eight points |
The point figures are described as drops; they do not establish separate absolute Codex scores. The model’s ability to use supplied evidence is stronger than its ability to obtain that evidence through the default tools.
Replacing the search tool recovers much of that difference. Mixedbread uses late interaction for retrieval. Shakir reports that GPT-5 with Mixedbread comes within three points of the browsing oracle, while the OfficeQA Pro gap almost completely closes. Mixedbread’s published companion experiments use different precise figures, so these rounded talk results should be kept separate from those published runs. Better retrieval helps immediately—but the queries agents send to it reveal another limitation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A semantic search tool receives a pile of keywords
One query found during benchmarking reads: “Senator woman questions billionaires, not a company. Then okay, thank you staff. Will check hearing.” It contains fragments that might occur in a hearing transcript, but it does not clearly describe the evidence being sought. A retrieval system designed for natural questions receives guesses about word overlap instead.
Shakir attributes this behavior to three influences:
- Codebase exploration. Coding agents learn to use tools such as
grep, where matching expressions in a file is useful. They carry that habit into document search by guessing what strings the document might contain. - Web search habits. Human-oriented web tools encourage agents to imitate familiar keyword queries.
- Benchmark incentives. Shakir argues that entity-heavy queries in BEIR and NanoBEIR structurally favor BM25, reinforcing lexical matching rather than a description of semantic intent.
These are the speakers’ explanations for the observed behavior. Their design target is clear: teach the agent to describe the evidence it needs, while retaining exact matching for tasks that require it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four tools inside a bounded search loop
Lichtenberg’s agent is designed for knowledge work beyond code search, with precision, speed and cost as explicit goals. Its harness runs on Mixedbread and separates four retrieval needs instead of asking one tool to serve every purpose.
| Tool | What the agent receives or does |
|---|---|
| Overview search | Summaries of up to 50 retrieved chunks |
| Semantic search | Full payloads of the top 10 chunks |
| Filter chunks | Finds and sorts chunks using metadata facets |
grep | Matches keywords or exact patterns |
Overview search exposes breadth without loading every full chunk into context. Semantic search supplies detail once the agent has a more focused intent. Metadata filtering and grep preserve structured and lexical access where those are the appropriate operations.
The harness permits at most four search rounds, with parallel searches inside each round. Its sequence is:
- Start with the user’s query, results from an initial semantic search on that query, and hints about available metadata facets.
- Use that corpus preview to split the search intent into at most four queries covering separate aspects.
- Choose the appropriate tool for each intent and run the searches in parallel.
- Deduplicate returned chunks before adding them to the agent’s context.
- Continue searching if necessary; once the evidence is sufficient, submit the relevant chunks in a plausible ranking.
The output is a ranked evidence list. The search agent’s responsibility is to find and order useful material, rather than necessarily compose the final answer itself.
Deduplication makes parallel exploration cheaper in context space: two searches can discover the same chunk without making the agent read it twice. A compact TypeScript implementation of that operation keeps a set of chunk IDs across rounds:
typescript
type Chunk = { id: string; text: string };
function collectUnseen(
batches: readonly (readonly Chunk[])[],
seen: Set<string>,
): Chunk[] {
const fresh: Chunk[] = [];
for (const batch of batches) {
for (const chunk of batch) {
if (seen.has(chunk.id)) continue;
seen.add(chunk.id);
fresh.push(chunk);
}
}
return fresh;
}
const seen = new Set<string>(["hearing-17"]);
const fresh = collectUnseen([
[
{ id: "hearing-17", text: "The senator questions the witness." },
{ id: "hearing-18", text: "The chair thanks the staff." },
],
[{ id: "hearing-18", text: "The chair thanks the staff." }],
], seen);
// fresh contains hearing-18 once.
In this hearing-themed example, hearing-17 is already in context and hearing-18 arrives through two searches. Only one new chunk is retained; this collection step does not replace the agent’s final relevance ranking.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Ask for evidence before asking for a query
The tool menu alone does not ensure good queries. Lichtenberg describes five harness choices that shape how the agent uses it:
- Goal. Require the agent to articulate the evidence it needs before writing a query.
- Tool. Distinguish a semantic aspect search from an exact keyword match. Use semantic retrieval for the former and
grepfor the latter. - Prompt. Ask for “one concise sentence describing what it wants to find,” rather than telling the model to write a search query. The wording is intended to avoid triggering its familiar BM25-style habits.
- Examples. Demonstrate good queries and how to divide an input into separate aspects worth investigating.
- Seed. Supply initial semantic results so the agent can learn the corpus’s language and decide where to dig deeper.
Together, these choices move query writing after an explicit decision about what evidence would help. The agent is no longer asked to guess a bag of terms before it has seen the corpus.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reward both the retrieved list and the search that produced it
The next layer is training. Mixedbread chooses a small language model for speed, then trains its search strategy: tool selection, semantic-query quality, exploration, ranking and efficiency. Training begins with supervised fine-tuning using a larger teacher model, followed by on-policy reinforcement learning with a custom search reward.
That reward combines two views of success:
- Retrieval reward: evaluate the final ranked list using recall and nDCG, alongside an LLM judge. Its rubrics assess relevance of the returned chunks and whether their ranking is plausible.
- Trajectory reward: use an LLM judge to assess the search process, including tool choice, efficiency, natural-sentence query formulation and the amount of exploration. Both insufficient and excessive exploration matter.
The distinction lets training address the result and the behavior separately. A useful final list does not, by itself, show that the agent chose efficient tools or wrote good semantic queries. Conversely, a well-formed search trajectory still needs to retrieve relevant evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From parallel exploration to reported benchmark results
The example trajectory starts with the initial search and metadata hints, launches four parallel searches in its first round, then performs a single grep search in its second. Its long, rambling input comes from the Congress Hearings task in OBLIQ-Bench, called Oblique Congress in the presentation. The semantic queries are sentences describing what the agent wants to find; the grep queries remain keyword patterns. That difference is intentional: the agent changes query style with the tool rather than enforcing natural language everywhere.
At the time of the presentation, the trained agent is unreleased. Lichtenberg reports an intermediate Oblique Congress nDCG@10 of 0.4, compared with 0.18 for the benchmark paper’s GPT multi-hop agent. The paper identifies that baseline as GPT-5.2 Multi-Hop Agent and reports 0.183 in its first version. The Mixedbread figure remains a preliminary reported result; the presentation does not provide enough experiment detail to establish identical evaluation conditions.
A separate beta, Mixedbread agentic search, is already running in production. Lichtenberg reports first place on Snowflake’s MADQA benchmark with 93.4% accuracy when Gemini 3.5 Flash, as named in the presentation, uses Mixedbread agentic search as its search tool. She also reports lower effort than comparable systems using other search agents, without giving a numerical effort measure, and points to the benchmark leaderboard for comparison. These are the team’s reported results, rather than an independently established historical leaderboard snapshot.
The ending returns to the original constraint: capable models still have room to improve through their access to knowledge. Better retrieval tools recover missing performance; a dedicated search agent adds tool selection, query formulation and controlled exploration around those tools. The production beta and the unreleased trained agent represent different stages of that work, with evidence quality and the effort needed to obtain it both part of the objective.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Code and evaluation materials for deep research over a fixed corpus of approximately 100,000 documents.
Databricks' document-reasoning benchmark, with scoring code and instructions for accessing Treasury Bulletin data.
Five retrieval tasks involving implicit relevance, including recovering congressional-hearing passages from imperfect recollections.
An introduction to multimodal document question answering and evaluation of answer accuracy alongside search effort.
Further reading
Mixedbread's published retrieval comparisons across BrowseComp-Plus, MADQA and OfficeQA-Pro.
Updates since the talk
Current instructions for enabling hosted agentic search, configuring ranked results and streaming search traces.
Full transcript
Loading transcript…