AI Engineer World's Fair 2025
RAG Evaluation Is Broken! Here's Why (And How to Fix It)
Loading slides…
Read the talk
RAG Evaluation Is Broken! Here's Why (And How to Fix It)
Local-answer benchmarks reward finding the right passage. Aggregate questions expose a different requirement: complete evidence, structured extraction, and explicit query semantics.
From a talk by Yuval Belfer and Niv Granot
Before you start: Familiarity with RAG retrieval and basic SQL will help; the article explains the aggregation query.
Local answers hide system failures
Read a long document, find a passage containing a fact, and write a question whose answer is that fact. This natural way to build a benchmark embeds a consequential assumption: the answer lives in a local chunk. A retrieval-augmented generation system can score well by finding that chunk and restating it, without demonstrating that it can answer questions about the corpus as a whole. This is the opening problem Yuval Belfer and Niv Granot of AI21 Labs identify beneath the apparent maturity of RAG.
Multi-hop questions try to escape that locality by requiring several connected facts. Belfer points to Google’s FRAMES, then paraphrases a dataset question that constructs a future wife’s name from facts about a US first lady’s family. His objection is about realism: a manufactured chain of lookups may exercise reasoning without resembling a customer’s work. FRAMES explicitly targets end-to-end factuality, retrieval, and reasoning; the criticism here is that those capabilities alone do not establish representative question design.
Other evaluations isolate one part of the pipeline:
| Evaluation | What it tests | What it assumes |
|---|---|---|
| Retrieval only | Finding relevant chunks | The answer is in the selected chunks |
| Generation only | Answering from supplied context | The necessary context is already in the prompt |
Neither view by itself tests the entire path from raw documents to a useful answer. Parsing and chunking can affect what evidence survives, and some questions do not point toward a particular location at all. Real customer corpora are messy and differ from one another, so success on a convenient benchmark need not generalize.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The optimization loop preserves the assumption
Teams need something to optimize, so they build a RAG system around the available benchmark. Scores rise, the team celebrates, and the result looks like progress. Then users ask questions over customer data, and the system struggles. The team builds a replacement benchmark and optimizes again—but if the replacement still consists of local questions with local answers, the same failure repeats. Changing the benchmark does not help if its underlying assumption stays fixed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Aggregate questions need complete evidence
Financial documents make the missing question types concrete. Starting from corpora such as FinanceBench, SEC filings, or similar financial material, Granot proposes questions a user might reasonably ask:
- Repeated comparisons: Which company reported the highest quarterly revenue the most times?
- Threshold counts: In how many fiscal years did Apple exceed $100 billion in annual revenue?
- Exhaustive lists: List every entity satisfying a condition.
These are illustrative questions, not identified FinanceBench items. The financial slide adds a specific list example: Fortune 500 companies that ranked in the top three but never first. Each question requires considering a set of records, not simply locating a sentence.
A conventional pipeline retrieves the top K chunks and asks a model to compile an answer. That can supply relevant examples without supplying all the evidence required for a count or an exhaustive list. If ten chunks mention Fortune 500 companies, an eleventh or twelfth may introduce another qualifying company. For any chosen N, relevant evidence may remain in chunk N + 1. Increasing K can improve coverage, but the cutoff itself does not establish completeness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A small World Cup test exposes the gap
To probe this failure, the speakers assembled a corpus of 22 documents from Wikipedia’s historical FIFA World Cup pages. They asked questions such as which team had won the tournament most often and how many World Cups Brazil had participated in. Reference answers came from Wikidata and Kaggle knowledge bases. The task was to answer questions across the tournament corpus, rather than extract answers from selected local passages.
The speakers reported the following results on their aggregate-question test:
| System | Questions answered correctly |
|---|---|
| Common introductory RAG pipeline | 5% |
| OpenAI Responses | 11% |
The first baseline is described as the kind of introductory pipeline found in LangChain or LlamaIndex. These are historical results from this particular experiment, not general product accuracy figures. The talk does not supply the question count, scoring procedure, model versions, retrieval settings, or exact OpenAI Responses configuration, and it does not report accuracy for the structured approach introduced next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Extract structure before answering
The proposed response is to convert the unstructured corpus into a data structure and ask questions over that structure. Counts, maxima, minima, and calculations are natural database operations. Once the relevant records exist, the system need not reconstruct an aggregate from whichever passages retrieval happens to return. As in ordinary RAG, the architecture separates ingestion from inference: invest computation while preparing the corpus so that answering can be fast.
Ingestion proceeds through four stages:
- Cluster the documents into sub-corpora. Financial records and World Cup pages, for example, belong to different groups.
- Identify a schema for each sub-corpus. Choose attributes that represent its recurring information.
- Populate the schema from every document. Extract records across the corpus rather than only from documents retrieved for one question.
- Store the results in a SQL database. The extracted representation becomes the basis for later queries.
For World Cup pages, the schema can include the year, winner, top three teams, top scorer, and other attributes. Granot stresses that populating it is the job of an LLM-based pipeline, not merely a single model call. The ingestion process must turn each document into a record that fits the shared representation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Translate the question into an aggregation
At query time, the system first identifies the relevant schema: is this a financial question, a World Cup question, or something else? It then applies text-to-SQL to the extracted data and returns the final answer. Retrieval of a few passages is replaced, for these questions, by a formal query over the prepared records.
The question about the team with the most World Cup wins becomes a grouping and counting operation. With a world_cups table containing one row per tournament and a winner column, the core SQL is:
sql
SELECT
winner,
COUNT(*) AS wins
FROM world_cups
GROUP BY winner
ORDER BY wins DESC
LIMIT 1;
GROUP BY gathers tournaments by winner, COUNT(*) counts the rows in each group, and descending order puts the largest count first. LIMIT 1 selects one leading team. The aggregation considers the stored tournament rows instead of only the top-ranked text chunks; its correctness still depends on the records extracted during ingestion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Structure creates its own correctness requirements
This approach requires a useful relational representation. Not every corpus has homogeneous attributes, not every collection implies a shared schema, and not every question belongs in a relational database. The speakers encounter suitable corpora among their clients, but they do not present structured extraction as a universal replacement for RAG.
Even the World Cup example contains normalization decisions that change the meaning of an answer:
- Historical identity: When counting Germany’s wins, should West Germany count as Germany? Grouping raw names and grouping a normalized identity can produce different totals.
- Field cardinality: A host-country field cannot always hold a single country. South Korea and Japan hosted jointly, so the representation must accommodate multiple hosts.
These decisions affect both ingestion and inference. The stored representation and the interpretation of the user’s question must agree about what is being counted.
Schema selection also needs a way to reject a poor fit. A question about whether Real Madrid won a 2006 final is not a World Cup question, even though it contains familiar football terms and a year. An LLM inclined to satisfy the user may try to force it into the available schema and produce an unsupported answer. Granot names ambiguity and abstention as challenges; the talk does not supply an implementation for resolving them.
There is also an ingestion trade-off. Finer-grained clustering and schema inference can represent more distinctions, but they increase complexity and the computation invested before a query arrives. At inference time, more complex schemas make text-to-SQL itself harder. Moving aggregation into SQL therefore changes the engineering problem: schema design, extraction, routing, and query generation all become part of answer correctness.
Evaluation and architecture must match the client’s questions. The familiar sequence of chunking, embedding, retrieval, and re-ranking is insufficient for many tasks, and benchmarks that omit those tasks conceal the limitation. Where the work demands counts, comparisons, and complete lists over recurring records, answering may require a structured representation beyond standard RAG. The benchmark must exercise those same requirements if its scores are to guide useful improvements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original paper introducing a benchmark for factuality, retrieval and multi-hop reasoning in end-to-end RAG.
Browse benchmark questions, answers and supporting Wikipedia links, including the future-wife example.
Public financial QA examples with annotated answers, evidence, source documents and an evaluation notebook.
Podcast episode page for Yuval and Niv's discussion of evaluation failures, chunking and structured RAG.
Updates since the talk
Later research coauthored by Niv Granot formalizes S-RAG and introduces HOTELS and WORLD CUP aggregation datasets.
Full transcript
Loading transcript…