AI Engineer World's Fair 2025
open-rag-eval: RAG Evaluation without "golden" answers.
Loading slides…
Read the talk
Open RAG Eval: Evaluating Retrieval and Answers Without Golden References
Open RAG Eval captures a pipeline’s retrieved passages and answers, then evaluates relevance, information coverage, citation support, and response consistency without hand-written golden references.
From a talk by Ofer Mendelevitch
Before you start: Familiarity with RAG—retrieving passages and using them to generate an answer—is helpful.
The golden-reference bottleneck
How do you evaluate a RAG system when you have useful questions but no curated set of correct answers or ideal retrieved passages? Creating those references adds work every time the evaluation set grows. Open RAG Eval, introduced here by Ofer Mendelevitch of Vectara developer relations, addresses that bottleneck: evaluate retrieval and generation without manually supplied golden chunks or golden answers. The opening slide’s QR code provides access to the repository.
The project draws on research conducted with the University of Waterloo’s Jimmy Lin Lab. Its approach replaces the requirement for curated references with automated assessments of what the pipeline actually retrieves and generates. Understanding those assessments matters: each measures a different part of the RAG system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From queries to evaluation files
Start with 10, 100, or 1,000 queries that matter to your RAG application. These are example evaluation-set sizes, not throughput claims. The workflow then follows the data through three stages:
- Capture the pipeline’s behavior. A RAG connector collects the actual retrieved chunks and generated answers. The demonstrated integrations include Vectara, LangChain, and LlamaIndex.
- Evaluate the captured outputs. The resulting RAG outputs feed metrics grouped into evaluators. This separates collecting a pipeline’s behavior from assessing it.
- Write evaluation files. Evaluators emit files containing the results for inspection and comparison.
The connector is therefore the boundary between your application and the evaluation machinery: it supplies what the system did, without requiring you to author the ideal answer first.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieval relevance with UMBRELA
The package introduces four complementary assessments: UMBRELA for retrieval, AutoNuggetizer for generation, citation faithfulness, and hallucination detection using a Vectara model. Keeping these dimensions separate helps distinguish retrieving useful evidence from producing a well-supported answer.
UMBRELA scores individual passages against the query, without requiring golden chunks. Its relevance scale runs from 0 to 3:
| Score | Meaning described in the talk |
|---|---|
| 0 | Passage has nothing to do with the query |
| 3 | Passage is dedicated to the query and contains the exact answer |
The endpoints distinguish irrelevant retrieval from a passage that directly supplies the requested information.
Mendelevitch reports that UMBRELA correlates well with human judgment, citing the Waterloo research. That supports using automated relevance assessment without golden chunks, but the consequential distinction is between agreement in evaluating systems and identical labels for every passage. The associated research supports system-ranking agreement; the talk supplies no correlation coefficient or experimental conditions for its broader qualitative claim.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Answer coverage with AutoNuggetizer
AutoNuggetizer evaluates generation by breaking the information an answer should cover into nuggets, or atomic information units. Instead of comparing the generated response with a single golden answer, it checks coverage of selected units:
- Create nuggets. Produce atomic units of information for the evaluation.
- Prioritize them. Assign each nugget a
vitalorokayimportance rating, order the nuggets, and select the top 20. - Judge answer coverage. An LLM judge examines the RAG response to determine whether each selected nugget is fully or partially supported by the answer.
The top-20 limit is a selection setting, not an accuracy result. The mechanism measures information coverage: an answer can cover one important point completely and another only partly.
For a small Python implementation of the importance-ordering step, retain each nugget’s text and rating together, then select the first 20 after sorting. This helper represents selection only; nugget creation and support judgments remain separate operations.
python
from typing import Literal, TypedDict
class Nugget(TypedDict):
text: str
importance: Literal["vital", "okay"]
def select_nuggets(nuggets: list[Nugget]) -> list[Nugget]:
priority = {"vital": 0, "okay": 1}
ordered = sorted(
nuggets,
key=lambda nugget: priority[nugget["importance"]],
)
return ordered[:20]
The original nugget records survive the selection, so the subsequent judge can assess each retained unit against the generated answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Citation support and whole-response consistency
Mendelevitch points to the papers for fuller method details before moving to two support checks. These operate at different scopes:
| Assessment | Scope | Question |
|---|---|---|
| Citation faithfulness | A citation and its associated claim | Does the cited passage support the claim? |
| Hallucination detection | The entire response | Does the response align with retrieved content? |
Citation faithfulness distinguishes full support, partial support, and no support. A citation’s presence alone does not establish that its passage substantiates what the answer says.
For the response-level check, HHEM, Vectara’s hallucination detection model, assesses alignment between the answer and retrieved evidence. Here, support is relative to that evidence: a statement can be true elsewhere yet unsupported by the passages supplied to the model. This complements nugget coverage by asking whether the answer stays grounded in what was retrieved.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspecting the results
Once evaluation finishes, you can inspect the result files directly. For easier comparison, the talk shows a hosted interface and describes dragging the evaluation files onto open-evaluation.ai. The populated view lists queries alongside retrieval and generation scores, with groups of colored results arranged side by side. This makes the separate evaluation dimensions visible together without requiring manual inspection of every file.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tuning a pipeline and extending its connectors
The intended use is to optimize and tune a RAG pipeline while keeping its evaluation methods inspectable. Open source provides access to the metric implementations, so a score need not remain an opaque judgment: you can examine how the assessment works.
Vectara, LangChain, and LlamaIndex provide the demonstrated integration paths. For a custom pipeline, the extension point is the connector that captures retrieved chunks and generated answers. Mendelevitch invites issues and pull requests for additional connectors, allowing other pipelines to feed the same evaluator workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Python toolkit with RAG connectors, evaluation metrics, examples and contribution instructions. Update from the repository, checked 2026-08-28: its viewer link points to openevaluation.ai and JSON is the default viewer format. Viewer functionality was not tested.
Original paper on automated retrieval relevance judgments using GPT-4o.
Automatic information-nugget creation and answer coverage evaluation, with initial comparisons against human assessment.
Model documentation and usage examples for assessing whether generated text is supported by supplied evidence.
Further reading
Project introduction coauthored by Ofer Mendelevitch, explaining the workflow and four evaluation dimensions.
TREC 2024 study comparing system rankings derived from automatic and manual relevance judgments.
Updates since the talk
Current metric inputs, scoring categories, formulas and configuration details.
Full transcript
Loading transcript…