AI Engineer World's Fair 2025

open-rag-eval: RAG Evaluation without "golden" answers.

Loading slides…

Read the talk

Open RAG Eval: Evaluating Retrieval and Answers Without Golden References

Open RAG Eval captures a pipeline’s retrieved passages and answers, then evaluates relevance, information coverage, citation support, and response consistency without hand-written golden references.

From a talk by Ofer Mendelevitch

Before you start: Familiarity with RAG—retrieving passages and using them to generate an answer—is helpful.

The golden-reference bottleneck

How do you evaluate a RAG system when you have useful questions but no curated set of correct answers or ideal retrieved passages? Creating those references adds work every time the evaluation set grows. Open RAG Eval, introduced here by Ofer Mendelevitch of Vectara developer relations, addresses that bottleneck: evaluate retrieval and generation without manually supplied golden chunks or golden answers. The opening slide’s QR code provides access to the repository.

The project draws on research conducted with the University of Waterloo’s Jimmy Lin Lab. Its approach replaces the requirement for curated references with automated assessments of what the pipeline actually retrieves and generates. Understanding those assessments matters: each measures a different part of the RAG system.

0:000:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From queries to evaluation files

Start with 10, 100, or 1,000 queries that matter to your RAG application. These are example evaluation-set sizes, not throughput claims. The workflow then follows the data through three stages:

  1. Capture the pipeline’s behavior. A RAG connector collects the actual retrieved chunks and generated answers. The demonstrated integrations include Vectara, LangChain, and LlamaIndex.
  2. Evaluate the captured outputs. The resulting RAG outputs feed metrics grouped into evaluators. This separates collecting a pipeline’s behavior from assessing it.
  3. Write evaluation files. Evaluators emit files containing the results for inspection and comparison.

The connector is therefore the boundary between your application and the evaluation machinery: it supplies what the system did, without requiring you to author the ideal answer first.

Architecture diagram linking queries, a RAG connector and pipeline, RAG outputs, evaluators with metrics, model boxes, and RAG evaluations.Open full source frame
Open-RAG-Eval connects a RAG pipeline to evaluators, metrics, and models.
0:430:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:43 · section reference included

Retrieval relevance with UMBRELA

The package introduces four complementary assessments: UMBRELA for retrieval, AutoNuggetizer for generation, citation faithfulness, and hallucination detection using a Vectara model. Keeping these dimensions separate helps distinguish retrieving useful evidence from producing a well-supported answer.

UMBRELA scores individual passages against the query, without requiring golden chunks. Its relevance scale runs from 0 to 3:

ScoreMeaning described in the talk
0Passage has nothing to do with the query
3Passage is dedicated to the query and contains the exact answer

The endpoints distinguish irrelevant retrieval from a passage that directly supplies the requested information.

Mendelevitch reports that UMBRELA correlates well with human judgment, citing the Waterloo research. That supports using automated relevance assessment without golden chunks, but the consequential distinction is between agreement in evaluating systems and identical labels for every passage. The associated research supports system-ranking agreement; the talk supplies no correlation coefficient or experimental conditions for its broader qualitative claim.

1:341:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:34 · section reference included

Answer coverage with AutoNuggetizer

AutoNuggetizer evaluates generation by breaking the information an answer should cover into nuggets, or atomic information units. Instead of comparing the generated response with a single golden answer, it checks coverage of selected units:

  1. Create nuggets. Produce atomic units of information for the evaluation.
  2. Prioritize them. Assign each nugget a vital or okay importance rating, order the nuggets, and select the top 20.
  3. Judge answer coverage. An LLM judge examines the RAG response to determine whether each selected nugget is fully or partially supported by the answer.

The top-20 limit is a selection setting, not an accuracy result. The mechanism measures information coverage: an answer can cover one important point completely and another only partly.

Three-step AutoNuggetizer slide: create nuggets with vital or okay ratings; the slide labels selection as “Top 20 vital nuggets”; an LLM judge checks full or partial answer support.Open full source frame
AutoNuggetizer creates nuggets, orders them by importance, selects up to 20, and checks answer support.

For a small Python implementation of the importance-ordering step, retain each nugget’s text and rating together, then select the first 20 after sorting. This helper represents selection only; nugget creation and support judgments remain separate operations.

python

from typing import Literal, TypedDict

class Nugget(TypedDict):
    text: str
    importance: Literal["vital", "okay"]


def select_nuggets(nuggets: list[Nugget]) -> list[Nugget]:
    priority = {"vital": 0, "okay": 1}
    ordered = sorted(
        nuggets,
        key=lambda nugget: priority[nugget["importance"]],
    )
    return ordered[:20]

The original nugget records survive the selection, so the subsequent judge can assess each retained unit against the generated answer.

2:523:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:52 · section reference included

Citation support and whole-response consistency

Mendelevitch points to the papers for fuller method details before moving to two support checks. These operate at different scopes:

AssessmentScopeQuestion
Citation faithfulnessA citation and its associated claimDoes the cited passage support the claim?
Hallucination detectionThe entire responseDoes the response align with retrieved content?

Citation faithfulness distinguishes full support, partial support, and no support. A citation’s presence alone does not establish that its passage substantiates what the answer says.

For the response-level check, HHEM, Vectara’s hallucination detection model, assesses alignment between the answer and retrieved evidence. Here, support is relative to that evidence: a statement can be true elsewhere yet unsupported by the passages supplied to the model. This complements nugget coverage by asking whether the answer stays grounded in what was retrieved.

3:253:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:25 · section reference included

Inspecting the results

Once evaluation finishes, you can inspect the result files directly. For easier comparison, the talk shows a hosted interface and describes dragging the evaluation files onto open-evaluation.ai. The populated view lists queries alongside retrieval and generation scores, with groups of colored results arranged side by side. This makes the separate evaluation dimensions visible together without requiring manual inspection of every file.

Slide showing the Open Evaluation interface with query rows, two groups of colored metric scores, and aggregate scores above them.Open full source frame
The openevaluation.ai interface displays evaluation results side by side.
3:544:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:54 · section reference included

Tuning a pipeline and extending its connectors

The intended use is to optimize and tune a RAG pipeline while keeping its evaluation methods inspectable. Open source provides access to the metric implementations, so a score need not remain an opaque judgment: you can examine how the assessment works.

Vectara, LangChain, and LlamaIndex provide the demonstrated integration paths. For a custom pipeline, the extension point is the connector that captures retrieved chunks and generated answers. Mendelevitch invites issues and pull requests for additional connectors, allowing other pipelines to feed the same evaluator workflow.

Summary slide listing Apache 2.0 licensing, Vectara, LangChain and LlamaIndex connectors, an invitation to contribute connectors, and no need for golden chunks or answers.Open full source frame
Open-RAG-Eval summarizes its open-source license, connectors, and evaluation without golden references.
4:264:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:26 · section reference included

Resources

From the talk

  • Python toolkit with RAG connectors, evaluation metrics, examples and contribution instructions. Update from the repository, checked 2026-08-28: its viewer link points to openevaluation.ai and JSON is the default viewer format. Viewer functionality was not tested.

  • UMBRELAPaper1:44

    Original paper on automated retrieval relevance judgments using GPT-4o.

  • Automatic information-nugget creation and answer coverage evaluation, with initial comparisons against human assessment.

  • Model documentation and usage examples for assessing whether generated text is supported by supplied evidence.

Updates since the talk

Transcript download
Full transcript

Loading transcript…