AI Engineer World's Fair 2024
Navigating RAG Optimization with an Evaluation-Driven Compass
Loading slides…
Read the talk
Navigating RAG Optimization with an Evaluation-Driven Compass
A documentation question-answering system improves through ten experiments that separate missing evidence from noisy context, then use reranking and hybrid search to address distinct failures.
From a talk by Atita Arora and Deanna Emery
Before you start: Familiarity with embeddings, vector search, and language-model prompts will help you follow the experiments.
Start with the simplest retrieval pipeline
Split documents into chunks, embed those chunks, and store their vectors. When a user asks a question, retrieve the relevant chunks and add them to a prompt for a language model. This is the starting point for retrieval-augmented generation, or RAG: the model answers with information drawn from an external knowledge source. The implementation is straightforward to describe, but each step introduces a choice that can change the answer.
More advanced pipelines add query expansion or rewriting before retrieval, then reranking or fusion afterward. Agentic versions can route questions to task-specific agents and incorporate optimization systems such as DSPy. These additions create more opportunities to improve results—and more components whose contribution needs to be measured.
The architectures Atita Arora presents share a vector database. For knowledge management, that layer must support relevant retrieval, large document collections, and efficient resource use. The demonstration uses Qdrant, an open-source vector search database written in Rust.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An incorrect answer can begin upstream
Before changing the model, trace the evidence through the pipeline. Information might be absent from the source collection or lost during extraction. Either gap can produce an incomplete answer even when retrieval and generation behave as intended. During ingestion, chunk boundaries determine which facts remain together, while the embedding model determines how well the system represents the terminology in those chunks.
Retrieval adds three closely related decisions: which documents are relevant, how many to retrieve, and in what order to present them. Generation can inherit the resulting omissions or distractions, then introduce another failure by straying from the supplied context. A fluent answer therefore does not tell you which stage succeeded.
The query itself can also be vague or ambiguous. Beyond an individual answer, an application must maintain coherent conversations, handle hundreds or thousands of concurrent users, and meet security and compliance requirements. These are separate concerns from whether one retrieved passage contains the right fact, but they remain part of productionizing RAG.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Improve what enters the context
The first interventions follow the path of the data.
- Extraction and cleaning: Recover information that was missed and remove defects before indexing it.
- Embedding choice: Start with a general-purpose model, then test whether a model that understands the domain's terminology improves retrieval.
- Metadata: Attach useful document attributes and filter on them to exclude irrelevant material.
- Chunking: Test smaller chunks or semantic boundaries instead of assuming the initial split is appropriate.
These changes affect what evidence is available and how precisely it can be retrieved.
Context size requires its own experiment. Lost in the Middle motivates examining where useful evidence appears within a long context; simply supplying more text need not make that evidence easier for the model to use. Arora also points to HNSW, BM25, and graph-based retrieval approaches as options to explore. Once candidates have been retrieved, reranking can prioritize the documents that deserve space in the final context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose changes with evidence
Generation offers additional levers, including prompt design and chain-of-thought prompting. Retrieval can combine dense vectors for semantic matching with sparse vectors when exact terms matter, over one field or several. Different language models may produce different answers from the same context, while agents can handle task-specific information fetching or access to custom data.
A menu of techniques is not a sequence of required upgrades. With so many possible changes, the practical problem is deciding which failure to address next. Evaluation supplies the feedback: establish how the system behaves, make a targeted change, and measure whether it moves the application toward its goal.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build an evaluation around the documents
Because a RAG system depends on its underlying documents, Deanna Emery starts with evaluation data tailored to the application and domain. The dataset should contain realistic inputs and expected outputs. Quotient supports running these experiments and can help generate an initial evaluation dataset when one is not already available.
The evaluation workflow connects retrieval to answer generation:
- Submit the evaluation questions to the populated Qdrant database.
- Attach the returned contexts to the evaluation examples.
- Submit those examples to Quotient, which orchestrates prompt formatting, language-model execution, and metric computation.
This makes the retrieved context part of the experiment rather than treating the final answer as the only observable result.
The companion walkthrough uses notebook outputs executed before the presentation. Its application is question answering over Qdrant's documentation, intended to help users find answers quickly. That concrete use case determines what the experiments should optimize.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Distinguish missing evidence from noisy evidence
A documentation assistant should be helpful, but inaccurate instructions can actively misguide a user. Emery therefore makes faithfulness the principal optimization metric, with the aim of minimizing hallucinations.
The metrics answer different questions in this demonstration:
| Metric | Diagnostic question |
|---|---|
| Context relevance | Does the retrieved context contain the information needed to answer? |
| Chunk relevance | How much retrieved information is useful rather than noise? |
| Faithfulness | Does the answer remain faithful to the supplied context? |
| General text quality | How does the generated text perform on the accompanying quality measures? |
The distinction between the first two is crucial. A context can contain the necessary answer while surrounding it with a large amount of irrelevant material. The experiment needs to detect both conditions. These are the speakers' working metric definitions; no numerical formula is needed to follow the subsequent diagnosis.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate chunk size from retrieval window
The initial configuration is deliberately simple: an embedding model, a chunking strategy, a retrieval window, and Mistral Instruct for generation. The first question is whether the model needs more context. Experiment two increases the chunking parameters to supply larger chunks.
Emery reports that larger chunks slightly improve text quality and increase context relevance, but substantially reduce faithfulness and chunk relevance. More of the necessary information is now present, yet a smaller portion of the retrieved material is useful. The additional context has not produced a more faithful answer.
That combination suggests a narrower intervention: restore the smaller chunks, but retrieve more of them. This changes the retrieval window without retaining the larger chunk size. Among the first three experiments, smaller chunks with a larger retrieval window achieve the best relevance and faithfulness scores. The useful lesson is the distinction between larger chunks and more chunks: both add context, but they change its composition differently.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Change models, then inspect individual failures
The next two experiments test a different embedding model and a different generation model. The alternative embedding model does not improve the result in this experiment. In experiment five, switching to GPT-3.5 while retaining the same RAG configuration improves all reported metrics. This establishes a stronger naive baseline before adding another retrieval stage.
The aggregate scores still conceal an uneven experience. A large variance in context relevance indicates that some questions retrieve useful evidence much less reliably than others. Emery opens the two examples with the worst hallucination performance and inspects their retrieved documents.
Those documents appear to overemphasize recurring words such as Qdrant, support, and search. They repeat terms from the questions but contain unrelated information. This is a diagnosis drawn from the retrieved text: matching familiar vocabulary is not sufficient to recover the passage that answers a particular question.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve broadly, then rerank
Expanding the retrieval window could improve the chance of finding the missing evidence, but passing every additional document to the generator would also add noise. Reranking separates those decisions: retrieve a broader candidate set, score those candidates again, and send only the top few onward. The candidate window can grow without forcing the final context to grow with it.
The selection step can be expressed directly in Python. Here, scores contains one relevance score per retrieved candidate, with larger scores indicating greater relevance; scoring happens before this function is called.
python
from collections.abc import Sequence
def select_context(
candidates: Sequence[str],
scores: Sequence[float],
final_count: int,
) -> list[str]:
if len(candidates) != len(scores):
raise ValueError("Each candidate needs one score")
if final_count < 1:
raise ValueError("final_count must be positive")
ranked = sorted(
zip(candidates, scores),
key=lambda item: item[1],
reverse=True,
)
return [text for text, _ in ranked[:final_count]]
The separate final_count makes the boundary explicit: retrieval decides which documents are eligible, while reranking decides which eligible documents reach generation.
The next three experiments compare rerankers from mixedbread, Cohere, and Jina ColBERT against the best naive configuration. Emery reports that reranking generally improves context relevance and faithfulness, with Cohere achieving the best scores on those two metrics in this comparison. The chart also displays chunk relevance and the accompanying text-quality measures: ROUGE-L, BERTScore, and BERT sentence similarity.
Returning to the same two difficult examples reveals where that improvement occurred. For the second previously failing example, reranking retrieves documents containing the answer, and context relevance and faithfulness are close to one. Looking at the documents alongside the scores connects the metric movement to a concrete change in available evidence.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Recover evidence that reranking cannot find
The first difficult example still performs poorly. Even after expanding the candidate window and reranking it, the system has not identified the relevant documents. Reranking cannot promote evidence that retrieval never supplies. Qdrant's documentation contains specialized terminology, jargon, and acronyms, making the limitations of a general-purpose embedding model a plausible next explanation.
Fine-tuning or training an embedding model could address that mismatch, but requires time and money. The next experiment instead changes how candidates are found: hybrid search combines dense vectors with sparse vectors that help capture documents sharing the query's terminology. Dense retrieval supplies semantic matches; sparse retrieval adds a route to evidence whose exact technical vocabulary matters.
Emery tests hybrid retrieval both with and without reranking. Hybrid search with Cohere reranking achieves the best reported performance across all metrics except chunk relevance. That exception preserves an important distinction: the configuration that performs best overall does not necessarily produce the highest proportion of useful retrieved material.
On both inspected difficult examples, hybrid search with reranking brings context relevance and faithfulness close to one. These are results for those two examples, not the dataset aggregate. The retrieved material now better contains the information needed to answer, supporting the diagnosis that domain-specific terminology was a significant retrieval obstacle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the experiment loop tied to the application
Across ten experiments on Qdrant documentation question answering, Emery reports that faithfulness rises from 0.76 to just under 0.85, while the generic question-answering prompt remains unchanged. Retrieval settings, retrieval methods, and the generation model do change. The scores describe this demonstration's evaluation, rather than answer-accuracy percentages.
The progression is an experimental method: make incremental changes, evaluate a combination of metrics, inspect patterns in the underlying examples, form a hypothesis, and repeat. Aggregate scores identify a direction; individual retrieved documents help explain why the system is failing and what to test next.
Continued improvement depends on keeping that evidence relevant. Domain understanding helps identify useful interventions, and an up-to-date evaluation dataset keeps the tests aligned with the application. The closing constraint is to avoid over-engineering: add complexity when a carefully chosen combination of metrics and observed failures gives it a purpose.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Companion repository linking the talk's hybrid retrieval and reranking example, alongside other RAG evaluation workshops.
Experiments showing how the position of relevant evidence affects multi-document question answering and key-value retrieval.
Rust-based vector database with vector search, payload filtering and client libraries.
Current guides to composing language-model programs and optimizing them against evaluation metrics.
Further reading
Notebook and saved experiment files covering chunking, embedding models, generation models, rerankers and hybrid retrieval.
Updates since the talk
Current instructions for combining dense and sparse searches and composing retrieval stages through Qdrant's Query API.
Full transcript
Loading transcript…