# Building Production-Ready RAG Applications

Jerry Liu

AI Engineer Summit 2023

[Talk page](https://ai.engineer/talks/TRjq7t2Ms5I-production-ready-rag) · [Video](https://www.youtube.com/watch?v=TRjq7t2Ms5I)

## Production-Ready RAG Starts With Measurement, Not More Context

Jerry Liu explains how to diagnose retrieval failures, evaluate complete RAG pipelines, improve precision through chunking and metadata, and selectively introduce agents or fine-tuning.

![Selected presentation frame from Building Production-Ready RAG Applications: Jerry Liu at 493 seconds](https://ai.engineer/api/talk-essay-media/2923710baec033b198c6e4de312ce4ba91f2155c92af16b619e83c21524cdd4b)

Production-Ready RAG Starts With Measurement, Not More Context

### The baseline architecture—and why it breaks

[0:14](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=14s)

Jerry Liu, co-founder and CEO of LlamaIndex, presents two broad approaches for helping language models work with data outside their training: **retrieval augmentation**, which supplies external context in the prompt while keeping the model fixed, and **fine-tuning**, which incorporates new knowledge through updates to model weights or an adapter. A basic **Retrieval Augmented Generation (RAG)** system consists of data ingestion followed by querying, which itself includes retrieval and synthesis. [0:14](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=14s) [1:09](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=69s)

This architecture underlies familiar applications that answer questions over PDFs and other unstructured data, but a working prototype is not necessarily production-ready. If retrieval misses relevant material, the language model never receives the context required to answer correctly. **Low precision** introduces irrelevant chunks, unnecessary material, hallucination risk, and lost-in-the-middle problems; **low recall** means necessary information never appears in the retrieved set. Outdated information, irrelevant outputs, toxicity, and bias introduce additional limitations beyond retrieval itself. [2:04](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=124s) [3:01](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=181s)

Improvement opportunities span the entire pipeline: the underlying data and chunk boundaries, embedding representations, retrieval algorithms, and the final synthesis stage. Liu also argues that language models can contribute before answer generation by breaking complicated questions into simpler questions, routing requests across data sources, and supporting more sophisticated reasoning. [3:59](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=239s) [4:55](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=295s)

![Selected presentation frame from Building Production-Ready RAG Applications: Jerry Liu at 267 seconds](https://ai.engineer/api/talk-essay-media/6f56ee551b0a3a8d9a1f86d66b3c933aa4721a2510e4903e84dd6e8eee86f590)

The baseline architecture—and why it breaks

- RAG keeps model weights fixed and supplies external context, while fine-tuning updates model weights or an adapter.
- The baseline pipeline ingests data, retrieves relevant information, and synthesizes an answer.
- Low precision adds irrelevant chunks and hallucination risk; low recall omits information needed to answer.
- Improve data preparation, embeddings, retrieval, and synthesis; use LLM reasoning to decompose queries or route across sources.

### Build evaluation before optimizing the pipeline

[4:55](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=295s)

A production improvement strategy needs a **task-specific benchmark** before architectural changes can be meaningfully compared. Liu distinguishes between evaluating the complete query-to-response experience and evaluating individual components. When retrieval appears to be the bottleneck, measuring final answer quality alone does not adequately explain whether a proposed retrieval change actually returns more relevant evidence. [4:55](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=295s) [5:50](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=350s)

For retrieval evaluation, construct a dataset linking input queries to the document identifiers relevant to each query. Those labels can come from human annotation, production user feedback, or synthetic generation. Once the dataset exists, standard ranking measures such as **success rate**, **hit rate**, **MRR**, and **NDCG** make retrieval quality measurable; Liu frames this component as an information-retrieval problem rather than an exclusively language-model problem. [6:45](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=405s)

For end-to-end evaluation, assemble queries alongside human annotations, user feedback, reference answers, or synthetic examples generated with GPT-4. Run those examples through both retrieval and synthesis, then apply language-model-based evaluations, with or without reference labels. This separates the question of whether the system found useful source material from whether it ultimately produced a useful answer. [6:45](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=405s) [7:44](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=464s)

![Selected presentation frame from Building Production-Ready RAG Applications: Jerry Liu at 434 seconds](https://ai.engineer/api/talk-essay-media/0d2b0845283a5bc54b53ffc0276a5f06689d4f8f0a94e166d02381f742b68ecc)

Build evaluation before optimizing the pipeline

- Define a task-specific benchmark before changing retrieval or synthesis.
- Evaluate retrieval separately from end-to-end answers to identify which component needs improvement.
- Pair queries with relevant document IDs and score retrieval using success rate, hit rate, MRR, or NDCG.
- Evaluate complete responses using human annotations, user feedback, reference answers, or GPT-4-generated examples.

### Start with chunking, parsing, and metadata

[8:34](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=514s)

Liu recommends beginning with **table-stakes RAG techniques** rather than immediately adopting the most complex interventions. Practical starting points include better parsing instead of uniformly splitting documents, adjusting chunk sizes, using hybrid search where supported, and applying metadata filters. More advanced options such as reranking and recursive retrieval can follow, but their value should be established against the benchmark rather than assumed. [8:34](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=514s)

Chunk size is particularly consequential because **more retrieved tokens do not necessarily improve answer quality**. Information located in the middle of a long context window can be harder for the model to use, and reranking retrieved material does not automatically improve the final generated response. Liu describes observing a dataset-specific optimal chunk size and cases where reranking increased error metrics, reinforcing the need to evaluate synthesis outcomes as well as retrieval changes. [9:30](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=570s) [10:26](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=626s)

**Metadata filtering** adds structured context to otherwise unstructured chunks. Useful metadata can include page numbers, document titles, summaries of adjacent chunks, or generated questions that a chunk could answer. For a question about risk factors in a 2021 SEC 10Q document, semantic search alone might retrieve documents from other years; inferring a structured filter equivalent to year equals 2021 and combining it with semantic search narrows the candidate set and improves precision. [10:26](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=626s) [11:20](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=680s)

![Selected presentation frame from Building Production-Ready RAG Applications: Jerry Liu at 703 seconds](https://ai.engineer/api/talk-essay-media/54ba5fcdf49f39de027669647fb5ca7997fdce9fd4f9fdd26bd3ae4b0a0ed935)

Start with chunking, parsing, and metadata

- Begin with better parsing, chunk-size adjustments, hybrid search, and metadata filters before advanced retrieval.
- More retrieved tokens and reranking do not necessarily improve generated answers; test dataset-specific chunk sizes.
- Attach structured metadata such as page numbers, document titles, adjacent-chunk summaries, or generated questions.
- Combine semantic search with inferred filters such as year equals 2021 to exclude irrelevant documents and improve precision.

### Retrieve narrowly, then expand the evidence

[12:18](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=738s)

**Small-to-big retrieval** addresses a mismatch between what makes text easy to retrieve and what makes it useful for answer synthesis. Embedding a large passage can blur the specific information a query targets because the representation includes surrounding material. Instead, index smaller units, potentially down to individual sentences, and retrieve those more specific units first. [12:18](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=738s)

After identifying the relevant small unit, expand to a larger surrounding window before passing evidence to the language model. This preserves precise matching while still providing enough context to synthesize a coherent answer. Liu describes using a smaller retrieval count, such as K equals 2, instead of retrieving K equals 5 large chunks that may flood the context window and bury the relevant material. [13:05](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=785s)

A related approach embeds a **reference to a parent chunk** through a smaller excerpt, a summary, or a generated question associated with that larger passage. The retrieval representation is optimized for matching likely queries, while the returned parent context supplies the information needed for synthesis. The key design principle is that the object optimized for embedding-based retrieval does not have to be identical to the evidence ultimately delivered to the model. [13:51](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=831s)

![Selected presentation frame from Building Production-Ready RAG Applications: Jerry Liu at 810 seconds](https://ai.engineer/api/talk-essay-media/0fe09e9b98069a40f64af7c2c141a9070304c64172adcb083d4cf12e2ce1e8c8)

Retrieve narrowly, then expand the evidence

- Embed sentences or other small units to match specific queries more precisely than large, unfocused chunks.
- After retrieving a small unit, expand its surrounding context before answer synthesis.
- A smaller top K can reduce excess context and lost-in-the-middle problems compared with retrieving more large chunks.
- Retrieve using a smaller excerpt, summary, or generated question, then return the associated parent context for synthesis.

Small-to-big retrieval
Match granular evidence first, then expand context for answer synthesis.
- Small units — Embed sentences or smaller chunks.
- Precise retrieval — Retrieve granular matches for the query.
- Expanded context — Return the surrounding window or parent chunk.
- Answer synthesis — Use broader context to generate the response.
- Small units → Precise retrieval (match specific context)
- Precise retrieval → Expanded context (expand retrieved evidence)
- Expanded context → Answer synthesis (supply sufficient context)

### Use agents and fine-tuning when the task justifies them

[4:55](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=295s)

Some questions exceed the capabilities of a single top-K retrieval pass. A task might require multiple reasoning steps, summarizing an entire document, or comparing several documents. Liu describes an exploratory **multi-document agent** architecture in which each document exposes tools for summarization and fact-specific question answering; because an agent cannot directly access unlimited tools, a retrieval layer first selects the relevant document tools before the agent acts on them. [13:51](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=831s) [14:52](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=892s) [15:47](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=947s)

Agents are potentially more expressive, but Liu explicitly notes the tradeoffs: they can be harder to implement and may increase **latency and cost**. Their distinguishing feature is that the system retrieves executable capabilities rather than merely inserting retrieved text into a context window. This combination of retrieval and tool use is presented as a direction for more complex analysis, not as the default replacement for simpler RAG pipelines. [9:30](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=570s) [14:52](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=892s) [15:47](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=947s)

**Embedding fine-tuning** can adapt retrieval to domain-specific questions when pretrained representations return inappropriate information. Liu describes generating synthetic queries from raw text chunks and using those examples to fine-tune either the base embedding model or an adapter. Adapter-based tuning does not require access to the base model’s weights, and tuning only the query side can avoid re-indexing the entire document corpus. [15:47](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=947s) [16:44](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1004s)

Fine-tuning can also target the language model responsible for synthesis. Liu describes exploring synthetic training examples produced by a larger model such as GPT-4 and distilling those capabilities into 3.5 Turbo, with the aim of improving reasoning, longer responses, and structured outputs. These possibilities are presented as areas of exploration rather than guaranteed production results, and their usefulness still depends on the task-specific evaluations established earlier. [4:55](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=295s) [17:33](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1053s)

![Selected presentation frame from Building Production-Ready RAG Applications: Jerry Liu at 1018 seconds](https://ai.engineer/api/talk-essay-media/fee338923c248bb086ce14cff3fa13959c41cc7282c30a321af2c920566d4753)

Use agents and fine-tuning when the task justifies them

- Use multi-document agents for tasks requiring sequential reasoning, document summarization, or cross-document comparison.
- Retrieve relevant document tools before executing their summarization or fact-specific question-answering capabilities.
- Agents can increase implementation complexity, latency, and cost; justify their use against task-specific evaluation.
- Fine-tune embeddings with synthetic queries; adapters avoid requiring base weights, and query-only tuning avoids corpus re-indexing.

Multi-document agent execution
Retrieve relevant document capabilities, then execute them for more complex analysis.
- Documents — Represent each document through available capabilities.
- Document tools — Expose summarization and fact-specific question answering.
- Tool retrieval — Select relevant tools from a larger collection.
- Agent execution — Act on retrieved tools instead of inserting raw text.
- Documents → Document tools (model as capabilities)
- Document tools → Tool retrieval (retrieve relevant tools)
- Tool retrieval → Agent execution (execute selected tools)

## Key takeaways

- Define a **task-specific benchmark** before changing the pipeline, and evaluate retrieval quality separately from complete query-to-answer performance. [4:55](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=295s) [5:50](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=350s) [6:45](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=405s)
- Start with **better parsing, chunk-size tuning, hybrid search, and metadata filters** before introducing more complex retrieval or agent architectures. [8:34](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=514s) [10:26](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=626s) [11:20](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=680s)
- More retrieved context and reranking do not automatically improve answers; measure their effects because context overload and **lost-in-the-middle** behavior can worsen results. [9:30](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=570s) [10:26](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=626s) [13:05](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=785s)
- Use **small-to-big retrieval** to match precise, compact evidence first and then expand to the broader context needed for synthesis. [12:18](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=738s) [13:05](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=785s) [13:51](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=831s)
- Reserve **multi-document agents** and **fine-tuning** for needs that justify their added complexity, latency, or cost; query-side adapter tuning can improve retrieval without re-indexing the full corpus. [9:30](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=570s) [14:52](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=892s) [16:44](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1004s)

## Slides

### Building Production-Ready RAG Applications

Jerry Liu introduces his talk on building production-ready RAG applications.

[0:20](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=20s) [Title slide for Jerry Liu’s talk on production-ready RAG applications.](https://assets.ai.engineer/sites/aiecode2025/sha256/faed4908b080cb46103ef863fc7d349884974eaffc488a3a75734a41198e3a8f.webp)

Building Production-Ready RAG Applications

Jerry Liu, LlamaIndex co-founder/CEO

### GenAI Enterprise Use Cases

GenAI can extract document fields, answer questions from a knowledge base, and support conversations. The workflow example reads inbox messages and sends an email suggesting next steps.

[0:29](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=29s) [Diagrams illustrate document processing, knowledge search, conversational agents, and workflow automation.](https://assets.ai.engineer/sites/aiecode2025/sha256/e2d06bf0f9298f0efea0bbbcbdb80716b9ca212dc32c055c6db82930a7605d3a.webp)

GenAI - Enterprise Use-cases

Document
Topic:
Summary:
Author:
Document Processing
Tagging & Extraction

Knowledge Base
Answer:
Sources: ...
Knowledge Search & QA

Agent: ...
Human: ...
Agent: ...
Conversational Agent

Inbox
read
Workflow:
• Read latest messages from user A
• Send email suggesting next-steps
write
Email
Workflow Automation

### Retrieval Augmentation

Retrieval augmentation keeps the model fixed and puts source text into the prompt alongside the question before sending it to the LLM.

[0:45](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=45s) [Source text is added to an input prompt before it is sent to an LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/93f62f0ae598176bdc1f844d5069d0eb89b51ffca5c7002214282ef2bb08f310.webp)

Paradigms for inserting knowledge

Retrieval Augmentation - Fix the model, put context into the prompt

Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep...

Input Prompt

Here is the context:
Before college the two main things...

Given the context, answer the following question:
{query_str}

LLM

### Fine-Tuning

Fine-tuning incorporates knowledge by training on new data and updating the model’s weights or an adapter.

[1:13](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=73s) [A diagram connects source text and an LLM to illustrate fine-tuning model weights.](https://assets.ai.engineer/sites/aiecode2025/sha256/6825e5516a82381c39161bad98d56f91cdffab3be5bd11f2638cca0acffebc0e.webp)

Paradigms for inserting knowledge

Fine-tuning - baking knowledge into the weights of the network

N

Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep...

LLM

RLHF, Adam, SGD, etc.

### RAG Stack

Introduces retrieval-augmented generation and the stack for building a question-answering system.

[1:38](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=98s) [Section slide titled RAG Stack.](https://assets.ai.engineer/sites/aiecode2025/sha256/149869d42ee9a2af4ce946eef2af3f1c0fb424460845987d4f70a013f333b04b.webp)

RAG Stack

### Current RAG Stack for building a QA System

Ingestion splits a document into chunks and stores them in a vector database. Querying retrieves chunks for an LLM to synthesize an answer. LlamaIndex can set this up in around five lines of code.

[1:43](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=103s) [A document is split into chunks stored in a vector database; retrieved chunks are sent to an LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/4bcdc7a94619006633d3c88b4c3ddf2801ddef12193f30d73b4b868377ddfe48.webp)

Current RAG Stack for building a QA System

Data Ingestion
Data Querying (Retrieval + Synthesis)

Doc
Chunk
Chunk
Chunk
Chunk
Vector Database
Chunk
LLM

5 Lines of Code in LlamaIndex!

[1:49](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=109s) [Diagram showing documents split into chunks, stored in a vector database, and retrieved for an LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/2a6a9ca6410e2f464bfb6637b76b7012f5033173d7efd6d9fe86baae50cc50ce.webp)

Current RAG Stack for building a QA System

Data Ingestion
Data Querying (Retrieval + Synthesis)

Doc
Chunk
Chunk
Chunk
Chunk
Vector Database
Chunk
LLM

5 Lines of Code in LlamaIndex!

### Challenges with Naive RAG

The speaker introduces response quality as a challenge in bringing naive RAG applications to production.

[2:22](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=142s) [Section title: Challenges with Naive RAG.](https://assets.ai.engineer/sites/aiecode2025/sha256/0b618ccf4c6fa2ce5d3387fab2a0bbdf1475650b72207bb544c2dd749e86c8cd.webp)

Challenges with “Naive” RAG

### Challenges with Naive RAG (Response Quality)

Low precision brings irrelevant chunks into the context; low recall leaves needed information out. Even with retrieval, the LLM can produce hallucinations, irrelevant answers, or toxic or biased responses.

[3:42](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=222s) [Slide lists retrieval problems and response-generation problems, including hallucination, irrelevance, and toxicity or bias.](https://assets.ai.engineer/sites/aiecode2025/sha256/712597ebcc22021668659e2cd158250f06fabf73d758a0b6a0206139b52a7fa2.webp)

Challenges with Naive RAG (Response Quality)

• Bad Retrieval
  ◦ Low Precision: Not all chunks in retrieved set are relevant
    ▪ Hallucination + Lost in the Middle Problems
  ◦ Low Recall: Now all relevant chunks are retrieved.
    ▪ Lacks enough context for LLM to synthesize an answer
  ◦ Outdated information: The data is redundant or out of date.
• Bad Response Generation
  ◦ Hallucination: Model makes up an answer that isn’t in the context.
  ◦ Irrelevance: Model makes up an answer that doesn’t answer the question.
  ◦ Toxicity/Bias: Model makes up an answer that’s harmful/offensive.

### What Do We Do?

RAG can improve at four stages: enrich document data, optimize embeddings, retrieve beyond top-k lookup, and use LLMs to break down questions and route them to data sources.

[3:57](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=237s) [Slide asks how to improve data, embeddings, retrieval, and synthesis, with a RAG pipeline diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/dc05eef9f6f4a5ad59485396e017d8b15e79ef7353f7ad5f27b5be84c047c468.webp)

What do we do?

• Data: Can we store additional information beyond raw text chunks?
• Embeddings: Can we optimize our embedding representations?
• Retrieval: Can we do better than top-k embedding lookup?
• Synthesis: Can we use LLMs for more than generation? v

Doc → Chunk → Vector Database → Chunk → LLM
Doc → Chunk → Vector Database

Data
Embeddings
Retrieval
Synthesis

### Evaluation

Before trying RAG techniques, establish a task-specific way to measure performance.

[5:17](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=317s) [Section title: Evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/1cd4f2f7253d53833ecea32ffc2a5770d1dad9e6aa2373b692b5acca6c029d7a.webp)

Evaluation

### Evaluation

Evaluate RAG end to end from query to response, and assess retrieval and synthesis separately to identify which component needs improvement.

[5:52](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=352s) [Evaluate RAG components in isolation and end to end; a diagram shows retrieval from a vector database followed by synthesis with an LLM.](https://assets.ai.engineer/sites/aiecode2025/sha256/a160d016b511f34921861e1e55428b344a489cf02b349b4d9b22c287452de11c.webp)

Evaluation

• How do we properly evaluate a RAG system?
  ○ Evaluate in isolation (retrieval, synthesis)
  ○ Evaluate e2e

Vector Database → Chunk → LLM
Retrieval
Synthesis

### Evaluation in Isolation (Retrieval)

A retriever ranks document IDs for each query. An evaluator compares them with expected relevant IDs and scores the ranking using metrics such as MRR, Precision@K, and NDCG.

[6:29](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=389s) [Retrieval evaluation uses queries and relevant documents as a dataset, then measures ranking metrics against expected IDs.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4d26c16f7e42738c35ba21014fbf313a4f9d3fe5ab2bc6854e910bca93eb9e0.webp)

Evaluation in Isolation (Retrieval)

• Evaluate quality of retrieved chunks given user query
• Create dataset
  ○ Input: query
  ○ Output: the “ground-truth” documents relevant to the query
• Run retriever over dataset
• Measure ranking metrics
  ○ Success rate / hit-rate
  ○ MRR
  ○ Hit-rate

User Query → Retriever → Retrieved IDs:
node_24
node_2
node_5
node_8
node_50

Retrieved IDs:
node_24
node_2
node_5
node_8
node_50

Expected IDs: node_2, node_8

Retriever Evaluator
MRR
Precision@K
NDCG
...

### Evaluation E2E

Run queries through the full RAG pipeline, then evaluate generated responses without labels for qualities such as faithfulness and relevance, or compare them with reference answers for correctness.

[7:34](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=454s) [A RAG pipeline and evaluation workflow comparing label-free metrics with labeled correctness evaluation.](https://assets.ai.engineer/sites/aiecode2025/sha256/b4c717c81ef60394967197369dc1d42ad70566a886beea86318083de0f5ad040.webp)

Evaluation E2E

• Evaluation of final generated response given input
• Create Dataset
  ◦ Input: query
  ◦ [Optional] Output: the “ground-truth” answer
• Run through full RAG pipeline
• Collect evaluation metrics:
  ◦ If no labels: label-free evals
  ◦ If labels: with-label evals

User Query → RAG Pipeline → Generated Response
Context

Generated Response
[Optional] Context
→ Label-free Evaluator
Faithfulness
Relevancy
Adheres to Guidelines
Toxicity-free

Generated Response
Actual Response
→ With-Label Evaluator
Correctness
etc.

### Optimizing RAG Systems

With an evaluation benchmark defined, the talk turns to improving the RAG system.

[8:13](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=493s) [Section title: Optimizing RAG Systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/9a9e649ae1dd0c626fb7853b7a88d7368c0d9187a7a7329432edf0625be92ca0.webp)

Optimizing RAG Systems

### From Simple to Advanced RAG

Start with parsing, chunk sizes, hybrid search and metadata filters. Advanced retrieval, agents and fine-tuning offer more expressive options, with potentially greater implementation effort, latency and cost.

[8:22](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=502s) [RAG approaches range from basic retrieval improvements to advanced retrieval, fine-tuning, and agentic behavior, with increasing complexity and cost.](https://assets.ai.engineer/sites/aiecode2025/sha256/e12693c3a7b69831d54e27b88c22d14b50beaa2bdbeeb414cec58aeb042edf68.webp)

From Simple to Advanced RAG

Table Stakes
Better Parsers
Chunk Sizes
Hybrid Search
Metadata Filters

Advanced Retrieval
Reranking
Recursive Retrieval
Embedded Tables
Small-to-big Retrieval

Fine-tuning
Embedding fine-tuning
LLM fine-tuning

Agentic Behavior
Routing
Query Planning
Multi-document Agents

Less Expressive
Easier to Implement
Lower Latency/Cost

More Expressive
Harder to Implement
Higher Latency/Cost

### Table Stakes: Metadata Filtering

Metadata adds structured context to text chunks, such as a page number or organization. It can help retrieval and response quality and can be used with vector database filters.

[9:49](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=589s) [Metadata examples and benefits for retrieval, response quality, and vector database filtering, illustrated with a text chunk.](https://assets.ai.engineer/sites/aiecode2025/sha256/74bb65ccbda2fb1fbd90aa67e5a5a7d70b44c1c4ac00e42e4820365b728ebe70.webp)

Table Stakes: Metadata Filtering

• Metadata: context you can inject into each text chunk
• Examples
  ◦ Page number
  ◦ Document title
  ◦ Summary of adjacent chunks
  ◦ Questions that chunk can answer (reverse HyDE)
• Benefits
  ◦ Can Help Retrieval
  ◦ Can Augment Response Quality
  ◦ Integrates with Vector DB Metadata Filters

Example of Metadata
{"page_num": 1, "org": "OpenAI"} — Metadata
We report the development of GPT-4, a large-scale, multimodal... — Text Chunk

[11:02](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=662s) [Examples and benefits of metadata attached to text chunks, with a metadata and text chunk diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/9aefed0ca8e954c47d49de7bff1ab0436ee0f7e0f956d6b1a78a15c1e09a375b.webp)

Table Stakes: Metadata Filtering

• Metadata: context you can inject into each text chunk
• Examples
  ◦ Page number
  ◦ Document title
  ◦ Summary of adjacent chunks
  ◦ Questions that chunk can answer (reverse HyDE)
• Benefits
  ◦ Can Help Retrieval
  ◦ Can Augment Response Quality
  ◦ Integrates with Vector DB Metadata Filters

Example of Metadata
{"page_num": 1, "org": "OpenAI"} — Metadata
We report the development of GPT-4, a large-scale, multimodal... — Text Chunk

### Metadata Filtering: Semantic Search Example

A search for 2021 risk factors returns top-4 chunks from 2019, 2020 and 2021. Combining semantic search with a year metadata filter can narrow results to 2021.

[11:29](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=689s) [A semantic search for 2021 risk factors returns document chunks from multiple years.](https://assets.ai.engineer/sites/aiecode2025/sha256/61f630f984b81dceab6c069dd8db8aacccdb4453bf4abbcaf9445a0967ab50c4.webp)

Table Stakes: Metadata Filtering

Question: “Can you tell me the risk factors in 2021?”

Raw Semantic Search is low precision.

query_str:
<query_embedding>

Single Collection of all 10Q Document Chunks

top-4
2020 10Q chunk 6
2021 10Q chunk 4
2019 10Q chunk 7
2021 10Q chunk 7

No guarantee you’ll return the relevant document chunks!

### Advanced Retrieval: Small-to-Big

Small-to-big retrieval embeds individual sentences, then expands the matched sentence into a surrounding window for LLM synthesis. The AMOC example highlights the match in red and its context in green.

[11:51](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=711s) [A sentence is retrieved for an AMOC question, then expanded to surrounding context for the language model.](https://assets.ai.engineer/sites/aiecode2025/sha256/47a3d98063af783697e737d82bbfb0c8ea4ed2b97901c513faf329a830688b19.webp)

Advanced Retrieval: Small-to-Big

Intuition: Embedding a big text chunk feels suboptimal.

Solution: Embed text at the sentence-level - then expand that window during LLM synthesis

Embed Sentence → Link to Expanded Window

Question: What are the concerns surrounding the AMOC?

Embedding Lookup

Continuous observation of the Atlantic meridional overturning circulation (AMOC) has improved the understanding of its variability (Frajka-Williams et al., 2019), but there is low confidence in the quantification of AMOC changes in the 20th century because of low agreement in quantitative reconstructed and simulated trends. Direct observational records since the mid-2000s remain too short to determine the relative contributions of internal variability, natural forcing and anthropogenic forcing to AMOC change (high confidence). Over the 21st century, AMOC will very likely decline for all SSP scenarios but will not involve an abrupt collapse before 2100.

3.2.2.4 Sea Ice Changes

Sea ice is a key driver of polar marine life, hosting unique ecosystems and affecting diverse marine organisms and food webs through its impact on light penetration and supplies of nutrients and organic matter (Arrigo, 2014).

What the LLM Sees
What the LLM Sees

### Agentic Behavior: Multi-Document Agents

A multi-document agent retrieves three document agents; the diagram shows vector search and summarization at the document level for questions requiring reasoning across documents.

[14:26](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=866s) [A multi-document agent selects document agents with search and summarization tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/5c5d398d2dcf983db23b334852e9a0f53ba7a4893aeda040736a903e87790b4f.webp)

Agentic Behavior: Multi-Document Agents

Intuition: There’s certain questions that “top-k” RAG can’t answer.

Solution: Multi-Document Agents
• Fact-based QA and Summarization over any subsets of documents
• Chain-of-thought and query planning.

Multi-Document Agent
Retrieve top_k=3

…
Document Agent
Document Agent
Document Agent
Document Agent
Document Agent
Document Agent
…

Vector Search
Summarization
…
…

### Agentic Behavior: Multi-Document Agents

A multi-document agent retrieves three document agents, which provide tools for fact-based questions and summaries across documents.

[14:53](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=893s) [A multi-document agent selects document agents with search and summarization tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/3475ea45a8ed9b7834d1a626033ec1aa693f726feed4f8eca733bf22c6626350.webp)

Agentic Behavior: Multi-Document Agents

Intuition: There’s certain questions that “top-k” RAG can’t answer.

Solution: Multi-Document Agents
• Fact-based QA and Summarization over any subsets of documents
• Chain-of-thought and query planning.

Multi-Document Agent
Retrieve top_k=3
… Document Agent | Document Agent | Document Agent | Document Agent | Document Agent | Document Agent …
Vector Search | Summarization
… …

### Fine-Tuning: Embeddings

An LLM generates queries for document chunks, creating labeled examples to fine-tune an embedding model for more relevant retrieval on the dataset.

[16:10](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=970s) [Slide proposes generating synthetic queries from document chunks to fine-tune an embedding model, with a query-generation diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/2896f632f281acb73f777c360009f132f2f0d74ed113d3a2e1618432c85ef9a5.webp)

Fine-Tuning: Embeddings

Intuition: Embedding Representations are not optimized over your dataset

Solution: Generate a synthetic query dataset from raw text chunks using LLMs

Use this synthetic dataset to finetune an embedding model.

The gist of using LLMs to generate labeled data

A few query and relevant document examples
For each doc in documents
Documents
You are an award winning relevance expert. Suggest relevant queries for this article $article queries:
LLM Query Generation Prompt
Large Language Model
GPT-x
BARD
Flan-T5
Synthetic queries for documents
“Labeled data”

9
Credits: Jo Bergum, vespa.ai

[16:15](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=975s) [Slide proposes generating synthetic queries from document chunks to fine-tune an embedding model, with a query-generation diagram.](https://assets.ai.engineer/sites/aiecode2025/sha256/e460d3e1ec31fcefdcd28653baa5155713c1b13117f2a3db9f25470c348957df.webp)

Fine-Tuning: Embeddings

Intuition: Embedding Representations are not optimized over your dataset

Solution: Generate a synthetic query dataset from raw text chunks using LLMs

Use this synthetic dataset to finetune an embedding model.

The gist of using LLMs to generate labeled data

A few query and relevant document examples
For each doc in documents
Documents
You are an award winning relevance expert. Suggest relevant queries for this article $article queries:
LLM Query Generation Prompt
Large Language Model
GPT-x
BARD
Flan-T5
Synthetic queries for documents
“Labeled data”

9
Credits: Jo Bergum, vespa.ai

### Fine-Tuning: LLMs

GPT-4 generates questions and answers from document chunks to create a dataset for fine-tuning a weaker LLM. The speaker describes this as an approach being explored.

[17:30](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1050s) [Slide depicts generating questions and answers from text chunks with a larger model, then fine-tuning an LLM on the resulting dataset.](https://assets.ai.engineer/sites/aiecode2025/sha256/7fa525f2ecc4a0daf0b4be81f6c810a690d96104fb4f6ccb70081f9f24328f9e.webp)

Fine-Tuning: LLMs

Intuition: Weaker LLMs are not bad at response synthesis, reasoning, structured outputs, etc.

Solution: Generate a synthetic dataset from raw chunks (e.g. using GPT-4). Help fix all of the above!

Llama 2 Paper
Chunk
Generate Questions from each Chunk
GPT-4
Question
GPT-4
Generate Answers with Questions/Context
Question-Answer Dataset
Fine-tuning
Fine-tuned Model

[17:43](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1063s) [Slide depicts generating questions and answers from text chunks with a larger model, then fine-tuning an LLM on the resulting dataset.](https://assets.ai.engineer/sites/aiecode2025/sha256/b987922fb85b55065e84e2e43d0611955dc0e0a619da4729625d6efa121c1803.webp)

Fine-Tuning: LLMs

Intuition: Weaker LLMs are not bad at response synthesis, reasoning, structured outputs, etc.

Solution: Generate a synthetic dataset from raw chunks (e.g. using GPT-4). Help fix all of the above!

Llama 2 Paper
Chunk
Generate Questions from each Chunk
GPT-4
Question
GPT-4
Generate Answers with Questions/Context
Question-Answer Dataset
Fine-tuning
Fine-tuned Model

## Full transcript

[0:00](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=0s) [on hold music] Hey everyone.

[0:15](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=15s) Uh, my name is Jerry, Co-founder and CEO of LlamaIndex, and today we'll be talking about how to build production-ready RAG applications. Um, I think there's still time for a raffle for the bucket hat, so if you guys stop by our booth, uh, please fill out the Google form.

[0:27](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=27s) Okay, let's get started. So everybody knows that there's been a ton of amazing use cases in GenAI recently, you know, um, knowledge search and QA, conversational agents, uh, workflow automation, document processing.

[0:39](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=39s) These are all things that you can build, uh, especially using the reasoning capabilities of LLMs, uh, over your data.

[0:47](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=47s) So if we just do a quick refresher, in terms of like paradigms for how do you actually get language models to understand data that hasn't been trained over, there's really like two main paradigms.

[0:58](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=58s) One is retrieval augmentation, where you like fix the model and you basically create a data pipeline to put context into the prompt from some data source into the input prompt of the language model.

[1:09](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=69s) Um, so like a vector database, uh, you know, like unstructured text, SQL database, et cetera.

[1:15](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=75s) The next paradigm here is fine-tuning. How can we bake knowledge into the weights of the network by actually updating the weights of the model itself, some adapter on top of the model, but basically some sort of training process over some new data to actually incorporate knowledge?

[1:30](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=90s) We'll probably talk a little bit more about retrieval augmentation, but this is just like to help you get, uh, started and really understanding the mission statement of, of the company.

[1:39](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=99s) Okay, let's talk about RAG, Retrieval-Augmented Generation. Um, it's become kind of a buzzword recently, but we'll first walk through the current RAG stack for building a QA system. This really consists of two main components, uh, data ingestion as well as data querying, which contains retrieval and synthesis.

[1:58](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=118s) Uh, if you're just getting started in LlamaIndex, you can basically do this in around like five-ish lines of code, uh, so you don't really need to think about it.

[2:04](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=124s) But if you do wanna learn some of the lower-level components, and I do encourage like every engineer, uh, AI engineer, to basically just like learn how these components work under the hood, um, I would encourage you to check out some of our docs to really understand how do you actually do data ingestion, uh, and data querying.

[2:18](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=138s) Like, how do you actually retrieve from a vector database, and how do you synthesize that with an LLM?

[2:25](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=145s) So that's basically the key stack that's kind of emerging these days. Like, for every sort of chatbot, like, you know, chat over your PDF, like over your unstructured data, um, a lot of these things are basically using the same principles of like, how do you actually load data from some data source and actually, you know, um, uh,

[2:43](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=163s) retrieve and query over it? But I think as developers are actually developing these applications, they're realizing that this isn't quite enough. Uh, like there's, there's certain issues that you're running into that are blockers for actually being able to productionize these applications.

[2:58](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=178s) And so what are these challenges with naive RAG? One aspect here is just like, uh, the response, and, and this is the key thing that we're focused on, like, the, the response quality is not very good.

[3:08](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=188s) You run into, for instance, like bad retrieval issues, like, uh, during the retrieval stage from your vector database. If you're not actually returning the relevant chunks from your vector database, you're not gonna be able to have the correct context actually put into the LLM.

[3:21](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=201s) So this includes certain issues like low precision, not all chunks in the retrieve set are relevant. Uh, this leads to like hallucination, like lost in the middle problems. You have a lot of fluff in the returned response.

[3:32](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=212s) This can mean low recall, like your top K isn't high enough, or basically like the, the, the set of like information that you need to actually answer the question is just not there.

[3:40](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=220s) Um, and of course there's other issues too, like outdated information. And many of you who are building apps these days might be familiar with some like key concepts of like just why the LLM isn't always, you know, uh, guaranteed to give you a correct answer.

[3:52](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=232s) There's hallucination, irrelevance, like toxicity bias. There's a lot of issues on the LLM side as well.

[3:59](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=239s) So what can we do? Um, what can we actually do to try to improve the performance of a retrieval-augmented generation application? Um, and, and for many of you, like you might be running into certain issues, and it really runs the gamut across like the entire pipeline.

[4:14](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=254s) There's stuff you can do on the data, like can we store additional information beyond just like the raw text chunks, right, that, that you're putting in the vector database?

[4:22](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=262s) Can you optimize that data pipeline somehow, play around with chunk sizes, that type of thing? Can you optimize the embedding representation itself? A lot of times when you're using a pre-trained embedding model, it's not really optimal for giving you the best performance.

[4:34](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=274s) Um, there's the retrieval algorithm. You know, the default thing you do is just look up the top K most similar elements from your vector database to return to the LLM.

[4:43](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=283s) Um, many times that's not enough, and, and what are kind of like both simple things you can do as well as hard things? Uh, and there's also synthesis, like, uh, why is there...

[4:51](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=291s) Yeah, there's like a V in the... Anyway, so, so can we use LLMs for more than generation? Um, and so basically, like you can, um, use the LLM to actually help you with like reasoning, um, as opposed to just like pure, um, uh, pure, uh, just like, uh, just pure generation, right?

[5:06](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=306s) You can actually use it to try to reason over, given a question, can you break it down into simpler questions, route to different data sources, a-and kind of like, uh, have a, a more sophisticated way of like querying your data.

[5:19](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=319s) Um, of course, like if you've kind of been around some of my recent talks, like I always say, before you actually try any of these techniques, you need to be pretty task-specific and make sure that you need a way to, that you actually have a way to measure performance.

[5:31](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=331s) So I'll probably spend like two minutes talking about evaluation. Um, Simon, my co-founder, just ran a workshop yesterday on really just like how do you evaluate, uh, build a data set, evaluate RAG systems, and help iterate on that.

[5:43](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=343s) Uh, if you missed the workshop, don't worry. I'll-we'll have the slides and, and materials, uh, available online so that you can take a look. Um, at a very high level, in terms of evaluation, it's important because you basically need to define a benchmark for your system to understand how are you going to iterate on and improve it.

[5:58](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=358s) Uh, and there's like a few different ways you can try to do evaluation, right? I think Anton from, from Chroma was, was just saying some of this, but like you basically need a way to, um, evaluate both the end-to-end solution, like you have your input query as well as the output response.

[6:12](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=372s) You also want to probably be able to evaluate like specific components. Like if you've diagnosed that the retrieval is, is like the portion that needs improving, you need like retrieval metrics to really understand how can you improve your retrieval system.

[6:24](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=384s) Um, so there's retrieval and there's synthesis.

[6:28](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=388s) Let's talk a little bit, just like thirty seconds on each one. Um, evaluation on retrieval, what does this look like? You basically want to make sure that the stuff that's returned actually answers the query and that you're kind of, you know, not returning a bunch of fluff, uh, and that the stuff that you return is relevant to

[6:44](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=404s) the question. Um, so first you need an evaluation data set. A lot of people are, uh, have, like, human-labeled data sets. If you're in, uh, building stuff in prod, you might have, like, user feedback as well.

[6:54](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=414s) If not, you can synthetically generate a data set. This data set is input, like query, and output the IDs of, like, the returned documents are relevant to the query.

[7:03](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=423s) So you need that somehow. Once you have that, you can measure stuff with ranking metrics, right? You can measure stuff like success rate, hit rate, MRR, NDCG, uh, a variety of these things.

[7:13](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=433s) Uh, and, and so, like, once you are able to evaluate this, like, this really isn't, uh, kind of like an LLM problem. This is like an IR problem, and this has been around for at least, like, a decade or two.

[7:23](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=443s) Um, but a lot of this is becoming be- like, you know, it's, it's still very relevant in the face of actually building these LLM apps.

[7:31](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=451s) The next piece here is, um, there's a retrieve portion, right? But then you generate a response from it, and then how do you actually evaluate the whole thing end-to-end?

[7:38](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=458s) So evaluation of the final response, uh, given the input. You still wanna generate some sort of data set, so you could do that through, like, human annotations, user feedback.

[7:47](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=467s) You could have, like, ground truth reference answers given the query that really indicates like, "Hey, this is the proper answer to this question." Um, and you can also just like, you know, synthetically generate it with, like, GPT-4.

[7:57](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=477s) Uh, you run this through the full RAG pipeline that you built, the retrieval and synthesis, uh, and you can run, like, LLM-based evals. Um, so label-free evals, with-label evals.

[8:07](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=487s) There's a lot of, uh, projects these days, uh, going on about how do you actually properly evaluate the outputs, uh, predicted outputs of a language model.

[8:16](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=496s) Once you've defined your eval benchmark, now you wanna think about how do you actually optimize your RAG systems. So I sent a teaser on this slide, uh, a few, uh, like yesterday, but the way I think about this is that when do you wanna actually improve your system, there's, like, a million things that you can do to

[8:32](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=512s) try to actually improve your RAG system. Uh, and, like, you probably don't wanna start with the hard stuff first, uh, just because, like, you know, part of the value of language models is how it's kind of democratized access to every developer.

[8:43](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=523s) It's really just made it easy for people to get up and running. And so if, for instance, you're running into some performance issues with RAG, I'd probably start with the basics, like I call it, like, table stakes RAG techniques.

[8:52](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=532s) Uh, better parsing, um, so that you don't just split by even chunks, like adjusting your chunk sizes, trying out stuff that's already integrated with a vector database like hybrid search, as well as, like, metadata filters.

[9:04](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=544s) There's also, like, advanced retrieval methods, uh, that you could try. This is, like, a little bit more advanced. Some of it pulls from, like, traditional IR. Some of it's more, like, kind of, uh, really, like, uh, new in the, in this age of, like, LLM-based apps.

[9:16](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=556s) There's, like, uh, re-ranking. Um, that's a traditional concept. There's also concepts in LlamaIndex like recursive retrieval, like dealing with embedded tables, like, uh, small-to-big retrieval, and a lot of other stuff that we have that help you potentially improve the performance of your application.

[9:31](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=571s) Uh, and then the last bit, like, this kinda gets into more expressive stuff that might be harder to implement, might incur a higher latency and cost, but is potentially more powerful and forward-looking, is, like, agents.

[9:40](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=580s) Like, how do you incorporate agents towards better, like, RAG pipelines to better answer different types of questions and synthesize information? And how do you actually fine-tune stuff?

[9:51](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=591s) Let's talk a little bit about the table stakes first. So chunk sizes. Tuning your chunk size can have outsized impacts on performance, right? Uh, if you've kind of, like, played around with RAG systems, uh, this may or may not be obvious to you.

[10:03](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=603s) What's interesting, though, is that, like, more retrieved tokens does not always equate to higher performance and that if you do, like, re-ranking of your retrieved tokens, it doesn't necessarily mean that your final generation response is gonna be better.

[10:15](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=615s) And this is, again, due to stuff like lost in the middle problems, where stuff in the middle of the LLM context window tends to get lost, whereas stuff at the end, uh, tends to be a little bit, uh, uh, more well-remembered by the LLM.

[10:26](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=626s) Um, and so I think we did a workshop with, like, Arize a few, uh, a week ago, where basically we showed, you know, uh, there is kind of, like, an optimal chunk size given your data set.

[10:34](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=634s) And a lot of times when you try out stuff like re-ranking, it actually increases your error metrics.

[10:41](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=641s) Metadata filtering. Uh, this is another, like, very table stakes thing that I think everybody should look into, and I think vector databases like, you know, Chroma, Pinecone, Weaviate, like, the- these, uh, vector databases are all implementing these, uh, capabilities under the hood.

[10:54](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=654s) Metadata filtering is basically just like, how can you add structured context, uh, to your, your chunks, like your text chunks? And you can use this for both, like, embeddings as well as synthesis, but it also integrates with, like, the meta- metadata filter capabilities of a vector database.

[11:09](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=669s) Um, so metadata is just like, again, structured JSON dictionary. It could be, like, page number. It could be the document title. It could be the summary of adjacent chunks.

[11:16](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=676s) You can get creative with it too. You could hallucinate, like, questions, uh, that the chunk answers. Um, and it can help retrieval. It can help augment your response quality.

[11:23](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=683s) It also integrates with the vector database filters.

[11:27](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=687s) So as an example, um, let's say the question, uh, is over, like, the SEC, uh, like, 10-Q document, and, uh, like, can you tell me the risk factors in 2021?

[11:36](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=696s) If you just do raw semantic search, typically it's very low precision. You're gonna return a bunch of stuff that may or may not match this. You might even return stuff from, like, other years if you have a bunch of documents from different years in the same vector collection.

[11:47](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=707s) Um, and so, like, you're kinda, like, rolling the dice a little bit.

[11:53](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=713s) But one idea here is basically, you know, if you have access to the metadata of the documents, um, and you ask a question like this, you basically combine structured query capabilities by inferring the metadata filters, like a WHERE clause in a SQL statement, like a year equals 2021, and you combine that with semantic search to return the

[12:10](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=730s) most relevant candidates given your query. And this improves the precision of your, uh, of your results.

[12:18](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=738s) Moving on to stuff that's maybe a bit more advanced, like advanced retrieval is one thing that we found generally helps is this idea of, like, small-to-big retrieval. Um, so what does that mean?

[12:28](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=748s) Basically, right now, when you embed a big text chunk, you, uh, also synthesize over that text chunk. And so it's a little suboptimal because what if, like, the embedding representation's, like, biased because, you know, there's a bunch of fluff in that text chunk that contains a bunch of ir- irrelevant information, you're not actually optimizing your retrieval

[12:44](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=764s) quality. So embedding a big text chunk sometimes feels a little suboptimal. One thing that you could do is basically embed text at the sentence level or on a smaller level, and then expand that window during synthesis time.

[12:55](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=775s) Um, and so this is contained in a variety of, like, LlamaIndex abstractions, but the idea is that you return, you retrieve on more granular pieces of information, so smaller chunks.

[13:05](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=785s) This makes it so that these chunks are more likely to be retrieved when you actually ask a query over these specific pieces of context. But then you wanna make sure that the LLM actually has access to more information to actually synthesize a proper result.

[13:18](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=798s) So this leads to, like, more precise retrieval, right? So, um, we, we tried this out. It, it helps avoid, like, some lost in the middle problems. You can set a smaller top-K value, like K equals two, uh, whereas, like, uh, over this data set, if you set, like, K equals five for naive retrieval over big text chunks,

[13:34](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=814s) you basically start returning a lot of context, and that kind of leads into issues where, uh, you know, maybe the relevant context is in the middle, but you're not able to find out, uh, or, or you're, like, the, the LLM is, is, is not able to kind of, uh, synthesize over that information.

[13:51](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=831s) A very related idea here is just, like, embedding a reference to the parent chunk, um, as opposed to the actual text chunk itself. So for instance, if you wanna embed, like, not just the raw text chunk or not the text chunk, but actually, like, a smaller chunk, um, or a summary or questions that the chunk answers, we

[14:08](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=848s) have found that that actually helps to improve retrieval performance a decent amount. Um, and it's, it, it kind of go- again, goes along with this idea, like, a lot of times you want to embed something that's more amenable for embedding-based retrieval, uh, but then you want to return enough context so that the LLM can actually synthesize over

[14:23](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=863s) that information. The next bit here is actually kind of even more advanced stuff, right? This goes on into agents, and this goes on into that last pillar that I, I had mentioned, which is how can you use LLMs for, for reasoning as opposed to just synthesis?

[14:42](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=882s) The intuition here is that, like, for a lot of RAG, if you're just using the LLM at the end, you're, one, constrained by the quality of your retriever, and you're really only able to do stuff like question answering.

[14:52](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=892s) And there's certain types of questions and more advanced a- uh, analysis that you might wanna launch that, like, top-K RAG can't really answer. It, it's not necessarily just a one-off question.

[15:01](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=901s) You might need to have, like, an entire sequence of reasoning steps to actually pull together a piece of information, or you want- might wanna, like, summarize a document and compare with, like, other documents.

[15:10](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=910s) So one kind of architecture we're, we're exploring right now is this idea of, like, multi-document agents. What if, like, instead of just, like, RAG, we moved a little bit more into agent territory?

[15:19](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=919s) We modeled each document not just as a sequence of text chunks, but actually as a set of tools that contains the ability to both, like, summarize that document as well as to do QA over that document over specific facts.

[15:31](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=931s) Um, and of course, if you wanna scale to, like, you know, hundreds or thousands or millions of documents, um, uh, typically an agent can only have access to a limited window of tools.

[15:41](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=941s) So you probably wanna do some sort of retrieval on these tools, similar to how you want to retrieve, like, text chunks from a document. The main difference is that because these are tools, you actually wanna act upon them.

[15:50](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=950s) You want to use them as opposed to just, like, taking the raw text and plugging it into the context window. So blending this combination of, like, uh, kind of, um, embedding-based retrieval or any sort of retrieval as well as, like, agent tool use is a very interesting paradigm that I think is really only possible with this age

[16:06](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=966s) of LLMs and hasn't really existed, uh, before this.

[16:13](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=973s) Another kind of advanced concept is this idea of fine-tuning. Um, and so fine-tuning, uh, you know, so some other presenters have talked about this as well, but the idea of, like, fine-tuning in a RAG system is that it, it really optimizes specific pieces of this RAG pipeline for you to kind of better, um, like, improve the performance

[16:32](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=992s) of either retriever or synthesis capabilities. So one thing you can do is fine-tune your embeddings. Um, I think, uh, Anton was talking about this as well. Like, if you just use a pre-trained model, the embedding representations are not gonna be optimized over your specific data, so sometimes you're just gonna re- retrieve the wrong, wrong information.

[16:48](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1008s) Um, if you can somehow tune these embeddings so that given any sort of, like, relevant question that the user might ask, that you're actually returning the relevant response, then you're going to have, like, better performance.

[16:59](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1019s) So, um, an idea here, right, is to generate a synthetic query data set from raw text chunks using LLMs and use this to fine-tune an embedding model. Um, and you can do this, like,

[17:10](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1030s) uh, if we go back really quick actually, uh, you can do this by basically, um, kind of fine-tuning the base model itself. You can also fine-tune an adapter on top of the model.

[17:19](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1039s) Um, and fine-tuning an adapter on top of the model has a few advantages in that, uh, you don't require the base model's weights to actually fine-tune stuff. And if you just fine-tune the query, you don't have to re-index your entire document corpus.

[17:33](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1053s) There's also fine-tuning LLMs, which of course, like, a lot of people are very interested in doing these days. Um, an intuition here specifically for RAG is that if you have a weaker LLM, like 3.5 Turbo, like Llama 2-7B, like, these weaker LLMs are bad, are, are, are not bad at, like, um, are...

[17:50](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1070s) Wait. Yeah. Weaker LLMs are, are maybe a little bit worse at, like, response synthesis, reasoning, structured outputs, et cetera, um, compared to, like, bigger models. So a solution here is what if you can generate a synthetic data set using a bigger model like GPT-4?

[18:04](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1084s) That's something we're exploring. And you actually distill that into 3.5 Turbo, so it gets better at chain of thought, longer response quality, um, better structured outputs, and a lot of other possibilities as well.

[18:15](https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=1095s) So all these things are in our docs. There's production RAG, uh, there's fine-tuning, and I have two seconds left. So thank you very much. [audience applauds] [upbeat music]
