Building Production-Ready RAG Applications

[on hold music] Hey everyone.

Uh, my name is Jerry, Co-founder and CEO of LlamaIndex, and today we'll be talking about how to build production-ready RAG applications. Um, I think there's still time for a raffle for the bucket hat, so if you guys stop by our booth, uh, please fill out the Google form.

Okay, let's get started. So everybody knows that there's been a ton of amazing use cases in GenAI recently, you know, um, knowledge search and QA, conversational agents, uh, workflow automation, document processing.

These are all things that you can build, uh, especially using the reasoning capabilities of LLMs, uh, over your data.

So if we just do a quick refresher, in terms of like paradigms for how do you actually get language models to understand data that hasn't been trained over, there's really like two main paradigms.

One is retrieval augmentation, where you like fix the model and you basically create a data pipeline to put context into the prompt from some data source into the input prompt of the language model.

Um, so like a vector database, uh, you know, like unstructured text, SQL database, et cetera.

The next paradigm here is fine-tuning. How can we bake knowledge into the weights of the network by actually updating the weights of the model itself, some adapter on top of the model, but basically some sort of training process over some new data to actually incorporate knowledge?

We'll probably talk a little bit more about retrieval augmentation, but this is just like to help you get, uh, started and really understanding the mission statement of, of the company.

Okay, let's talk about RAG, Retrieval-Augmented Generation. Um, it's become kind of a buzzword recently, but we'll first walk through the current RAG stack for building a QA system. This really consists of two main components, uh, data ingestion as well as data querying, which contains retrieval and synthesis.

Uh, if you're just getting started in LlamaIndex, you can basically do this in around like five-ish lines of code, uh, so you don't really need to think about it.

But if you do wanna learn some of the lower-level components, and I do encourage like every engineer, uh, AI engineer, to basically just like learn how these components work under the hood, um, I would encourage you to check out some of our docs to really understand how do you actually do data ingestion, uh, and data querying.

Like, how do you actually retrieve from a vector database, and how do you synthesize that with an LLM?

So that's basically the key stack that's kind of emerging these days. Like, for every sort of chatbot, like, you know, chat over your PDF, like over your unstructured data, um, a lot of these things are basically using the same principles of like, how do you actually load data from some data source and actually, you know, um, uh,

retrieve and query over it? But I think as developers are actually developing these applications, they're realizing that this isn't quite enough. Uh, like there's, there's certain issues that you're running into that are blockers for actually being able to productionize these applications.

And so what are these challenges with naive RAG? One aspect here is just like, uh, the response, and, and this is the key thing that we're focused on, like, the, the response quality is not very good.

You run into, for instance, like bad retrieval issues, like, uh, during the retrieval stage from your vector database. If you're not actually returning the relevant chunks from your vector database, you're not gonna be able to have the correct context actually put into the LLM.

So this includes certain issues like low precision, not all chunks in the retrieve set are relevant. Uh, this leads to like hallucination, like lost in the middle problems. You have a lot of fluff in the returned response.

This can mean low recall, like your top K isn't high enough, or basically like the, the, the set of like information that you need to actually answer the question is just not there.

Um, and of course there's other issues too, like outdated information. And many of you who are building apps these days might be familiar with some like key concepts of like just why the LLM isn't always, you know, uh, guaranteed to give you a correct answer.

There's hallucination, irrelevance, like toxicity bias. There's a lot of issues on the LLM side as well.

So what can we do? Um, what can we actually do to try to improve the performance of a retrieval-augmented generation application? Um, and, and for many of you, like you might be running into certain issues, and it really runs the gamut across like the entire pipeline.

There's stuff you can do on the data, like can we store additional information beyond just like the raw text chunks, right, that, that you're putting in the vector database?

Can you optimize that data pipeline somehow, play around with chunk sizes, that type of thing? Can you optimize the embedding representation itself? A lot of times when you're using a pre-trained embedding model, it's not really optimal for giving you the best performance.

Um, there's the retrieval algorithm. You know, the default thing you do is just look up the top K most similar elements from your vector database to return to the LLM.

Um, many times that's not enough, and, and what are kind of like both simple things you can do as well as hard things? Uh, and there's also synthesis, like, uh, why is there...

Yeah, there's like a V in the... Anyway, so, so can we use LLMs for more than generation? Um, and so basically, like you can, um, use the LLM to actually help you with like reasoning, um, as opposed to just like pure, um, uh, pure, uh, just like, uh, just pure generation, right?

You can actually use it to try to reason over, given a question, can you break it down into simpler questions, route to different data sources, a-and kind of like, uh, have a, a more sophisticated way of like querying your data.

Um, of course, like if you've kind of been around some of my recent talks, like I always say, before you actually try any of these techniques, you need to be pretty task-specific and make sure that you need a way to, that you actually have a way to measure performance.

So I'll probably spend like two minutes talking about evaluation. Um, Simon, my co-founder, just ran a workshop yesterday on really just like how do you evaluate, uh, build a data set, evaluate RAG systems, and help iterate on that.

Uh, if you missed the workshop, don't worry. I'll-we'll have the slides and, and materials, uh, available online so that you can take a look. Um, at a very high level, in terms of evaluation, it's important because you basically need to define a benchmark for your system to understand how are you going to iterate on and improve it.

Uh, and there's like a few different ways you can try to do evaluation, right? I think Anton from, from Chroma was, was just saying some of this, but like you basically need a way to, um, evaluate both the end-to-end solution, like you have your input query as well as the output response.

You also want to probably be able to evaluate like specific components. Like if you've diagnosed that the retrieval is, is like the portion that needs improving, you need like retrieval metrics to really understand how can you improve your retrieval system.

Um, so there's retrieval and there's synthesis.

Let's talk a little bit, just like thirty seconds on each one. Um, evaluation on retrieval, what does this look like? You basically want to make sure that the stuff that's returned actually answers the query and that you're kind of, you know, not returning a bunch of fluff, uh, and that the stuff that you return is relevant to

the question. Um, so first you need an evaluation data set. A lot of people are, uh, have, like, human-labeled data sets. If you're in, uh, building stuff in prod, you might have, like, user feedback as well.

If not, you can synthetically generate a data set. This data set is input, like query, and output the IDs of, like, the returned documents are relevant to the query.

So you need that somehow. Once you have that, you can measure stuff with ranking metrics, right? You can measure stuff like success rate, hit rate, MRR, NDCG, uh, a variety of these things.

Uh, and, and so, like, once you are able to evaluate this, like, this really isn't, uh, kind of like an LLM problem. This is like an IR problem, and this has been around for at least, like, a decade or two.

Um, but a lot of this is becoming be- like, you know, it's, it's still very relevant in the face of actually building these LLM apps.

The next piece here is, um, there's a retrieve portion, right? But then you generate a response from it, and then how do you actually evaluate the whole thing end-to-end?

So evaluation of the final response, uh, given the input. You still wanna generate some sort of data set, so you could do that through, like, human annotations, user feedback.

You could have, like, ground truth reference answers given the query that really indicates like, "Hey, this is the proper answer to this question." Um, and you can also just like, you know, synthetically generate it with, like, GPT-4.

Uh, you run this through the full RAG pipeline that you built, the retrieval and synthesis, uh, and you can run, like, LLM-based evals. Um, so label-free evals, with-label evals.

There's a lot of, uh, projects these days, uh, going on about how do you actually properly evaluate the outputs, uh, predicted outputs of a language model.

Once you've defined your eval benchmark, now you wanna think about how do you actually optimize your RAG systems. So I sent a teaser on this slide, uh, a few, uh, like yesterday, but the way I think about this is that when do you wanna actually improve your system, there's, like, a million things that you can do to

try to actually improve your RAG system. Uh, and, like, you probably don't wanna start with the hard stuff first, uh, just because, like, you know, part of the value of language models is how it's kind of democratized access to every developer.

It's really just made it easy for people to get up and running. And so if, for instance, you're running into some performance issues with RAG, I'd probably start with the basics, like I call it, like, table stakes RAG techniques.

Uh, better parsing, um, so that you don't just split by even chunks, like adjusting your chunk sizes, trying out stuff that's already integrated with a vector database like hybrid search, as well as, like, metadata filters.

There's also, like, advanced retrieval methods, uh, that you could try. This is, like, a little bit more advanced. Some of it pulls from, like, traditional IR. Some of it's more, like, kind of, uh, really, like, uh, new in the, in this age of, like, LLM-based apps.

There's, like, uh, re-ranking. Um, that's a traditional concept. There's also concepts in LlamaIndex like recursive retrieval, like dealing with embedded tables, like, uh, small-to-big retrieval, and a lot of other stuff that we have that help you potentially improve the performance of your application.

Uh, and then the last bit, like, this kinda gets into more expressive stuff that might be harder to implement, might incur a higher latency and cost, but is potentially more powerful and forward-looking, is, like, agents.

Like, how do you incorporate agents towards better, like, RAG pipelines to better answer different types of questions and synthesize information? And how do you actually fine-tune stuff?

Let's talk a little bit about the table stakes first. So chunk sizes. Tuning your chunk size can have outsized impacts on performance, right? Uh, if you've kind of, like, played around with RAG systems, uh, this may or may not be obvious to you.

What's interesting, though, is that, like, more retrieved tokens does not always equate to higher performance and that if you do, like, re-ranking of your retrieved tokens, it doesn't necessarily mean that your final generation response is gonna be better.

And this is, again, due to stuff like lost in the middle problems, where stuff in the middle of the LLM context window tends to get lost, whereas stuff at the end, uh, tends to be a little bit, uh, uh, more well-remembered by the LLM.

Um, and so I think we did a workshop with, like, Arize a few, uh, a week ago, where basically we showed, you know, uh, there is kind of, like, an optimal chunk size given your data set.

And a lot of times when you try out stuff like re-ranking, it actually increases your error metrics.

Metadata filtering. Uh, this is another, like, very table stakes thing that I think everybody should look into, and I think vector databases like, you know, Chroma, Pinecone, Weaviate, like, the- these, uh, vector databases are all implementing these, uh, capabilities under the hood.

Metadata filtering is basically just like, how can you add structured context, uh, to your, your chunks, like your text chunks? And you can use this for both, like, embeddings as well as synthesis, but it also integrates with, like, the meta- metadata filter capabilities of a vector database.

Um, so metadata is just like, again, structured JSON dictionary. It could be, like, page number. It could be the document title. It could be the summary of adjacent chunks.

You can get creative with it too. You could hallucinate, like, questions, uh, that the chunk answers. Um, and it can help retrieval. It can help augment your response quality.

It also integrates with the vector database filters.

So as an example, um, let's say the question, uh, is over, like, the SEC, uh, like, 10-Q document, and, uh, like, can you tell me the risk factors in 2021?

If you just do raw semantic search, typically it's very low precision. You're gonna return a bunch of stuff that may or may not match this. You might even return stuff from, like, other years if you have a bunch of documents from different years in the same vector collection.

Um, and so, like, you're kinda, like, rolling the dice a little bit.

But one idea here is basically, you know, if you have access to the metadata of the documents, um, and you ask a question like this, you basically combine structured query capabilities by inferring the metadata filters, like a WHERE clause in a SQL statement, like a year equals 2021, and you combine that with semantic search to return the

most relevant candidates given your query. And this improves the precision of your, uh, of your results.

Moving on to stuff that's maybe a bit more advanced, like advanced retrieval is one thing that we found generally helps is this idea of, like, small-to-big retrieval. Um, so what does that mean?

Basically, right now, when you embed a big text chunk, you, uh, also synthesize over that text chunk. And so it's a little suboptimal because what if, like, the embedding representation's, like, biased because, you know, there's a bunch of fluff in that text chunk that contains a bunch of ir- irrelevant information, you're not actually optimizing your retrieval

quality. So embedding a big text chunk sometimes feels a little suboptimal. One thing that you could do is basically embed text at the sentence level or on a smaller level, and then expand that window during synthesis time.

Um, and so this is contained in a variety of, like, LlamaIndex abstractions, but the idea is that you return, you retrieve on more granular pieces of information, so smaller chunks.

This makes it so that these chunks are more likely to be retrieved when you actually ask a query over these specific pieces of context. But then you wanna make sure that the LLM actually has access to more information to actually synthesize a proper result.

So this leads to, like, more precise retrieval, right? So, um, we, we tried this out. It, it helps avoid, like, some lost in the middle problems. You can set a smaller top-K value, like K equals two, uh, whereas, like, uh, over this data set, if you set, like, K equals five for naive retrieval over big text chunks,

you basically start returning a lot of context, and that kind of leads into issues where, uh, you know, maybe the relevant context is in the middle, but you're not able to find out, uh, or, or you're, like, the, the LLM is, is, is not able to kind of, uh, synthesize over that information.

A very related idea here is just, like, embedding a reference to the parent chunk, um, as opposed to the actual text chunk itself. So for instance, if you wanna embed, like, not just the raw text chunk or not the text chunk, but actually, like, a smaller chunk, um, or a summary or questions that the chunk answers, we

have found that that actually helps to improve retrieval performance a decent amount. Um, and it's, it, it kind of go- again, goes along with this idea, like, a lot of times you want to embed something that's more amenable for embedding-based retrieval, uh, but then you want to return enough context so that the LLM can actually synthesize over

that information. The next bit here is actually kind of even more advanced stuff, right? This goes on into agents, and this goes on into that last pillar that I, I had mentioned, which is how can you use LLMs for, for reasoning as opposed to just synthesis?

The intuition here is that, like, for a lot of RAG, if you're just using the LLM at the end, you're, one, constrained by the quality of your retriever, and you're really only able to do stuff like question answering.

And there's certain types of questions and more advanced a- uh, analysis that you might wanna launch that, like, top-K RAG can't really answer. It, it's not necessarily just a one-off question.

You might need to have, like, an entire sequence of reasoning steps to actually pull together a piece of information, or you want- might wanna, like, summarize a document and compare with, like, other documents.

So one kind of architecture we're, we're exploring right now is this idea of, like, multi-document agents. What if, like, instead of just, like, RAG, we moved a little bit more into agent territory?

We modeled each document not just as a sequence of text chunks, but actually as a set of tools that contains the ability to both, like, summarize that document as well as to do QA over that document over specific facts.

Um, and of course, if you wanna scale to, like, you know, hundreds or thousands or millions of documents, um, uh, typically an agent can only have access to a limited window of tools.

So you probably wanna do some sort of retrieval on these tools, similar to how you want to retrieve, like, text chunks from a document. The main difference is that because these are tools, you actually wanna act upon them.

You want to use them as opposed to just, like, taking the raw text and plugging it into the context window. So blending this combination of, like, uh, kind of, um, embedding-based retrieval or any sort of retrieval as well as, like, agent tool use is a very interesting paradigm that I think is really only possible with this age

of LLMs and hasn't really existed, uh, before this.

Another kind of advanced concept is this idea of fine-tuning. Um, and so fine-tuning, uh, you know, so some other presenters have talked about this as well, but the idea of, like, fine-tuning in a RAG system is that it, it really optimizes specific pieces of this RAG pipeline for you to kind of better, um, like, improve the performance

of either retriever or synthesis capabilities. So one thing you can do is fine-tune your embeddings. Um, I think, uh, Anton was talking about this as well. Like, if you just use a pre-trained model, the embedding representations are not gonna be optimized over your specific data, so sometimes you're just gonna re- retrieve the wrong, wrong information.

Um, if you can somehow tune these embeddings so that given any sort of, like, relevant question that the user might ask, that you're actually returning the relevant response, then you're going to have, like, better performance.

So, um, an idea here, right, is to generate a synthetic query data set from raw text chunks using LLMs and use this to fine-tune an embedding model. Um, and you can do this, like,

uh, if we go back really quick actually, uh, you can do this by basically, um, kind of fine-tuning the base model itself. You can also fine-tune an adapter on top of the model.

Um, and fine-tuning an adapter on top of the model has a few advantages in that, uh, you don't require the base model's weights to actually fine-tune stuff. And if you just fine-tune the query, you don't have to re-index your entire document corpus.

There's also fine-tuning LLMs, which of course, like, a lot of people are very interested in doing these days. Um, an intuition here specifically for RAG is that if you have a weaker LLM, like 3.5 Turbo, like Llama 2-7B, like, these weaker LLMs are bad, are, are, are not bad at, like, um, are...

Wait. Yeah. Weaker LLMs are, are maybe a little bit worse at, like, response synthesis, reasoning, structured outputs, et cetera, um, compared to, like, bigger models. So a solution here is what if you can generate a synthetic data set using a bigger model like GPT-4?

That's something we're exploring. And you actually distill that into 3.5 Turbo, so it gets better at chain of thought, longer response quality, um, better structured outputs, and a lot of other possibilities as well.

So all these things are in our docs. There's production RAG, uh, there's fine-tuning, and I have two seconds left. So thank you very much. [audience applauds] [upbeat music]