Specialized RAG Agents: Lessons learned from deploying complex AI systems in production

[on-hold music] My name is Douwe Kiela.

I'm the CEO at Contextual AI, and I'm here to talk to you about RAG in production, RAG agents specifically. Um, and I'll, I'll share some of the lessons that I've learned.

So my background is in AI research, uh, but after that, I became the CEO of a AI company focused on enterprise. Uh, so I thought I would share some of my learnings with you, uh, in the hope that that's useful.

So if you, uh, look at enterprise AI, uh, if you work in this space, you probably notice that there's a huge opportunity, uh, ahead of us, right? Everybody wants to grab that opportunity.

There's, there's these huge numbers flying around. Four point four trillion dollars is the, is the estimated added value to the global economy, according to McKinsey. So we have this giant opportunity.

But at the same time, if you actually look at what's happening in enterprises, you see a lot of frustration. It's probably even true for some of the people in the audience right here.

If you're a VP of AI, then you're probably under some pressure right now. It's like, "Where's the ROI? We're investing all this money in AI, but where is it actually leading us to?

Are we getting something out of this?" So, uh, Forbes has this interesting study where they showed that only one in four businesses actually get value from AI. So why is that happening, right?

It feels a bit like a paradox. Uh, so to, to explain it, we can look, uh, at a paradox that might be familiar to you. It's, it's something called Moravec's paradox.

It's from robotics. And in robotics, they were very surprised when they found out that it's actually much easier to beat humans at chess than to have a robot that can vacuum clean your house or have a self-driving car.

Um, so the, the, the paradox here is really that things that seem hard are actually much easier for computers than you would expect, and things that seem easy actually turn out to be much harder, right?

So there's something very similar happening right now in, in enterprise AI specifically, and this is around context. So on the one hand, we have these amazing language models, right?

You've, you've-- That's why we're all here, basically, because we see this revolution happening right in front of our eyes. So language models can generate code much better than most humans.

They can solve mathematical problems much better than, than most of us here can do, uh, and we're pretty smart. Um, so it's really amazing what they can do. But one of the things that they really still struggle with, and that's one of the things that as humans we are very good at, sort of without effort, is putting

things in the right context, right? So as humans, we build on our expertise, we build on our intuition that we've developed over the years, especially if we're a specialist.

This is something that is very easy for us to do, is to put something in the right context and, um, and in the right situation so that you can make sense of the information or the, the problem that you're solving.

So I would argue that this is really the key observation, this, this context paradox, um, for unlocking ROI with AI. And the reason for that is that where we are right now here is, is in the bottom left, right?

So we're, we're mostly focused on convenience. We have general-purpose assistants. They're very useful. Mostly, if you're lazy, they help you sort of solve your problems faster. But where you really want to get to is differentiated value.

If you're an enterprise, i-it's nice that you can make, uh, things more convenient. You probably can make people more efficient and more productive. That's great. But where you want to get to is this business transformation ideal, right?

That's what all the CEOs are probably telling you as a VP of AI. Like, "I want to change my entire business. How am I going to do that?" So getting to that differentiated value, that's where you want to get to.

But the problem is that the higher you go on that axis, the further you go on the context axis. So the, the better you need to be at handling the context, uh, uh, that exists within your enterprise.

Um, so what should we do about that? Um, so that observation is really why I started the company that I'm currently the, the CEO of, Contextual AI. Um, and we started this two years ago to try to help bridge this gap, um, and we've learned some lessons along the way that I thought I would share with you,

uh, in the hope that they're also useful for you.

So the first observation i-is really that language models are awesome, but often they're only twenty percent of a much bigger system. Um, so if you have an enterprise AI deployment, usually that means it's a RAG system.

Uh, so I, I think everybody here probably has heard of RAG. Uh, RAG is something that I, uh, originally pioneered with my team at Facebook AI Research when I was there.

Uh, so RAG is, is really kind of the standard way that you get gen AI to work on your data. So what happens very often these days is a new language model comes out, everybody goes, "Whoa, lu- new language model.

It's great." Everybody starts to think just about the language model, but very few people actually think about the system around the language model, and that system needs to solve the problem, right?

So you can have a relatively mediocre language model, but an amazing RAG pipeline around it, and that's going to be much better than an amazing language model with a terrible RAG pipeline around it.

So the basic observation here, or the lesson, is that you should be thinking about systems, not about models. The model is only a small part of the system, and the system is the thing that solves the problem.

The next observation is that if you're in an enterprise, expertise is really your fuel, right? So, uh, one of the, the things that you want to be able to do as an enterprise is unlock all of that expertise.

So you have all of this i-institutional knowledge in your company. How do you get it out? Um, so one way to try to do that is, is using these generalist, kind of general-purpose assistants, but it's very hard to get them to, to, uh, to match the expertise of people in your company.

So ideally, what you want to do is to specialize so that you can capture that expertise much better. So, uh, at my company, we call this specialization over AGI.

AGI is great. There are lots of use cases for it. If you really want to solve a very difficult problem that is very domain-specific where you understand the use case, you want to specialize for it, and you'll get much further.

So that's, I guess, uh, pretty counterintuitive if you look at the sort of broader, uh, interests, right? Most people are much more excited about AGI, but solving real problems is much easier with specialization.

The next lesson is, uh, at an enterprise scale is your moat. So if you think about what a company really is, is a company maybe its people? Probably a little bit, right?

But over time, what the company really is or what makes a company a company is its data, because even people are transient, right? So the data that a company owns, that is the company in the long term.

So now as an enterprise, you need to think how you can unlock all of that potential, right? And so, uh, one of the, the big issues that we see a lot is that enterprises think that, uh, you need to scrub the data and clean it and invest a lot of time in, in, uh, making your data accessible

with AI. But what you really want to do is make sure that AI can work on your noisy data at scale. And doing that is incredibly difficult, but if you succeed in doing that, that's how you get to differentiated value.

That's how you get that moat, because the data makes, makes your company your company, and so that data is really your moat.

Uh, one observation, uh, and this is really a, a hard truth that, that we've learned, and I, I think that many of you might have learned already or that you're about to find out, uh, if you're earlier in, in your journey, is that pilots are very easy.

Uh, building a demo is not very difficult these days, right? If you want to build a RAG system, you take one of the frameworks, you put in some documents, you have a working solution.

It's great. You give it to your ten users, they all tell you it's fantastic. And then you show it to the CEO, and he says, "Okay, we're going to fire half the customer support team, and we're going to replace them with AI, and we're gonna do that in three months."

And now you're on the hook for productionizing something that is actually much, much harder. And so getting this to work at tens of thousands or hundreds of thousands or millions of documents, you can't do that with any existing tools, uh, that are out there on the open source market.

It's very, very difficult to do that. Making this scale to thousands of users is very hard. Um, making it work for lots of different use cases. If you're an enterprise, maybe you have twenty thousand different use cases that you want to cover.

So how do you scale if that's the problem that you're solving? And then there's, of course, enterprise requirements around security and, and compliance. So bridging that gap is much harder than you think.

And, and the, the right way to deal with that is to really focus on production from day one. So don't design for the pilot, design for production, uh, and that can save you a lot of time.

And that brings me to the next observation, is that speed is really much more important than perfection. What we see, um, in, in terms of production roll-outs of, uh, RAG agents, it's all about speed.

Um, and, and what that means is, uh, you need to give it to your users relatively early, real users, not, not sort of, uh, testers who are, are kind of friendly.

You wanna give it to real users to get their feedback. You wanna do that early. It doesn't have to be perfect. It just needs to be barely functional. And if you do that, then you can hill climb to actually get to this level where it's good enough.

If you don't do that, and you wait too long, and then you try to design something that is perfect, it's gonna be very hard to, to bridge that gap from pilot to production.

So iteration is really the key to a lot of, uh, successful, uh, production AI deployments in, in enterprises.

Next observation is, is related to this too, which is that, uh, if you want your engineers to be fast, and if you want to follow that, that speed maxim I just talked about, then you don't want them woring-- working on boring stuff.

Uh, sounds kind of obvious, but it turns out that engineers are working on a lot of very boring stuff. Um, and, and so one of, one of the, the things that they have to worry about, for example, is what is the optimal chunking strategy for my RAG system?

And it's different for every use case, and it's different for every framework. And then they have to think about what the right prompt is or really basic things that ideally they don't have to think about too much because you really want your engineers to think about, how am I going to deliver business value, right?

How, how do I make sure I have this differentiated value and that I'm actually better than my competitors? Um, so make sure that your engineers spend time on the things that matter and not on the chunking strategy or, or things that, that can be abstracted away, uh, very well these days by, by state-of-the-art platforms for, for RAG

agents. Next one is, is about making AI easy to consume. So what, what I mean by that is we actually see, uh, this happen quite often where companies have gen AI running in production.

And then the next question I often ask them is, "Okay, how many people are actually using it?" And, and surprisingly often, the answer is zero. Almost nobody's actually using it.

They did all this work, but they had to make sure it, it came through, uh, sort of model risk a-and, and, uh, teams like that. So it was really like kneecapped almost, and now it's barely useful.

Uh, so that's one scenario, or, or very often people just don't actually know how to use the technology. So it, it really is a journey that you are on, uh, and the easier you can make your solutions to consume, the better it is.

And what that, what that means for most enterprises is not just thinking about your enterprise data and how you make AI work on it, but also how you integrate it into their workflows.

So the closer you can integrate it into a workflow that already exists in your enterprise, the more successful you're going to be with real production usage.

Uh, next one is, is related, uh, to the, to the previous one as well, uh, where it's really about getting usage. It's about sort of being sticky and, and so this sounds maybe a, a little obvious-

But the, the quicker you can wow users or get this sort of spark where they, they suddenly get it, like this for, for me as a CEO of a AI company, that's really the special moment when people suddenly go like, "Wow, I didn't know that it could do this."

Um, so you can try to design your experiences for onboarding users around this observation too, right? So where, where they get to the wow as quickly as possible. So for us, we have this really nice example with someone at Qualcomm.

So we're, we're running in production globally with Qualcomm, with thousands of customer engineers, and one of them became so happy when they found this document. It was seven years old.

It was hidden away somewhere. They didn't know it existed. They had all these questions, and they just never knew what the answers were, and suddenly because they asked our system, they got these answers, and like their, their world was never the same again after that.

Um, so these are the, the small, the small wins sort of, uh, that, that really matter for, for, uh, evangelizing, uh, production in AI.

Uh, so that brings me to the, the penultimate learning, which is that it's not even really about accuracy anymore. So accuracy is almost table stakes, right? Uh, so I, I think as AI practitioners, we probably know that getting a hundred percent accuracy is very hard, if not impossible.

Getting ninety-five percent accuracy, maybe you can get there, or ninety percent. But what enterprises are, are thinking about much more these days is, what about the missing five percent?

Or what about the missing ten percent? How do I deal with the things that might go wrong, right? Um, so there's a minimum requirement for accuracy, but beyond that, it's really about inaccuracy, and the way to deal with that is through observability.

So you wanna be very careful with how you evaluate these systems. You wanna be very careful with making sure you have proper audit trails, especially if you work in a regulated industry.

This is incredibly important, right? Making sure that you have an audit trail that says, "This is why I generated this answer. It's because I found it here in this document."

Basic things like that. So attribution essentially in a RAG system actually becomes very, very important for dealing with the inaccuracies. And similarly, what you can do is you can check the claims that your system generates.

So do a lot of post-processing to ensure that you have proper attributions, uh, that, that you can really back up, uh, as evidence. [coughs]

Struggling with the clicker a bit. So, uh, fi-final one, uh, that I wanna end on, and this sounds maybe a little bit cheesy, but it, it really is, is true, is be ambitious.

We, we actually see a lot of projects fail, not because people are aiming too high, but because people are aiming too low, where, where folks are going like, "I have gen AI running in production," and then, "What does it do?"

"It answers basic questions about who your 401(k) provider is, or how many days of vacation I get." Like, that's not really where the ROI of AI is, right? So you want to aim for really ambitious things where if you solve them, you actually have, have ROI, and you don't just have a gimmick that people don't really, uh,

use anyway. So try to be, be ambitious because we really live in special times. We have the, the astronaut here on the slide. Um, so, so I, I think it was a pretty special time to be alive during the, the Moon landing and when all of that happened, right?

We're in a, in a similar moment right now, where AI is, is really going to change everything, is going to change our entire society in the next couple of years.

Um, and so you have an opportunity being in the role that you're in to, to really, uh, uh, affect that change in society yourself. Uh, so, so be am- ambitious when you do that.

Uh, and don't aim for the, the low-hanging easy fruit. Uh, aim for, for the sky.

So, uh, that's really, uh, what, what my lessons were for you here. Uh, this context paradox is not going away, um, but by understanding these lessons that I, I shared with you, hopefully you can turn some of these challenges, uh, that we see everywhere in enterprise AI into opportunities for yourself.

Um, so, so it's really build better systems, think about systems, not models, focus on your expertise and specialize for it. Don't settle for general solutions. Specialize for, for the expertise that you have in your company, and be ambitious, and then you'll be very successful.

Thank you. [outro music]