[
  {
    "startMs": 12700,
    "text": "Uh, I'm leading AI engineering solutions at Langfuse, uh, by ClickHouse, and, uh, we are an a-agent observability and eval platform that a lot of users use, open source. And, uh, I'm here to talk to you about an experience that we had with one of, uh... a top five bank, helping them getting through how they can put s-safety, uh, guardrails whenever they were shipping AI applications into production and, uh, to make sure that they are running with the e-proper evals on that. I, I like to"
  },
  {
    "startMs": 42460,
    "text": "start with, uh, you know, something that we learned with the experience with this customer is that regulated industries, they don't necessarily care about the public benchmarks. I know that every week, uh, whenever, whenever, anytime a new model comes out, um, we see all the benchmarks. You know, uh, Opus five is now leading the benchmarks in a, in a multitude of dimensions, and that's great. Um, that tells us what the models are really good at in terms of like the technical capabilities and maybe"
  },
  {
    "startMs": 72340,
    "text": "using sometimes like open datasets that are out there. But sometimes that doesn't mean that it will translate effectively to internal data and use cases that you have in, in your organization. And specifically around, you know, financial services, where banks are very, very sensitive to the-- their data, their use cases, and for the most part, they like to believe that that's part of their secret sauce, the way that they, you know, build their own models, build their own, uh, datasets, and also not exposing"
  },
  {
    "startMs": 102340,
    "text": "a lot of that data to, uh, to the public. Um, so that's why we needed to figure out a way that the, you know, financial services and this bank in particular, that they can then say, \"Look, we don't necessarily care what the model can do. What we care about is really whether this application that we're trying to build can ship into production.\" A good example would be like a ten K analyst that they were trying to build for their investment bankers. You know, every time an earnings"
  },
  {
    "startMs": 131860,
    "text": "call is, is out there, they wanna make sure that their teams can accelerate and get leverage from all the information that is out there. And as soon as an earnings call is done, they can get an analysis on all the transcripts and all that. However, that is not just a simple task of just taking the, the call transcript and just like summarizing it and whatnot, because there's a lot of knowledge that the analyst has and a lot of context about how the bank and the firm operates that is not necessarily ingrained or trained in the"
  },
  {
    "startMs": 161560,
    "text": "models out of the box. So how then, then you bring that knowledge and that context into the model? And then furthermore, once you've done that, how then you reliably are able to evaluate that the outputs that it's giving you are the, the, the correct ones and that you can kinda clear those quality gates that you have put in place. So that was, that was, uh, again, a kind of a, a big learning for us that they and they told us, uh, directly, \"Look, we don't certify models here. We certify applications.\""
  },
  {
    "startMs": 191600,
    "text": "Right? So it, it, it could be a ten K analyst, it could be an investment ana-analysis and whatnot. The other, uh, learning from this experience is that in regulated industries and, and f-finance in particular, there are three main requirements when it comes to b-building these solutions. The first one that more, more often than not, they want it to stay on their own premises. Um, and this could be either their own data centers because, uh, it's, it's funny how things are kinda coming"
  },
  {
    "startMs": 221060,
    "text": "back to, to, to center because we have done-- we have been doing all this movement into the cloud, but now because of AI and how sensitive the information that we're feeding these models is, then especially banks are rethinking and repurposing a lot of their data centers that they had and actually standing it up again and, and kinda standing up GPU infrastructure to run models locally because of the sensitivity of the data that they are dealing with, uh, on these models. The"
  },
  {
    "startMs": 251120,
    "text": "other thing is, the other requirement is that it has to be gated. It cannot be just going out to the internet obvious-- for obvious reasons. They don't wanna have even the, uh, the appearance that some model provider will train on their data, even if there is like zero data retention and all, and all these, you know, guarantees. But banks are very sensitive to that, and they want to make sure that that stays within their control. And sometimes you have even banks that actually have, uh, kinda research divisions that actually spend"
  },
  {
    "startMs": 280920,
    "text": "time and have PhDs training their own models and fine-tuning them and all that. And the third aspect is, with all of this, is that the deployment needs to be air-gapped, right? So that only the people that needs to see that data and is within their infrastructure, that is very, very critical. The, the other thing is when, when we were talking to them, it's like, okay, now you have, um, all this platform, we have all these models, but then we need to prove that the application is actually working to the way"
  },
  {
    "startMs": 310980,
    "text": "that we want, one. Second, that is actually meeting the compliance requirements that we have as a regulated industry. And third, that our users are not gonna come to us complaining that it is not usable. Because then, you know, it becomes a situation where you don't get user adoption, and then there is a dissatisfaction and a bad sentiment about using AI-enabled solutions. So we needed to find a way to help them h-come up with a framework and a methodology for them to have like a stamp of approval, pass, fail if an application or"
  },
  {
    "startMs": 341040,
    "text": "an AI agent had any change on their configuration. That could be the system prompt, that could be the model th-they're using, a new tool. But then how do I give it, okay, this is ready to go into production or not based on, on these changes that, um, we have given them? And then if we go, like, very high level, how does that work mechanically, right? The first thing that we need is to build a golden data set. That's the first thing."
  },
  {
    "startMs": 370872,
    "text": "That's gonna be anchor. Um, a lot of these-- Good, good news is a lot of these banks, they typically already have a golden data set that they want to test against. But sometimes what is missing is how they can operationalize the testing, uh, of the test cases against that data set. But that's to me, typically is the anchor. The way that I like to think about this, if you're coming from a software engineering background, is is these are your test cases, right? You will never push anything into production without passing it through QA,"
  },
  {
    "startMs": 400912,
    "text": "right? So this is the same principle. Then once you run this, um, and, and you operationalize it, you're gonna generate a bunch of traces, and the traces are the evidence of what the model was trained on or, sorry, what the model was evaluated on, right, against this data set. So you have that evidence on the trace. And then at the end, with the trace as the, as the, uh, uh, as the, kind of the, uh, as that, you know, evidence that you have, you can then generate the"
  },
  {
    "startMs": 430412,
    "text": "scores that are gonna give you the pass-fail across different dimensions that you, uh, select. When we started out this journey with this customer, it was-- we first started with a couple-- with a bunch of models, um, small, large models with, with different levels of intelligence, and we noticed something interesting, right? So this one is, is a particular example of two numbers that stood out for us. The, the first one, ninety-seven percent in groundedness is that this model was really"
  },
  {
    "startMs": 460292,
    "text": "good at extracting information from, you know, internal sources and grounding the responses on actual data that the bank had. Perfect. Ninety-seven percent. That is a passing bar. Amazing. But then when it came down to give, you know, a-accurate numbers, then the number was trip-- the, the model was tripping up. Because again, LLMs inherently are not great at math. They're just next token predictors. And then what happened was that even though it was grounded the"
  },
  {
    "startMs": 489971,
    "text": "responses on actual information from the bank, whenever it needed to give financial numbers or just do simple math, the model was failing. And then I'll show you in a minute how we were able to go about that. And this leads us to then, again, what I said before, that they don't approve models, they approve applications. So how they can put a harness or controls around this model, so irrespective of if tomorrow I"
  },
  {
    "startMs": 519872,
    "text": "swap a model, then the quality of my application will still remain the same."
  },
  {
    "startMs": 526672,
    "text": "Now we are expanding into kind of the process that we, uh, help them establish. Starting from the left with the golden data set, then you go into the inside of the agent. The agent is going to do a bunch of tasks. It's gonna have a planning phase. It's going to retrieve data from your official, official sources. And then you see the little box that is, is circled in blue, is gonna calculate. In this case, what we did, we gave the model a tool for those areas where the"
  },
  {
    "startMs": 556492,
    "text": "model was com-coming up short, right? The sixty percent where it was not scoring well, then we said, \"You know what? This is a task that the model doesn't need to do on its own. We can just give it a tool.\" Because math, you know, is deterministic. It's, it's gonna be the same all the time. And then we can improve the, the accuracy of the model. But at the same time, if you notice all the dimensions that we have, we are evaluating the application against a multitude of dimensions, not just one. We're not"
  },
  {
    "startMs": 586392,
    "text": "going to say, \"Well, because everything in the average is giving me above ninety percent, then it's going to pass.\" That might not be the case. You might have different degrees or thresholds for the different categories that you have here in terms of the accuracy, regulatory compliance and whatnot. And then at the end is where you define your gate. You see, uh, not-- anything that doesn't clear the gate based on your thresholds doesn't get promoted into production. This is just a, an, a snippet of, of the code"
  },
  {
    "startMs": 616192,
    "text": "and, and at, at the end, a link to the repo and to the blog post that you can read more in depth about the work that we did here. But it's effectively you want to build your own benchmark instead of relying on the public ones that are available there. And the, the first thing you need to do that is to build a data set, like I said before. This is an example of what a typical data set would look like. It's, uh, there are key pairs. You have inputs, expected outputs, and some metadata that you are then"
  },
  {
    "startMs": 646112,
    "text": "u-uh, using to evaluate how the agent is performing, both in terms of quality, behavior, and the, the, the responses that it's giving you as well. This is how it looks after we have, uh, scored or evaluated one agent run. And you notice that I circled the calculate tool call because in this case now this is how we were able to get the model from sixty percent in numerical accuracy all the"
  },
  {
    "startMs": 676092,
    "text": "way down to like ninety-five percent. Because we gave it a tool, anytime there's a number that needs to be calculated, just use this little Python code and off you go. And in that way, you can improve the, the results that you're giving back to your, to your customers. The other step on, on this is about scoring. How do you score each of the dimensions that I was telling you before? We have a couple of ways that you can do this. You can have deterministic scores that, again, like the name"
  },
  {
    "startMs": 706112,
    "text": "implies, is, you know, very, very simple. This is, this is not LLM-based. You just have a code that is running, and it's for the same inputs, it's gonna generate the same outputs all the time. And what we've noticed is with these solutions, as they get more complicated, you need a blend of scores and evaluators that are deterministic and non-deterministic, because otherwise, relying one hundred percent just on LLM judges is not gonna cut it. Um, and then the, the second"
  },
  {
    "startMs": 736044,
    "text": "one, the path, is you also want to figure out if the agent is taking the right, you know, route or path to get to the answer. Because it might be that the answer might look okay or complete, uh, or semi-complete, but then maybe the route it took to get there, it might not be the appropriate one. It might be missing steps. So you also can evaluate that. And the third one is about how you use an LLM as a judge to judge some of the more qualitative"
  },
  {
    "startMs": 765644,
    "text": "aspects of that answer. And by doing all of this, then we got to taking this little model that was failing with sixty percent, we got it to over ninety percent, and then we are clearing the eighty-five percent gate that the bank had for this particular use case by adding that tool, but also by adding this evaluation framework on top that then we can measure the accuracy and the quality of the agent across many different dimensions."
  },
  {
    "startMs": 795664,
    "text": "And this is something very important because now you can go back and show the evidence of how you actually measured the quality of the agent, but also more importantly, you can link it to the changes that were made into the agent, into the configuration, system prompt, model, tools that were used, and then you say, \"Oh, but look at how by adding this tool, changing this model, we went from sixty percent accuracy over to ninety-five, ninety-plus"
  },
  {
    "startMs": 825264,
    "text": "percent accuracy.\" And now this is where you start defining your gates, right? And this is where working with your domain experts becomes very, very critical. You don't want to just rely on technical kind of ability for this. You want to make sure that if you're del-delivering, uh, an, a risk use case that is gonna be NI, AI-enabled, that you have the risk people involved to tell you what is the bar that it"
  },
  {
    "startMs": 855204,
    "text": "needs to meet for them to feel comfortable putting this either internally or in front of customers. And that bar might be different depending on the use case. So that is why you need to have first the dimensions or the scores, uh, on which you're going to evaluate this application. And then for each of those and for each use case or application, you end up with different thresholds that the agent needs to meet in order for it to have, quote-unquote, that stamp of approval, uh, for it to go into production."
  },
  {
    "startMs": 886924,
    "text": "And this is what it looks like when you-- we have like a full end-to-end run across, you know, we had one dataset that it was like one hundred and fifty questions. Another one was like fifteen hundred. So but now you can measure the configuration of the agent across these different dimensions and vertices, and you see how even though it is passing on groundedness, completeness, tool use, you might say, \"Well, if I take the average of all of"
  },
  {
    "startMs": 916864,
    "text": "these, then it should be okay.\" But in regulatory compliance, it's getting a zero. So that is one that is just binary. It's just either zero or one hundred. You don't want to put anything, if you're a bank, out there that is gonna violate any regulatory compliance requirement that you have because you might get penalties, you might incur into reputational damage and all of that. So it might be that one dimension actually completely shuts down the gate for you to move this into production."
  },
  {
    "startMs": 946944,
    "text": "So that's why it's critical to have a multi-pronged view on how you evaluate your agents. And this is where when you start to kinda picking up all these failures, or we call them failure modes, this is where bringing back to the main experts is really critical. You can curate for them what we would call an annotation queue. So then say anytime that there is a failure on regulatory compliance, I want to curate those specific traces or evidence, right, from my"
  },
  {
    "startMs": 976944,
    "text": "evaluation and put it in a place that my domain expert can come in and look at the trace, look at what happened, and then give me feedback. And with that feedback, I can then go back and, you know, feed back into the agent to improve it. And this is what it would look like connecting it to the process before, but including that human expert into the middle. If you notice down below that anytime a couple of things are happening here, anytime a prompt gets"
  },
  {
    "startMs": 1006904,
    "text": "promoted or a change in the configuration, then it triggers a rerun of the gate clearance process or the evaluation process. But then anytime a rewriter or a human expert comes in and gives me feedback, that is a potential candidate to go back into my golden dataset. Because now we have evidence from runs of the agent. It have been reviewed, and now we can say, \"Oh, actually, this one piece is a better, you know, bar to"
  },
  {
    "startMs": 1036784,
    "text": "evaluate this particular dimension on the agent.\" And this is kind of the whole process end to end to make sure that we are including the, the domain experts, but also making sure that we're clearing our, our quality gates. Then this got us to a place where we were able to help them go from weeks to minutes when it comes to, you know, evaluating AI applications, especially in like risk and compliance, which typically take even months sometimes. So but this process,"
  },
  {
    "startMs": 1067064,
    "text": "it becomes very it-iterative. We can be very specific and focused on the areas that we want on the agent. Then there's no template for this. Um, you know, this industry's been, uh, being developed right now, so typically you need to define what is it that you want to evaluate and, and, and kind of the metrics that matter to you. And the three things that I'd like to give you with is you need to write your own benchmark, because like I said, the public ones don't really matter for your data because they were not"
  },
  {
    "startMs": 1096944,
    "text": "done on your data per se. The second one you got-- you have to gate the application, not the model. That gives you more flexibility if tomorrow you want to swap a model. You have already all the controls in the application and, and the configuration of the agent, and then it becomes easier for you to adjust. And then the third, let ma-making sure the domain experts come, come in and set the bar or, or the thresholds that you are deemed, uh, reasonable in order for you to promote this into production. And finally,"
  },
  {
    "startMs": 1127084,
    "text": "you know, use your data. It's your bar. It's your gates, especially in regulated industries where you want to keep things in your premises, and you're probably not relying on publicly available information. So thank you for the time. I know I'm a little bit over, but I wanted to give you with, uh, the detailed blog post of this work that we did. It also links to the repo that you can reproduce this yourself and, uh, I'm around if you have any questions. So thank you."
  }
]
