[
  {
    "startMs": 12700,
    "text": "Hi guys. So my name's Jonathan Osborne. I am a lead AI engineer from NatWest Group. I'll tell you more about what that is in a second. Uh, and today I'm gonna be talking about taking agents from prototype to production, customer-facing that is, in a regulated banking environment. And I guess maybe some spoilers. With the access to coding assistance and tools that we have today, the actual creation of the agent, that's kind of the easy part. The much harder part, I think, is proving that it's safe. So let me tell you a"
  },
  {
    "startMs": 42640,
    "text": "bit about Nafwe-- NatWest Group. So NatWest is the second-largest retail bank in the UK, and currently we have somewhere in the range of nineteen million retail customers. Nineteen million, hold onto that number because it's really what makes everything else in this talk matter. Cora is our customer's primary point of contact, so it's our digital assistant, and it serves somewhere in the realm of a hundred thousand conversations a day. And until now, the vast, vast bulk of those answers were scripted. So if a customer comes to"
  },
  {
    "startMs": 72500,
    "text": "us and they ask a conversation, then if we haven't thought to script that, then, uh, you know, we're not gonna be able to answer. Customer's gonna get frustrated. They're probably gonna have to queue to speak to a human agent. That's changing. So a-as of this year, we are now working to bring approximately a third of the journeys within Cora to AI-generated answers backed by RAG and our knowledge bases and agents. Agents that are able to take action on the behalf of customers. So for example, helping them make a payment or change their address."
  },
  {
    "startMs": 104020,
    "text": "I guess the, the scary part here really isn't the model. Like by and large, models are quite good. They provide good answers now. Uh, it's the blast radius. So with nineteen million customers, uh, if we get a few of these utterances wrong, uh, that isn't a bug ticket. That isn't kind of a maybe a funny headline. It's a real customer, and they're real money. It's likely very difficult questions from the regulator, and it's reputational damage for us as a company. So this talk is about what we had to go through to get the approval to go live. A-and"
  },
  {
    "startMs": 134020,
    "text": "I guess what it took us to maybe trust some of our own metrics as well because as I say, the interesting part is not the agent, it's proving and making it safe. Let's maybe make that more concrete by considering an example here. So kind of same question, two very different conversations. So customer comes, they ask, \"Can I overpay my mortgage without being charged?\" Today, in the scripted solution, that's gonna recognize that keyword mortgage, and we're gonna present the customer with a menu. So apply rates, my mortgage. Customer"
  },
  {
    "startMs": 164020,
    "text": "is gonna have to select none of the above. Maybe we'll ask them to rephrase their question, but more than likely, game over. This customer is gonna have to queue, speak to a human agent. They're gonna be frustrated. On the right-hand side, bit of a different story. So AI has been able to answer that question using our knowledge bases. So you can overpay up to ten percent of your balance in a year and no early repayment charges. Shall I check your deal? But then the customer does something which is very normal but is can be frustrating, so they change the topic. So actually, can I close the account instead?"
  },
  {
    "startMs": 194460,
    "text": "This happens all the time, and a traditional decision tree is really, really bad at this type of digression. It's worth mentioning that digressions are gonna come back, so we'll hold on to that point. I guess the key point though is if we build an agent that can answer any question, it can also answer any question wrong. So that ten percent, if that doesn't apply to this particular customer's deal, it's a real customer, it's their real mortgage, and it's their real money at stake. So why is it so much harder to make an agentic-backed so-solution, you know,"
  },
  {
    "startMs": 224640,
    "text": "more, uh, you know, how do we protect against it as opposed to that scripted bot? Well, it's because it reasons, it decides, it generates. And there are at least kind of four failure mode here, which we wouldn't have in that scripted solution. One, it can be confidently wrong. Hallucination against a fictitious source and sound absolutely sure of itself while it does it. Two, it can be talked into things. Prompt injection. The input itself can become adversarial. Three, right but unhelpful. So it can provide accurate on-policy answers that just go"
  },
  {
    "startMs": 254660,
    "text": "round and around and around and never actually answer the customer's conversation. Incredibly frustrating. And four, it can be non-deterministic. Same inputs, completely different output. That means that your traditional software engineering assertions just don't really work very well here. And as a consequence of that, regression tests that have been built up over years by the existing Cora team, they really no longer apply. A scripted system's failure surface is completely bounded. We can write out every p- situation that might occur. We can test it, and we can be"
  },
  {
    "startMs": 284640,
    "text": "really confident it's gonna do what we want it to. With an agent, that's really not possible, and we can't really traditionally test our way to safety. So how do we deal with this? Well, the first part for us was shrinking the attack surface before we actually then thought about hardening it. Introducing what we call the routing funnel. So every customer conversation comes in at the top here. First, we apply a contextual filter, and that allows us to remove kind of common events like button clicks. That's about twenty percent. Next, we have some"
  },
  {
    "startMs": 314500,
    "text": "deterministic preclassification rules. So with these rules, we're kind of using the customer's utterance and some metadata to route them to a destination with no ML or AI involved at all at this point. It's only utterances that make it past these first two gates that we then actually start the models on. First, there's a relevancy and rephrasing step. That's kind of gated. It helps with those really detailed multi-context questions. And then our intent classifier. This is actually a neural network over embeddings. This isn't genitive AI, which is able to use a customer's utterance to predict the right"
  },
  {
    "startMs": 344540,
    "text": "backend. Finally, there's some policy rules to ensure that this customer should actually be getting into that particular backend. Why'd we build it this way? Well, four reasons, a lot of them about being a bank. Firstly, access control, so we need to enforce that to ensure the customer should be getting to this ac-- this, this backend. Auditability, so every decision here needs to carry a code. It needs to be fully auditable and reproducible. The regulator ensures this. Resilience. In the case that a model or the gen AI isn't available, we still need to provide"
  },
  {
    "startMs": 374520,
    "text": "the customer an answer. And I guess governance. So all of this is version config. We can make changes to it without a full redeployment. That's really important because it means that we can react fast to changing market conditions. We didn't just take a language model, slap some context into it, and point it at ninety million customers. We first thought really hard about what actually needed a model to solve. This brings us on to sort of the really important question of is this actually working? And the key metric that they looked at in kind of traditional Cora was accuracy when doing this. So if I told you this"
  },
  {
    "startMs": 404484,
    "text": "intent writer is ninety-four percent accurate, that probably sounds really good, but does that actually answer the important question? So is it grounded in a real source? Did the customer's question actually get resolved? Did it know when it was out of its depth, when to stop, bring in a human? You can't embody that with one single score, and that's why we've moved away from quality of the score, started treating it more as a vector. Uh, every score-- every kind of answer we score across three different categories. So safety, could it harm the customer? Task, did it actually do its"
  },
  {
    "startMs": 434484,
    "text": "job? Experience, how did it feel to deal with? All of these can be monitored and alarmed independently, and they allow us to differentiate between is it hallucinating versus is it right but just insufferable to deal with. There are different answers, and they require really different solutions. It's where-- The key point here is a router that's ninety-four percent accurate but a hundred percent confident the six percent of the time it's wrong is way more dangerous than one that's eighty-five percent confident and knows it. The first thing we"
  },
  {
    "startMs": 464404,
    "text": "did here before we worried about automating this is what we always do, and we started off with some human validation. Uh, the numbers that you see here are from June validation that we did on our payment agent. Hundred and eleven conversations over thirty-seven different payment journeys, four brands, three independent human scorers per conversation. Headline here, eighty-three percent pass rate across the board. But as ever, the headline's pretty useless. Uh, let's dig into some of that data. So ninety-seven percent at-- on the happy path scenarios. That's great. Seventy-six percent edge cases."
  },
  {
    "startMs": 494224,
    "text": "Hmm, not quite so good. Topic changes, like that mortgage example that we discussed earlier, only forty-three percent. So that single eighty-three percent score was hiding that fifty-four percent spread. The second point on the right is probably more nuanced and less talked about, and that's do human validators actually agree with each other? On something objective like bias and fairness, you do actually see a ninety-five percent agreement rate, which is great. But on something more subjective, intent understanding, so, you know, are-- does the bot actually understand what"
  },
  {
    "startMs": 524184,
    "text": "the, what the user wants? Only a sixty-four percent agreement rate. The common thing when we start automating this, bringing LLM as a judge, is that we want to make our judge match our human validators. But what are they meant to be matching here? There's only a sixty-four agreement re- agreement rate between humans, so that can't simply be the target here, essentially. One, one caveat, by the way, like, the two biggest causes of failure here actually turned out to be engineering issues, not a problem with the model at all. But nevertheless, that proves useful value of the human validation because it caught"
  },
  {
    "startMs": 554124,
    "text": "them. So let's turn to actually automating this then, so LLM as a judge. So Cora Plus, that's not live yet, so I'm gonna turn to another internal project. So this is an internal complaints agent that we have. Essentially, the goal here is that customers will occasionally complain. This will take the complaint. It has access to a load of tools to then be able to write a summary for complaint handlers to essentially explain why has this particular user complained and how we can help them with it. It went live in February, and we've got around fourteen thousand of"
  },
  {
    "startMs": 584124,
    "text": "these cases which have been scored by both a human validator and an LLM as a judge. If we dive into the data, so this is the faithfulness score here on the right. So you can see upper bar in the red there. That's the LLM as a judge. Mean point nine seven, right up at the top, very little spread. Human, uh, mean point eight three, much, much wider spread. Key point here is that if you look at that average error versus the human, it's point two. You might conclude that the judge is doing okay, but the correlation is almost zero."
  },
  {
    "startMs": 615264,
    "text": "Uh, in, in this case, like, a judge which doesn't discriminate between the actually good cases and the bad cases is basically worthless. You don't want your judge to be generous, and that is what LLMs want to do by default. Maybe one other point to make here is that, so that amber dotted line, that represents the threshold for this particular metric. For this judge model, you can see we're comfortably above that. However, we made a change to that model. Didn't change anything else. Same input, same output. Just changed the actual model behind it. And that dropped all the way from point eight"
  },
  {
    "startMs": 645144,
    "text": "above to point two above. That means that the judge model itself actually becomes part of your configuration, and it can lead to breaking changes. A, a key point we took here, by the way, is that we moved our judge to a different model family from that of the, uh, you know, the actual agent. So judge and judged, different blind spots, and that's had to be really effective in pinning the two scores closer together. Coming back to the core example. So how does this scoring actually run? So live conversations between the agents and its guardrails, all of those are captured"
  },
  {
    "startMs": 675024,
    "text": "via open source standards like O- OTEL and OpenInference. All of that detail is then sampled. So by default, we take five percent random sample, but we also want to look at everything interesting. By interesting, I mean conversations that hand off, fail, those are particularly long, one for the user of angry. We then score these with our LLM as a judge. So that's that raft and dimension of metrics that I talked about earlier. And then absolutely all of this ends up in Comet Obiq. So traces, scores, experiments. It's governed by a couple of bank constraints here. So we work,"
  },
  {
    "startMs": 705064,
    "text": "work in a completely air-gapped environment, so no internet access to anything. So judge model is reached over a private path. We have an internal GenAI gateway. Comet, a SaaS product. We're consuming that as a signed partner app via AWS. Second around personally identifiable information. We have a zero-tolerance policy for any PII leaving our network. So that means a lot of this evaluation happens without that present. Different cadences here. So traces are being captured as they happen. We're doing offline scoring every four hours. And then regress-- This is also built"
  },
  {
    "startMs": 735104,
    "text": "into our CI/CD. So if we make changes to prompts or to context, that can have, like, an outsized impact on what the agent is actually responding. So we need to C-c-constantly be comparing versus a known good baseline. We do that in CICD. They have a hard regression gate, so changes can't be merged in without us understanding exactly what's going to occur. I guess key point is that continuous evaluation is part of your production infrastructure. It's not a notebook that you run a couple of times when your model goes live."
  },
  {
    "startMs": 764264,
    "text": "So there's different models here, and they've got different failure modes. So on the right-hand side, we've got the agents, we've got the generative AI. So generative AI regresses. So as we've just talked about, somebody can make changes to a prompt, and that can have outsized impact on the behavior. That's why we have the CICD in place. You can see here a real OpenTrace, which is looked from our payment agent. Every node in the Lang graph, uh, becomes a different span. And when the agent is slow or is breaking in a particular place, this allows us or an agent to jump in and investigate exactly why."
  },
  {
    "startMs": 794624,
    "text": "On the classical s-- so the intent classifier is traditional ML, and thus we've got to monitor it in a more classical way. It drifts over time, right? So human conversation will naturally evolve from the point at which the model was trained, so we need to look at more traditional things like input feature drift, F1 score based off human scores... Sorry, human labels. And then, you know, we have RAG thresholds defined as to exactly when we need to trigger retraining for that model. On the right, you can see an NPM data drift chart. I, I really wouldn't underestimate, like, input feature drift here. It's a really good"
  },
  {
    "startMs": 824144,
    "text": "canary as to when your model is starting to de-decline before your F1 score does, implying the need for retraining. So two different problems, two different instruments. But what we don't want is two different observability and evaluation frameworks. Comet's a really useful layer because it brings in both Open for the GenAI and NPM model performance monitor for the ML. So one platform. How do we actually deploy this then without kind of breaking the bank and creating a load of headlines? We start off with a shadow deployment. So what I mean by that is we put this"
  },
  {
    "startMs": 854084,
    "text": "live, uh, we use it to start classifying customer utterances, but the existing scripted solution is what will actually respond to the customer. And that means that we can get a sense of the real date, you know, the, the input, uh, the input space for different customer queries with almost zero risk. I think this shadow mode is like one of the more transferable bits of this project for us because often we have this deadlock in a regulated industry where obviously we can't go live and get production data without the approval to do so. But equally, to"
  },
  {
    "startMs": 884044,
    "text": "actually enable us to go live in many cases, our model risk colleagues increasingly want to see how our models are going to react to that actual production data. This allows us to kind of break that apart so we can break apart, observe safely from actually answer customer queries. Uh, and it takes one really big decision point, a big bang release, into a series of smaller ones, which we can make confidently. We then go brand by brand, so we have like percentage-based rollout within the brand, and we have kind of KPIs which allow us to determine go, no-go"
  },
  {
    "startMs": 914004,
    "text": "at various points along that. Some of those decisions are automated, and some of those require manual approval. Um, I think the... And yeah, some of those require manual approval. What kind of makes that possible is that we u- make a lot of use of feature flagging here. So there's like a global flag that we can trigger to send everything back to the scripted solution. But also every single back end can be isolated either globally or on a particular dimension, such as brand. And all of that can be done in seconds. Before I wrap up, let me"
  },
  {
    "startMs": 943884,
    "text": "maybe just talk very briefly and pull this back to that complaints project. So just to remind you, that was the colleague-facing agent which writes investigation summaries for, summaries for complaint handlers. In that project, human feedback is a gate. It's not a thumbs up. It's not a tick box activity. So if a complaint handler wants to rerun their investigation, they can't do so until they provided human feedback. We want to understand why they felt the need to do that rerun. But also it's allowed us to build up this quality data set of fourteen-- fourteen thousand customer of, uh,"
  },
  {
    "startMs": 973924,
    "text": "of colleague labels, I guess, in this case. And that's really valuable. In a lot of projects, we struggle to get a tenth of that number. And I think the projects have also traded nicely here. So, you know, they found that judge problem in production. We have judge calibration and regression checks because of that. But equally, the architecture that we built, which relies heavily on Comet Open, they don't use that yet, and they're gonna be adopting that for their production monitoring. I guess it's a glimpse inside an enterprise, right? So the, the agent with the cheap part here, that's code that we can produce to solve the business problem. The valuation"
  },
  {
    "startMs": 1003844,
    "text": "spine is the bit that we can then reuse again and again. So to wrap up, like, if you are going to be shipping customer-facing agents in a regulated environment, then these are probably the points I would give you. I'm not gonna go through everything or maybe just pick a couple. So three and four here, they're the, they're the points with the actual production judge data, right? So that judge pinned near the top is a broken instrument. Your judges are gonna want to be overly positive. You need to fight against that. Compare the spans, not just the averages."
  },
  {
    "startMs": 1033824,
    "text": "Ensure the judge is actually matching the human reviewer. Fourth, the judge model itself does become part of that threshold config, right? So if you are going to be making changes to the judge model, then, uh, you know, you can introduce a breaking change without knowing it. Uh, Comet is obviously absolutely key here. So we have that for both halves, both the generative AI and the traditional ML, which we do still use a little bit. So, and I guess the, the final point here is next month I'll find out how much of this is actually true when we go live. So thanks"
  },
  {
    "startMs": 1063864,
    "text": "a lot for listening. It was really a pleasure to get to speak to all of you. So I'll be at the Comet booth after this. If you do have any questions, you know, disagreements, et cetera, really happy to hear them. But thanks a lot for your time."
  }
]
