Building the AI Agent Is Easy. Proving It's Safe Isn't - Jonathan Osborne, NatWest

Loading slides…

Read the talk

How NatWest Tests Whether Cora Can Safely Answer a Customer

A mortgage question, 111 payment conversations and 14,000 complaint investigations show how NatWest decides which requests an agent can handle and how much confidence to place in the scores.

From a talk by Jonathan Osborne

At a glance

Ideas worth remembering

  • Route button clicks and recognizable requests deterministically. For requests that need a classifier, check the customer’s permission after it predicts a backend.

  • Report results by scenario and by scoring criterion. NatWest’s 83% overall payment pass rate included a 43% pass rate on topic changes, and reviewers agreed much less often on intent than on fairness.

  • Calibrate a judge on paired human and model scores for individual cases. Check the spread, the correlation and the margin above the threshold whenever the judge model changes.

  • Use shadow classification to inspect routing on real requests, then expand customer exposure by brand and percentage with a tested route back to the scripted service.

A new answer at the scale of 100,000 conversations a day

Jonathan Osborne, a lead AI engineer at NatWest Group, could build an agent that answered banking questions. The harder engineering problem was deciding what evidence would justify letting it answer customers. NatWest, the UK’s second-largest retail bank, had roughly 19 million retail customers. Its digital assistant, Cora, was their primary point of contact and handled around 100,000 conversations a day. At that volume, even an infrequent mistake could affect someone making a decision about real money.

Most of Cora’s answers were scripted. If the team had anticipated a question, it could write and test the path to an answer. A customer whose question fell outside those paths might have to rephrase it or queue for a human. NatWest was working to bring generated answers and agents to approximately a third of Cora’s journeys. Retrieval-augmented generation (RAG) would draw on the bank’s knowledge bases. Agents could also take actions on a customer’s behalf, such as helping make a payment or change an address.

That combination widened what Cora could do and the consequences of getting it wrong. A plausible answer could cost a customer money, leave the bank answering difficult questions from its regulator and damage its reputation. To seek approval for the customer-facing service, the engineers first had to establish which of their own measurements they could trust.

0:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

One mortgage question exposes four ways an agent can fail

“Can I overpay my mortgage without being charged?” Scripted Cora recognizes the word “mortgage” and offers choices such as applying, checking rates or going to the customer’s mortgage. The customer can select “none of the above,” but the system has made little progress. Asking the customer to phrase the question differently is another turn spent trying to reach the answer. Queuing for a bank employee is the likely outcome.

In Osborne’s agent example, Cora retrieves guidance and says the customer can overpay up to 10% of their balance in a year without an early repayment charge. It offers to check their deal. That is a much more direct response, but the order matters. The customer has already heard a specific allowance before Cora has established that it applies to their mortgage. Then the customer asks whether they can close the account instead. A change of subject is ordinary customer behavior, even if it breaks the sequence the engineer had in mind.

Two illustrative chats start with the same mortgage overpayment question. Watson offers a menu and the customer rejects the choices. The generated answer mentions a usual 10% allowance, then the customer asks about closing the account.Open full source frame
We are changing what happens next

Osborne identified four failure modes. First, an agent can sound certain about a fabricated answer and even cite a fictitious source. Second, the input can become adversarial through prompt injection, with someone trying to talk the agent into doing something it should resist. Third, an answer can be accurate and within policy while the conversation goes in circles. A customer who still cannot get their question answered experiences a failure that a factuality check could miss.

Fourth, the same input can produce a different output on the next run. The Cora team had accumulated years of regression tests for a decision tree whose possible paths it could enumerate. An assertion about the response expected at a particular point worked for that system. With an agent choosing a course of action and generating its reply, passing that assertion on one run offers limited assurance about the next.

2:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:14 · section reference included

Cora filters known events before it starts a model

Every incoming conversation enters a routing funnel. The first gate is a contextual filter for events such as button clicks. If a customer has just selected a known option, the event itself conveys their choice. This filter handles about 20% of incoming traffic. The second gate applies deterministic preclassification rules to the customer’s utterance and accompanying metadata. When those rules identify a destination, Cora can route the request directly. Up through this point, routing requires no model inference.

The remaining utterances can enter a gated relevance and rephrasing step, useful for detailed questions that depend on several pieces of context. Next comes the intent classifier, a neural network over embeddings that predicts which backend should handle the utterance. This is a traditional machine-learning component. A final set of policy rules checks whether this particular customer should have access to the predicted backend. A classifier can correctly identify what someone wants to do while the access decision still has to be made.

Horizontal bars narrow from 100% of customer turns to 80% at the contextual filter and about 45% at each of four routing stages. A side panel explains how the rules support access control, auditability, resilience, and governance.Open full source frame
The routing funnel

Four operational requirements shaped the funnel. The policy gate enforces access control. For auditability, every routing decision carries a code so the bank can reproduce and account for the route it took. For resilience, the scripted service gives Cora a way to answer when a model is unavailable. For governance, the team keeps routing rules in versioned configuration. It can change a rule in response to changing conditions without redeploying the whole application.

4:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:44 · section reference included

Routing accuracy leaves three questions about the answer

Suppose the intent router is 94% accurate. That tells an engineer how often it picked the right destination. It says little about whether the eventual answer was grounded in a real source, whether the customer’s problem was resolved or whether Cora recognized when to bring in a human. Osborne contrasted a router that reports 100% confidence even on the 6% of requests it misroutes with one whose 85% confidence is well calibrated. The latter gives Cora a signal it can use when deciding whether to proceed or hand off.

NatWest evaluates each answer across three categories. Safety asks whether the response could harm the customer. Task completion asks whether it accomplished the job. Experience asks what it felt like to deal with Cora. Each can have its own monitoring and alerts. An invented mortgage term merits investigation into the source and the answer. Repeatedly returning accurate guidance to a customer who is trying to do something else calls for a different investigation. An overall quality score could send both incidents to an engineer looking for the same defect.

6:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:30 · section reference included

Payment reviews uncover a 54-point gap on changes of subject

Before automating the scoring, NatWest ran a human validation exercise in June. Three people independently reviewed each of 111 conversations covering 37 payment journeys across four brands. The payment agent passed 83% overall. Split the conversations by what the customer did, however, and the result changes substantially. Happy-path scenarios passed 97% of the time. Edge cases passed 76%. When the customer changed topic, the pass rate fell to 43%.

The gap between following the expected path and changing the subject was 54 percentage points. The customer who asks about closing an account partway through discussing a mortgage illustrates the kind of digression at issue. Testing only whether an agent could carry a cooperative customer through a payment would have missed much of the difficulty. Human review also helped locate faults outside the model. The two biggest causes of failure in the exercise turned out to be engineering issues.

Independent scoring exposed another variable in the evaluation: how consistently people could apply each criterion. Reviewers agreed 95% of the time on bias and fairness. They agreed only 64% of the time on intent understanding, the judgment about whether the agent understood what the customer wanted. That complicates the instruction to make an automated judge match a human. On intent, choosing a different human as the reference could change whether the very same judgment counted as a match.

The June baseline shows pass rates of 97% for happy paths, 76% for edge cases, and 43% for topic changes. A separate panel shows pairwise scorer agreement ranging from 95% on bias and fairness to 64% on intent understanding.Open full source frame
We had humans score it first
7:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:44 · section reference included

A judge averaging 0.97 could barely distinguish good cases from bad

Cora Plus had yet to go live, so Osborne drew the production evidence for automated judging from a separate system. NatWest’s internal complaints agent, live since February, receives a complaint, uses tools to investigate and writes a summary for a complaint handler. The summary should help the handler understand why the customer complained and how the bank can help. Around 14,000 cases had scores from both a human validator and an LLM judge, allowing the team to compare the assessments case by case.

On faithfulness, the judge gave a mean score of 0.97, with scores packed near the top of the scale. Humans averaged 0.83 and used a much wider range. Osborne reported an average error against the humans of about 0.2, a number that might look tolerable in isolation. The correlation was almost zero. When handlers marked some summaries down and gave others high scores, the judge scarcely distinguished between them. An engineer using its score to find summaries worth investigating would have little to go on.

Two bars compare faithfulness scores for the same outputs. The LLM judge’s narrow red bar sits above the 0.89 pass mark. The wider green bar for human scores crosses it. The reported correlation is 0.04.Open full source frame
A broken judge looks like a good judge

A model change exposed a second problem. The team kept the input and the agent’s output fixed and changed only the model doing the judging. In Osborne’s threshold comparison, the reported margin above the cutoff fell from 0.8 to 0.2. The work being assessed was identical, yet the bank’s apparent headroom had narrowed. The judge version therefore belongs in the configuration alongside the threshold. NatWest also moved the judge to a different model family from the agent being evaluated. Osborne said the different blind spots helped bring judge and human scores closer together.

9:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:14 · section reference included

Trace each conversation, score samples every four hours and gate prompt changes

Cora’s evaluation pipeline captures activity from the agent and its guardrails using OpenTelemetry and OpenInference. It takes a 5% random sample, then also selects conversations that hand off to a human, fail, run particularly long or show signs of an angry customer. The two selection methods serve different purposes. The random sample includes ordinary interactions. The targeted selection gives engineers more chances to examine behaviors that would appear infrequently in that sample. The LLM judge scores the selected cases across the quality dimensions, and traces, scores and experiments go into Comet Opik.

A four-stage flow follows a conversation through sampling and LLM scoring into Comet Opik. Panels below describe private connections, PII masking, and the timing of traces, offline scoring and regression checks.Open full source frame
How the scoring runs

The bank runs in an air-gapped environment without internet access. Evaluation reaches the judge over a private path through an internal GenAI gateway. NatWest consumes Comet, a SaaS product, as a signed partner app through AWS. The bank also has zero tolerance for personally identifiable information leaving its network, so much of the evaluation takes place with that information absent.

Trace capture, scoring and release decisions operate on different clocks. Cora records traces as events happen. Offline scoring runs every four hours. A change to a prompt or the context supplied to the agent triggers a comparison against a known-good baseline in CI/CD. A hard regression gate holds the merge until the team understands the effect. This matters even for a small text edit because its effect on the agent’s responses can be disproportionately large.

11:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:05 · section reference included

Trace spans locate agent failures while feature drift warns of classifier decay

In a trace from the payment agent, each LangGraph node appears as its own span. If the conversation is slow or breaks at a particular step, the engineer can open that part of the trace and investigate what happened there. The conversation-level score identifies an interaction worth inspecting. The spans give the investigator a place to start inside the agent.

The intent classifier needs another instrument. Customers’ language evolves after training, so the distribution of inputs can shift even while the classifier stays fixed. NatWest monitors input feature drift alongside F1 calculated from human labels, with thresholds for triggering retraining. Osborne found feature drift particularly valuable because a shift in the inputs could show up before a decline in F1. The team uses Comet for both kinds of work, with Opik for generative AI and model performance monitoring for traditional ML. Engineers can observe both parts of a journey while measuring each according to how it fails.

On the left, an Opik trace and node diagram show a LangGraph payment agent spending 61.6 seconds in its outbound guardrail. On the right, a multicolored area chart shows classifier feature drift rising and falling during 21–28 Sep.Open full source frame
Two models, two failure modes
12:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:44 · section reference included

Shadow routing gives risk reviewers production evidence before customers get generated replies

Model-risk colleagues wanted to know how Cora would handle actual customer language before approving a customer-facing release. Yet the team needed access to production requests to gather that evidence. Shadow deployment separated two decisions that would otherwise have had to happen together. The candidate classified live utterances while existing scripted Cora supplied the replies customers actually saw. Reviewers could inspect the real range of queries and where the classifier would send them. Whether a generated reply would resolve a customer’s problem remained a question for the next stage.

After shadow routing, the team planned to roll out customer-facing replies brand by brand, gradually increasing the percentage of traffic exposed within each brand. At each stage, performance indicators would help determine whether to expand further. Some decisions could be automated and others required manual approval. Feature flags gave operators three levels of intervention. They could isolate a single backend for one brand, isolate that backend globally or use the global flag to send every request back to scripted Cora. Each change could take effect within seconds.

Four boxes show a gated rollout from shadow classification, while Watson answers, to customer-facing routing at successively larger brands. Three panels list the requirements to advance, live alerts and rollback options.Open full source frame
Shadow first, then brand by brand
14:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:07 · section reference included

A required reason for reruns turns complaint handling into evaluation data

The complaints team collected feedback at a point where a colleague had a specific judgment to offer. A handler who wanted the agent to rerun an investigation first had to explain why the original attempt needed another pass. Capturing that reason before the rerun helped build the set of roughly 14,000 colleague labels. Osborne said other projects struggled to collect even a tenth as many. Those labels made it possible to discover how generously the production judge was grading the work.

The two projects exchanged what they learned. Cora incorporated judge calibration and regression checks after the complaints team found the scoring problem. The complaints team, in turn, planned to adopt Cora’s Opik-based production monitoring. Each agent addressed a different banking task, but the machinery for recording what it did, comparing assessments and detecting regressions could be reused. At the time Osborne spoke, the complaints agent had supplied the live evidence. Cora Plus was due to face its own customer rollout the following month.

15:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:43 · section reference included

Transcript download
Full transcript

Loading transcript…