← All speakers

Jonathan Osborne

Conference affiliation: NatWest

Jonathan Osborne works on the engineering needed to move a banking assistant from scripted replies to agents that can answer questions and act for customers. A lead AI engineer at NatWest Group at the time of his 2026 talk, he described his team’s work on Cora: deciding which requests need a model, measuring failures that an accuracy score conceals, and gathering evidence before customers depend on a new capability.

On this page

A mortgage question makes the stakes concrete. A scripted assistant might recognize “mortgage” and return a menu without answering whether an overpayment incurs a charge. An agent can retrieve a specific allowance, but an answer that is correct for one mortgage deal can cost another customer money. Jonathan’s concern is the connection between a plausible answer and the particular customer entitled to rely on it.

Cora’s development is a wider collaborative effort. NatWest introduced the assistant in 2017 and announced a Cora+ pilot developed with IBM in June 2024. Jonathan’s account concerns preparations for further customer-facing agent capabilities, including payments and address changes; those capabilities were awaiting rollout when he spoke.

Measuring failures that averages hide

Jonathan separates answer quality into three dimensions: safety, whether it could harm the customer; task completion, whether it resolved the request; and experience, what getting help felt like. Each needs its own monitoring because a fabricated answer and an accurate conversation that goes nowhere require different repairs.

A June payment-agent evaluation showed why the distinctions matter. Three reviewers independently scored each of 111 conversations covering 37 payment journeys across four brands. The overall pass rate was 83%, but it reached 97% when customers followed the expected steps, fell to 76% on edge cases and dropped to 43% when they changed the subject. The two biggest causes of failure were engineering issues. Improving the model alone would have missed them.

Human review also exposed uncertainty in the evaluation itself. Reviewers agreed 95% of the time on bias and fairness, but only 64% on whether the agent understood the customer’s intent. For Jonathan, that complicates the instruction to make an automated judge match human reviewers: the team first needs to understand where the reviewers disagree.

A separate NatWest agent that drafts investigation summaries for complaint handlers supplied a sharper warning. Across roughly 14,000 cases, a language-model judge’s faithfulness scores averaged 0.97, while human scores averaged 0.83 and varied much more widely. Case-by-case correlation was almost zero. Adjusting the judge’s average would not make it identify the summaries people rated poorly.

Lessons from that project informed Cora’s judge calibration and regression checks. Jonathan also described using a different model family for the judge and the agent to reduce shared blind spots. Changing the judge model could shift scores even when the material being assessed stayed the same, so the judge’s identity had to be treated as part of the evaluation configuration. The complaints project made human feedback a prerequisite for rerunning an investigation, giving the team both a reason for the rerun and a growing set of labels to examine.

Controlling exposure before rollout

Jonathan’s approach combines evaluation with decisions about where agents operate:

  • Route selectively. Cora filters routine events and applies deterministic routing rules before invoking models. An embedding-based intent classifier selects a destination, and policy checks enforce access to banking systems. Recorded routing decisions support audits; versioned configuration lets the team change routing without a full deployment.
  • Make changes testable. Traces let engineers inspect individual steps in an agent’s execution. Automated scoring and regression gates compare prompt or context changes against a known baseline. Jonathan distinguishes these regressions from the gradual drift of the traditional intent classifier, which requires monitoring changes in inputs and performance against human labels.
  • Observe before customers depend on it. In shadow deployment, the new system processes live requests while the existing scripted service continues to answer. Model-risk colleagues can examine behavior on real traffic before approving customer-facing use. Rollout then proceeds by brand and percentage, with feature flags that can disable an individual backend or return the service to scripted responses within seconds.

Jonathan emphasizes the reusable work around the agent: tracing, human feedback, calibrated scoring and regression checks. The complaints project supplied evidence of a broken judge; Cora’s team used that lesson in its evaluation design, while the complaints team planned to adopt the monitoring architecture developed for Cora. That exchange makes the infrastructure useful beyond a single assistant.

1 conference talk

Loading more talks and highlights…

References