What a $900M Bank Error Teaches Us About AI Agents - John Feminella, Two Sigma

Loading slides…

Read the talk

What a $900M Bank Error Teaches Us About AI Agents

A registry, risk scoring, runtime guardrails, shared evaluations and runtime protections form a combined control system.Open full source frame
A registry, risk scoring, runtime guardrails, shared evaluations and runtime protections form a combined control system.

John Feminella uses Citibank’s mistaken Revlon payment to explain five composable controls for autonomous agents: make them visible, review them by risk, enforce purpose-based access, share evaluation infrastructure, and reserve human attention for consequential decisions.

From a talk by John Feminella

At a glance

Ideas worth remembering

  • A warning and multiple approvers can still miss the consequential distinction: Citi intended an interest payment, while the software sent principal too. Controls must help distinguish intended actions from dangerous ones.

  • Make registration a low-friction part of building agents. Ownership, declared tools and endpoints support both governance and reuse; a heavy universal gate can push work out of sight.

  • Review agents by risk and enforce access by purpose. In managed environments, registry policies become restrictions on endpoints, hosts and resources rather than instructions the model is expected to remember.

  • Share the common evaluation machinery while leaving teams responsible for agent-specific expectations. A model change is a dependency change whose effects need evaluation.

  • Use deterministic rules for settled cases, independent classifiers for remaining routine decisions, human review for consequential ambiguity and outright denial for clearly dangerous calls.

The warning was correct. The payment was wrong.

On August 11, 2020, Citibank was supposed to distribute about $8 million in accrued interest to Revlon’s lenders. Instead, it sent about $900 million—the outstanding principal of the loan. A warning appeared, and three approvers accepted the transaction. John Feminella, who introduces himself as working on AI architecture and engineering at Two Sigma, starts here because the failure happened inside a process with controls, not in their absence. His views in the talk are his own, not Two Sigma’s.

Revlon’s 2016 term loan was divided among more than 300 lenders, including hedge funds, credit funds, collateralized loan obligations and banks. Citi served as the administrative agent: Revlon paid Citi, and Citi distributed payments to the lenders. The planned transaction combined an interest payment with changes to the loan arrangements. Some principal would roll into a new facility through an exchange of paper; the remaining principal would stay put. Only interest was supposed to leave the bank as cash.

The processing software, FLEXCUBE, made that seemingly simple requirement awkward. In Feminella’s account, the easiest way to calculate the correct interest payments was to record the transaction as though Revlon were paying off the entire loan. The principal portion then had to be redirected into an internal wash account—an account owned by the bank—so that it was booked without actually leaving the bank’s perimeter. The software’s representation of a full payoff depended on a separate routing configuration to produce an interest-only cash payment.

Which configuration separated those two outcomes? The fields labeled FRONT, FUND and PRINCIPAL needed the wash-account number and their associated checkboxes selected. In the screen Feminella shows, only PRINCIPAL has an account number and a checked box. FRONT and FUND are blank and unchecked. FLEXCUBE warned that money was about to leave the bank, but money was supposed to leave: that was the point of paying the interest. The three approvers accepted the warning, and the principal went to the lenders instead of the wash account. The crucial distinction was between an expected interest outflow and an unintended principal outflow; a general warning about money leaving did not make that distinction useful to the reviewers.

The FLEXCUBE screen highlights FRONT, FUND and PRINCIPAL settings in the wash-account routing example.Open full source frame
The FLEXCUBE screen highlights FRONT, FUND and PRINCIPAL settings in the wash-account routing example.

The recipients received what they would have been owed on a full repayment, although Revlon had not intended to repay the loan. That mattered because its troubled debt was trading below face value. Feminella illustrates the incentive with debt worth sixty cents on the dollar suddenly repaid in full. Lenders resisting repayment to Citi argued that they had simply received money they were owed, albeit earlier than expected. He connects the incident to clauses subsequently added to commercial loan agreements to address mistaken payments.

The agent-safety question follows directly: how do you give a system real autonomy over consequential software without relying on warnings that people can reasonably misunderstand? Adding a procedure and multiple approvals had not ensured that the actual cash movement matched the intended transaction. Agents will encounter the same gap between what a workflow appears to mean and what its tools actually do.

0:122:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Give agents useful access without trusting instructions alone

An agent’s behavior cannot be known completely in advance. Feminella’s useful comparison is with human workers: people also make unpredictable decisions and mistakes, yet organizations still let them communicate, use tools and act independently. Boundaries, visibility and supporting systems make that autonomy workable. The design goal is to make useful activity safer, rather than treating safety and useful activity as opposite ends of a single slider.

Three tempting approaches fall short for different reasons:

  • Lock down everything. A sandbox with no access to real systems limits exposure, but also removes much of the agent’s practical value.
  • Allow everything. Broad freedom provides capability with weak isolation, which Feminella considers unsuitable for highly regulated enterprises.
  • Put policy in the prompt. A policy document can tell an agent what it should do, but that instruction does not guarantee enforcement when it calls a tool.

The alternative is a set of composable controls, each doing a particular job. Shared capabilities matter as much as restrictions: a control should help teams build and operate agents without forcing every experiment to wait for a new approval process. The five controls that follow address visibility, review, access, evaluation and runtime decisions.

5:135:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:13 · section reference included

1. Register agents without creating a queue for every idea

The first problem is proliferation. Large teams run many agents; individuals run notebook experiments, chatbots, web tools and combinations of MCP servers. Across that variety, basic operational questions become hard to answer: who owns an agent, which tools can it use, what is its purpose, and what should it never do?

An agent registry records declared tools and metadata, connects agents to a product catalog and supports human and programmatic discovery.Open full source frame
An agent registry records declared tools and metadata, connects agents to a product catalog and supports human and programmatic discovery.

Missing answers hurt reuse as well as governance. A team cannot choose an existing capability it cannot discover, so it may build another solution to the same problem. Requiring every agent to complete a heavy registration-and-approval process before anything runs creates another failure: a small notebook experiment waits behind high-risk work. People either abandon the experiment or move it onto a less visible, less governed path.

An agent registry makes those facts discoverable without making registration itself an onerous review. It records declared tools, intended endpoints, the product the agent belongs to and its owner. Humans, programs and other agents can use the registry to discover what is available and retrieve the information needed for later decisions. “You can’t govern what you can’t see” also means you cannot reliably reuse it.

Registration happens automatically when a team uses one of the firm’s approved agent-building frameworks; manual registration remains available. That integration is the important operational choice. Visibility becomes a normal byproduct of building an agent through the supported path, rather than extra paperwork that authors have an incentive to avoid.

7:348:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:34 · section reference included

2. Spend review attention where the risk is

Once the agents are visible, their differences become obvious. An agent summarizing Jira or Linear tickets does not pose the same operational risk as an agent deploying changes to a running production system. Sending both through the same review process can leave a trivial summarizer waiting weeks—or, in Feminella’s example, two months—for the same sign-offs as the deployer.

An illustrative summarizer waits through four committee reviews, showing the cost of reviewing every agent identically.Open full source frame
An illustrative summarizer waits through four committee reviews, showing the cost of reviewing every agent identically.

The cost is more than slower experimentation. Reviewers spend a limited pool of attention on low-risk requests while high-risk requests compete for that same attention. A process intended to protect production can become less useful at doing so when everything is treated as equally deserving of scrutiny. Slow queues also recreate the incentive to work around the process.

Risk-based review uses what an agent declares up front—its tools, endpoints and applied controls—to distinguish these cases. Purpose-built enterprise agents make that assessment more tractable: a ticket summarizer has a known job, whereas an open-ended coding agent such as Claude Code does not know what the next user will ask it to do. The proposed control scores agents so human review goes to the appropriate places. Feminella does not specify the scoring formula or thresholds, so this is an architectural approach rather than a reproducible risk rubric.

9:3410:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:34 · section reference included

3. Turn declared purpose into enforced tool access

An agent can act in several capacities: call an internal API, search the web or use permissions attached to a role account. Those capabilities create different exposures. Reading a public webpage and acting through a privileged account should not be treated as interchangeable forms of tool use. The useful access question is how finely permissions must be divided for the agent to do its job without receiving unrelated powers.

A short, universal allowlist avoids making those distinctions, but can leave agents unable to search or call the APIs their work requires without asking someone first. Teams then give up or find a way around the restriction. Purpose-based access instead asks authors to declare the tools their particular agent needs. Feminella’s example is modest: an author may need four tools out of five hundred available, not blanket permission to use the entire collection.

What changes when two agents request the same tool? A research summarizer can use a web tool to retrieve research because that access fits its declared purpose. A release deployer has no corresponding reason to use that tool in the example, so it does not receive access by default. The comparison makes permission depend on the relationship between a tool and a job, rather than treating a tool as universally safe or universally forbidden.

The same tool receives different permissions for a research summarizer and a release deployer, according to their purposes.Open full source frame
The same tool receives different permissions for a research summarizer and a release deployer, according to their purposes.

The managed runtime turns that declaration into a restriction the agent cannot satisfy merely by promising to behave. At initialization, the environment looks up the agent’s applicable policies in the registry. Those policies become enforceable statements about reachable endpoints, hosts and other resources. The registry therefore supplies information to the runtime; the runtime applies the access limits.

Within this model, registration and conformance to the applicable risk score allow teams to operate through the governed path without opening an IT ticket for every access change. That low-friction path is conditional on using the managed environment and fitting the policy. It is how the architecture aims to keep enforcement strong without turning routine development into repeated negotiations with another team.

11:4312:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:43 · section reference included

4. Share evaluation infrastructure, keep agent-specific tests

The next inconsistency appears in evaluation. Some teams build elaborate evaluation stacks; others build nothing. A model is a runtime dependency of the application, and changing it can change both the agent’s outputs and their quality. Without evaluations, a team lacks a way to judge those effects. Feminella treats this as analogous to shipping software without unit tests.

A shared evaluation framework provides common processing, integration and trace ingestion while agent authors retain agent-specific logic.Open full source frame
A shared evaluation framework provides common processing, integration and trace ingestion while agent authors retain agent-specific logic.

Centralizing every evaluation decision would recreate the review bottleneck. If a complete suite and committee sign-off are required for every prompt tweak or one-line change, iteration slows even when the change does not warrant that amount of human attention. The shared element should be the evaluation capability, rather than a queue through which every development decision must pass.

The firm supplies the common parts of running evaluations, avoiding separate teams rebuilding many of the same features. Agent authors still write what is true about their own agents: the task-specific expectations against which behavior should be checked. Shared infrastructure removes duplicated machinery; it does not remove the need to define what a good result means for the particular job.

14:4315:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:43 · section reference included

5. Route consequential decisions to humans

Human-in-the-loop approval becomes a different system at production scale. Parallel agents can generate requests faster than a person can review them. Repeatedly accepting routine requests trains the reviewer to click yes by reflex, so the presence of an approval step stops guaranteeing meaningful scrutiny. The bank example has already shown why counting approvers is not enough.

A classifier routes tool calls to allow, human judgment or blocking according to their consequences.Open full source frame
A classifier routes tool calls to allow, human judgment or blocking according to their consequences.

A separate failure can remove the approval requirement altogether. Feminella recounts an incident in which an AI research-lab director asked an email agent to suggest deletions and wait for permission. In his account, the inbox was large enough to trigger context compaction, which dropped the instruction to wait. The agent then deleted hundreds of emails and did not stop when instructed from a phone. The anecdote illustrates a context-held instruction being lost; it does not establish the reliability of any particular classifier or runtime control.

The proposed alternative separates routine permission decisions from the cases where human judgment matters:

  • Explicit rules: Use deterministic fast paths when an applicable rule already settles the request.
  • Routine calls without a settling rule: Let independent, conservative classifiers make the decision instead of repeatedly asking a person for the same predictable answer.
  • Unclear or potentially dangerous calls: Escalate requests where ambiguity or consequences make human judgment useful.
  • Clearly dangerous calls: Deny them outright rather than treating every prohibited action as another opportunity to click approve.

The independence matters: the agent doing the work is not the only mechanism deciding whether its action should proceed. Classification belongs in the governed runtime, where it can affect execution, rather than existing only as advice elsewhere. These controls address two distinct problems—the volume of routine approvals and the fragility of an approval instruction stored in conversation context.

How does a request reach a human instead of becoming another automatic yes? Feminella explains that an applicable explicit rule takes the fast path. The diagram below shows what happens when no rule settles the request. A classifier allows routine calls, sends requests needing judgment to a human and blocks clearly dangerous calls. The branches preserve human attention for a decision that could change the outcome. The talk supplies this routing logic, but no classifier accuracy results or error rates with which to quantify its protection.

Rules first; classification when rules do not settle the call

Agent action requestA runtime tool call needs a permission decision.
Applicable explicit rule?decision — Use a deterministic fast path where a rule settles the request.
Independent conservative classifierdecision — Assess calls that an explicit rule does not settle.
Permitoutcome — Routine permitted actions proceed without another human approval.
Human reviewoutcome — Review unclear, ambiguous or potentially dangerous requests.
Denyoutcome — Clearly dangerous actions do not proceed.
Arrows show decision routing, not a sequence of approvals. Human review is one branch, alongside permission and denial.
Read the diagram as text
  • Agent action request. A runtime tool call needs a permission decision.
  • Applicable explicit rule?. decision — Use a deterministic fast path where a rule settles the request.
  • Independent conservative classifier. decision — Assess calls that an explicit rule does not settle.
  • Permit. outcome — Routine permitted actions proceed without another human approval.
  • Human review. outcome — Review unclear, ambiguous or potentially dangerous requests.
  • Deny. outcome — Clearly dangerous actions do not proceed.
  • Agent action request → Applicable explicit rule?: Check explicit rules.
  • Applicable explicit rule? → Permit: Rule permits.
  • Applicable explicit rule? → Deny: Rule prohibits.
  • Applicable explicit rule? → Independent conservative classifier: No rule settles the request.
  • Independent conservative classifier → Permit: Routine permitted call.
  • Independent conservative classifier → Human review: Unclear or potentially dangerous.
  • Independent conservative classifier → Deny: Clearly dangerous.
3:133:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:17 · section reference included

Controls that let agents go faster

The five controls do different jobs and connect at specific points. The registry makes agents visible and supplies policy information. Risk scoring directs review attention. Managed environments enforce what agents may reach. Shared evaluations and metrics help teams judge behavior without rebuilding the common infrastructure. Runtime rules and classifiers govern individual calls. This is a set of cooperating capabilities, not five consecutive sign-offs that every action must collect.

Brakes let cars go fast and stop safely—the speaker’s analogy for controls that enable useful agent autonomy.Open full source frame
Brakes let cars go fast and stop safely—the speaker’s analogy for controls that enable useful agent autonomy.

Feminella closes with a question: why do cars have brakes? Stopping is only part of the answer. Brakes let a driver go fast while retaining a way to stop safely. Agent controls should serve the same purpose. The ambition is real autonomy with enforceable limits and useful intervention—not an agent that can do nothing, or a person approving everything until approval no longer means anything.

9:0413:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:47 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…