Your Financial Agent Needs to Show Its Work - Anirudh Veeraragavan, Fieldguide
Loading slides…
Read the talk
Why Was This Evidence Sufficient? Building a Financial Agent That Can Show Its Work
Fieldguide’s Anirudh Veeraragavan follows a bank reconciliation from draft findings to inspection, showing what an agent must preserve so another auditor can examine a decision years later.
From a talk by Anirudh Veeraragavan
At a glance
Ideas worth remembering
Check the row’s review state when the agent writes. Route new evidence about a signed finding into a proposed revision.
Record each step’s ID, inputs and outputs, and bind cited passages to a document ID, version, content hash and source.
Sample apparent passes when auditing an eval. Agreement with an obsolete reference answer can conceal a real failure.
Give reviewers the discrepancy they need to investigate, such as an eleven-business-day clearing interval, when asking them to decide.
Build the reconciliation around the opinion a reviewer will sign
In audit, the opinion is the product, says Anirudh Veeraragavan, lead security engineer at Fieldguide. The value of that opinion depends on the basis for it. Consider a bank reconciliation. A client reports a cash balance on its books, and the bank reports a balance for the same client. Those numbers can legitimately differ. A deposit in transit can appear on the books before it reaches the bank. An outstanding check can reduce the balance on the books before it clears. The auditor tests whether those explanations account for the difference.
Veeraragavan sketches a reasonable first version of an agent for this high-volume work. Give it the client’s books, the list of reconciling items and the supporting evidence. Supply the audit methodology that tells it how to match the items. Prior-year audits can serve as worked examples, and previously reviewed audits can supply expected answers for an evaluation. The agent drafts its findings and presents them to a human reviewer for sign-off. In the scenario he asks the audience to imagine, the metrics look good, users are happy and the team ships the agent to general availability.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An inspection narrows the question to one piece of evidence
Then an audit is selected for inspection. Someone picks a particular test procedure, looks at one piece of evidence and asks, “Why was this sufficient?” The evaluation score describes performance across cases. The inspector wants the basis for the conclusion in this case. Even a satisfied reviewer who signed the finding leaves that question open for the next person who has to examine the work.
Veeraragavan cites Peregrine Financial Group, an Iowa brokerage, to put the stakes in perspective. For more than twenty years, the firm committed bank account fraud. On paper it claimed more than $200 million. At the bank there was only $5 million. His concern here is the distance between a reported financial position and the cash that someone could actually substantiate.
Audit practice gives him an acceptance test for the agent. An experienced auditor coming to the engagement fresh should be able to understand the tests performed, the evidence collected and the decision reached. To see what that requires, he introduces Calder Freight Lines, a fictional client closing its books for 2026. Its bank shows about $2.9 million, its books show about $2.7 million, and its reconciliation lists 41 items to explain the gap. He follows four of those items, two checks and two deposits.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Authorization depends on whether this row has been reviewed
A reviewer flags one of the checks as a possible exception and signs the row. A few days later, additional information explains the discrepancy and triggers another agent run. The agent has permission to edit the finding. It reads the new evidence correctly and changes the row to “no exception.” An inspector opening the row now sees a cleared check bearing the signature of the person who flagged it.
The error occurred at the write. The new finding is sound, but the reviewer’s signature now appears to endorse a conclusion they never saw. Checking whether the agent may call the editing tool gives too coarse an answer. Authorization has to take account of who is making the call, which object it targets and the state of that object at the moment of the write.
For an unreviewed row, let the agent draft a finding under its own identity. For a signed row, give its new conclusion the status of a suggestion. That lets the reviewer consider the additional evidence while preserving what the earlier sign-off covered. The arrival of better evidence is a good reason to reopen a question. It gives the agent grounds to propose a change, while the decision about revising signed work stays with a person.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Execution control makes a disagreement between runs inspectable
Suppose the procedure runs in 2027 and returns no exception. A year or two later, during a review, someone reruns it and gets a different finding. Both runs have logs. Each says the agent fetched evidence, processed it and returned an answer. Read side by side, those logs describe the same sequence. They reveal little about where the second run departed from the first. Running the entire procedure again would give the auditor a third result to explain.
Veeraragavan would record the run as a chain of steps. Each step has an ID, inputs and outputs. The record distinguishes operations performed by deterministic code, such as fetching an item or applying a matching rule, from operations entrusted to the model, such as extracting a field from a PDF or arriving at the final conclusion. An auditor can then ask a narrower question at each boundary. Did the runs fetch the same item? What value did each extraction return? What reached the matching step?
Once the engineer locates a difference, they can replay that individual operation with its recorded inputs and compare its output with the earlier result. In Veeraragavan’s example, the original run was correct and the newer one was wrong. That direction matters. The inspection raised a legitimate question about signed work, and the record gives the reviewer a way to defend the original finding by examining how the agent reached it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evidence provenance ties page two to a particular bank statement
Another finding clears an item against the wrong account. Its citation points to page two of a January bank statement. The document can be real and the cited material can be on that page. A large engagement may contain several files called January bank statement. Even if the filename is unique, an auditor still needs to determine which version the model read and who supplied it. Checking that a file exists would miss the account error.
Give the citation a specific document ID, a version and a hash of the contents. Keep the page reference so the reviewer can inspect the passage in that exact document and check which account it describes. Record where the document came from as well. Identifying the contents and establishing their source answer two different audit questions.
Provenance also gives the agent a basis for enforcing evidence requirements. A procedure may require a statement obtained directly from the bank. A client-uploaded copy could have been altered. With a source record, the system can flag that copy for review even when the model finds the expected amount on the expected page.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluation integrity requires reopening some of the passes
The first version of the agent treated last year’s reviewed audits as gold data. Veeraragavan gives three reasons to inspect those reference answers. A later decision may have superseded the finding selected for the dataset. A human may have relied on out-of-band evidence that the agent never received. Or the audit methodology may have changed, making a historically sound finding the wrong expected answer for the current procedure. Each calls for a different inquiry into the decision, the available evidence or the applicable rule.
His illustrative evaluation reports 97%. Start with the nine apparent misses. In four of them, the evaluation itself had the wrong answer and penalized the agent. A team that investigated only disagreements could reasonably come away thinking its agent performed better than the headline score suggested.
Now inspect the agreements. In twelve cases, the evaluation expected no exception and the agent received credit for clearing the item. Changes to business rules or audit methodology made those clearances wrong. These failures would never enter a queue populated only by failed tests. The 97% combines credit and penalties assigned for different reasons. To interpret it, the team has to examine what justified the labels on both sides of the pass line.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Escalate the deposit that took eleven business days to clear
Several events already encountered in the reconciliation give the agent a concrete reason to stop. It needs to revise signed work. A fresh run disagrees with the earlier finding. A document fails the procedure’s evidence requirements. In each case the handoff should identify the point that needs judgment. For the deposit Veeraragavan singles out, the relevant fact is that it took eleven business days to clear. Calling attention to that timing discrepancy tells the reviewer what to investigate before deciding whether to accept the explanation.
Veeraragavan wants escalations to be easy to verify, even when approval takes time. Frequent prompts encourage automation bias as reviewers get used to clicking yes. He uses an illustrative 80/20 split: if the agent can handle 80% of the work, spend the reviewer’s attention on the 20% that warrants intervention. The time a person spends investigating an exceptional deposit is part of the design.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The same four controls apply to a loan decision or an AML alert
Each control covers a failure the others leave open. Authorization preserves the relationship between a signature and the finding reviewed. Execution control lets someone reconstruct a correct result and investigate a conflicting one. Evidence provenance lets that person trace a mistaken clearance back to the account and document the agent used. Evaluation integrity determines what the team can infer from its 97% score. A reconciliation can pass any three checks and still fail the fourth.
In credit, the decision to examine might be an approval or a decline. In anti-money-laundering work, it might be the closure or escalation of an alert. The object under review changes, but an experienced person arriving later still needs to recover the evidence, follow the operations that produced the finding and see who had authority to act. That is what lets the institution stand behind a consequential decision after the people who made it have moved on.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full transcript
Loading transcript…