Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait — Al Chen, Cisco
Read the talk
Shipping AI Into 5 Major Banks Without a 12-Month Compliance Wait
Al Chen explains how his team turns vulnerability remediation, disaster recovery and sensitive-data handling into work that bank security teams can inspect.
From a talk by Al Chen
At a glance
Ideas worth remembering
CVE remediation needs an account of image versions, bank policies and upstream dependencies; a package-version change can be supplemented with an exploit test.
Backup and restore logs show job status; row counts and replica convergence help assess the data itself. Cross-region recovery also requires a decision about where records may reside.
A judge’s reasoning can expose sensitive data even when the UI is redacted. Separating scores from explanations and using a second judge on redacted input adds protection at the cost of another model call.
Deployment speed depends on the bank’s readiness for image approvals, GPUs, security review and access configuration as well as the vendor’s engineering.
Production requires showing the work
A bank-facing AI application can handle requests from millions of customers, making production a different proposition from a prototype or an internal productivity agent. Al Chen of Cisco describes deploying AI observability software into that environment, where a conventional SaaS connection with outbound internet access can create a route for sensitive information to leave the bank. 1:14
Chen compares compliance review to his tenth-grade math tests: the answer mattered, but most credit came from the proof. His team aims to shorten a process that can take a year to a couple of weeks by making three areas inspectable: how images reach a bank’s cluster, how data survives a regional failure, and how sensitive information stays away from unauthorized employees and outside the bank’s virtual private cloud (VPC). The timing depends on the bank’s own readiness, as the later examples make clear.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make CVE remediation work across banks
A single common vulnerability and exposure (CVE) can be researched in a vulnerability database. The operational problem is the multiplication: a platform has several images, each image can contain several CVEs, and banks run different image versions. Their scanners, severity policies and remediation deadlines can also differ. 2:42
Some vulnerabilities sit in transitive dependencies of open-source projects. Chen’s team follows upstream release notes and explains to bank security teams when a fix depends on a maintainer. For issues it can address, it may upgrade a package or rebuild against a base Docker image. Even then, one bank may require package version 3.8 while another prefers an older version because it sees too many CVEs in the newer one; the resulting software still has to function.
The team uses an agent skill to help research and report these cases. Its table tells engineers which CVEs to fix and gives bank security teams explanations for issues blocked by dependencies. The useful distinction is between work the team can undertake and risk it must account for while waiting on an upstream project.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test the exploit, not just the package version
Chen describes scanners such as Sysdig, Wiz and Prisma checking whether a package has reached a specified version. That check alone does not demonstrate whether a particular exploit still works against the application. His proposed extra evidence is a Harbor task: a harness containing agent instructions, the application’s Docker images and tests. 5:23
In the example, version 2.6.2 of a package is supposed to remediate a CVE. The agent is asked to write an exploit based on the older version, try it against the newer one, and write a test to check whether the exploit succeeds. Harbor can run trials in parallel using containers or sandboxes, giving the team repeated attempts to inspect. A failed attempt is evidence about that tested exploit, not proof that every possible exploit is absent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A successful restore job still needs data checks
For disaster recovery, Chen describes a ClickHouse deployment where the bank must arrange cross-region recovery itself. His pattern backs up a primary-region database to a bucket, then runs a restore job into a disaster-recovery region, which may contain several replica pods. The procedure uses clickhouse-backup, an open-source project maintained by Altinity. Chen’s assessment of managed cross-region failover is specific to the situation he describes in the talk. 7:22
The backup tool’s logs can show that the jobs ran, but that status does not establish what the data looks like now. A bad disk or node, a migration problem, or replication lag can leave data stale. Chen adds checks for table row counts and whether replicas within a region have converged. Those checks ask about the restored data rather than only the job’s completion.
Recovery design also makes a data-residency decision. A bucket can be placed in Oregon or Virginia, but the location of individual records can matter under jurisdictional rules. One option Chen describes is to replicate trace IDs and metadata across regions while keeping the core trace data in its permitted region. That helps preserve regional restrictions, while limiting what the remote region actually holds.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A judge can put sensitive information back into its explanation
Redaction at an SDK, gateway or middleware can make the customer-facing UI look clean. An LLM-as-a-judge creates another path: it scores an agent’s work and produces an explanation for a human reviewer. Chen points to a seemingly innocuous instruction intended to make the explanation verifiable; if the judge sees raw traces, its reasoning may reproduce personally identifiable information (PII) or material nonpublic information (MNPI). He recommends inspecting judge prompts and traces for that leakage. 10:07
The proposed flow separates the score from the reasoning. The score persists for evaluation. Application code checks whether a record contains sensitive information: reasoning can persist when the check finds none, and is suppressed when it finds sensitive information. The diagram shows those branches and the alternative for cases where a bank still wants an explanation.
That alternative runs a second judge on input with sensitive information redacted and displays its explanation. It costs another LLM call on the same request. Chen presents redacted input as a reason for confidence in the second output; the protection depends on the redaction actually covering the sensitive information.
Separate the score from explanations
Read the diagram as text
- Judge on raw trace. raw — Produces a score and reasoning.
- Score persists. raw — Available in the UI for evaluation.
- Sensitivity check. raw — Application code checks the record for sensitive information.
- Show reasoning. raw — Persist reasoning when the check finds no sensitive information.
- Suppress reasoning. raw — Do not show the raw-data judge’s reasoning when the check finds sensitive information.
- Redacted input. alternative — Sensitive information is removed before the second judge sees it.
- Second judge. alternative — Generates an explanation for display using the redacted input.
- Judge on raw trace → Score persists: Score.
- Judge on raw trace → Sensitivity check: Reasoning.
- Sensitivity check → Show reasoning: No sensitive information found.
- Sensitivity check → Suppress reasoning: Sensitive information found.
- Redacted input → Second judge: Input.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The deployment clock depends on the bank
A vendor’s promise of deployment in a few weeks means little until it is checked against the bank’s infrastructure, security posture and cluster. Chen looks for practical signals: is there already a process for pulling images from a registry, and are GPUs sourced and ready for inference? 13:06
The timeline can extend to months when formal security reviews are needed or the bank is building a custom LLM gateway or role-based access control configuration. It can stretch further if image-pull approval and GPU procurement have no established process. Chen finds that many of these delays come from organizational alignment as much as technology.
His team treats a customer’s problem as its own, dividing the work among engineering, workflow, and human and organizational streams, then automating where it can. The three pillars in the talk sit within a larger production checklist. The stakes explain the methodical pace: checking accounts, mortgages and credit cards leave little room for an answer that merely looks plausible.
Chen closes with the other half of his school-test analogy. After a test, he and his friends reconstructed problems at the lunch table and compared how they solved them. He offers the conference the same role: a place to exchange approaches to difficult customer problems, including the slow, messy organizational work that makes production possible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
A way to follow up on Chen’s invitation to discuss deployment pillars and his team’s production checklist.
Related talks
- You Can't Prompt the Room: The Last Skill AI Won't Replace
Explores stakeholder alignment and requirements work that can determine whether AI projects reach adoption.
- Why We Deleted Our MCP Server and Rebuilt It — Abhi Arya, Reducto
Examines how people can inspect agent workflows and detect consequential mistakes.
Full transcript
Loading transcript…