Memory-Aware Agent Harnesses for Anti-Money Laundering — Nacho Martinez, Oracle

Read the talk

Memory-Aware Agent Harnesses for Anti-Money Laundering

Nacho Martinez builds an AML workshop around a configurable agent loop, selective retrieval and persistent memory, with a human reviewing the transactions the agent flags.

From a talk by Nacho Martinez

At a glance

Ideas worth remembering

  • The AML agent flags simulated transactions for human inspection; its harness controls the data, memory, tools and loop around the model.

  • Selective retrieval keeps a large tool and skill library available without loading it all into every context. The workshop searches separate stores with K = 3.

  • Start memory simply: save useful information outside the current conversation, then add similarity-based recall when the task warrants it.

  • Choose storage, query interfaces, retrieval and loop limits for the workload. A bounded bank-monitoring exercise and an unattended research agent need different patience and context management.

A bank agent that flags, then waits for a human

A customer receives $1 million and then makes $50,000 ATM withdrawals over three days. That pattern is suspicious enough to investigate, but a flag is not a finding of wrongdoing. In Nacho Martinez’s workshop, an agent watches a fictional mid-sized bank’s transactions and identifies cases for a bank employee to inspect. The aim is to reduce discovery work while leaving consequential action to a person. 5:09

The model supplies reasoning, but it cannot by itself retain an unlimited investigation history or connect to the bank’s data. The harness supplies the surrounding machinery: inputs, memory, tools and an agent loop that decides what to bring into context and what to do next. Martinez focuses on these components because an application developer can configure them even when using a third-party model whose weights they cannot change. 6:10

Martinez describes an agentic system as a combination of automation, supplied by workflows, and autonomy, supplied by an agent that can choose actions. A different harness configuration can change how the same model performs on a task: storage, retrieval, available actions and the contents of each request all affect what it can use. That makes the harness a practical place to experiment, although the talk gives no comparative performance measurements for the AML agent.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:39 · section reference included

Choose storage for the work, then decide what to remember

Multiple agents writing the same file create a coordination problem: which write wins, and what happens to a reader that saw an earlier version? An audience member suggests worktrees, which give agents separate working copies. Martinez contrasts that approach with database transactions and their atomicity, consistency, isolation and durability properties. Transactions can coordinate supported database operations; separate worktrees still need their changes reconciled. Files remain useful for unstructured material, so the choice depends on the workload. 13:39

He refers to an on-screen example of multiple agents writing to the same file.Open full source frame
He refers to an on-screen example of multiple agents writing to the same file.

Memory is the machinery for retaining, refining and recalling information when it becomes useful. Martinez treats the current context window as short-term memory and proposes three kinds of longer-term memory:

  • Episodic: previous conversations, summaries and tool logs—the record of what happened.
  • Procedural: skills and workflows—how to perform recurring work.
  • Semantic: facts and preferences, such as a user’s favorite framework.

The working context has a finite capacity, and long contexts can make relevant information harder to use. Martinez makes a stronger claim that shorter contexts always perform better; the recording does not establish that for every model and task. His practical design question is narrower: how can an agent keep useful history available without putting all of it in every request?

He distinguishes two approaches. Summarization replaces a conversation with a shorter account that can seed the next one. His example of compaction stores conversation chunks in a file or database with identifiers, so the agent can retrieve a needed chunk later, including by offset. A summary saves space by condensing details; stored chunks retain a route back to details that the agent chooses to fetch. 18:57

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:09 · section reference included

Load the tools and skills a task needs

The agent loop assembles context from sources such as previous conversation, files, databases, tools and skills, then calls a model to decide what to do. It can repeat, but the workshop caps it at eight iterations and up to 60 seconds of reasoning per iteration. Martinez calls this a limit on the agent’s “patience”: it bounds resource use, and it is configurable. 19:27

A large skill library creates its own context problem. Martinez’s example has around 100 skills, each perhaps 3,000–5,000 tokens; loading them all would crowd out the actual conversation. His proposed toolbox and skillbox patterns embed the available tools and skills, search for those similar to the current task, load relevant candidates, and offload them after use. Retrieval saves space, but the choice of what to retrieve becomes part of the agent’s behavior. 20:57

Skill distillation adds reusable procedures to that library. For example, a successful front-end session might yield a skill.md recording a preferred coloring style. Once embedded and stored, that skill can be retrieved on a later front-end task that refers to the style. The model’s weights have not changed; the later request can benefit from a procedure saved outside them. Retrieval is still a selection step, not a guarantee that the right skill will be chosen.

Martinez places this alongside two other approaches to continual learning: parametric changes modify model weights, for example through fine-tuning or low-rank adaptation; representational work changes how information is embedded; nonparametric memory supplies external information to the model. The workshop concentrates on the last category, while using embeddings to find that information.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:27 · section reference included

Follow the simulated investigation

Instructions and retrieved material also arrive in different parts of a model request. Martinez describes an instruction hierarchy in which system instructions carry more weight than a user request, while tools and skills can be supplied as task-specific material. How those roles are represented and prioritized depends on the model interface; the harness must decide what to place where. 25:59

What does the investigator see? The workshop application generates transactions from fake data and adds suspicious ones to a list. It also offers views of the agent’s context and database customers, plus a visualization of the 30 most recent bank events. The agent uses available tool calls and searches vector-backed knowledge as it examines evidence and works through suspicious activity reports. These views connect the flag to the surrounding investigation; they do not establish how accurately the agent detects suspicious activity. 30:29

The intended deployment runs in the background, connected to the database, and produces decisions for a human to inspect rather than automatically blocking an account. The exercise itself lives in student.ipynb; the workshop registration page accepts a GitHub handle for access to the repository and Codespaces exercise.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:59 · section reference included

Control the context budget at retrieval time

In the Q&A, an audience member asks whether episodic memory can confuse the agent with material from previous conversations. Martinez’s answer is architectural: decide which memory sources matter, how to retrieve them and how much weight to give them. He suggests embedding previous episodes when similarity-based recall is useful. Separating memory components makes their roles clearer, though separation alone does not prove that retrieved memories cannot conflict. 33:17

How does the workshop prevent a large toolbox from filling the request? Its loop searches one vector store for tools and another for skills, with K = 3 for each. It assembles context around the model’s assessment of the task and dispatches tools as needed. Martinez proposes a reranker as an additional refinement: it could order the retrieved candidates and allow a task that needs only one skill to use fewer than three. The workshop does not use that reranker. 38:53

The budget includes more than the user’s message: system and developer instructions, tool definitions, tool outputs and retrieved documents all consume space. Martinez also discusses caching repeated request content as a cost consideration. His percentages and cost savings are illustrative claims, not general measurements; actual request composition, caching behavior and prices depend on the service and configuration.

Two searches feed context assembly

Current taskThe model determines what the request needs.
Tool storeretrieval — Similarity search retrieves up to three relevant tools.
Skill storeretrieval — Similarity search retrieves up to three relevant skills.
Context assemblyThe loop builds the working context and dispatches tools.
The workshop retrieves up to three candidates from each store. A reranker is a proposed addition, not part of the demonstrated setup.
Read the diagram as text
  • Current task. The model determines what the request needs.
  • Tool store. retrieval — Similarity search retrieves up to three relevant tools.
  • Skill store. retrieval — Similarity search retrieves up to three relevant skills.
  • Context assembly. The loop builds the working context and dispatches tools.
  • Current task → Tool store: Find relevant tools.
  • Current task → Skill store: Find relevant skills.
  • Tool store → Context assembly: Supply candidates.
  • Skill store → Context assembly: Supply candidates.
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

33:17 · section reference included

Start with two memory stores and a recall decision

Martinez recommends beginning with just short-term and long-term memory. When the agent identifies something worth keeping from its working context, save it as text in long-term storage. The text can later be embedded for retrieval; the initial version does not need a large taxonomy of memory components. 40:52

One possible extension inserts recall between observation and reasoning. A query searches past episodes; if a result is sufficiently relevant under the chosen similarity criterion, the loop includes it in context. Otherwise it discards the result. That threshold gives the harness a way to avoid treating every vaguely related memory as useful, although it also means a relevant episode can be missed.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

40:52 · section reference included

Build the harness around the problem’s constraints

A custom harness need not reproduce an entire general-purpose coding agent. Martinez imagines a consultant building a focused financial agent, then adapting the surrounding machinery for another customer with different data and constraints. Starting small makes it possible to remove components that the specific job does not need. 47:57

The stopping rule illustrates the choice. The AML workshop limits the loop to eight iterations to control resource use. A long-running mathematical research agent with no human intervention may instead need automatic summarization or compaction and much greater patience. Treating the model connection as a replaceable part of the harness keeps attention on those task-specific decisions.

Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

47:57 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…