Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j

Loading slides…

Read the talk

Turning Agent Memory Into Skills That Work

Selected presentation frame from Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j at 128 secondsOpen full source frame
Lyon presents agent memory as a connected graph with short-term, long-term, and reasoning memory.

Will Lyon explains how Neo4j connects conversations, entities and decision traces, then distills that memory into reusable procedures whose supporting evidence can be checked as it changes.

From a talk by William Lyon

At a glance

Ideas worth remembering

  • Entity resolution and a shared ontology turn retrieved mentions into connected domain knowledge with canonical identities and explicit types.

  • Decision traces preserve evidence, policies, tool calls, results and execution costs so experience can be shared across agents.

  • Skill distillation needs grounding, coverage and coherence: supporting some claims is insufficient if other steps lack evidence or the input combines unrelated topics.

  • Connections from skill components to their supporting memory let curation detect when changed, removed or contradictory data makes a procedure stale.

  • The patient-intake demonstration turns an earlier conversation into steps through charting, grounded in tool calls and entities and packaged with SKILL.md and references.

A successful run can still leave the next agent starting over

An agent reasons, acts and successfully completes a task. Then it forgets what it learned. Will Lyon, a product manager at Neo4j, opens with this “amnesic loop”: finishing an attempt does not necessarily make the next attempt better. A memory system may preserve the conversation without preserving a usable account of the people, actions and decisions involved.

Embedding-based memory addresses part of that problem. It stores text, finds similar chunks when a new request arrives, and inserts those chunks into the context window. The agent must then work out what they mean and how to use them. Similarity retrieves relevant wording; it does not itself establish that several descriptions refer to the same thing.

The opening example is a provider described in several ways: by name, by an abbreviated name and as a cardiologist. Retrieving those mentions leaves an identity question for the agent to solve. A canonical representation gives the mentions a common referent. That is the first requirement behind Lyon’s formulation that agent knowledge should be “connected, typed, and traceable.” Retrieval remains useful, but the memory must also describe what its contents refer to and how they fit together.

0:120:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:12 · section reference included

Turn mentions into entities with a shared domain model

Neo4j’s context graph connects three kinds of memory: short-term, long-term and reasoning memory. The first transformation starts with user and assistant messages. Entity extraction identifies the things discussed and their relationships; the graph records those relationships with explicit types. The result carries more structure than a collection of text passages.

Selected presentation frame from Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j at 160 secondsOpen full source frame
The slide shows messages passing through extraction into a typed knowledge graph.

Two distinct mechanisms make that structure useful:

  • Entity resolution: Different mentions must resolve to a canonical entity. For the provider example, the graph can associate the person with a name and role rather than leaving each mention as an unrelated fragment.
  • Shared ontology: A domain model describes the kinds of entities and relationships the system recognizes. In the later healthcare demonstration, that vocabulary includes providers and encounters. It gives extraction a common structure to work toward.

The distinction matters: identifying a provider-shaped mention and deciding which provider it denotes are separate jobs. The ontology describes the kind of thing; resolution establishes its identity. Lyon places particular weight on resolution because the connected graph depends on getting that common identity right.

2:122:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:12 · section reference included

Preserve the decisions and tool results behind the outcome

Messages and extracted entities describe what the conversation contained. Reasoning memory adds the actions taken during the run. Its organizing unit is a decision trace: a decision linked to supporting evidence, explicitly modeled policies, the execution plan, tool calls and their results. Token use and elapsed time belong in that record too.

This makes a completed run useful beyond its final answer. A later agent can encounter both the domain entities and the recorded route through the tools. Lyon’s intended setting includes hundreds or thousands of agents with overlapping tool access. Persisting their traces in one organizational context graph gives them a shared record from which to learn, rather than confining each run’s experience to one agent.

What information must remain connected for another agent to understand a decision? The diagram separates conversational memory from execution memory, then joins them in the shared graph. That connection preserves both what was discussed and what was done with it.

Conversations and execution traces meet in shared memory

User and assistant messagesconversation — The conversational record supplies mentions and relationships.
Typed, resolved entitiesconversation — Extraction, canonical identity and a shared ontology organize domain knowledge.
Decision tracesexecution — Decisions connect to evidence, policies, tool calls, results, token use and timing.
Shared context graphShort-term, long-term and reasoning memory remain connected.
Other organizational agentsAgents can use the persisted experience of earlier runs.
The context graph connects domain knowledge with evidence about agent actions, making both available across agents.
Read the diagram as text
  • User and assistant messages. conversation — The conversational record supplies mentions and relationships.
  • Typed, resolved entities. conversation — Extraction, canonical identity and a shared ontology organize domain knowledge.
  • Decision traces. execution — Decisions connect to evidence, policies, tool calls, results, token use and timing.
  • Shared context graph. Short-term, long-term and reasoning memory remain connected.
  • Other organizational agents. Agents can use the persisted experience of earlier runs.
  • User and assistant messages → Typed, resolved entities: Extract and resolve.
  • Typed, resolved entities → Shared context graph: Store domain knowledge.
  • Decision traces → Shared context graph: Persist agent runs.
  • Shared context graph → Other organizational agents: Share memory.
4:094:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:09 · section reference included

Give reusable skills explicit execution steps

Memory records what happened. A skill packages guidance for what an agent should do next. The skill format described here starts with metadata explaining its purpose, then uses progressive disclosure: more detailed Markdown and reference material enter context when the agent needs that part of the procedure. Packaging makes the guidance reusable across agents without loading every reference upfront.

Prose still leaves interpretation work. A procedure can refer ambiguously to an entity, describe a step loosely or make its output difficult to debug. Neo4j’s research addresses these problems by extending the skill representation with metadata that treats the procedure as a typed execution graph. Steps become nodes, and a strict schema governs their descriptions and how they are actionable.

Selected presentation frame from Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j at 544 secondsOpen full source frame
The slide represents a skill as linked execution steps, including validation and recommendation.

The graph representation makes the procedure’s parts explicit while retaining the skill as a shareable package. It also creates a more structured object to inspect: the system can reason about individual steps instead of treating the entire procedure as one block of prose. Lyon reports an increase in successfully completed SkillsBench tasks when the research protocol was applied to human-curated skills. The recording gives no numerical result or evaluation details, so this supports a reported benefit in that comparison rather than a quantified expectation for other workloads.

6:086:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:08 · section reference included

Check the evidence across the skill—and keep checking it

Skill distillation starts from a context graph scoped to a workspace or project and turns observed experience into procedural guidance. The goal is to retain well-understood steps and their supporting data. That support also gives the system something to monitor: when the memory changes, the procedure derived from it may need to change.

Three checks ask different questions about a candidate skill:

  • Grounding: Does the skill draw on data actually present in memory?
  • Coverage: Are the steps and descriptions supported throughout the skill, rather than only in one part?
  • Coherence: Does the selected subgraph describe a sufficiently unified topic, or has it combined material that should become multiple skills?
Selected presentation frame from Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j at 672 secondsOpen full source frame
The slide groups grounding, coverage, and coherence as checks for a skill.

These checks constrain what gets packaged. Evidence for some statements does not establish support for every step, and a well-supported collection can still mix several topics. Governance continues after generation: a dynamically loaded, governed skills registry can provide an up-to-date skill and expose whether it has become stale or drifted. The useful unit is therefore a procedure with maintained connections to its evidence.

9:5910:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:59 · section reference included

Choose a scope before synthesizing the procedure

The Neo4j Agent Memory Service, abbreviated NAMS, implements this pattern as a Neo4j Labs service. A workspace-scoped graph holds the three memory types. Background workers distill skills, while background curation surfaces skills whose supporting memory has changed.

The first consequential choice is scope. A skill can start from the subgraph around an entity or from a specific conversation. That selection determines which experience the system will attempt to turn into a procedure. Lyon describes most pipeline steps as deterministic; the LLM synthesizes some claims and generates text for the skill description. Deterministic processing surrounds the language-generation work, rather than making the entire process deterministic.

Selected presentation frame from Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j at 800 secondsOpen full source frame
The pipeline diagram begins with scope selection and continues through skill distillation stages.

Each component retains support in the underlying graph. Curation can then watch for three consequential changes: supporting data changes, disappears or becomes contradictory. Coherence uses graph algorithms such as community detection. If the selected material spans several topic communities, that can indicate that the proposed skill should be decomposed. It is a signal for deciding how to divide the material, rather than a claim that every cross-community procedure is invalid.

How does a change in memory reach an already generated skill? The flow below makes the evidence connection visible. Distillation produces the procedure, but curation follows the supporting data afterward; packaging does not end the skill’s relationship with memory.

Distill from evidence, then monitor that evidence

Selected memory scopeStart from an entity subgraph or a conversation.
DistillationMostly deterministic processing, with an LLM synthesizing claims and descriptive text.
Grounded skill graphIndividual components remain supported by underlying memory data.
Skill packageSKILL.md and references support progressive disclosure.
Background curationMonitor changed, removed or contradictory support and surface stale skills.
The skill graph retains support for its components, allowing later memory changes to trigger staleness signals.
Read the diagram as text
  • Selected memory scope. Start from an entity subgraph or a conversation.
  • Distillation. Mostly deterministic processing, with an LLM synthesizing claims and descriptive text.
  • Grounded skill graph. Individual components remain supported by underlying memory data.
  • Skill package. SKILL.md and references support progressive disclosure.
  • Background curation. Monitor changed, removed or contradictory support and surface stale skills.
  • Selected memory scope → Distillation: Define the input.
  • Distillation → Grounded skill graph: Construct the procedure.
  • Grounded skill graph → Skill package: Package for reuse.
  • Selected memory scope → Background curation: Observe supporting-data changes.
  • Grounded skill graph → Background curation: Identify affected components.
12:1512:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:15 · section reference included

From a healthcare conversation to a patient-intake skill

The demonstration starts with memory already ingested. NAMS exposes REST APIs and MCP tools for ingesting, retrieving and working with the three memory types. Its dashboard provides a graph view that can be traversed, a view of flagged entities and the domain ontology. Here, the healthcare ontology organizes data about encounters and providers, and the workspace contains healthcare-agent conversations.

The next action narrows that collection to a skill-sized input. The interface offers an entire workspace, an entity, specific conversations or a class in the ontology as possible scopes. Lyon selects the most recent conversation. That choice queues a distillation job, which fetches the relevant data, runs a seven-stage pipeline, constructs the skill graph and packages it.

While that job runs, the demonstration opens an earlier result derived from a patient-intake conversation. The observable change is from a record of one interaction to a procedure covering patient intake through charting. The extracted steps remain grounded in the actual tool calls and entities that supplied their components. The earlier conversation therefore contributes both the procedural sequence and inspectable support for it. This is a previously generated result; the demonstration does not establish completion of the newly queued job or show the resulting skill executing a fresh intake.

Selected presentation frame from Turning Agent Memory Into Skills That Work — Will Lyon, Neo4j at 1024 secondsOpen full source frame
The NAMS interface shows a patient-intake skill beside its conversation detail.

The output includes SKILL.md; a downloaded package would also contain references following the progressive-disclosure pattern. The graph representation and the familiar skill package serve different parts of the same workflow: structured steps retain their evidence, while the package carries guidance that another agent can load as needed. The talk ends by pointing toward slides, service documentation and open-source tooling for implementing these patterns.

14:4615:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:46 · section reference included

Resources

Transcript download
Full transcript

Loading transcript…