# Citation Needed: Provenance for LLM-Built Knowledge Graphs

Daniel Chalef, Zep AI

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/H7puB0RwJMM-citation-needed-provenance-llm-built-knowledge) · [Video](https://www.youtube.com/watch?v=H7puB0RwJMM)

## Citation Needed: Provenance for LLM-Built Knowledge Graphs

A fact in agent memory is only useful if its origins survive synthesis, entity merges, contradictions and deletion. Graphiti models that changing evidence as part of the graph.

Familiarity with LLM extraction and basic graph concepts—nodes, edges and traversal—is helpful; no Graphiti experience is required.

### Where did this fact come from?

[0:20](https://www.youtube.com/watch?v=H7puB0RwJMM&t=20s)

When an LLM produces a fact that never appeared verbatim in its inputs, how can you trace where it came from? A summary, extracted fact or structured record can combine several sources through nondeterministic interpretation. That synthesis can erase the paper trail. **Provenance** preserves how an artifact was built and why—information needed for debugging, evaluating source trust, deciding what to delete and supporting legal compliance.

Daniel Chalef’s team built [Graphiti](https://github.com/getzep/graphiti), an open-source temporal graph framework underlying Zep’s enterprise agent memory infrastructure. Its inputs extend beyond chat: voice transcripts, email and business data all contribute context. Once these sources become agent memory, two questions recur: where did a fact come from, and how trustworthy is its evidence?

Consider the talk’s stylized healthcare example. An agent retrieves a confident statement: the patient has a penicillin allergy. Its context was synthesized from a lengthy electronic health record, a PDF lab report and an AI intake chat. If the allergy information came from the patient’s own chat message, presenting it to a treating doctor without that origin can mislead. The presence of clinical documents elsewhere in the input does not establish that they supplied this particular fact. The retrieval needs to expose both the exact source and the basis for trusting it.

![Slide titled “The retrieval you can't trust” shows a patient allergy statement, synthesized facts, and a question mark above EHR record, lab report, and intake chat sources.](https://ai.engineer/api/talk-essay-media/5320cfa68f87457ef8f962bb531d94ff58991530075f845ec9c3a33d927390e2)

A penicillin-allergy fact with its source trail missing.

### A source pointer becomes an evolving set

[2:58](https://www.youtube.com/watch?v=H7puB0RwJMM&t=178s)

A `source_id` works well when a warehouse or data-lake pipeline copies or deterministically transforms a value from known inputs. An LLM context pipeline introduces several different changes:

- **Synthesis:** each extracted fact may depend on one or more of the sources supplied in a prompt.
- **Identity resolution:** Jay Smith and John Smith may become one entity whose facts originate in different places.
- **Invalidation:** new information may contradict an old fact, changing what the store knows beneath an existing pointer.

Chalef argues that an append-only change log becomes difficult to manage as these mutations accumulate. The requirement is an evolving set of source relationships that survives the mutations themselves.

### Keeping lineage correct as the graph changes

[4:13](https://www.youtube.com/watch?v=H7puB0RwJMM&t=253s)

Graphiti calls source records **episodes**. In the healthcare example, three episodes supply the input data. The extracted patient and penicillin entities are connected by an edge; together, those two entities and their relationship form a triple that can be rendered as the allergy fact. Relationships back to the episodes make source attribution a graph walk.

Creating those links on the first write is straightforward. Preserving them requires rules for every mutation. When two entities merge, the merged entity must retain the union of their source links; otherwise, identity resolution silently removes evidence. When new information contradicts a fact, the graph must preserve the lineage of that change as well. The described invalidation operation adds a date to the affected edge and records the episodes responsible for invalidating it.

![Slide titled “Keeping lineage correct as the graph mutates” contains three diagram panels: Multi-source facts, Entity resolution, and Temporal invalidation.](https://ai.engineer/api/talk-essay-media/e195058900ae131bde92cfdb6b55bc1e1d460b40a089f825e191d83487b69b98)

Preserving lineage through multiple sources, entity merges, and temporal invalidation.

The talk calls the invalidation date `invalidAt`. Current [Zep fact documentation](https://help.getzep.com/facts) uses `invalid_at` for when the fact ceased to hold in the world and `expired_at` for when the system learned the change. These are different clocks: the event and its later discovery need not coincide. The architectural point is that both derived facts and changes to those facts remain connected to their source data.

### Source metadata does not decide policy

[6:16](https://www.youtube.com/watch?v=H7puB0RwJMM&t=376s)

**Metadata projection** carries source classifications into derived context. At ingestion, an episode originating in an EHR receives an EHR tag. Entities and facts derived from that episode inherit the tag, allowing retrieval to filter for the relevant clinical-source classification while traversing the graph. One tagging operation at ingestion supports later evaluation of the evidence.

Multiple parents make the policy more interesting. Suppose a fact has three parent episodes and one is unverified. Whether that fact is sufficient for an action depends on the consequences of being wrong.

| Context | Dangerous mistake | Policy in the example |
| --- | --- | --- |
| Penicillin allergy | Missing an allergy warning | Any source reporting the allergy should block the prescription |
| Consent on file | Acting on unverified consent | Every parent must be verified |

The allergy policy retains a warning even when its evidence is patient-reported. The consent policy withholds authorization unless all supporting parents satisfy verification. The same graph shape therefore supports opposite decision rules.

**The graph exposes evidence; the agent applies the business rule.** Graphiti can expose which parent episodes have a tag, but it does not determine the appropriate risk policy for every application. This distinction also matters when implementing filters: current [Zep metadata projection documentation](https://help.getzep.com/v3/episode-metadata-projection) says a search result matches when at least one associated episode satisfies the filter. Combining predicates with AND requires those predicates to hold within that episode; it does not establish that every parent is verified. An all-parent consent rule needs that separate evaluation.

### Deleting a source without deleting every derived fact

[8:54](https://www.youtube.com/watch?v=H7puB0RwJMM&t=534s)

Retention policies and right-to-be-forgotten requests introduce another mutation: deleting source data. Lineage identifies the facts affected by that deletion, but a fact can have other surviving support. In the deletion example, the intake chat is one of three episodes supporting the allergy fact. Removing that chat leaves the EHR and lab-report episodes, so the allergy fact survives. A contact-preference fact supported only by the intake chat disappears. **Delete a fact only when no remaining episodes support it.**

Proposed deletion of the intake chat
Removing one source preserves facts with other support and removes facts left without any.
Constructed example: The identifiers ehr, lab_report, intake_chat, penicillin_allergy and contact_preference are teaching labels for the talk's source episodes and facts. The comparison models the proposed operation, not an executed deletion.
Deletion request — unchanged: Delete episode intake_chat.
Operation: Remove intake_chat and its support associations; delete facts whose remaining support set is empty.
- EHR episode
  - Before: Before proposed deletion: Present
  - After: Result under the support-set rule: Present
- Lab-report episode
  - Before: Before proposed deletion: Present
  - After: Result under the support-set rule: Present
- Intake-chat episode
  - Before: Before proposed deletion: Present
  - After: Result under the support-set rule: Not present
- penicillin_allergy
  - Before: Before proposed deletion: Supported by: ehr, lab_report, intake_chat
  - After: Result under the support-set rule: Supported by: ehr, lab_report
- contact_preference
  - Before: Before proposed deletion: Supported by: intake_chat
  - After: Result under the support-set rule: Not present
Removing one source preserves facts with other support and removes facts left without any.

The support-set operation can be expressed directly in Python. This models the proposed deletion of the intake chat and the resulting fact associations:

```python
supports = {
    "penicillin_allergy": {"ehr", "lab_report", "intake_chat"},
    "contact_preference": {"intake_chat"},
}
deleted_episodes = {"intake_chat"}

remaining_supports = {
    fact: parents - deleted_episodes
    for fact, parents in supports.items()
}
retained_facts = {
    fact: parents
    for fact, parents in remaining_supports.items()
    if parents
}
deleted_facts = set(supports) - set(retained_facts)

assert retained_facts == {
    "penicillin_allergy": {"ehr", "lab_report"}
}
assert deleted_facts == {"contact_preference"}
```

This rule concerns support associations. Current [Zep deletion documentation](https://help.getzep.com/deleting-data-from-the-graph) specifies additional limits: shared-node names and summaries are not regenerated and can retain information from a deleted episode. Deleting an episode that invalidated a fact also does not restore that fact. Association-based cleanup therefore does not, by itself, establish complete erasure.

### Build lineage into the data structure

[10:19](https://www.youtube.com/watch?v=H7puB0RwJMM&t=619s)

Because deriving context is lossy and generative, lineage needs to be engineered into the data structure instead of reconstructed from logs afterward. Graphiti retains source content verbatim and links derived artifacts back to it. Those links support source verification, debugging and deletion decisions, as well as compliance workflows. Most of the capabilities described in the prepared talk are available in Graphiti’s repository.

![Key insights slide lists three takeaways about engineered lineage, verbatim sources, and provenance benefits, with github.com/getzep/graphiti below.](https://ai.engineer/api/talk-essay-media/7d807da07c00a25c4be45685c9cde2c0eaa6c2ac44907a17e1f5dea0dd87b8f6)

Engineer lineage into the graph and link derived data back to verbatim sources.

Supporting compliance is not the same as guaranteeing it. Chalef’s [companion article](https://blog.getzep.com/how-zep-tracks-provenance-in-agent-memory/) places request handling, verification and auditing around deletion propagation. The graph provides the dependency information those processes need.

That information has a construction cost. Chalef describes Graphiti-style graph construction as expensive and says the team has invested in reducing the cost and latency of generating graph artifacts. The recording supplies no quantitative cost or latency result. His closing slide offers Zep on the left and Graphiti on the right before the discussion turns to implementation questions.

### Where provenance lives and how agents reach it

[12:34](https://www.youtube.com/watch?v=H7puB0RwJMM&t=754s)

A question about changing relevance weights exposes a boundary in the graph-centric design. Chalef says that capability belongs to Zep, not Graphiti, and that its provenance tracing uses a separate data structure outside the graph. For this particular problem, not all provenance is represented by graph relationships.

Asked about entity and edge creation, Chalef first describes how agents retrieve graph artifacts:

- **Vector similarity search** finds textual artifacts by semantic similarity.
- **Full-text search** finds textual matches.
- **Graph operations**, including breadth-first search, follow relationships.

The available operations depend on the underlying graph database. Whichever route leads an agent to an artifact, its provenance can then be inspected. This explains access to evidence, rather than specifying how entity or edge types are chosen.

### From an episode to a changing fact

[14:29](https://www.youtube.com/watch?v=H7puB0RwJMM&t=869s)

The next question goes underneath episode ingestion. Episodes are themselves graph nodes, and other derived artifacts can also be nodes because they need their own lineage. Adding an episode runs a pipeline whose high-level sequence is:

1. **Extract structure:** identify entities, relationships and candidate facts. A hydrated fact triple has a subject, verb and object.
2. **Deduplicate entities:** reconcile newly extracted entities with identities already in the graph.
3. **Deconflict facts:** compare new facts with existing ones and invalidate those that the new evidence changes.

The pipeline must therefore handle both additions and mutations; extracting triples alone is insufficient.

Chalef illustrates deconfliction with a preference. Initially, Daniel loves Adidas shoes. Three months later, the shoes fall apart, he returns them and includes an angry message. The new evidence yields facts that Daniel returned the shoes and was unhappy with Adidas. It also invalidates the older preference. Retaining both statements without accounting for the change would leave the agent with stale context.

Although much of the pipeline uses LLMs, the team tries to avoid them where conventional techniques suffice. Chalef names information retrieval, traditional NLP, entropy-related techniques and SimHash among approaches used for deduplication. He describes these alternatives as cheaper, faster and more deterministic than LLM processing, without reporting numerical comparisons.

### The provenance burden of file-based memory

[17:41](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1061s)

Files, wikis and Markdown knowledge bases offer another way to represent agent memory. The audience asks whether Zep is pursuing that direction and whether these provenance ideas transfer. Chalef identifies a specific difficulty: after an agent mutates lines in a file, understanding why those changes occurred and which sources justified them becomes hard.

The burden grows in multi-agent, multi-user and multi-source settings. Chalef says Markdown works well for desktop usage and sometimes for server-based agentic applications. His objection concerns the enterprise problems his team encounters, particularly maintaining provenance at scale; it is not a claim that file memory fails in every setting. The answer does not establish a Zep roadmap for file-based memory.

### Extract together, then examine why things changed

[19:19](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1159s)

The final question asks how memory is written back: explicitly or implicitly, through a conversation summary or at every turn, and by which component. Chalef describes the extraction design rather than specifying its trigger cadence. A single-shot LLM extraction produces entities, relationships and facts together.

Chalef attributes inexpensive extraction to that consolidation. Reflection then checks accuracy and enriches the lineage: the goal is to capture why an artifact changed, beyond recording that two artifacts are related. He places the described functionality partly in Graphiti and partly in Zep without assigning each individual step to a product. The resulting memory needs to preserve not only the latest fact, but the evidence and change history that make that fact intelligible.

## Resources

- [Graphiti](https://github.com/getzep/graphiti) — Open-source temporal knowledge graph framework with installation instructions, examples and hybrid retrieval.
- [How Zep tracks provenance in agent memory](https://blog.getzep.com/how-zep-tracks-provenance-in-agent-memory/) — Chalef's companion article explains episode lineage, metadata projection and deletion using the talk's healthcare example.
- [Episode metadata projection](https://help.getzep.com/v3/episode-metadata-projection) — Current Zep documentation explains source associations, effective metadata and per-episode filtering semantics.
- [Deleting data from the graph](https://help.getzep.com/deleting-data-from-the-graph) — Deletion operations and their limits, including retained shared-node summaries and unchanged invalidation status.
- [Adding episodes to Graphiti](https://help.getzep.com/graphiti/core-concepts/adding-episodes) — Examples for ingesting messages, text and JSON, with a warning about bulk ingestion and fact invalidation.
- [Zep: A Temporal Knowledge Graph Architecture for Agent Memory](https://arxiv.org/abs/2501.13956) — The team's original architecture paper introducing Graphiti-backed temporal agent memory.

## Slides

### Citation needed

LLM synthesis can obscure how an output was built. Daniel Chalef introduces provenance as a way to trace its origins for debugging, source trust, and deletion decisions.

[0:18](https://www.youtube.com/watch?v=H7puB0RwJMM&t=18s) [Title slide introducing provenance for LLM-built knowledge graphs.](https://assets.ai.engineer/sites/aiecode2025/sha256/c3f74469d8e7518c1d4768b58b2717b2e145d11a06217b3171fa8f5422230826.webp)

zep

AI ENGINEER WORLD'S FAIR 2026 · GRAPHS TRACK

Citation needed

Provenance for LLM-built knowledge graphs

Daniel Chalef
FOUNDER, ZEP AI · GRAPHITI

### Graphiti

Graphiti builds temporal knowledge graphs using LLM extraction and powers Zep's agent memory. The lineage structure ships in Graphiti; Zep built the scoping patterns on top.

[1:12](https://www.youtube.com/watch?v=H7puB0RwJMM&t=72s) [Overview of Graphiti as an open-source temporal knowledge graph framework, with supported databases.](https://assets.ai.engineer/sites/aiecode2025/sha256/5b62c949aa8645328a31cfb3b79da946ac42e41e111e3afda1194f75faeb46dd.webp)

OPEN SOURCE
Graphiti
getzep/graphiti  ★ 28.2k

Temporal knowledge graphs, built from your data by LLM extraction.

The engine behind Zep's agent memory.

SHIPS IN GRAPHITI TODAY
The lineage structure in this talk ships in Graphiti today.

PATTERN — BUILD ON IT
The scoping patterns are what Zep has built on Graphiti.

RUNS ON
Neo4j
FalkorDB
Amazon Neptune
König
Zep's graph DB

zep
Citation Needed · AIE World's Fair 2026

### The retrieval you can't trust

A synthesized allergy fact reaches an agent without showing which of three sources produced it. The speaker warns that hiding a patient-reported source could mislead a doctor.

[1:53](https://www.youtube.com/watch?v=H7puB0RwJMM&t=113s) [A synthesized patient fact is shown without a clear link to the records that produced it.](https://assets.ai.engineer/sites/aiecode2025/sha256/2daf2ef898abd2cb13ad986bd2a4e532bfef4fb66ecdf1f814c3f54c95cd2627.webp)

THE PROBLEM
The retrieval you can't trust

agent · tool result
"Patient has a penicillin allergy."

Synthesis is everywhere: summaries, extracted facts, agent memory. The output doesn't appear verbatim in any source.

Which sources produced this fact? By retrieval time, that's gone.

memory · synthesized facts
"Patient's primary physician is Dr. Osei"
"Patient has a penicillin allergy"
"Patient is prescribed lisinopril 10 mg daily"
"Patient reports intermittent chest pain since March"
"Patient prefers contact by chat"

?

EHR record
ehr_export.json

Lab report
lab_results.pdf

Intake chat
self-report

zep
Citation Needed · AIE World's Fair 2026

### Why a source link on the fact isn't enough

A single source_id can identify a verbatim ETL copy. LLM-derived facts can combine sources, merge identities, and be superseded, so their lineage must track a changing set of sources.

[2:56](https://www.youtube.com/watch?v=H7puB0RwJMM&t=176s) [A comparison shows why a single source identifier can fail when facts combine sources, entities merge, or data changes.](https://assets.ai.engineer/sites/aiecode2025/sha256/af785e3a8443d2e1be532acb8bae5b7778649ffd17eb275d65eee9f0b30590e7.webp)

THE PROBLEM
Why a source link on the fact isn't enough
The obvious fix: store a source_id on the fact.

ETL — WHERE IT WORKS

doc-17 · orders.csv
↓
fact — value copied verbatim
source_id: doc-17 ✓

One source, one value, copied. Lineage is one-to-one and deterministic.

LLM EXTRACTION — WHERE IT BREAKS

many → one: Several sources produce one fact. Its wording appears in none.
merge: "J. Smith" and "John Smith" merge into one identity.
invalidate: Newer data supersedes the old fact.

source_id: ? [crossed out]
Lineage must be set-valued, survive mutation, answer queries.

zep
Citation Needed · AIE World's Fair 2026

### Provenance is a graph problem

A patient and penicillin form an allergy fact linked to three source episodes stored verbatim. Following those provenance links lets a reader trace the derived fact back to its sources.

[4:10](https://www.youtube.com/watch?v=H7puB0RwJMM&t=250s) [A derived patient-allergy fact links to three source episodes stored below it.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4bbafd14d4181c2f3defd323295dab555d901fe7ad1bb376291bace76cb1a93.webp)

THE STRUCTURE
Provenance is a graph problem

DERIVED — ENTITIES AND FACTS
Patient — HAS_ALLERGY — Penicillin
fact: "Patient has a penicillin allergy"

Legend: provenance link; episode; derived

GROUND TRUTH — EPISODES, STORED VERBATIM

ep-4f21  ehr_export.json
{ "allergy": "penicillin",
  "recorded_by": "Dr. Osei" }

ep-7c88  lab_results.pdf
"...positive skin test,
 penicillin G..."

ep-9a03  intake_chat
"I think I'm allergic
 to penicillin?"

derived layer above, ground truth below

zep
Citation Needed · AIE World's Fair 2026

### Keeping lineage correct as the graph mutates

A fact retains links to every source episode. When entities merge, their source links must be combined; when new data invalidates a fact, the graph records the superseding episode and timing.

[4:59](https://www.youtube.com/watch?v=H7puB0RwJMM&t=299s) [Three diagrams show lineage for multi-source facts, entity resolution, and temporal invalidation.](https://assets.ai.engineer/sites/aiecode2025/sha256/9b474a1000c902df00fcba8c989a625265a102670c9be701acc9a7ff7fb5249b.webp)

THE STRUCTURE
Keeping lineage correct as the graph mutates

Multi-source facts
one fact edge
ep-1
ep-2
ep-3
A fact keeps every episode that produced it.

Entity resolution
J. Smith
John Smith
John Smith · merged
ep-a1
ep-b7
✕ dropped = orphaned
Merging entities must union their lineage.

Temporal invalidation
fact — valid_at 2025-11-02
invalidated_by
ep-new · 2026-03-14
The graph records which episode superseded this fact, and when — valid_at / invalid_at.

zep
Citation Needed · AIE World's Fair 2026

### Deletion is the inverse problem

Deleting the intake chat removes the contact preference it alone supports. The penicillin allergy fact survives because two other episodes still support it.

[8:51](https://www.youtube.com/watch?v=H7puB0RwJMM&t=531s) [Deleting an episode removes a fact supported only by that episode while a fact with two remaining sources survives.](https://assets.ai.engineer/sites/aiecode2025/sha256/67d1ff871ee944be7657db471fa1df0776cfe14216d176f72d4d8474bb010324.webp)

PATTERNS
Deletion is the inverse problem
Remove a source and the facts derived from it.

fact: "Patient has a penicillin allergy"
episodes: [ep-4f21, ep-7c88, ep-9a03]
kept — 2 parents remain

fact: "Patient prefers contact by chat"
episodes: [ep-9a03] — removed, no remaining support

GROUND TRUTH — EPISODES

ep-4f21 ehr_export.json
{ "allergy": "penicillin", … }

ep-7c88 lab_results.pdf
"…positive skin test…"

ep-9a03 intake_chat
"I think I'm allergic…"

remove_episode("ep-9a03")

delete an episode → shared facts survive, single-source facts go

zep
Citation Needed · AIE World's Fair 2026

### Key insights

Graphiti stores sources verbatim and links derived graph data back to them, supporting verification, debugging, and deletion. The speaker notes that graph construction and provenance are expensive.

[10:19](https://www.youtube.com/watch?v=H7puB0RwJMM&t=619s) [Three insights recommend engineering lineage into graphs, preserving sources, and linking derived data to those sources.](https://assets.ai.engineer/sites/aiecode2025/sha256/afd66c0f71268298f98d89feb45daf21142a94c0fae27a021c9218d32926c5ef.webp)

CLOSING
Key insights

01 LLM graph construction is lossy and generative — lineage must be engineered in, not logged after.
02 Store sources verbatim; link every derived node and edge back to them.
03 The same edges give you compliance, verification, debugging, filtering, and deletion.

github.com/getzep/graphiti

zep
Citation Needed · AIE World's Fair 2026

### Questions

Asked about changes in edge relevance, the speaker says Zep uses a separate data structure to trace provenance across edges. The slide offers QR codes for Zep and Graphiti.

[12:00](https://www.youtube.com/watch?v=H7puB0RwJMM&t=720s) [Questions slide with QR codes for Zep and Graphiti and a professional contact address.](https://assets.ai.engineer/sites/aiecode2025/sha256/61f85fd09b9bd64556e8fa1530d4e17f461d9667f6db025d250ee919d325be46.webp)

Questions

Zep
getzep.com

Graphiti
github.com/getzep/graphiti

[redacted]

zep

## Full transcript

[0:00](https://www.youtube.com/watch?v=H7puB0RwJMM&t=0s) [upbeat music] So LLMs are really great at pulling together data from many sources, uh, but they do so dem- d- non...

[0:20](https://www.youtube.com/watch?v=H7puB0RwJMM&t=20s) Sorry. They do so non-deterministically. They interpret and synthesize data, generating a summary, an extracted fact, uh, a structured record, and this output artifact may not appear verbatim in the source inputs.

[0:36](https://www.youtube.com/watch?v=H7puB0RwJMM&t=36s) Synthesis often destroys the paper trail of how these outputs were originated. And I'm going to be talking today about provenance, which is tracing how an artifact was built and why.

[0:51](https://www.youtube.com/watch?v=H7puB0RwJMM&t=51s) Legal compliance often demands provenance, but it's also useful for debugging, deciding which sources you trust, and which artifacts to delete. And solving this at scale presents a real engineering challenge.

[1:10](https://www.youtube.com/watch?v=H7puB0RwJMM&t=70s) If I can get my next slide going here.

[1:14](https://www.youtube.com/watch?v=H7puB0RwJMM&t=74s) So my team and I built Graphiti, uh, the open source temporal graph framework, and Zep, our enterprise agent memory infrastructure, is built on Graphiti. Our customers derive context or agent memory from many user touchpoints.

[1:32](https://www.youtube.com/watch?v=H7puB0RwJMM&t=92s) Those could be chat, but not only chat. Often it's voice transcripts, email, business data, and our customers have struggled with provenance. Where did this fact come from? What is the veracity of this fact?

[1:47](https://www.youtube.com/watch?v=H7puB0RwJMM&t=107s) And over the next few f- slides, I'll share how we engineered solutions to this problem.

[1:55](https://www.youtube.com/watch?v=H7puB0RwJMM&t=115s) So h- here's an stylized failure mode. An agent retrieves context about a patient, so this is a healthcare scenario,

[2:06](https://www.youtube.com/watch?v=H7puB0RwJMM&t=126s) and what comes back is a clean, confident fact, patient has a penicillin allergy.

[2:13](https://www.youtube.com/watch?v=H7puB0RwJMM&t=133s) And the context was synthesized from three sources: a lengthy EHR record, Electronic health record, a PDF lab report, and something a patient typed into an AI intake chat.

[2:30](https://www.youtube.com/watch?v=H7puB0RwJMM&t=150s) If the agent presents the fact to a doctor in treat, in a treatment scenario without clearly indicating the source was from the patient themselves, it may mislead the doctor.

[2:44](https://www.youtube.com/watch?v=H7puB0RwJMM&t=164s) When an agent retrieves context, can we point to the exact source and its veracity?

[2:51](https://www.youtube.com/watch?v=H7puB0RwJMM&t=171s) For complex agent applications, the answer is often no.

[2:58](https://www.youtube.com/watch?v=H7puB0RwJMM&t=178s) So I, I can imagine you're probably thinking, um, "But can't we just store like a source ID on the fact?"

[3:07](https://www.youtube.com/watch?v=H7puB0RwJMM&t=187s) This can work well in structured data warehouses or data lakes. A pipeline outputs one value copied or mutated deterministically, and the sources are known and easily marked. But with context pipelines run by LLMs, this breaks in several ways.

[3:26](https://www.youtube.com/watch?v=H7puB0RwJMM&t=206s) You prompt an LLM with several sources. Many facts are each synthesized from one or more of the sources.

[3:35](https://www.youtube.com/watch?v=H7puB0RwJMM&t=215s) Somebody like Jay Smith and John Smith are merged into a single entity, one identity, and John's facts are derived from many different places.

[3:47](https://www.youtube.com/watch?v=H7puB0RwJMM&t=227s) So new data might invalidate old facts. The store keeps changing underneath your pointer. And an append-only log, which often might, you know, might come to mind here,

[4:00](https://www.youtube.com/watch?v=H7puB0RwJMM&t=240s) gets very hard to manage at scale as there's so many changes occurring. So lineage needs to be an evolving set and survive mutation.

[4:13](https://www.youtube.com/watch?v=H7puB0RwJMM&t=253s) So sets of links between facts and their sources can be modeled on a graph as relationships, so provenance in a context store containing facts is a knowledge graph.

[4:27](https://www.youtube.com/watch?v=H7puB0RwJMM&t=267s) We have three... In this example, we have three source data. In Graphiti, they're termed episodes. We have two entities extracted from the episodes, patient and penicillin,

[4:41](https://www.youtube.com/watch?v=H7puB0RwJMM&t=281s) and an edge between them. This graph triple, the two entities and the edge, can be hydrated as a fact, patient has a penicillin allergy.

[4:54](https://www.youtube.com/watch?v=H7puB0RwJMM&t=294s) Tracing a fact to its source is just a graph walk.

[5:01](https://www.youtube.com/watch?v=H7puB0RwJMM&t=301s) So it's pretty simple and easy to map source to fact on the first write.

[5:09](https://www.youtube.com/watch?v=H7puB0RwJMM&t=309s) But keeping it r- correct while the graph changes can be really hard.

[5:16](https://www.youtube.com/watch?v=H7puB0RwJMM&t=316s) When new data, uh, so for example, when two entities merge, the merged entity needs to keep all source links from both. Otherwise, we silently drop a source and we lose lineage.

[5:32](https://www.youtube.com/watch?v=H7puB0RwJMM&t=332s) And when new data contradicts existing data, mutating it, we need to capture this lineage too.

[5:41](https://www.youtube.com/watch?v=H7puB0RwJMM&t=341s) In the right-most card, a fact is rendered invalid by new data, and in Graphiti, an invalidAt date is added

[5:51](https://www.youtube.com/watch?v=H7puB0RwJMM&t=351s) to the mutated edge. And the source episodes that resulted in the edge mutating are noted against the fact

[6:06](https://www.youtube.com/watch?v=H7puB0RwJMM&t=366s) So again, in Graphiti, the relationship between source data and derived artifacts, such as facts, is easily modeled on the graph.

[6:16](https://www.youtube.com/watch?v=H7puB0RwJMM&t=376s) With metadata pro-projection, we can also model classifications that span many different episodes and facts derived from them. And so I'll give you an example here.

[6:30](https://www.youtube.com/watch?v=H7puB0RwJMM&t=390s) In, in the prior healthcare scenario, episodes may originate from an EHR record and have an EHR tag associate-associated with them, but not all records are. And so on ingestion, we tag the episodes with the EHR tag.

[6:50](https://www.youtube.com/watch?v=H7puB0RwJMM&t=410s) All subsequent entities and facts derived from the episode inherit the tag.

[6:58](https://www.youtube.com/watch?v=H7puB0RwJMM&t=418s) So if the agent wants to retrieve only facts from verified clinical sources, it's very simple to filter for the appropriate tag as we walk the graph.

[7:11](https://www.youtube.com/watch?v=H7puB0RwJMM&t=431s) So one tagging action at ingestion supports evaluating the veracity of a fact.

[7:22](https://www.youtube.com/watch?v=H7puB0RwJMM&t=442s) But what if the fact has three parents or more? Here we have a verified flag as our metadata, and in this case, one parent is not verified.

[7:34](https://www.youtube.com/watch?v=H7puB0RwJMM&t=454s) So is the fact verified? For the allergy flag, which could be a life and death situation, the agent missing it, missing that particular flag,

[7:47](https://www.youtube.com/watch?v=H7puB0RwJMM&t=467s) could be a deadly mistake, so not retrieving the fact. And any source of the three should block that prescription being issued.

[7:58](https://www.youtube.com/watch?v=H7puB0RwJMM&t=478s) But for something like a consent on file for procedure fact, the mistake is operating on unv-- on an unverified consent, so the patient hasn't actually given their consent. And every parent needs to be verified, so every single episode should have that tag.

[8:18](https://www.youtube.com/watch?v=H7puB0RwJMM&t=498s) So the facts have very similar shapes. They have three, three parent episodes, but opposite policies. And here,

[8:28](https://www.youtube.com/watch?v=H7puB0RwJMM&t=508s) Graphiti or the underlying store exposes that choice. It exposes which of the episodes have the gra-- the particular tag.

[8:39](https://www.youtube.com/watch?v=H7puB0RwJMM&t=519s) But your agent needs to execute or apply your business rules. So that's not necessarily something that we bake into the graph. It's situational.

[8:54](https://www.youtube.com/watch?v=H7puB0RwJMM&t=534s) Another situation where lineage is really important, we may have to delete source data due to retention policies or right to be forgotten requests, so privacy compliance. And this is really tricky because if we have context derived from multiple sources,

[9:14](https://www.youtube.com/watch?v=H7puB0RwJMM&t=554s) how do we do so? Mapping lineage here is really useful. We know which facts are derived from the source data we intend on deleting. But what if only some of the source data needs to be deleted, but not all?

[9:31](https://www.youtube.com/watch?v=H7puB0RwJMM&t=571s) So in this example, we need to delete the intake chat da-data, so what the patient filled in, which is only one of three source data. In Graphiti's model,

[9:47](https://www.youtube.com/watch?v=H7puB0RwJMM&t=587s) the allergy fact survives the deletion, and that's because there are two parents, parent episodes still supporting the fact. However, the contact preference fact

[10:00](https://www.youtube.com/watch?v=H7puB0RwJMM&t=600s) is deleted, as it was derived solely from the deleted source data.

[10:07](https://www.youtube.com/watch?v=H7puB0RwJMM&t=607s) So the rule is pretty simple here, and it's easier to apply because the link exists. A fact is only deleted if no remaining episodes support it.

[10:19](https://www.youtube.com/watch?v=H7puB0RwJMM&t=619s) So to sum it all up, deriving context is lossy and generative.

[10:26](https://www.youtube.com/watch?v=H7puB0RwJMM&t=626s) Lineage needs to be built into the data structure, engineered into the data structure, which is a graph, not logged afterwards.

[10:37](https://www.youtube.com/watch?v=H7puB0RwJMM&t=637s) And in Graphiti, we keep the sources verbatim, and we link everything back, everything derived from those sources back to the source.

[10:47](https://www.youtube.com/watch?v=H7puB0RwJMM&t=647s) And provenance offers many benefits to users of Graphiti. You have compliance built in, which makes your chief compliance officer very happy.

[10:59](https://www.youtube.com/watch?v=H7puB0RwJMM&t=659s) You can verify a fact based on its sources, so you understand veracity. Should I trust this fact?

[11:08](https://www.youtube.com/watch?v=H7puB0RwJMM&t=668s) It's easy to debug where something came from, so why do I have this fact? How is it generated? And also determining what to delete. And most of what I've covered today is in the Graphiti framework, so you can go to, uh, the Graphiti repo on GitHub, and I have a little, uh, QR cod-QR code later that you

[11:32](https://www.youtube.com/watch?v=H7puB0RwJMM&t=692s) can zap, uh, and try it out. So by the way, as an aside,

[11:41](https://www.youtube.com/watch?v=H7puB0RwJMM&t=701s) lineage and provenance is expensive. Graph construction is really expensive, uh, in the way that Graphiti does it. And so we've put significant effort into reducing cost and latency of generating graph artifacts.

[11:57](https://www.youtube.com/watch?v=H7puB0RwJMM&t=717s) I'd be happy to speak to how we've done that, uh, in, in the Q&A.

[12:01](https://www.youtube.com/watch?v=H7puB0RwJMM&t=721s) So thanks for attending. Um, if you'd like to learn a little bit more about Zep or Graphiti, you can zap the QR codes. Uh, Zep is on the left and Graphiti on the right.

[12:13](https://www.youtube.com/watch?v=H7puB0RwJMM&t=733s) And I don't know if we're doing Q&A here or outside.

[12:18](https://www.youtube.com/watch?v=H7puB0RwJMM&t=738s) You can do some.

[12:19](https://www.youtube.com/watch?v=H7puB0RwJMM&t=739s) Okay, happy to do Q&A.

[12:21](https://www.youtube.com/watch?v=H7puB0RwJMM&t=741s) Yeah. Just repeat the question.

[12:23](https://www.youtube.com/watch?v=H7puB0RwJMM&t=743s) Yeah.

[12:24](https://www.youtube.com/watch?v=H7puB0RwJMM&t=744s) We have time.

[12:25](https://www.youtube.com/watch?v=H7puB0RwJMM&t=745s) Anybody have a question? Yep. One right from the front.

[12:34](https://www.youtube.com/watch?v=H7puB0RwJMM&t=754s) How do you go about mutating the graph

[12:38](https://www.youtube.com/watch?v=H7puB0RwJMM&t=758s) at the edge to account for weight changes in relevancy? How do we mutate the graph- At the edge ... at the edge. Uh, uh-

[12:41](https://www.youtube.com/watch?v=H7puB0RwJMM&t=761s) To account for weight changes in relevancy. Oh, to account for weight changes in relevancy. Um, that is a structure that we've actually built into Zep, not into Graphiti. And, um, what we do is we do have for that a-- some tracing that we do, which is kind of X of the graph.

[13:02](https://www.youtube.com/watch?v=H7puB0RwJMM&t=782s) So not all of the, um, provenance is in the graph. I go to like fifty edges. Sorry, you? I go to like fifty edges. If you have fifty edges?

[13:12](https://www.youtube.com/watch?v=H7puB0RwJMM&t=792s) And mine go to fifty edges. Uh, sorry, is-- was that a question? If you have fifty? Oh, there you go.

[13:19](https://www.youtube.com/watch?v=H7puB0RwJMM&t=799s) I said I, I use fifty.

[13:22](https://www.youtube.com/watch?v=H7puB0RwJMM&t=802s) You use fifty?

[13:23](https://www.youtube.com/watch?v=H7puB0RwJMM&t=803s) Yeah.

[13:24](https://www.youtube.com/watch?v=H7puB0RwJMM&t=804s) Fifty edges.

[13:26](https://www.youtube.com/watch?v=H7puB0RwJMM&t=806s) Yeah, edges.

[13:26](https://www.youtube.com/watch?v=H7puB0RwJMM&t=806s) Um, so, so, you know, Zep is able to, uh, look at provenance across those, but using a separate data structure from the graph.

[13:36](https://www.youtube.com/watch?v=H7puB0RwJMM&t=816s) Oh, okay. Thank you.

[13:38](https://www.youtube.com/watch?v=H7puB0RwJMM&t=818s) Great.

[13:38](https://www.youtube.com/watch?v=H7puB0RwJMM&t=818s) For that particular problem.

[13:42](https://www.youtube.com/watch?v=H7puB0RwJMM&t=822s) You're next.

[13:43](https://www.youtube.com/watch?v=H7puB0RwJMM&t=823s) So, uh, does the agent also create the edges, edge types and the entities itself, or how does it resolve those edge-

[13:50](https://www.youtube.com/watch?v=H7puB0RwJMM&t=830s) Yeah

[13:51](https://www.youtube.com/watch?v=H7puB0RwJMM&t=831s) ... entity types?

[13:51](https://www.youtube.com/watch?v=H7puB0RwJMM&t=831s) So in Graphiti, you can search across the entire graph. Uh, it has, um, vector similarity search against various textual artifacts, um, full-text search, as well as graph relational operations, things like BFS.

[14:08](https://www.youtube.com/watch?v=H7puB0RwJMM&t=848s) Um, it depends on the underlying, uh, graph database that's used. And so your agent can walk the graph, it can search semantically, et cetera.

[14:19](https://www.youtube.com/watch?v=H7puB0RwJMM&t=859s) And obviously, from anywhere you hit in the graph, you're then able to understand the provenance of a particular artifact that you've hit.

[14:29](https://www.youtube.com/watch?v=H7puB0RwJMM&t=869s) Uh, I find it very, uh, fascinating this, uh, temporal, uh, support out, right out of the Graphiti database. Can you, uh, tell just a little more, um, under the hood, what are those really the, the episodes are nodes in the graph, just like the other nodes?

[14:49](https://www.youtube.com/watch?v=H7puB0RwJMM&t=889s) And how do you extract-- So I saw the API that, let's say I give some, uh, add, add episode. How do we extract information from the add episode call under the hood, like large language models and, and edges?

[15:04](https://www.youtube.com/watch?v=H7puB0RwJMM&t=904s) Yeah. Yeah. So, uh, yes, episodes are an entity on the graph, uh, or a node on the graph. Um, it makes sense to model them that way. Um, in Zep and Graphiti, we have various odd derived artifacts that are, um, nodes on the graph as well, because they too need to have lineage, and

[15:29](https://www.youtube.com/watch?v=H7puB0RwJMM&t=929s) we need to understand how they were derived. Um,

[15:34](https://www.youtube.com/watch?v=H7puB0RwJMM&t=934s) and in terms of how the add episode method works,

[15:39](https://www.youtube.com/watch?v=H7puB0RwJMM&t=939s) there's a pretty complicated, uh, pipeline that gets run on, um, uh, episode ingestion. And I'll just give it the very high-level, uh, uh, outline for you. So there's a structured extraction,

[15:56](https://www.youtube.com/watch?v=H7puB0RwJMM&t=956s) extracting entities and the relationships between them, and candidate facts. And those are the materialized or hydrated fact triples. So two entities and a fact. Um, and it's structured as, uh, a fact is structured as, um, um, a subject, well, ob-- subject, uh, verb, object.

[16:21](https://www.youtube.com/watch?v=H7puB0RwJMM&t=981s) Um, and after that, there is a, uh, deconfliction process that runs, a deduplication and deconfliction process. We deduplicate entities, and we deconflict facts, 'cause there might be existing facts in the graph that are gonna be mutated by a new learned fact.

[16:41](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1001s) So Daniel loves Adidas shoes. Three months later, Daniel's shoes fell apart. He sends it back to-- the, the shoes back to the return application, and he sends a nasty gram along with it.

[16:53](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1013s) Well, now Daniel returned shoes is a fact, and Daniel was unhappy about Adidas. We need to invalidate the Daniel loves Adidas shoes fact. And so that is part of that pipeline as well.

[17:06](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1026s) A lot of what we do uses LLMs, but we try very hard not to use LLMs in this process as well. So where we're able to deploy more traditional information retrieval techniques, more traditional NLP techniques, uh, looking at things like entropy and a bunch of other, you know, using, um, SimHash and a bunch of other approaches to

[17:30](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1050s) dedupe, uh, we do so. Uh, far cheaper, far faster, deter-- far more deterministic.

[17:38](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1058s) Hopefully, that answers your question.

[17:41](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1061s) Yep. Uh, thank you for the great talk. Um, just wanted to ask a question. So it seems like a common theme these days in, uh, memory systems is more file-based memory and wikis and knowledge bases.

[17:56](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1076s) And, uh, I'm just wondering, have-- is Zep working on something like that? And also, uh, could some of the ideas here be represented in that paradigm?

[18:05](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1085s) Yeah. Uh, Markdown suffers from provenance. File-based, um- File-based, uh, memory starts to break down with provenance. It's very difficult when you mu- mutate lines in a file to understand the lineage or the provenance of why those changes occurred.

[18:30](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1110s) Um, not only that, but in multi-agent, multi-user, and multi-source scenarios, it can be very challenging to manage markdown files at scale. I think they work really well, um, for desktop usage.

[18:47](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1127s) Uh, they sometimes work well in, uh, agentic use cases that are server-based, not necessarily desktop or single user, single agent scenarios. Um,

[18:59](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1139s) but what we found is that it's, um, they just break down with the types of enterprise problems that we're solving and particularly-- in particular provenance as an example. Does that answer your question?

[19:14](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1154s) Oh, more. I don't know how much time we have left, but maybe one more.

[19:17](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1157s) Just one.

[19:18](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1158s) Yeah.

[19:19](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1159s) Uh, Daniel, thank you. Um, question on the write back. How do you do that explicitly or implicitly? Um, how do you create the facts? Do you ask the LLM to summarize the conversation-

[19:31](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1171s) Yeah

[19:31](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1171s) ... or at, at every turn you do that? And, and-

[19:34](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1174s) Yeah

[19:34](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1174s) ... which component does it, Zep or, uh-

[19:37](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1177s) Yeah. So, so we do, um, as pa- we actually as part of the extraction, we've managed to get, um, a single shot extraction working that extracts entities and the relationships between them and facts.

[19:55](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1195s) And we're able to do so really cheaply as a consequence. Um, and so yes, we're using an LLM for that. Uh, we do have a reflection step or some reflection built in to ensure that the things that we've retrieved, um, are actually accurate, um, as well as to do some other stuff around, uh, uh,

[20:20](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1220s) more, more richness to the lineage. So why did something change? Not just this was related, but also why did it change?

[20:30](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1230s) Uh, where was that?

[20:31](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1231s) Uh, th- that's, uh, partly in Graphiti, partly in Zep.

[20:35](https://www.youtube.com/watch?v=H7puB0RwJMM&t=1235s) Yeah. All right. Well, thank you everybody. [outro music]
