# AI on Your Lakehouse: Context Comes in Shapes, Not Queries

Zach Blumenfeld · Ben Squire · Ryan Knight

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/kRkcNOsRyYg-ai-on-your-lakehouse-context-comes) · [Video](https://www.youtube.com/watch?v=kRkcNOsRyYg)

## AI on Your Lakehouse: Context Comes in Shapes, Not Queries

A repair copilot needs more than matching passages and valid SQL. Warehouse connections, navigable documents, and link communities give it different ways to assemble the context a question requires.

Basic SQL, Python, and command-line familiarity will help with the exercises; the article introduces the graph concepts as they appear.

### The repair history and the manual are only the beginning

[2:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=128s)

An auto-repair copilot can retrieve the right records and still give the wrong answer. Its warehouse contains structured repair history; its document store contains manuals, safety bulletins, and recalls. Text-to-SQL and vector search make those sources accessible, but access alone does not ensure that an agent sees the relationships it needs.

Consider Autofix, the fictional national repair chain used throughout the workshop. A technician has a vehicle in a bay and needs to know how to repair it. Leadership wants a copilot to support that work, and the AI engineer has to connect the technical library to the history of repairs already performed. The demonstration uses cloud-stored documents and BigQuery, with patterns intended to extend to Databricks and Snowflake. As the number of similar tables and documents grows, relevant context can fall through the cracks even when individual retrieval calls work correctly.

The harder questions concern the whole operation: Which incoming vehicles lack documentation? Which documents are never used? Which repairs keep failing? Which recalls form recurring groups? A similarity search returns things resembling a query; establishing a missing connection requires examining coverage and relationships. Even a seemingly ordinary warehouse question can depend on choosing the correct joins among many lookalike tables.

**The context an agent receives should fit the question it is answering.** The workshop introduces three shapes:

| Shape | Context supplied | Useful question |
|---|---|---|
| Table of Contents | Hierarchy and document cross-links | What should I read next? |
| Themes | Groups inferred from interlinking | What patterns span the library? |
| Connections | Warehouse metadata and join paths | How do these records relate? |

The implementation starts with Connections, then builds document navigation and Themes.

![Slide outlining search limitations, three graph shapes, and agent context, with a portability note below.](https://ai.engineer/api/talk-essay-media/758d52101718b0b6015f7f7b4b0a13b1dbeedc0c749d134ebb891b712c73ad7f)

Context comes in shapes, not queries.

### A graph, an agent, and a runnable workspace

[7:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=457s)

The underlying model is a **property graph**. Nodes represent things such as people and cars; relationships represent associations such as a person owning or driving a car. Both can carry properties, including strings, numbers, dates, and vectors. Blumenfeld compares the connected structure to pre-joined tables: once relationships exist, a query can follow them instead of reconstructing each association from scratch.

The [GraphAcademy workshop](https://graphacademy.neo4j.com/courses/workshop-lakehouse) organizes the exercises into seventeen lessons. To follow along:

1. Sign in, enroll in the course, and open its Environment lesson.
2. Launch the preconfigured GitHub Codespace in your own account. A local clone is also possible through the repository’s Bash setup script, but requires the local dependencies, including Claude Code.
3. Wait for initialization, then populate the environment file with Anthropic access, BigQuery credentials, and your individually provisioned Neo4j sandbox credentials.
4. Start Claude Code. In the demonstration, opening a fresh terminal resolves the case where it asks for a subscription instead of recognizing the API key. Accept the configured MCP server and ask the agent to check its Neo4j connection.

Blumenfeld’s graph already contains data from an earlier run; a new participant’s sandbox begins empty.

The coding workflow uses the [Neo4j CLI](https://github.com/neo4j-labs/neo4j-cli) to execute database queries, a Cypher skill to guide query construction, and a Graph Data Science skill for the later algorithms. Rather than handwrite every query, participants give the agent specifications for the outputs they want.

The repository separates those concerns: `.claude` contains skills; outline, search, and themes scripts contain implementation placeholders; a small Python SQL runner calls BigQuery; and Markdown specifications describe the required outputs. Solution scripts provide a recovery path when generated queries fail. The repository also includes data-loading material, Databricks directions for comparisons with Genie and AI Search, source PDFs, and a readable Markdown corpus. Cursor is a possible local alternative, though Blumenfeld has not tested the workshop with it.

### Connections: use a graph to choose the SQL

[21:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1317s)

The warehouse schema centers on work orders, with vehicles, diagnostic trouble codes (DTCs), procedures, work-order parts, and parts connected through primary and foreign keys. The first task is to make those relationships legible to the agent. [NeoCarta](https://github.com/neo4j-labs/neocarta/blob/main/README.md), a Neo4j Labs project, supplies a graph representation of warehouse metadata. It can also represent business terminology and processes, although the workshop does not use those extensions.

Blumenfeld distinguishes an ontology, which helps interpret and reason about data, from a semantic layer, which supplies agreed terms for querying it accurately. A separate virtual-graph approach can expose SQL data through a federated Cypher interface; the exercise instead uses an MCP ingestion operation to build the metadata graph. The workshop load is described as producing six tables and five reference join paths. Its hierarchy runs from database to schema to table to columns, with optional representative values.

The first question is concrete: Which vehicle received part `IC2042`? The agent reads the NeoCarta metadata through MCP, identifies the relevant join path, and uses the Python runner to query BigQuery. **The graph guides warehouse execution without replicating the warehouse wholesale.** Representative values may accompany metadata, but the repair records remain in BigQuery and SQL retrieves the answer.

During the setup recap, one detail matters for anyone following along: an absent `.env`, or a workspace containing only `.env.example`, can mean initialization is still running. The setup script populates files asynchronously. Each participant must use their own sandbox credentials alongside the service credentials. Once the query completes, the agent’s Python SQL calls perform the selections and joins, and the displayed answer groups vehicles by make and vehicle count. The demonstrated join paths come from BigQuery’s primary/foreign-key metadata; NeoCarta can also use query logs or manually supplied joins, terminology, and metrics.

![GraphAcademy environment lesson with an Open in GitHub Codespaces button and a folder overview.](https://ai.engineer/api/talk-essay-media/a2c71ceab9d840ee92900f4220fd7842885b5e67eb41f3dbc8f08a2ea918f922)

Opening the workshop Codespace.

### When metadata is enough—and when to import records

[34:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2066s)

A graph can improve execution and it can improve an agent’s understanding of a schema. Supply-chain shortest paths and variable-length traversals illustrate the first use. Choosing a short join path from hundreds of candidate tables illustrates the second: the final SQL may be simple even when discovering the right relationship is difficult. NeoCarta addresses that discovery problem while preserving existing warehouse access. [Neo4j Virtual Graph](https://neo4j.com/blog/auradb/neo4j-virtual-graph-is-now-in-public-preview/), described in the recording as a preview capability, moves the graph interface down to query execution through Cypher pushdown. Related context-graph work is mentioned, but is not part of this implementation.

Why not copy the transactional data into Neo4j? The answer depends on what the copy buys:

- **Keep records in the warehouse:** continuously updated datasets require synchronization, selective copying can require custom ETL, and security policies may prohibit moving sensitive records into another system.
- **Import records for graph workloads:** large recursive joins, graph algorithms, graph embeddings, and clustering can justify native graph storage. Unstructured documents also benefit when ingestion gives them a useful structure.

The metadata graph itself persists. Starting NeoCarta does not rebuild it: the build-connections script explicitly invokes ingestion, after which later queries reuse the stored metadata. Blumenfeld expects the load to be idempotent but does not rerun it to demonstrate that property.

The Cypher skill is maintained internally and distributed alongside skills for agent memory, cloud infrastructure, and language drivers. Its purpose is to supply current query guidance instead of letting the agent rely on old web examples. Blumenfeld does not describe the exact process used to author its `SKILL.md`; the dev-container installation file shows where the CLI and skills are installed.

![GitHub displays the Cypher SKILL.md with description, compatibility, When to Use, When NOT to Use, and Pre-flight sections.](https://ai.engineer/api/talk-essay-media/ec16dba8fc3f3d9abf40685f587638698c902afa1786f2e05a687336e10f045e)

The Neo4j Cypher skill’s scope and usage guidance.

A question about bronze, silver, and gold data layers receives a roadmap answer rather than an additional demonstration. NeoCarta is customer-driven; at the time of the workshop, a Databricks connector is under development, with metric views and document support among the possible extensions being discussed.

### Table of Contents: preserve the structure documents already have

[43:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2620s)

The document side contains simulated manuals, bulletins, and recalls, available as PDFs and Markdown. An ABS manual, for example, refers to platform codes and diagnostic troubleshooting material elsewhere in the library. Those references already carry useful structure. The loader reads local files for the exercise, standing in for the cloud-stored collection.

The Table of Contents shape lets the agent navigate that structure. Blumenfeld invokes [PageIndex](https://github.com/VectifyAI/PageIndex) as an analogy for reading a document hierarchy; it is not a workshop dependency. The outline begins with a library, descends through folders, documents, and sections, and includes cross-links to other documents. Every location has a hierarchical URI. An agent can use a URI to retrieve text, inspect a subtree, or follow a reference into another part of the library. Unlike Connections, this shape imports document content into the graph.

A familiar GraphRAG ingestion pipeline extracts entities—vehicles or parts, for example—and connects them to source chunks. Here the first graph is lighter: a deterministic parser preserves containment, section order, and existing named links, resembling the references in an Obsidian vault. Hyperlinks can supply the same kind of cross-document relationship.

For documents with consistent structure and substantial interlinking, this avoids an LLM extraction pass. The loader is designed to be idempotent, and the hierarchy becomes part of each node’s URI, from library through folder and file to section. Inspecting the loaded graph reveals the three relationships the tools will use: containment through `HAS`, section ordering, and cross-document references through `LINKS_TO`.

### Build an outline that can stop at the right depth

[52:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3168s)

The outline tool starts as a query placeholder inside the Autofix skill’s script. A Markdown specification describes the human-readable shape and optional arguments, and the agent uses the Cypher skill to implement it. The central operation is a variable-length traversal over `HAS`; the demonstration shows a bound of `*0..25`, with the script controlling traversal depth. A second query retrieves cross-links.

The containment operation can be expressed directly in Cypher. Given a root URI, this query returns the root and descendants through the demonstrated bound, together with their depth:

```cypher
MATCH (root {uri: $rootUri})
MATCH path = (root)-[:HAS*0..25]->(node)
RETURN node.uri AS uri,
       length(path) AS depth
ORDER BY depth, uri;
```

The zero-length path includes the root itself. This is the traversal core; the complete outline also formats the hierarchy and attaches the separately retrieved references.

Without a root URI or depth restriction, the script returns the entire graph. Blumenfeld can do that with the workshop’s small corpus, but discourages it for larger collections. Setting depth to one gives a shallow view; supplying a URI moves the root. The live example starts at the Falcon 2.0 document, then takes a coil-identification URI from a link and uses it to request another outline. The output of one navigation step becomes the input to the next.

To check whether the agent actually uses these tools, inspect its tool-call history, including NeoCarta MCP calls. Asking the agent to describe its steps is useful for teaching, but production verification belongs in execution logs. The ingestion code is similarly inspectable: the corpus parser uses regex for these consistently structured documents. Messier sources may require additional processing, including **[GLiNER](https://arxiv.org/abs/2311.08526)** for entity extraction or an LLM; that does not make either a complete document-structure parser. NeoCarta’s MCP endpoint, its CLI, and the skills approach are not inherently tied to a particular model.

### Keep the vocabulary useful to the query author

[1:03:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3789s)

A single `HAS` relationship across all containment levels makes the variable-length query simple. Separate types such as `HAS_FOLDER`, `HAS_DOCUMENT`, and `HAS_SECTION` would require more cases. The same reasoning keeps `LINKS_TO` general. More specific relationships can express richer distinctions, but hundreds of types become harder for both people and agents to manage. On the warehouse side, semantic metadata serves a related purpose: it reconciles business terminology with the physical schema.

Navigation complements search. A full-text or vector query can locate a starting point; an outline can expose its neighbors and the larger structure around it. Repeated similarity searches may miss a document that matters when investigating an absence or a coverage gap. Explicit traversal provides a different way to inspect the collection, rather than requiring every next step to resemble the original question.

The workshop also leaves some entities implicit. Instead of creating a dedicated part node for every mention, the agent can read a part from a document and use it to query the warehouse. Blumenfeld sees stronger agents making this practical in some cases. A domain such as life sciences may still require explicit molecule and medication ontologies because precise relationships are essential to valid inference. The decision is about the domain’s required semantics, not a universal requirement to extract everything—or nothing—during ingestion.

### Search the text, then restrict it to a subtree

[1:08:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4127s)

The next placeholder implements search using Neo4j’s Lucene full-text index, without vector search. The index covers document and section nodes containing text, not folders, and returns matches with scores. Because each match carries a hierarchical URI, the query can apply a subtree restriction after full-text retrieval: keep only nodes whose URI begins with the requested prefix.

**Semantic expansion** supplies some vocabulary flexibility without an embedding pipeline. If a user describes an engine shuddering, the agent can expand that description to terms such as misfire or rough idle, then combine them with Lucene’s `OR`. The demonstration also searches for coils only under recalls. The URI encodes enough containment information to scope those results without an additional graph traversal.

The complete callable procedure is [`db.index.fulltext.queryNodes()`](https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/full-text-indexes/). The following Cypher shows the search-then-filter sequence, with the configured index name, Lucene expression, URI scope, and result limit supplied as parameters:

```cypher
CALL db.index.fulltext.queryNodes($indexName, $lucene)
YIELD node, score
WHERE node.uri STARTS WITH $scopeUri
RETURN node.uri AS uri, score
ORDER BY score DESC
LIMIT $limit;
```

For the symptom example, `$lucene` can contain `misfire OR "rough idle"`. For a recall-only coil search, it contains the coil query and `$scopeUri` identifies the recalls subtree. The prefix condition filters the returned nodes; it does not change the full-text index into a separate index for that folder.

### Names, freshness, and references are part of retrieval quality

[1:14:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4478s)

An agent can help create a missing full-text index, but it still needs a sensible indexing contract. Neo4j supports indexing multiple labels and properties, and that flexibility creates choices the agent may misunderstand. Blumenfeld thinks he may have generated the workshop index with an agent, but does not recall that construction step.

Document names matter because an outline exposes names before content. A poor title gives the agent less evidence for deciding where to navigate; search can compensate by looking inside the document. Outdated or drifted content is another problem. Blumenfeld proposes last-updated or authority metadata as possible additions, and emphasizes descriptive link aliases: they explain why a document points elsewhere. LLM enrichment during ingestion can help when that descriptive information is missing.

A deterministic load can rebuild the same document graph from unchanged sources. Incremental updates are possible too, but a document is not isolated: renaming it changes the URI that other documents may reference. Partial synchronization therefore needs to update or clean the affected subtree and account for incoming links, rather than simply replacing one node.

An audience question about selectively applying PageIndex leads to a useful sequencing clarification. There is no mandatory outline-first order. An agent can inspect an outline and then search within a promising area, or search first and then follow the matching document’s links. The optional practice material explores the Cypher details further, though the live workshop moves on to Themes.

### Themes: discover groups through document links

[1:21:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4903s)

Looking only at `LINKS_TO` relationships reveals another shape. Manuals, recalls, and bulletins refer to one another; a guide may be the shared target of many repair procedures. Zooming out reveals groups of highly interconnected documents. These groups can expose subjects that were never explicitly tagged in the library.

![GraphAcademy lesson beside a graph result showing a central node connected to surrounding nodes and a results overview.](https://ai.engineer/api/talk-essay-media/0250d960b543173d8378cadb71af22b1df977c9b405a28031be07112271f0d43)

Exploring links between documents and sections.

The workshop uses **[Leiden community detection](https://neo4j.com/docs/graph-data-science/current/algorithms/leiden/)** to assign documents to groups. The aim resembles the global-versus-local distinction in [Microsoft GraphRAG](https://arxiv.org/abs/2404.16130), but the input here is existing document structure rather than an LLM-extracted entity graph. Blumenfeld describes Leiden as a successor to Louvain with more efficient behavior. Neo4j Graph Data Science runs it on an in-memory projection of the relevant graph. His discussion of millions or billions of nodes concerns the intended scale of graph analytics, not a measured workshop run; the projection still requires adequate memory.

The agent receives a compact view of each group rather than a graph visualization:

| Field | What it conveys |
|---|---|
| Theme ID | The community assignment |
| Interlinking description | Conductance-based separation from other groups |
| Top shared targets | References common to the group |
| Most-linked documents | Central documents to inspect next |

Conductance relates connections across the community boundary to connectivity within the graph, supporting descriptions such as tightly or loosely interlinked. Link labels and document titles make the groups interpretable: the examples expose a BCM/bus cluster and a braking cluster containing rotor pads and hydraulic lines, without generating new theme names.

The implementation again begins with a specification and a placeholder, this time in `themes.py`. The agent uses the Cypher and GDS skills to build the projection, while the script calls Leiden through the GDS library. Before projecting, it collapses section-to-section references to their owning documents using the URI structure. That produces document-level relationships, so the resulting communities describe how documents group together rather than fragmenting the view across individual sections.

Blumenfeld reports 13 groups at the default `gamma=1` and 14 themes at `gamma=2` on the simulated workshop corpus. Gamma controls resolution, allowing a more finely divided view; the appropriate setting depends on the corpus. The output continues to use source link labels and document names. Returned URIs let the agent move from a theme into an outline or scoped search, supporting questions about concentrated issues, thin documentation, or where new material belongs.

### Community descriptions, changing data, and trust

[1:33:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5603s)

Leiden supplies group IDs; describing those groups is a separate choice. The workshop displays source metadata, which avoids generated labels but depends on good document titles and link names. Microsoft GraphRAG-style processing can instead add entity extraction and hierarchical LLM summaries, at additional cost and latency, with potentially varying generated text. Source-derived descriptions avoid that generation variability, but unchanged documents alone do not guarantee identical Leiden assignments: reproducibility also depends on randomness controls and execution settings.

For changing data, rerun the projection and retain snapshots of the communities and their descriptions. Those snapshots let an application examine how groups evolve over time. Blumenfeld points to chargeback fraud and anti-money-laundering analysis as established uses of temporal community analysis, where repeated clustering helps reveal changing patterns.

Large theme views can be filtered by community size or conductance, but small communities may still be important. Filtering the view also does not validate the source. The deterministic parser trusts existing references: a wrong link remains a wrong link, and a malformed section may survive ingestion. Agent traversal could help audit such problems, but the workshop does not demonstrate a complete cleaning solution.

### From a diagnostic code to a repair recommendation

[1:39:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5959s)

Predefined shapes do not have to cover every possible question. The Neo4j CLI can retrieve the schema and execute custom Cypher, letting a coding agent compose a query when it needs to connect information in an unanticipated way. Blumenfeld reports a better text-to-Cypher experience with the CLI and maintained skills, without presenting a measured comparison. The live CLI exercise is abbreviated, but the schema inspection shows the information available to the agent.

![Codespaces terminal shows schema output with index and constraint tables beneath an open theme-format document.](https://ai.engineer/api/talk-essay-media/57d79399f8d03a83c50afd61c921a40ebebc9153117273789bf33d0dbcdd566e)

Inspecting graph schema through the Neo4j CLI.

The first combined demonstration returns to the technician: for a given VIN and diagnostic code, what has fixed similar vehicles? The operational goal is to reduce the **comeback ratio**—customers returning because the previous repair did not work. Warehouse history records those outcomes, while manuals, recalls, and bulletins describe the parts and procedures. A prewritten domain skill exposes the outline, search, and themes scripts alongside the SQL runner and NeoCarta MCP access.

The answer is assembled through several operations:

1. Search for the diagnostic code and related symptoms, including misfire or rough idle.
2. Locate the relevant engine manual and inspect its linked procedures.
3. Retrieve the warehouse join paths.
4. Query the vehicle and comparable work orders involving the diagnostic code and part.
5. Combine the repair history with the documentation to recommend replacing the old ignition coil with its revised part.

The graph supplies the document-to-procedure route; the warehouse supplies what happened in previous repairs.

Blumenfeld’s comparison with vector retrieval is about the path through the evidence. A search-based system can answer this question too, but repeatedly searching for each related procedure introduces opportunities to retrieve a misfire passage about the wrong part. Following the manual’s explicit links gives the agent a more constrained route before it returns to the warehouse.

### Find the mismatch between the library and the field

[1:45:31](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6331s)

The supervisor’s question is broader: Are documented procedures consistent with field problems, what documentation is missing, and what documentation is not being used? This requires comparing the technical library with work-order history. A search result by itself cannot establish that the relevant documentation is absent; the analysis needs some account of what exists on both sides. The agent works across document information, diagnostic codes, and warehouse queries to investigate the gaps.

For this exercise, Blumenfeld explicitly disables the Neo4j CLI so the predefined shapes remain visible. With unrestricted CLI access, the agent might prefer custom Cypher instead. The intended investigation compares field codes, repair choices, and comeback history against the library: repeated unsuccessful repairs can suggest that the correct procedure is not being applied, while documentation with no corresponding field activity raises a different coverage question.

The live agent takes a different route from the one requested. It performs extensive semantic-expansion full-text searches instead of using the outline. Blumenfeld reports procedure mismatches and two field DTC codes with no code-level documentation. The hierarchical URIs still help it navigate the search results, but the demonstration does not show the intended outline traversal. Blumenfeld attributes the extra delay to the comprehensive searches and suggests that following explicit links could have reduced the work. **A tool being available does not mean the agent will choose it.**

### Theme coverage and the join the agent still performs

[1:52:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6730s)

The final demonstration asks for common patterns across bulletins and recalls, together with how many cars each affects. The agent calls Themes to identify groups, retrieves fixes from warehouse history, and associates the work orders with the relevant themes. This connects the library’s subject structure to operational activity.

A question about when the shapes are defined exposes the design sequence: first specify what the agent should receive, then derive the queries needed to produce it, and use those queries to inform the data model. The underlying document graph remains the `HAS` containment tree plus cross-links. The completed answer uses Themes and Connections without search, grouping cars and work orders under theme types and presenting percentages.

Asked how accuracy holds up as the knowledge base grows, Blumenfeld says he has not benchmarked this exact implementation. Customers evaluate their own ontologies and query patterns against alternatives such as ordinary vector search on their own datasets. These exercises establish the mechanisms and show the tool choices made during the live run; evaluating accuracy and improving the skill’s routing guidance remain application-specific work.

The last architectural question uses financial statements and CEO commentary as another example of structured and unstructured sources. In this workshop, **the warehouse metadata graph and the document graph have no explicit edges connecting them**. The agent extracts a part or diagnostic code from one source and uses it to look up information in the other, performing the cross-source match at query time. Stored links could make that mapping more deterministic. Blumenfeld leaves them out to keep the initial implementation easy to build and model-agnostic.

The online course remains available for repeating the exercises. The shared credentials are temporary: Blumenfeld plans to withdraw the Anthropic key and leave the BigQuery key available only for a while. Anyone returning to the workshop should be prepared to supply their own access.

## Resources

- [Follow the GraphAcademy workshop](https://graphacademy.neo4j.com/courses/workshop-lakehouse) — Seventeen lessons build warehouse connections, document outlines and community views using a fictional auto-repair dataset.
- [NeoCarta](https://github.com/neo4j-labs/neocarta/blob/main/README.md) — Neo4j Labs semantic-layer project that exposes warehouse metadata and join context through MCP.
- [Neo4j CLI](https://github.com/neo4j-labs/neo4j-cli) — Agent-oriented command-line tooling with installable Neo4j skills.
- [Leiden configuration and execution](https://neo4j.com/docs/graph-data-science/current/algorithms/leiden/) — GDS reference for community detection, resolution, randomness controls and memory estimation.
- [Neo4j full-text indexes](https://neo4j.com/docs/cypher-manual/current/indexes/semantic-indexes/full-text-indexes/) — Create and query full-text indexes, select analyzers and interpret returned scores.
- [PageIndex](https://github.com/VectifyAI/PageIndex) — Document retrieval using hierarchical indexes and model-guided tree navigation.
- [From Local to Global](https://arxiv.org/abs/2404.16130) — Microsoft's GraphRAG research on answering global questions using graph communities and generated summaries.
- [GLiNER](https://arxiv.org/abs/2311.08526) — Original research on a compact model for extracting entities of arbitrary types.
- [Virtual Graph public-preview announcement](https://neo4j.com/blog/auradb/neo4j-virtual-graph-is-now-in-public-preview/) — July 2026 announcement of public-preview access and supported lakehouse sources.

## Slides

### AI on Your Lakehouse

Neo4j’s hands-on workshop frames context as shapes that help agents answer and act across a lakehouse.

[1:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=69s) [Workshop title slide introducing context as shapes rather than queries.](https://assets.ai.engineer/sites/aiecode2025/sha256/4f17fb11f48fc65d897ac365c1736e97a6d6ef6b4a3cb75858e75a1eabd1c226.webp)

AI on Your Lakehouse

Context comes in shapes, not queries.

Give your agent the shapes it needs to answer and act accurately - across any lakehouse.

neo4j
HANDS-ON WORKSHOP · 2026
Neo4j · GraphAcademy

### Who's here

Zach Blumenfeld introduces the workshop team; Ben Squire and Ryan Knight can help attendees who run into setup trouble.

[1:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=79s) [Portraits and roles of workshop team members Zach Blumenfeld, Ben Squire, and Ryan Knight.](https://assets.ai.engineer/sites/aiecode2025/sha256/68d2a32a64c9aeb13718bf34386f598447167786eb65b4711f1f60dfca15948e.webp)

Who's here
The team running today's workshop.

Zach Blumenfeld
AI Research Engineer

Ben Squire
Senior Developer Advocate

Ryan Knight
Senior Partner Architect

Neo4j
YOUR TEAM
AI on Your Lakehouse · 2026

### Context comes in shapes, not queries

Vector search and Text2SQL each provide a slice, leaving agents without a view of what is relevant or connected. The workshop builds graph shapes from warehouse data and documents to give agents context.

[2:06](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=126s) [Workshop overview covering retrieval limitations, three graph shapes, agent context, and a portable data model.](https://assets.ai.engineer/sites/aiecode2025/sha256/95a30ae070cbbfc2cecda2670a161537aba5a7eadc00dd740323ef9ad95e7806.webp)

neo4j
TODAY YOU'LL LEARN

Context comes in shapes, not queries.

Vector search and Text2SQL each hand your agent a slice — not the view to know what's relevant, or how it connects.

01
See where they fall short
Vector search and Text2SQL, side by side.

02
Build three graph shapes
Trees, themes, and connections from your lakehouse data.

03
Give your agent context
So it can answer reliably — and act.

PORTABLE BY DESIGN
Agnostic data model — the same code runs on BigQuery, Snowflake, Databricks, or wherever your data already lives.

AI on Your Lakehouse · 2026

### Three jobs, one goal

At fictional repair chain AutoFix, Dani needs a reliable fix, Morgan wants a copilot technicians can trust, and Sam has to build it.

[3:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=188s) [AutoFix use case showing a technician and describing technician, operations, and AI engineer roles.](https://assets.ai.engineer/sites/aiecode2025/sha256/7276f839bb80cc59249e376ea53a8322c18670dd4e32457245f61dc395045c46.webp)

neo4j
THE USE CASE · AUTOFIX GROUP

Three jobs, one goal.

AutoFix Group is a national repair chain — hundreds of bays, a library of bulletins and recalls, and a warehouse of every repair ever logged. Three people share one goal: answers they can trust, the first time.

DR
Dani Reyes
Master technician
The car's in her bay. She needs the fix that works — no comeback.

MT
Morgan Tao
VP, Service Operations
Wants a copilot every tech can ask in plain language — and trust.

SO
Sam Okafor
AI engineer
Has to build that copilot — and make it reliable.

AI on Your Lakehouse · 2026

### Reaching the data isn't the problem

Vector search returns similar passages, while Text2SQL can be subtly wrong across multi-hop joins. Each gives AutoFix’s agent a slice without the context to identify relevant facts and connect them.

[4:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=241s) [Comparison of vector search over documents and Text2SQL over warehouse data, illustrating missing context and connections.](https://assets.ai.engineer/sites/aiecode2025/sha256/ed16bd4cc28c33705bca81797b68ad82832bc359e9211a90612595441569547c.webp)

neo4j
THE CHALLENGE

Reaching the data isn't the problem.

AutoFix's agent can already reach the documents and the warehouse. But the big questions — what's really going wrong, what we never wrote down, how it all connects — get answered from a single slice. Confident, and wrong.

UNSTRUCTURED · VECTOR SEARCH
Documents
Manuals, bulletins, recall notices

Right meaning, wrong shape
A pile of similar passages — not the relevant, connected answer.

STRUCTURED · TEXT2SQL
Warehouse
Vehicles, work orders, parts, procedures

Quietly wrong
Fine on a few tables — subtly wrong across many multi-hop joins.

Each hands the agent a slice — never the view to know what's truly relevant, or how the right facts connect.

A context problem, not a query problem.

AI on Your Lakehouse · 2026

### Confident, and wrong

Semantic search can miss absent documentation, top-k retrieval can miss patterns outside its sample, and Text2SQL can struggle to join look-alike tables.

[4:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=297s) [Three columns describe limitations in finding missing information, identifying broad patterns, and connecting records.](https://assets.ai.engineer/sites/aiecode2025/sha256/c8fd8c3149dfcee478cc4bdfbe125d7273d3829211888573564c3a99201355a3.webp)

neo4j
WHERE IT BREAKS

Confident, and wrong.
Reaching the data was never the hard part. Three questions where vector search and Text2SQL look right — until you check.

01
ASK WHAT'S MISSING
It searches when you need to navigate.
Semantic search only reaches what resembles your query. Whatever you'd reach by following structure — including what isn't there — stays invisible.
Fault codes with zero documentation came back as “documented, just buried.”

02
ASK FOR PATTERNS ACROSS EVERYTHING
It samples when you need all of it.
Top-k retrieval hands back the nearest few and skips the rest — blind to what it left out.
112 bulletins & recalls in — ~7 patterns out, with its own gaps flagged.

03
ASK HOW THE RECORDS RELATE
It guesses the connections.
On a big, look-alike schema, Text2SQL picks the wrong table, loops, and runs up cost — or gives up.
Hundreds of near-identical tables, no map of how they join.

Each gap is a missing shape.
Not a model problem. Not a vendor problem.

AI on Your Lakehouse · 2026

### The three shapes

A tree-like table of contents navigates documents, communities reveal unnamed themes, and paths show how warehouse tables join.

[6:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=376s) [Diagrams introduce table of contents as trees, themes as communities, and connections as paths.](https://assets.ai.engineer/sites/aiecode2025/sha256/70157245a821014c524e44caa54c391c41175fd49547ad0f86b2c76f7505ef6e.webp)

The three shapes
The foundation of everything you build today.

CONTEXT COMES IN SHAPES

TREES
Table of contents
Navigate the documents.

COMMUNITIES
Themes
Surface patterns nobody named.

PATHS
Connections
How the warehouse tables join.

neo4j
AI on Your Lakehouse · 2026

### Graph Basics

A property graph connects people and a car with named relationships. Properties describe both nodes and relationships; the expanded network shows how adding data creates interconnected paths between nodes.

[8:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=494s) [A property graph diagram labels nodes, relationships, and properties using two people and a car as examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/12dc9a6861cccd444c461e5dfcce3dabaa302f43c50713a5d169cb108853abf1.webp)

neo4j

Graph Basics

Property Graph Data Model

Node
Relationship
Property

Person
Name:Terry

Person
Name:Sam

BROTHER_OF
SISTER_OF

OWNS
Since:10/Jan/11

DRIVES
Since:10/Feb/11

Car
Brand:Volvo

[8:35](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=515s) [A property graph illustrates people, a car, relationships and properties alongside a connected network.](https://assets.ai.engineer/sites/aiecode2025/sha256/ecde6046d475e50a688aea8b39c15ee1e74af7be80fefb4b06fde54215d94df4.webp)

neo4j

Graph Basics

Property Graph Data Model

Node
Relationship
Property

Person
Name:Terry

Person
Name:Sam

BROTHER_OF
SISTER_OF

OWNS
Since:10/Jan/11

DRIVES
Since:10/Feb/11

Car
Brand:Volvo

Connected Data

### Codespace stopped

The presenter pauses to restart a stopped GitHub Codespace before continuing the workshop.

[9:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=554s) [A GitHub page reports that the Codespace is stopped.](https://assets.ai.engineer/sites/aiecode2025/sha256/ab8832ffa41aa55a8b2185f0db72f383ac964d3815d82043d425fe403bc480ae.webp)

Codespace is stopped

Interface text:
AI on Your Lakehouse: Cont...
Codespaces
Codespaces - bookish mem...
workshop-credentials - Go...
Home - Neo4j Sandbox
Neo4j Browser
AI Engineer World's Fair 20...
Ask Gemini
github.com/codespaces/[illegible]

### Workshop course page

The course page organizes the workshop around connections, document navigation, and themes. First-time participants need to enroll; the presenter’s page instead shows Continue Course and 93% completion.

[9:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=558s) [The GraphAcademy course page shows the workshop description, progress and lesson sections.](https://assets.ai.engineer/sites/aiecode2025/sha256/6c655e785b99ef5fb49abee52ddb6bb77c0bc5a53956986139345a1a1139028e.webp)

AI on Your Lakehouse: Context Comes in Shapes, Not Queries

2 hours | Developing Practitioner Skills in Neo4j

Give your agent the shapes its answers need - across documents in Neo4j and a warehouse in BigQuery

Continue Course
Favorite Course

Continue where you left off
93% completed
Next Lesson: The Context Problem →

Course Description

Your agent can reach your data but still can’t use it reliably. Vector search and Text2SQL each hand it a slice - similar passages, or a guessed SQL join - not the view to know what is truly relevant and how it connects. Answers come back confident but wrong. The problem isn’t caused by a bad model or bad query, but rather a lack of context, and context comes in shapes.

In this hands-on workshop, you give an agent three reusable shapes:
• Connections (Paths) - how structured warehouse data joins and connects
• Table of Contents (Trees & Links) - navigate documents
• Themes (Communities) - surface patterns nobody named

These shapes help answer the questions many modern stacks struggle with and get quietly wrong -

01. The Context Problem
2 Lessons
Workshop Overview
Your Environment

02. Connections - the structured shape
2 Lessons
The Connections Shape
Build the Connections Graph

Interface text:
neo4j | GraphAcademy
All Courses
GraphRAG
Certifications
Zachary Blumenfeld
graphacademy.neo4j.com/courses/workshop-lakehouse/

### Workshop Overview

The AutoFix scenario pairs repair documents with BigQuery tables. The workshop builds connections, navigation, and theme tools; the presenter scrolls to the summary and directs attendees to Your Environment.

[9:59](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=599s) [The workshop overview describes an auto-repair scenario with the course navigation menu open.](https://assets.ai.engineer/sites/aiecode2025/sha256/beb1f10383db854712733759faf4f607beb192426b857e950d193e19b6e9dc78.webp)

Lesson
Workshop Overview

Your agent can reach your data but still cannot use it reliably. In this workshop, you will fix that by building three reusable graph shapes from lakehouse data - so your agent can answer the questions that actually run the shop.

The scenario - AutoFix Group

AutoFix Group is a national auto-repair chain running on a cloud lakehouse:
- Repair manuals, service bulletins, and recall notices live as PDFs in cloud storage
- Every repair, part, and vehicle lives in BigQuery tables (vehicles, work_orders, work_order_parts, parts, procedures, dtc_codes)

[Embedded diagram: small text illegible]

Interface text:
Your work here is done!
Next Lesson: Your Environment →
AI on Your Lakehouse: Context...
The Context Problem
Workshop Overview
Your Environment
Connections - the structured shape
The Connections Shape
Build the Connections Graph
Navigate What's There - Table of Contents
The Outline Shape
Build the Outline and Search Tools
Navigate the Shape
Optional Library Practice (Optional)
Surface Themes - Communities
Patterns Nobody Named
Build the Themes Tool
Reading Themes Like an [illegible]

graphacademy.neo4j.com/courses/workshop-lakehouse/1-context-problem/1-workshop-overview/
https://graphacademy.neo4j.com/courses/workshop-lakehouse/2-connections/1-connections-shape/

[10:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=605s) [Course summary lists the connections, navigation, and theme tools built in the workshop.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d0c35721583d255dc838ace769e6f32228c4ac629b0bdb2bf212bb7ee90fbef.webp)

Lesson Completed

Summary

In this workshop, you will:
- Build the connections shape so the agent follows the warehouse joins instead of guessing them
- Build navigation tools over a document tree parsed from real PDFs
- Build theme tools with community detection - patterns nobody named
- Watch your agent answer operational and estate-level questions using the shapes.

In the next lesson, you will open your workshop environment and load the AutoFix lakehouse.

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › The Context Problem
Workshop Overview
The Context Problem
Your Environment
Was this lesson helpful? Yes No

graphacademy.neo4j.com/courses/workshop-lakehouse/1-context-problem/1-workshop-overview/
https://graphacademy.neo4j.com/courses/workshop-lakehouse/1-context-problem/2-your-environment/

### Open the workshop environment

Open the GitHub Codespace to use the preinstalled workshop tools, then connect the environment to the Neo4j sandbox and BigQuery warehouse.

[10:11](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=611s) [The environment lesson describes connecting the workshop and provides a button to open a GitHub Codespace.](https://assets.ai.engineer/sites/aiecode2025/sha256/a76d938554fdba3a419b466b369f4042459fc614fd0a3386de5d85194f1129f8.webp)

Lesson
Your Environment

You saw questions that challenged the agent in the previous lesson. You will answer them the way you would on the job: by building a service-advisor skill - the shaped tools your coding agent runs to cover, navigate, and connect the AutoFix lakehouse.

In this lesson, you will open your workshop environment and connect it to both halves of the lakehouse: the Neo4j sandbox and the BigQuery warehouse.

Open your Codespace

The workshop environment runs in a GitHub Codespace - a cloud development environment in your browser with everything preinstalled: Python, the Neo4j CLI with its agent skills, neocarta, and the Claude Code coding agent.

Open in GitHub Codespace

While it builds (a few minutes), look at what is inside:

Folder | What

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › The Context Problem
Your Environment
Your work here is done!
Next Lesson: Connections - the structured shape →
Query
neo4j$
graphacademy.neo4j.com/courses/workshop-lakehouse/1-context-problem/2-your-environment/
https://github.com/codespaces/new/zach-blumenfeld/genai-workshop-lakehouse

### Create a Codespace

The presenter starts a new Codespace for the workshop repository, using the main branch, US West region, and a 2-core machine.

[10:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=618s) [GitHub displays the Codespace creation form while the Codespace is being created.](https://assets.ai.engineer/sites/aiecode2025/sha256/dffd7c5d7abda04af07da4503a9125f7fd68a8e848bf205981b444d808fa4c91.webp)

Create a new codespace

Repository
To be cloned into your codespace
zach-blumenfeld/genai-workshop-lakehouse
Codespace usage for this repository is paid for by zach-blumenfeld.

Branch
This branch will be checked out on creation
main

Dev container configuration
Your codespace will use this configuration
AI on Your Lakehouse — AutoFix wo...

Region
Your codespace will run in the selected region
US West

Machine type
Resources for your codespace
2-core

Creating codespace

Interface text:
github.com/codespaces/new/zach-blumenfeld/genai-workshop-lakehouse
Codespaces
Type / to search
Single sign-on to see results in the Neo4j, Neo4j Examples, and Neo4j Contrib, and 6 more organizations
Single sign-on
© 2026 GitHub, Inc. Terms Privacy Security Status Community Docs Contact Manage cookies Do not share my personal information

### Workshop environment in Codespaces

The Codespace is still running its setup script. The README describes a service-advisor agent that connects a Neo4j document graph with warehouse data in BigQuery.

[13:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=789s) [Codespaces displays the workshop README, including its data overview and quick start, while environment setup runs in the terminal.](https://assets.ai.engineer/sites/aiecode2025/sha256/26bfdfb188b39be202961881f9f12393735cc45c4d429f31ed42470942491beb.webp)

AI on Your Lakehouse — workshop environment

Companion environment for the GraphAcademy workshop AI on Your Lakehouse: Context Comes in Shapes, Not Queries.

You build a service-advisor skill for AutoFix Group: a playbook + scripts that let your coding agent answer questions about the technical library and the repair history — grounded in a graph that spans the lakehouse's documents and warehouse tables.

The data comes in two halves that share keys (part numbers, trouble codes):
- Documents — repair manuals, service bulletins, and recall notices (PDFs), parsed into a Neo4j document graph (a navigable tree plus Leiden themes).
- Warehouse — vehicles, work orders, and parts in BigQuery. The rows stay in BigQuery; neocarta reads only their metadata (the foreign-key join paths) into a Neo4j connections graph, and the agent federates across the two.

Quick start

In a Codespace (recommended)

Open the repo in a GitHub Codespace — .devcontainer/post-create.sh installs everything for you: Python dependencies, the Neo4j CLI with its agent skills, and Claude Code.

→ Open in GitHub Codespace

Locally

[illegible]

Interface text:
[Preview] README.md
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES...]
.claude
.devcontainer
bigquery
corpus
databricks
docs
load
skill
solutions
sources
tools
.env.example
.gitignore
.mcp.json
CLAUDE.md
README.md
requirements.txt
run-mcp.sh

PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 1
Use Cmd/Ctrl + Shift + P -> View Creation Log to see full logs
Finishing up...
Running postCreateCommand...
  › .devcontainer/post-create.sh

CHAT
Build with Agent
AI responses may be inaccurate
Describe what to build
Auto
Local
Default Approvals
OUTLINE
TIMELINE
Codespaces: glowing funicular
main
Sign In
Layout: U.S.

### Workshop environment setup

While the environment starts, the workshop page shows an empty sandbox and asks participants to start Claude in a terminal. The connections graph comes next; the document graph is loaded in Module 3.

[13:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=807s) [Workshop page describing the empty sandbox and starting a coding agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/117d09850b7a07f903bedff8aa75f9446784c2fc1bea77d1607b06b0cc0e699c.webp)

GCP_PROJECT_ID and BIGQUERY_DATASET_ID pointing at the workshop’s read-only AutoFix dataset. You never write to it - you only read, and only its metadata crosses into the graph.

Your sandbox starts empty

Your sandbox is empty for now. You build the connections graph in the next module and load the document graph in Module 3 - each at the point you need it.

Meet your agent

Open a new bash terminal and start your coding agent in the repository root.

shell: Start claude
claude

You should be prompted for an Anthropic key. Select yes.

text: The prompt you should see

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › The Context Problem
Your Environment
Query
neo4j$
Copy
https://graphacademy.neo4j.com/courses/workshop-lakehouse/1-context-problem/2-your-environment/

### Workshop course navigation

The course menu previews the connections and table-of-contents modules. The speaker plans to start with structured data, then move to unstructured data and communities.

[13:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=820s) [Course navigation menu showing the connections and table-of-contents modules.](https://assets.ai.engineer/sites/aiecode2025/sha256/19293eefcead12482f608ba75022216247919faee379587804f71e2b17185329.webp)

AI on Your Lakehouse: Context...

The Context Problem
Workshop Overview
Your Environment

Connections - the structured shape
The Connections Shape
Build the Connections Graph

Navigate What’s There - Table of Contents
The Outline Shape
Build the Outline and Search Tools
Navigate the Shape
Optional Library Practice (Optional)

Background page, partially obscured:
[illegible]UERY_DATASET_ID pointing at the workshop’s
[illegible] you never write to it - you only read, and only its
[illegible]aph.
[illegible]s empty
[illegible]w. You build the connections graph in the next
[illegible]nt graph in Module 3 - each at the point you need
[illegible]d start your coding agent in the repository root.
[illegible] an Anthropic key. Select yes.

Query
neo4j$

Interface text:
graphacademy.neo4j.com/courses/workshop-lakehouse/1-context-problem/2-your-environment/
https://graphacademy.neo4j.com/courses/workshop-lakehouse/3-navigate-documents/
Your Environment | Gr...
[Preview] README.md
Codespaces
.env — genai-worksho...
workshop-credentials
Home - Neo4j Sandbo...
Neo4j Browser
AI Engineer World's Fa...
Ask Gemini
Copy

### Neo4j skills repository

Neo4j’s team developed these skills internally. The repository lists skills for Cypher, agent memory, Aura, and drivers; the speaker says they can be downloaded through the Neo4j CLI.

[40:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2430s) [GitHub repository listing Cypher, agent memory, Aura, and driver skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/2ec4e3b958d72f83112571525e06f12258f32cc4d3d4f474f40f628a3427dc70.webp)

neo4j-skills
Public

.claude-plugin — fix: restore skills.sh listing — inline mcp description, add n... — last month
.codex-plugin — Added versioning — last month
.github — release action for versioning — last month
evals — Add multi-agent plugin registration + rename vector-index... — 2 months ago
neo4j-agent-memory-skill — Added versioning — last month
neo4j-aura-agent-skill — Added versioning — last month
neo4j-aura-graph-analytics-skill — skill-refresh: update skills against Neo4j release notes 202... — 3 weeks ago
neo4j-aura-provisioning-skill — Added versioning — last month
neo4j-cli-tools-skill — Added versioning — last month
neo4j-cypher-skill — skill-refresh: update skills against Neo4j release notes 202... — 3 weeks ago
neo4j-document-import-skill — Added versioning — last month
neo4j-driver-dotnet-skill — Added versioning — last month
neo4j-driver-go-skill — Added versioning — last month
neo4j-driver-java-skill — Added versioning — last month
neo4j-driver-javascript-skill — Added versioning — last month
neo4j-driver-python-skill — Added versioning — last month
neo4j-gds-skill — skill-refresh: update skills against Neo4j release notes 202... — 3 weeks ago

About
Neo4j Skills for Coding and other Agents including Cypher
neo4j.com/llms.txt

gemini-cli
agent-skills
claude-code
codex-cli
claude-plugin
skill-md
skills-sh
claude-plugins-marketplace

Readme
MIT license
Activity
Custom properties
88 stars
1 watching
33 forks
Report repository
Contributors 13
Languages
Python 100.0%

Interface text:
github.com/neo4j-contrib/neo4j-skills
neo4j-contrib / neo4j-skills
Code
Issues
Pull requests 8
Agents
Actions
Projects
Security and quality
Insights
Watch 1
Fork 33
Star 88
main
Branches
Tags
Go to file
Add file
Code
github-actions[bot] — skill-refresh: update skills against Neo4j release notes ... — 6d44d31 — 3 weeks ago
85 Commits

### Neo4j Cypher skill overview

Neo4j’s Cypher skill covers query writing, patterns, search, and schema inspection for Cypher 25. The speaker says Neo4j’s internal teams created the skills and recommends loading them through the Neo4j CLI.

[40:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2448s) [GitHub directory and README describing the Cypher skill and its covered topics.](https://assets.ai.engineer/sites/aiecode2025/sha256/553b3218ffee6c643e8ed7b0aac24b93ff7c2c58a41ed4b79a471a5969ec9c0a.webp)

neo4j-skills / neo4j-cypher-skill /

Name | Last commit message | Last commit date
references | skill-refresh: update skills against Neo4j release notes 2026-06-08 (#33) | 3 weeks ago
scripts | Add file-based schema and synonym mapping to neo4j-cypher-skill (#32) | 3 weeks ago
README.md | Fix neo4j-cypher-skill: -- comments to // and correct GQL clause stat... | 2 months ago
SKILL.md | skill-refresh: update skills against Neo4j release notes 2026-06-08 (#33) | 3 weeks ago

README.md

neo4j-cypher-skill

Generates, optimizes, and validates Cypher 25 queries for Neo4j 2025.x and 2026.x.

Topics covered

Query writing — reads, writes, subqueries, batch operations, LOAD CSV, schema inspection, EXPLAIN/PROFILE validation
Patterns — MATCH, OPTIONAL MATCH, WITH, UNION, MERGE (constrained-key rules), FOREACH, UNWIND, CALL IN TRANSACTIONS
Subqueries — EXISTS {}, COUNT {}, COLLECT {}, CALL (x) { }, OPTIONAL CALL
Path expressions — Quantified Path Expressions (QPEs), match modes (DIFFERENT RELATIONSHIPS, REPEATABLE ELEMENTS), path selectors (SHORTEST 1, ALL SHORTEST)
Search — vector search (SEARCH clause 2026.01+, procedure fallback for 2025.x), fulltext (db.index.fulltext)
Schema — db.schema.visualization, SHOW INDEXES/CONSTRAINTS/PROCEDURES, apoc.meta.schema() (preferred when APOC available)

Interface text:
github.com/neo4j-contrib/neo4j-skills/tree/main/neo4j-cypher-skill
neo4j-contrib / neo4j-skills
Code | Issues | Pull requests 8 | Agents | Actions | Projects | Security and quality | Insights
Files | main | Go to file
.claude-plugin
.codex-plugin
.github
evals
neo4j-agent-memory-skill
neo4j-aura-agent-skill
neo4j-aura-graph-analytics-skill
neo4j-aura-provisioning-skill
neo4j-cli-tools-skill
neo4j-cypher-skill
  references
  scripts
  README.md
  SKILL.md
neo4j-document-import-skill
neo4j-driver-dotnet-skill
neo4j-driver-go-skill
neo4j-driver-java-skill
neo4j-driver-javascript-skill
neo4j-driver-python-skill
neo4j-gds-skill
neo4j-genai-plugin-skill
Add file
GitHub Actions[bot] | skill-refresh: update skills against Neo4j release notes 2026-06-08 (#33) | 3 weeks ago | History
https://github.com/neo4j-contrib/neo4j-skills/blob/main/neo4j-cypher-skill/SKILL.md

### Cypher skill usage guidance

The Cypher skill covers query writing and optimization; it directs search combining vector with fulltext or other ranked sources to a separate skill. The speaker recommends the Neo4j CLI for loading current skills.

[40:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2453s) [SKILL.md describes Cypher query tasks, compatibility, when to use the skill, and when to use other skills.](https://assets.ai.engineer/sites/aiecode2025/sha256/54a4f555dc420d2683e894424ae1b34e847009db53797223beebb42e3a1765a3.webp)

name: neo4j-cypher-skill

description: Generates, optimizes, and validates Cypher 25 queries for Neo4j 2025.x and 2026.x. Use when writing new Cypher queries, optimizing slow queries, graph pattern matching, vector or fulltext search, subqueries, or batch writes. Covers MATCH, MERGE, CREATE, WITH, RETURN, CALL, UNWIND, FOREACH, LOAD CSV, SEARCH, expressions, functions, indexes, and subqueries. Does NOT handle driver migration or API changes — use neo4j-migration-skill. Does NOT cover DB administration or server ops — use neo4j-cli-tools-skill.

compatibility: Neo4j >= 2025.01 (safe baseline); Cypher 25
version: 1.0.1

When to Use
• Writing, optimizing, or debugging Cypher queries
• Graph pattern matching, QPEs, variable-length paths
• Vector/fulltext search, subqueries, batch writes, LOAD CSV

When NOT to Use
• Driver migration/API changes → neo4j-migration-skill
• DB admin (users, config, backups) → neo4j-cli-tools-skill
• Hybrid search that combines vector with fulltext or other ranked sources → neo4j-vector-index-skill

GQL conformance note: LET, FINISH, FILTER, and INSERT are valid Cypher 25 clauses (introduced via GQL conformance, mostly in Neo4j 2025.06). On older versions, fall back to WITH / (omit RETURN) / WHERE / CREATE. INSERT requires &-separated multi-labels and does not support dynamic labels/types.

Pre-flight
? | Known | Unknown
<db-name>-schema.json found in project | Use it directly — skip live inspection | —
Schema (from context or live DB) | Use directly | Run Schema-First Protocol

Interface text:
github.com/neo4j-contrib/neo4j-skills/blob/main/neo4j-cypher-skill/SKILL.md
github-actions[bot] skill-refresh: update skills against Neo4j release notes 2026-06-08 (#33)
Preview | Code | Blame | 389 lines (318 loc) · 19.4 KB | Raw
Files | main | Go to file
.codex-plugin
.github
evals
neo4j-agent-memory-skill
neo4j-aura-agent-skill
neo4j-aura-graph-analytics-skill
neo4j-aura-provisioning-skill
neo4j-cli-tools-skill
neo4j-cypher-skill
  references
  scripts
  README.md
  SKILL.md
neo4j-document-import-skill
neo4j-driver-dotnet-skill
neo4j-driver-go-skill
neo4j-driver-java-skill
neo4j-driver-javascript-skill
neo4j-driver-python-skill
neo4j-gds-skill
neo4j-genai-plugin-skill
neo4j-getting-started-skill
neo4j-graphql-skill
neo4j-graphrag-skill
neo4j-import-skill
neo4j-kafka-skill

### Codespace setup and vehicle results

The speaker confirms that the Neo4j CLI is loaded into the skill and points to the command in the dev-container install file. A vehicle-results table remains visible below.

[41:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2501s) [Codespace showing the start of a setup script above a vehicle results table.](https://assets.ai.engineer/sites/aiecode2025/sha256/b6400f029aa029fefd92658d00ad7798058a29da38fefbaa30cd0d7dca5e2bea.webp)

#!/usr/bin/env bash
# Codespace setup for the AI on Your Lakehouse workshop.
# Each step is independent - one failure must not block the others.

echo "--- Python dependencies"
pip install -r requirements.txt || echo "WARN: pip install failed - run it manually"

echo "--- Claude Code"
[illegible]

a specific engine or model (it's an ignition coil, so EVs receiving it is odd),
that's a signal worth digging into. Want me to narrow to a specific complaint/DTC,

* Cooked for 2m 49s

Make | Model | Year | Engine | Vehicles
Cascadia Motors | Falcon | 2023 | 2.0T | 18
Cascadia Motors | Heron | 2020 | 2.5 | 18
Cascadia Motors | Falcon | 2022 | 1.6 | 16
Northwind Trucks | Summit | 2020 | 5.0 V8 | 14
Cascadia Motors | Kestrel | 2019 | 1.5T | 14

Interface text:
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHO...
.claude
.devcontainer
  devcontainer.json
  post-create-local.sh
  post-create.sh
bigquery
corpus
databricks
docs
load
skill
solutions
sources
tools
.env
.env.example
.gitignore
.mcp.json
CLAUDE.md
OUTLINE
TIMELINE
[Preview] README.md
.env
post-create.sh
.devcontainer > post-create.sh
PROBLEMS OUTPUT DEBUG CONSOLE TERMINAL PORTS 2
claude
* Coun...
bash
Codespaces: glowing funicular
main
Ln 1, Col 1
Spaces: 2
UTF-8
LF
Shell Script
Sign In
Layout: U.S.

### Validate the connections graph

The course asks users to check the database after neocarta builds the metadata graph. The presenter then moves from structured data to documents.

[43:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2619s) [Course page instructing users to check the database after building the metadata graph.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b255a335e951e86963e7c1d5107e64611d76e60c15a8168d87ca061664a85b4.webp)

Validate the Connections Graph

Once neocarta has built the metadata graph, click the Check Database button to verify.

Check Database

1. Independent Text2SQL benchmarks: Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation (Luo et al., arXiv:2510.24762, 2025) reports 78.57% error on queries touching four or more tables - https://arxiv.org/abs/2510.24762; Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows (arXiv:2411.07763, ICLR 2025) reports frontier models at 10-17% on real enterprise schemas - https://arxiv.org/abs/2411.07763.

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › Connections - the structured shape
Build the Connections Graph
Query
neo4j$
The Connections Shape
Navigate What’s There - Table of Contents
https://graphacademy.neo4j.com/courses/workshop-lakehouse/2-connections/2-build-connections/
https://graphacademy.neo4j.com/courses/workshop-lakehouse/3-navigate-documents/

### Navigate what’s there

The presenter moves from structured data to documents. The module builds outline and search tools from specs, then navigates the library by tool, agent, and Cypher.

[43:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2623s) [Course module overview introducing document navigation, outline and search tools, and Cypher.](https://assets.ai.engineer/sites/aiecode2025/sha256/630aae5abb17f69d65a9c18f1979db536a1253b1c33ce619d2bc9a2a9e6de51c.webp)

Navigate What’s There - Table of Contents

Your work here is done!
Next Lesson: The Outline Shape →

Search finds what looks similar; navigation follows the structure to what you actually need — the complement that reads the table of contents before flipping through every page. In this module, you will read the spec for the outline shape, derive the graph reasoning it needs, and build your skill’s navigation tools - then put them to work.

By the end of this module, you will:
• Think shape-first: read the outline spec before writing any query
• Build the outline and search tools in your skill from their specs
• Navigate the library - by tool, by agent, and by Cypher in the sandbox

Ready? Let’s go →

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries
Navigate What's There - Table of Contents
Browser address: graphacademy.neo4j.com/courses/workshop-lakehouse/3-navigate-documents/
Link preview: graphacademy.neo4j.com/courses/workshop-lakehouse/3-navigate-documents/1-documents-are-graphs/
Browser tabs: Navigate What’s T…; post-create.sh - …; Codespaces; .env — genai-work…; workshop-credent…; Home - Neo4j San…; Neo4j Browser; AI Engineer World…; neo4j-skills/neo4j…
Ask Gemini

### The outline shape

The lesson introduces a graph outline for navigating AutoFix’s document library. It contrasts this structure with fixed-size chunks, which can lose a section’s location or split a procedure.

[43:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2626s) [Lesson page explaining how a graph outline helps an agent navigate a document library.](https://assets.ai.engineer/sites/aiecode2025/sha256/510ed68a2d4c47f6262f7ad3d1799ee1dc00369545d51412e381361331ff9d09.webp)

Lesson
The Outline Shape

AutoFix has a library of documents - service manuals, technical bulletins, and recall notices - and your agent needs to find its way around them.

The reflex move is to chunk every document, embed the chunks, and search by similarity. That hands the agent a slice that reads like the query. But a slice has no address: the agent can’t tell which document it came from, which section it sits in, or what comes before and after it. And the cut itself is lossy - a fixed-size chunk boundary can fall mid-procedure, and once a fact is split across two chunks at ingest, no reranker or bigger context window puts it back together.[1]

To navigate a library, an agent needs something a pile of chunks can’t give it: structure. That structure is an outline shape stored in a graph - an index of every document in the library and how they link to each other. It is built in two steps:

• Table of contents - the outline of a single document, so the agent can read it and jump straight to the section it needs.
• Library index - how those individual documents are organized and linked

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › Navigate What’s There - Table of Contents
The Outline Shape
Your work here is done!
Next Lesson: Build the Outline and Search Tools →
Query
neo4j$
graphacademy.neo4j.com/courses/workshop-lakehouse/3-navigate-documents/1-documents-are-graphs/

### Bulletin PDF in raw text

The bulletin PDF opens as raw text, exposing a link to a manual’s misfire-diagnosis section. The source folder also contains manuals and recalls.

[44:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2656s) [A bulletin PDF appears as raw PDF text in the editor, including a link to a manual section.](https://assets.ai.engineer/sites/aiecode2025/sha256/9da9d73032894369d2165a7af3ac255d76801ec4b09ba2329ba6805114c30453.webp)

sources > pdfs > bulletins > tsb-20-501.pdf

16 >>
17 endobj
18 4 0 obj
19 <<
20 /BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding /Name /F3 /Subtype /Type1 /Type [illegible]
21 >>
22 endobj
23 5 0 obj
24 <<
25 /A <<
26 /S /URI /Type /Action /URI (doc://man-falcon-engine#misfire-diagnosis)
27 >> /Border [ 0 0 0 ] /Rect [ 356.78 595 532.95 608 ] /Subtype /Link /Type /Annot
28 >>
29 endobj
30 6 0 obj
31 <<
32 /A <<
33 /S /URI /Type /Action /URI (doc://man-falcon-engine#misfire-diagnosis)

Interface text:
EXPLORER
GENAI-WORKSHOP-LAKEHO...
corpus
databricks
docs
load
skill
solutions
sources
  pdfs
    bulletins
    manuals
    recalls
  warehouse
tools
.env
.env.example
.gitignore
.mcp.json
CLAUDE.md
README.md
OUTLINE
TIMELINE

[Preview] README.md
.env
tsb-20-501.pdf
PROBLEMS
OUTPUT
DEBUG CONSOLE
TERMINAL
PORTS 2
* Cooked for 2m 49s
Make | Model | Year | Engine | Vehicles
claude
* Coun...
bash
Codespaces: glowing funicular
main
Ln 1, Col 1
Spaces: 2
UTF-8
LF
Plain Text
Sign In
Layout: U.S.

### Recall PDF and corpus folders

The recall PDF opens as raw PDF text. The presenter shows a Markdown corpus organized into bulletins, manuals, and recalls, whose documents link to sections in other documents.

[44:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2666s) [A recall PDF is open as raw text while the explorer shows bulletin, manual, and recall folders in the corpus.](https://assets.ai.engineer/sites/aiecode2025/sha256/d72eb0fe73ddd0646216934ba1aa4b38bb1e14130cbae24743555cf649010875.webp)

%PDF-1.3
%[illegible] ReportLab Generated PDF document (opensource)
1 0 obj
<<
/F1 2 0 R /F2 3 0 R /F3 4 0 R
>>
endobj
2 0 obj
<<
/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font
>>
endobj
3 0 obj
<<
/BaseFont /Helvetica-Oblique /Encoding /WinAnsiEncoding /Name /F2 /Subtype /Type1 /Ty[illegible]
>>
endobj
4 0 obj

Interface text:
The Outline Shape
rc-2021-01.pdf
Codespaces
.env — genai-work[illegible]
workshop-credent[illegible]
Home - Neo4j San[illegible]
Neo4j Browser
AI Engineer World[illegible]
neo4j-skills/neo4j[illegible]
Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHO...
.devcontainer
  post-create-local.sh
  post-create.sh
bigquery
corpus
  bulletins
  manuals
  recalls
databricks
docs
load
skill
solutions
sources
  pdfs
  warehouse
tools
.env
.env.example
OUTLINE
TIMELINE
[Preview] README.md
.env
rc-2021-01.pdf
sources > pdfs > recalls > rc-2021-01.pdf
PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 2
* Cooked for 2m 49s
Make  Model  Year  Engine  Vehicles
claude
* Coun...
bash
Codespaces: glowing funicular
main
Ln 1, Col 1
Spaces: 2
UTF-8
LF
Plain Text
Sign In
Layout: U.S.

### Linked Markdown manual

The Falcon ABS manual lists diagnostic codes and links to a diagnostic section in the Summit ABS manual, illustrating how the simulated vehicle documents refer to one another.

[44:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2681s) [A Markdown ABS manual lists diagnostic trouble codes and links to a section in another manual.](https://assets.ai.engineer/sites/aiecode2025/sha256/3eb515c8796f5fb9f8725a1aa8505617fc39a9083b753be6e166ec63ceb3e755.webp)

# Safety and Service Precautions
Never road-test a vehicle until the EBCM has completed its self-test and the ABS warning lamp has extinguished.

# Diagnostic Trouble Codes
The Falcon EBCM stores faults that fall into wheel speed, pump and relay, and configuration or communication categories. Wheel speed sensor circuit faults are reported as C0035, C0040, C0045, and C0050, one per corner. Pump and relay faults appear as C0110 and C0265, while configuration and network faults are stored as C0561 and U0121. Cross-reference equivalent platform codes in the [Summit Abs Service Manual :: Diagnostic Trouble Codes](doc://man-summit-abs#diagnostic-trouble-codes) when diagnosing a shared chassis.

## Wheel Speed Sensor Codes
Codes C0035, C0040, C0045, and C0050 map to the left front, right front, left rear, and right rear wheel speed sensor circuits respectively. An open or shorted sensor

Interface text:
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
.devcontainer
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
manuals
man-falcon-abs.md
man-falcon-charging.md
man-falcon-engine.md
man-falcon-fuel.md
man-harrier-abs.md
man-harrier-charging.md
man-harrier-cooling.md
man-harrier-engine.md
man-heron-emissions.md
man-heron-hvac.md
man-heron-ignition.md
man-heron-suspension.md
OUTLINE
TIMELINE
[Preview] README.md
.env
corpus > manuals > man-falcon-abs.md
PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 2
Cooked for 2m 49s
Make  Model  Year  Engine  Vehicles
claude
Coun...
bash
Codespaces: glowing funicular  main  Ln 1, Col 1  Spaces: 4  UTF-8  LF  Markdown  Sign In  Layout: U.S.

[45:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2715s) [An editor displays an ABS manual section with diagnostic codes and a link to another manual.](https://assets.ai.engineer/sites/aiecode2025/sha256/95e72347c286b9e843ff3dd26bdad4d8e59dce26d264d65f06b598219e73820a.webp)

# Safety and Service Precautions
Never road-test a vehicle until the EBCM has completed its self-test and the ABS warning lamp has extinguished.

# Diagnostic Trouble Codes
The Falcon EBCM stores faults that fall into wheel speed, pump and relay, and configuration or communication categories. Wheel speed sensor circuit faults are reported as C0035, C0040, C0045, and C0050, one per corner. Pump and relay faults appear as C0110 and C0265, while configuration and network faults are stored as C0561 and U0121. Cross-reference equivalent platform codes in the [Summit Abs Service Manual :: Diagnostic Trouble Codes](doc://man-summit-abs#diagnostic-trouble-codes) when diagnosing a shared chassis.

## Wheel Speed Sensor Codes
Codes C0035, C0040, C0045, and C0050 map to the left front, right front, left rear, and right rear wheel speed sensor circuits respectively. An open or shorted sensor

Interface text:
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
.devcontainer
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
manuals
recalls
databricks
docs
load
skill
solutions
sources
pdfs
warehouse
tools
.env
.env.example
OUTLINE
TIMELINE
[Preview] README.md
.env
man-falcon-abs.md
corpus > manuals > man-falcon-abs.md
PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 2
Cooked for 2m 49s
Make  Model  Year  Engine  Vehicles
claude
Coun...
bash
Codespaces: glowing funicular  main  Ln 1, Col 1  Spaces: 4  UTF-8  LF  Markdown  Sign In  Layout: U.S.

### Library outline

AutoFix’s outline nests bulletins, documents and sections in a containment tree. An arrow points from a bulletin section to a procedure in a different manual, adding a cross-document link.

[45:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2727s) [A course page explains a library containment tree and shows the names in an outline.](https://assets.ai.engineer/sites/aiecode2025/sha256/818f875408c2e8d1ec4c0e5800e59ce112895109f305f6efa0eeda496355980c.webp)

Build the library from trees.

One document is one tree. A library is many trees. Nest them - the library contains folders, folders contain documents, documents contain sections, sections nest into sub-sections - and one containment relationship, HAS, walks the whole thing top-down.

Here is that shape: AutoFix’s entire library rendered as a single navigable outline.

NAME                                      T  URI
AutoFix Technical Library                 L  technical-library
  bulletins/                              F  technical-library/bulletin[illegible]
    Falcon 2.0T: Cylinder misfire under load  D  technical-library/bulletin[illegible]
      Condition                           S  technical-library/bulletin[illegible]
      → Misfire Diagnosis                 →  technical-library/manuals/ma[illegible]
      Cause                               S  technical-library/bulletin[illegible]

Names on the left for a human to read; full URIs on the right that the agent copies back into a tool to drill in or pull the text.

But a real library is not a stack of independent documents, and this is where a plain tree runs out. The → row in the outline above points into a different document - the misfire-diagnosis procedure in a manual. AutoFix’s documents

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › Navigate What’s There - Table of Contents
The Outline Shape
Query
neo4j$
Copy
Browser address: graphacademy.neo4j.com/courses/workshop-lakehouse/3-navigate-documents/1-documents-are-graphs/

### Document relationship graph

HAS edges organize the library into folders, documents and sections. NEXT-SECTION connects sections in order, while LINKS_TO connects the bulletin’s Condition to a manual’s Misfire Diagnosis.

[47:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2877s) [A graph links a library to folders, documents and sections, with arrows for containment, section order and a cross-document link.](https://assets.ai.engineer/sites/aiecode2025/sha256/55506f2a95dba3f083c696a87dd9194ebdd10a3d75a6f682084eaf81deeb7e01.webp)

Library

HAS → bulletins/
HAS → manuals/

bulletins/ → HAS → tsb-20-501
manuals/ → HAS → man-falcon-engine

tsb-20-501 → HAS → Condition
tsb-20-501 → HAS → Repair Procedure
man-falcon-engine → HAS → Misfire Diagnosis
Condition → NEXT-SECTION → Repair Procedure
Condition → LINKS_TO → Misfire Diagnosis

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › Navigate What’s There - Table of Contents
The Outline Shape
graphacademy.neo4j.com/courses/workshop-lakehouse/3-navigate-documents/1-documents-are-graphs/
Copy
Query
neo4j$

### URIs carry the hierarchy

Each node’s URI follows the containment tree from library to folder, file and section. A subtree can be selected by its string prefix, while child URI pointers let the agent retrieve only the sections it needs.

[51:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3078s) [A workshop page explains hierarchical document URIs and section pointers.](https://assets.ai.engineer/sites/aiecode2025/sha256/7a4bdf871a498b5bc63c4055b2ef9d086984bb76ec73539c253141cff0f4bdff.webp)

URIs carry the hierarchy

Every node has a uri - a hierarchical slug that encodes its place in the tree:

technical-library
technical-library/bulletins
technical-library/bulletins/tsb-20-501.pdf
technical-library/bulletins/tsb-20-501.pdf#condition
technical-library/manuals/man-falcon-engine.pdf#misfire-diagnosis

Because URIs sort hierarchically, any subtree is a string prefix - scoping anything to the bulletins is STARTS WITH 'technical-library/bulletins'. Section fragments carry the full heading path, so a URI is always copy-pasteable back into any tool.

One more rule: a section’s content holds only its own body text, followed by uri: pointer lines for its children. Wherever the agent sees a uri: line, deeper content exists - one more lookup away, never duplicated upward. The outline shows the shape; the agent pulls the text only for the sections it actually needs.

Full-text now, vectors later - the shape is what matters

Interface text:
graphacademy.neo4j.com/courses/workshop-lakehouse/3-navigate-documents/1-documents-are-graphs/
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › Navigate What's There - Table of Contents
The Outline Shape
Copy
Query
neo4j$

### Document themes

The document structure groups 183 documents into 13 themes without AI tagging. One cluster centers on CAN bus communication; another centers on brakes and rotors.

[1:26:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5204s) [A course page shows clusters of technical documents, including communication and brake themes.](https://assets.ai.engineer/sites/aiecode2025/sha256/2149238967e7ad59fccaf01bdfda596f2cb91dd79a5b1be22820383757e273e0.webp)

THEMES  AutoFix Technical Library  183 docs · 183 grouped into 13 themes by shared [illegible]

T1  20 docs (11%) · tightly interlinked
  top shared targets  [CAN Bus Communication Diagnosis Procedure] in 11 docs · [illegible]
  most-linked docs    Kestrel Body Service Manual ............... D  technical[illegible]
                      Vega Electrical Service Manual ............ D  technical[illegible]
                      Kestrel Electrical Service Manual ......... D  technical[illegible]
                      (+17 more docs)
  links into T5 via   Kestrel headlamp assembly condensation and … D  techni[illegible]
  links into T2 via   Vega Electrical Service Manual ............. D  technical[illegible]
  links into T10 via  Ridgeline HD Body Service Manual ........... D  technical[illegible]

T2  16 docs (9%) · tightly interlinked
  top shared targets  [Front Brake Pad and Rotor Service Procedure] in 8 docs · [illegible]
  most-linked docs    Kestrel Brakes Service Manual ............. D  technical[illegible]
                      Vega Brakes Service Manual ................ D  technical[illegible]
                      Marlin EV Brakes Service Manual ........... D  technical[illegible]
                      (+13 more docs)
  links into T4 via   Marlin EV Brakes Service Manual ........... D  technical[illegible]
  links into T5 via   Kestrel Brakes Service Manual ............ D  technical[illegible]

T3  16 docs (9%) · tightly interlinked
  top shared targets  [Fuel Pressure and Sensor Faults] in 13 docs · [High-Press[illegible]

Interface text:
graphacademy.neo4j.com/courses/workshop-lakehouse/4-surface-themes/1-communities/
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › Surface Themes - Communities
Patterns Nobody Named
Copy
Query
neo4j$
MATCH p=()-[:LINKS_TO]->() RETURN p LIMI...
Graph  Table
Key  Value
name  "network-and-harness-sub-system-service/can-bus-communication-diagnosis-procedure"
sort_pos  14
uri  "technical-library/manuals/man-vega-electrical.pdf#network-and-harness-sub-system-service/can-bus-communicati[illegible]
1000 records returned

### Community detection concepts

Leiden assigns community IDs, while conductance describes how few links leave a cluster. Shared cross-references make clusters correspond to repair themes; the next challenge projects the document graph.

[1:27:04](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5224s) [A course page explains Leiden, conductance and why shared links form themes.](https://assets.ai.engineer/sites/aiecode2025/sha256/51d8a03f07f878b9426152cf3a864a9b3cb18f925764fcf6ce24825e9fc9b90f.webp)

externally

Leiden - the Graph Data Science (GDS) algorithm you will run to assign community IDs
Conductance - a metric for how few of a cluster’s links escape to other themes; it backs the cohesion word (like tightly interlinked) shown per theme
Why it works here - shared cross-reference links make clusters correspond to real repair themes, not noise

In the next challenge, you will project the document graph and run Leiden yourself.

Surface Themes - Communities
Build the Themes Tool
Was this lesson helpful? Yes No

Interface text:
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › Surface Themes - Communities
Patterns Nobody Named
Query
neo4j$
MATCH p=()-[:LINKS_TO]->() RETURN p LIMI...
Graph  Table
Key  Value
name  "network-and-harness-sub-system-service/can-bus-communication-diagnosis-procedure"
sort_pos  14
uri  "technical-library/manuals/man-vega-electrical.pdf#network-and-harness-sub-system-service/can-bus-communicati[illegible]"
1000 records returned

graphacademy.neo4j.com/courses/workshop-lakehouse/4-surface-themes/1-communities/
https://graphacademy.neo4j.com/courses/workshop-lakehouse/4-surface-themes/2-detect-themes/
Browser tabs: Patterns Nobody N…; search.py — genai…; Codespaces; .env — genai-work…; workshop-credent…; Home - Neo4j San…; Neo4j Browser; AI Engineer World; neo4j-skills/neo4j…; Ask Gemini

### Themes projection specification

The projection lifts section links to their owning documents, producing an undirected graph weighted by shared links. The presenter prepares to use the Cypher and GDS skills to build it.

[1:27:20](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5240s) [A course page describes an undirected, weighted document graph and shows a projection prompt.](https://assets.ai.engineer/sites/aiecode2025/sha256/ebf3eab92a6643586b651c577371048d0c613ae064929ae19b8eb9c66b4bc2f6.webp)

separate the document path from the section anchor. For example, technical-library/manuals/falcon-electrical#can-bus-diagnosis belongs to the document technical-library/manuals/falcon-electrical. Splitting on [illegible] and taking the first element is all the lookup needs:

Example fragment - <code>s</code> is a Section node already matched in the outer query
// s is a :Section node already in scope
MATCH (d:Document {uri: split(s.uri, '#')[0]})

Hand the projection spec to your agent and review the Cypher it generates. The query takes every LINKS_TO relationship between sections and lifts both endpoints to their owning Document - the same collapse you just saw with the URI split. The resulting graph is undirected (only the connection matters, not which document linked first) and weighted by how many links two documents share - the more shared targets, the stronger the edge between them. Same loop as the navigation tools - drive it from the spec. In this case, we also include the neo4j-gds-skill as Leiden is a Graph Data Science (GDS) algorithm.

Wire the themes projection from its spec
Use the neo4j-cypher-skill, neo4j-gds-skill, and the spec in docs/theme-format.md t[illegible]
anything in solutions/ for this exercise.

Interface text:
graphacademy.neo4j.com/courses/workshop-lakehouse/4-surface-themes/2-detect-themes/
AI on Your Lakehouse: Context Comes in Shapes, Not Queries › Surface Themes - Communities
Build the Themes Tool
Query
neo4j$
text: Copy
text: Copy

### Full-text search query

The search query uses a full-text index, optionally filters matches by URI prefix, and returns 140-character previews ranked by score.

[1:27:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5257s) [Editor showing a full-text search query with optional URI filtering and results ordered by score.](https://assets.ai.engineer/sites/aiecode2025/sha256/81625005795dd5049a00637e289e07302717636048d07644f490ced68e877125.webp)

// [illegible] BUILD FROM SPEC [illegible]
// Call db.index.fulltext.queryNodes('content_search', $lucene), keep rows
// where $under IS NULL OR node.uri STARTS WITH $under (the index ranks,
// the graph scopes), and return: label ('Document'|'Section'), uri,
// displayName, a 140-char content preview, score - top $k by score.

CALL db.index.fulltext.queryNodes('content_search', $lucene) YIELD node, score
WHERE $under IS NULL OR node.uri STARTS WITH $under
RETURN CASE WHEN node:Document THEN 'Document' ELSE 'Section' END AS label,
       node.uri AS uri,
       node.displayName AS displayName,
       left(node.content, 140) AS preview,
       score
ORDER BY score DESC
LIMIT $k
// [illegible]
"""

Interface text:
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHO...
.claude
skills / autofix-service-advisor
scripts
__pycache__
bq.py
db.py
outline.py M
run_sql.py
search.py M
themes.py
SKILL.md
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
OUTLINE
TIMELINE
[Preview] README.md
.env
search.py M
.claude > skills > autofix-service-advisor > scripts > search.py
PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 2
accept edits on (shift+tab to cycle) · ← for agents
In search.py
claude
bash
Codespaces: glowing funicular
main*
Ln 36, Col 45
Spaces: 4
UTF-8
LF
Python
Sign In
Python 3.12.13.final.0
Layout: U.S.

### Theme projection specification

The themes script specifies an in-memory graph projection: section links become weighted pairs of documents, with self-pairs removed. The projection query is still a placeholder.

[1:27:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5263s) [Editor showing instructions for projecting document links into a weighted graph and a placeholder query.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4550551449b485d45b8da3c7795dd51ff1a2dcf71f348b04bbc16a92b7509cc.webp)

from db import query

GRAPH = "autofix-themes"

PROJECT = """
// ================================================== BUILD FROM SPEC =====
// docs/theme-format.md "The reasoning to derive": project a doc-level graph
// from the cross-reference links (no Part/DTC glue nodes - they do not exist).
//   - match (s:Section)-[:LINKS_TO]->(tgt)
//   - collapse BOTH ends to their owning Document via split(uri, '#')[0]
//   - drop self-pairs, count(*) per pair AS weight
//   - RETURN gds.graph.project($graph, d, t,
//       { relationshipProperties: { weight: weight } },
//       { undirectedRelationshipTypes: ['*'] })
RETURN 0   // <-- replace
// ========================================================================
"""

LEIDEN = """

Interface text:
Build the Themes; themes.py; Codespaces; .env; workshop-credent...; Home - Neo4j San...; Neo4j Browser; AI Engineer World; neo4j-skills/neo4j...; Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHO...
.claude
skills / autofix-service-advisor
scripts
__pycache__
bq.py
db.py
outline.py M
run_sql.py
search.py M
themes.py
SKILL.md
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
OUTLINE
TIMELINE
[Preview] README.md; .env; search.py M; themes.py
.claude > skills > autofix-service-advisor > scripts > themes.py > ...
PROBLEMS; OUTPUT; DEBUG CONSOLE; TERMINAL; PORTS 2
accept edits on (shift+tab to cycle) · ← for agents
In search.py
claude; Coun...; bash
Codespaces: glowing funicular; main*; Ln 1, Col 1; Spaces: 4; UTF-8; LF; Python; Sign In; Python 3.12.13.final.0; Layout: U.S.

### Leiden community detection

The script leaves graph projection as a placeholder, then calls Leiden with relationship weights to assign theme IDs and return the community count.

[1:27:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5270s) [Editor showing a Leiden community detection call below the projection placeholder.](https://assets.ai.engineer/sites/aiecode2025/sha256/8fdf9cd3763a22db369928ff77fe52e584cde90616daccd0edc7fe29ac371be0.webp)

// - match (s:Section)-[:LINKS_TO]->(tgt)
// - collapse BOTH ends to their owning Document via split(uri, '#')[0]
// - drop self-pairs, count(*) per pair AS weight
// - RETURN gds.graph.project($graph, d, t,
//     { relationshipProperties: { weight: weight } },
//     { undirectedRelationshipTypes: ['*'] })
RETURN 0   // <-- replace
// ==========================================================================
"""

LEIDEN = """
CALL gds.leiden.mutate($graph, {
  mutateProperty: 'themeId', relationshipWeightProperty: 'weight',
  gamma: $gamma, randomSeed: 42, concurrency: 1
}) YIELD communityCount RETURN communityCount
"""

CONDUCTANCE = """
CALL gds.conductance.stream($graph, {

Interface text:
Build the Themes; themes.py; Codespaces; .env; workshop-credent...; Home - Neo4j San...; Neo4j Browser; AI Engineer World...; neo4j-skills/neo4j...; Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHO...
.claude
skills / autofix-service-advisor
scripts
__pycache__
bq.py
db.py
outline.py M
run_sql.py
search.py M
themes.py
SKILL.md
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
OUTLINE
TIMELINE
[Preview] README.md; .env; search.py M; themes.py
.claude > skills > autofix-service-advisor > scripts > themes.py > ...
PROBLEMS; OUTPUT; DEBUG CONSOLE; TERMINAL; PORTS 2
accept edits on (shift+tab to cycle) · ← for agents
In search.py
claude; Coun...; bash
Codespaces: glowing funicular; main*; Ln 1, Col 1; Spaces: 4; UTF-8; LF; Python; Sign In; Python 3.12.13.final.0; Layout: U.S.

### Projection and Leiden code

The projection placeholder describes how to turn document cross-references into weighted links. The Leiden call below uses those weights to assign theme IDs.

[1:27:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5276s) [Editor showing the projection instructions and Leiden call above a terminal with pasted text.](https://assets.ai.engineer/sites/aiecode2025/sha256/f35658d6ecce9cc50c44499fa5e986404aa4effb3ac9a2944db1868e81b6dabf.webp)

// BUILD FROM SPEC
// docs/theme-format.md "The reasoning to derive": project a doc-level graph
// from the cross-reference links (no Part/DTC glue nodes – they do not exist).
//   - match (s:Section)-[:LINKS_TO]->(tgt)
//   - collapse BOTH ends to their owning Document via split(uri, '#')[0]
//   - drop self-pairs, count(*) per pair AS weight
//   - RETURN gds.graph.project($graph, d, t,
//       { relationshipProperties: { weight: weight } },
//       { undirectedRelationshipTypes: ['*'] })
RETURN 0   // <-- replace
"""

LEIDEN = """
CALL gds.leiden.mutate($graph, {
  mutateProperty: 'themeId', relationshipWeightProperty: 'weight',
  gamma: $gamma, randomSeed: 42, concurrency: 1
}) YIELD communityCount RETURN communityCount
"""

Interface text:
Build the Themes
[Preview] README.md
.env
search.py M
themes.py
.claude > skills > autofix-service-advisor > scripts > themes.py
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES...]
.claude
skills / autofix-service-advisor
scripts
__pycache__
bq.py
db.py
outline.py M
run_sql.py
search.py M
themes.py
SKILL.md
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
OUTLINE
TIMELINE
PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 2
[Pasted text #1 +1 lines]
paste again to expand
In search.py
claude
Coun...
bash
Codespaces: glowing funicular
main*
Ln 1, Col 1  Spaces: 4  UTF-8  LF  Python
Sign In
Python 3.12.13.final.0
Layout: U.S.

### Requesting the projection query

The prompt asks Claude to use the Cypher and GDS skills with the theme spec to build a document-level projection. The outline collapses section links to documents and weights each pair by link count.

[1:28:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5296s) [Terminal showing a request to build the projection query from the specification, with the agent beginning to read files.](https://assets.ai.engineer/sites/aiecode2025/sha256/99f02d31119e950a9822eed264c03043cdb1264025be5dc623de7f38c196a11c.webp)

// ==================== BUILD FROM SPEC =====
// docs/theme-format.md "The reasoning to derive": project a doc-level graph
// from the cross-reference links (no Part/DTC glue nodes — they do not exist).
// - match (s:Section)-[:LINKS_TO]->(tgt)
// - collapse BOTH ends to their owning Document via split(uri, '#')[0]
// - drop self-pairs, count(*) per pair AS weight
// - RETURN gds.graph.project($graph, d, t,
//     { relationshipProperties: { weight: weight } },
//     { undirectedRelationshipTypes: ['*'] })

tested and working. Next is the themes.py query (Leiden community detection over document cross-references).

Use the neo4j-cypher-skill, neo4j-gds-skill, and the spec in docs/theme-format.md to fill in the BUILD FROM SPEC projection query in skill/scripts/themes.py. Do not reference anything in solutions/ for this exercise.

I'll read the spec and the current themes.py.

Reading 2 files… (ctrl+o to expand)
└ skill/scripts/themes.py

Seasoning… (7s · ↓ 165 tokens · thinking)
└ Tip: Use /memory to view and manage Claude memory

Interface text:
Build the Themes
Codespaces
Neo4j Browser
AI Engineer World
Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
.claude
skills / autofix-service-advisor
scripts
__pycache__
bq.py
db.py
outline.py
run_sql.py
search.py
themes.py
SKILL.md
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
OUTLINE
TIMELINE
[Preview] README.md
.env
search.py M
themes.py
.claude > skills > autofix-service-advisor > scripts > themes.py
PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 2
claude
Coun…
bash
Codespaces: glowing funicular
main*
Ln 1, Col 1
Spaces: 4
UTF-8
LF
Python
Sign In
Python 3.12.13.final.0
Layout: U.S.

### Building the theme projection query

The presenter asks Claude to build a graph projection using the theme-format spec and Neo4j skills. A GDS skill permission prompt appears; the intended projection aggregates section links at the document level.

[1:28:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5303s) [Editor showing the theme-format specification above the agent's projection request.](https://assets.ai.engineer/sites/aiecode2025/sha256/30b83a606bb83a14621df991a864f70797b00159995c7c5cc54417ec020d374b.webp)

# Theme format — the themes shape

**This is the spec you build `skill/scripts/themes.py` against.** It defines the evidence view an agent reasons over. The tool **never names a theme** — it ships evidence (shared targets, member titles); the agent does the naming.

## The shape

```

tested and working. Next is the themes.py query (Leiden community detection over document cross-references).

Use the neo4j-cypher-skill, neo4j-gds-skill, and the spec in docs/theme-format.md to fill in the BUILD FROM SPEC projection query in skill/scripts/themes.py. Do not reference anything in solutions/ for this exercise.

I'll read the spec and the current themes.py.

Reading 2 files… (ctrl+o to expand)
  └ skill/scripts/themes.py

Seasoning… (13s · ↓ 165 tokens · thinking)
  └ Tip: Use /memory to view and manage Claude memory

Interface text:
Build the Themes; theme-format.md; Codespaces; .env — genai-work…; workshop-credent…; Home - Neo4j San…; Neo4j Browser; AI Engineer World…; neo4j-skills/neo4j…; Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHO…
.claude
  settings.local.json
.devcontainer
  devcontainer.json
  post-create-local.sh
  post-create.sh
bigquery
corpus
  bulletins
  manuals
  recalls
databricks
docs
  outline-format.md
  theme-format.md
load
  build_connections.py
  load_documents.py
  parse_corpus.py
OUTLINE
TIMELINE
[Preview] README.md; .env; search.py M; theme-format.md
docs > theme-format.md > # Theme format — the themes shape
PROBLEMS; OUTPUT; DEBUG CONSOLE; TERMINAL; PORTS 2
claude; Coun…; bash
Codespaces: glowing funicular; main*; Ln 1, Col 1; Spaces: 2; UTF-8; LF; Markdown; Sign In; Layout: U.S.

[1:28:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5317s) [A coding assistant asks permission to use a Neo4j Graph Data Science skill.](https://assets.ai.engineer/sites/aiecode2025/sha256/e93e6b923db329952198c4507cc3507e18adac1ce82a970613c5ac358ef7efde.webp)

# Theme format — the themes shape

## The shape

```
THEMES  AutoFix Technical Library  183 docs · 183 grouped into 13 themes by shared

Claude may use instructions, code, or files from this Skill.

Neo4j Graph Data Science (GDS) embedded plugin via Python client or Cypher — covers GraphDataScience, gds.v2 plugin endpoints, gds.version, native projection, Cypher projection, graph catalog operations, stream/stats/mutate/write modes, memory estimation, PageRank, Louvain, WCC, FastRP, KNN, Node Similarity, ML pipelines, and cleanup. Use for Aura Pro, self-managed, local, or offline Neo4j DBMS with the GDS plugin installed. Does NOT cover Aura Graph Analytics GDS Sessions, AuraGraphDataScience, GdsSessions, gds.graph.project.remote, or AuraDB Cypher API projection/session management — use neo4j-aura-graph-analytics-skill. Does NOT handle Cypher authoring — use neo4j-cypher-skill. Does NOT cover driver setup — use neo4j-driver-python-skill or other driver skill.

Do you want to proceed?
1. Yes
2. Yes, and don't ask again for neo4j-gds-skill in /workspaces/genai-workshop-lakehouse
3. No

Esc to cancel · Tab to amend

Interface text:
Build the Themes; theme-format.md; Codespaces; .env — genai-work…; workshop-credent…; Home - Neo4j San…; Neo4j Browser; AI Engineer World…; neo4j-skills/neo4j…; Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES…
.claude
  settings.local.json
.devcontainer
  devcontainer.json
  post-create-local.sh
  post-create.sh
bigquery
corpus
  bulletins
  manuals
  recalls
databricks
docs
  outline-format.md
  theme-format.md
load
  build_connections.py
  load_documents.py
  parse_corpus.py
OUTLINE
TIMELINE
[Preview] README.md; .env; search.py M; theme-format.md
docs > theme-format.md > # Theme format — the themes shape
PROBLEMS; OUTPUT; DEBUG CONSOLE; TERMINAL; PORTS 2
claude; Coun…; bash
Codespaces: glowing funicular; main*; Ln 1, Col 1; Spaces: 2; UTF-8; LF; Markdown; Sign In; Layout: U.S.

### Projection signature confirmed

Claude checks the GDS projection signature before editing. The speaker explains that the projection aggregates links between sections to the document level, making document communities easier to identify.

[1:28:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5332s) [The assistant reports the graph projection signature and identifies the configuration arguments.](https://assets.ai.engineer/sites/aiecode2025/sha256/2e1c458c60c889944dd0f8c974178c2d85cb2e99f81a97651e97bdfefefca7eb.webp)

# Theme format — the themes shape

## The shape

```
THEMES  AutoFix Technical Library   183 docs · 183 grouped into 13 themes by shared

the GDS skill to confirm the exact gds.graph.project(...) signature before writing.

Skill(neo4j-gds-skill)
  Successfully loaded skill · 2 tools allowed

Confirmed. The skill verifies the Cypher-projection aggregation form: RETURN gds.graph.project(graphName, source, target, dataConfig, configuration) where relationshipProperties goes in the 4th (dataConfig) arg and undirectedRelationshipTypes: ['*'] goes in the 5th (configuration) arg. That matches the spec hint exactly. Let me fill it in.

Update(skill/scripts/themes.py)

Seasoning… (32s · ↓ 2.1k tokens)
  Tip: Use /btw to ask a quick side question without interrupting Claude's current work

Interface text:
Build the Themes; theme-format.md; Codespaces; .env — genai-work; workshop-credent[illegible]; Home - Neo4j San[illegible]; Neo4j Browser; AI Engineer World; neo4j-skills/neo4j[illegible]; Ask Gemini
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES...]
.claude
  settings.local.json
.devcontainer
  devcontainer.json
  post-create-local.sh
  post-create.sh
bigquery
corpus
  bulletins
  manuals
  recalls
databricks
docs
  outline-format.md
  theme-format.md
load
  build_connections.py
  load_documents.py
  parse_corpus.py
OUTLINE
TIMELINE
[Preview] README.md; .env; search.py M; theme-format.md
docs > theme-format.md > # Theme format — the themes shape
PROBLEMS; OUTPUT; DEBUG CONSOLE; TERMINAL; PORTS 2
claude; Coun...; bash
accept edits on (shift+tab to cycle) · esc to interrupt · ← for agents
Codespaces: glowing funicular; main*; Ln 1, Col 1; Spaces: 2; UTF-8; LF; Markdown; Sign In; Layout: U.S.

### Reviewing the proposed edit

Claude proposes a graph projection edit and asks for approval. The speaker explains that section links are aggregated to the document level, making document connections clearer and communities easier to find.

[1:28:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5335s) [A code diff is open while the assistant asks permission to edit the themes script.](https://assets.ai.engineer/sites/aiecode2025/sha256/916bbb7a0a508b53eb1198a1b62397db6c692c3e8dbdc4321c0587b1c839c0f2.webp)

// { undirectedRelationshipTypes: ['*'] })
- RETURN 0  // <-- replace
+ MATCH (s:Section)-[:LINKS_TO]->(tgt)
+ MATCH (d:Document {uri: split(s.uri, '#')[0]})
+ MATCH (t:Document {uri: split(tgt.uri, '#')[0]}) [partially obscured]

gds.graph.project(graphName, source, target, dataConfig, configuration) where relationshipProperties goes in the 4th (dataConfig) arg and undirectedRelationshipTypes: ['*'] goes in the 5th (configuration) arg. That matches the spec hint exactly. Let me fill it in.

Update(skill/scripts/themes.py)

Opened changes in Visual Studio Code
skill/scripts/themes.py
Symlink target: /workspaces/genai-workshop-lakehouse/.claude/skills/autofix-service-advisor/scripts/themes.py

Save file to continue...

Do you want to make this edit to themes.py?
1. Yes
2. Yes, allow all edits during this session (shift+tab)
3. No

Esc to cancel

Interface text:
Build the Themes
[Claude Code]
Codespaces
.env — genai-work
workshop-credent[illegible]
Home - Neo4j San[illegible]
Neo4j Browser
AI Engineer World
neo4j-skills/neo4j[illegible]
Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES...]
.claude
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
manuals
recalls
databricks
docs
outline-format.md
theme-format.md
load
build_connections.py
load_documents.py
parse_corpus.py
OUTLINE
TIMELINE
[Claude Code] themes.py (fcb026)
workspaces > genai-workshop-lakehouse > skill > scripts > themes.py
PROBLEMS
OUTPUT
DEBUG CONSOLE
TERMINAL
PORTS 2
claude
Coun...
bash
Codespaces: glowing funicular
main*
Ln 37, Col 1
Spaces: 4
UTF-8
LF
Python
Sign In
Layout: U.S.

### Building the theme projection

The projection rolls section links up to document pairs and counts them as weights. The script reports 183 documents grouped into 13 themes by shared cross-references, with none ungrouped.

[1:29:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5341s) [The terminal displays added query lines that project weighted document relationships.](https://assets.ai.engineer/sites/aiecode2025/sha256/2b398824653e65b0629fb006c91e9c21a8d73e22f389a32697ab59319a8b5ac5.webp)

# Theme format — the themes shape

## The shape

```
THEMES  AutoFix Technical Library  183 docs · 183 grouped into 13 themes by shared

39 +MATCH (t:Document {uri: split(tgt.uri, '#')[0]})
40 +WHERE d <> t
41 +WITH d, t, count(*) AS weight
42 +RETURN gds.graph.project($graph, d, t,
43 +  { relationshipProperties: { weight: weight } },
44 +  { undirectedRelationshipTypes: ['*'] })
45  // ================================================================
46  """

Seasoning… (35s · ↑ 2.6k tokens)
Tip: Use /btw to ask a quick side question without interrupting Claude's current work

Interface text:
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES...]
.claude
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
manuals
recalls
databricks
docs
outline-format.md
theme-format.md
load
build_connections.py
load_documents.py
parse_corpus.py
OUTLINE
TIMELINE
[Preview] README.md
.env
search.py M
theme-format.md
PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 2
claude
Coun...
bash
accept edits on (shift+tab to cycle) · esc to interrupt · ← for agents
high · /effort
In themes.py
Codespaces: glowing funicular
main*
Ln 1, Col 1  Spaces: 2  UTF-8  LF  Markdown  Sign In  Layout: U.S.

[1:29:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5347s) [A terminal reports 183 documents grouped into 13 themes.](https://assets.ai.engineer/sites/aiecode2025/sha256/934930213125fa1bf2c1584104de7d97e25f89fe787916abae4561648fca1490.webp)

# Theme format — the themes shape

## The shape

```
THEMES  AutoFix Technical Library  183 docs · 183 grouped into 13 themes by shared

44 +  { undirectedRelationshipTypes: ['*'] })
45  // ================================================================
46  """

Bash(python skill/scripts/themes.py 2>&1 | head -50)
  THEMES  AutoFix Technical Library  183 docs · 183 grouped into 13 themes by
  shared cross-references · 0 ungrouped
  … +61 lines (ctrl+o to expand)

Seasoning… (44s · ↓ 2.7k tokens · thinking)
Tip: Use /btw to ask a quick side question without interrupting Claude's current work

Interface text:
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES...]
.claude
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
manuals
recalls
databricks
docs
outline-format.md
theme-format.md
load
build_connections.py
load_documents.py
parse_corpus.py
OUTLINE
TIMELINE
[Preview] README.md
.env
search.py M
theme-format.md
PROBLEMS  OUTPUT  DEBUG CONSOLE  TERMINAL  PORTS 2
claude
Coun...
bash
accept edits on (shift+tab to cycle) · esc to interrupt · ← for agents
In themes.py
Codespaces: glowing funicular
main*
Ln 1, Col 1  Spaces: 2  UTF-8  LF  Markdown  Sign In  Layout: U.S.

### Testing theme granularity

The theme script groups 183 documents into 14 themes at gamma 2.0, up from 13. The agent raises the minimum from 8 to 15 documents to test how smaller themes fold into the ungrouped set.

[1:29:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5366s) [A terminal explains that a higher gamma produced 14 themes instead of 13 and starts another minimum-size test.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e787ed5e6abbe28235a72a10e437a4454d68728c115c933541baba37db92e2d.webp)

# Theme format — the themes shape

## The shape

```
THEMES  AutoFix Technical Library   183 docs · 183 grouped into 13 themes by shared

THEMES  AutoFix Technical Library   183 docs · 183 grouped into 14 themes by shared cross-references · 0 ungrouped
… +3 lines (ctrl+o to expand)

--gamma 2.0 produces finer themes (14 vs 13), confirming the projected weighted graph feeds Leiden's granularity dial. The --min-docs 8 fold didn't trigger because every theme already has ≥8 docs. Let me push the floor higher to actually exercise the fold-to-ungrouped reconciliation.

Bash(python skill/scripts/themes.py --min-docs 15 2>&1 | head -1)
  Running…

Interface text:
Build the Themes; theme-format.md; Codespaces; .env — genai-work; workshop-credent…; Home - Neo4j San…; Neo4j Browser; AI Engineer World…; neo4j-skills/neo4j…; Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES…]
.claude
  settings.local.json
.devcontainer
  devcontainer.json
  post-create-local.sh
  post-create.sh
bigquery
corpus
  bulletins
  manuals
  recalls
databricks
docs
  outline-format.md
  theme-format.md
load
  build_connections.py
  load_documents.py
  parse_corpus.py
OUTLINE
TIMELINE
[Preview] README.md; .env; search.py M; theme-format.md
docs > theme-format.md > # Theme format — the themes shape
PROBLEMS; OUTPUT; DEBUG CONSOLE; TERMINAL; PORTS 2
claude; Coun…; bash
Seasoning… (1m 0s · ↓ 3.6k tokens)
Tip: Use /btw to ask a quick side question without interrupting Claude's current work
accept edits on (shift+tab to cycle) · esc to interrupt · ← for agents
In themes.py
Codespaces: glowing funicular; main*; Ln 1, Col 1; Spaces: 2; UTF-8; LF; Markdown; Sign In; Layout: U.S.

### Technical-document search results

The terminal lists ranked excerpts from ignition-coil recalls as the presenter prepares to run the edited theme script in a new terminal.

[1:29:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5389s) [A terminal displays ranked excerpts from vehicle service bulletins and recalls.](https://assets.ai.engineer/sites/aiecode2025/sha256/a36a73c7c7cfa578c9a5b01ec4914f2aa16c4d0a693ce3c0540277ac02a8cdb2.webp)

# Theme format — the themes shape
## The shape

```
THEMES  AutoFix Technical Library  183 docs · 183 grouped into 13 themes by shared

sitation, or stumble under load accompanied
4.983  S  technical-library/bulletins/tsb-27-101.pdf#condition
1.714  S  technical-library/recalls/rc-2023-02.pdf#defect-description
       Defect Description: On affected Heron 2.5 vehicles, the application-specific ignition coil IC-2044-A may develop an internal short
1.526  S  technical-library/recalls/rc-2021-01.pdf#defect-description
       Defect Description: Affected vehicles were built with ignition coil IC-2042-A whose electrical connector can overheat under sustai
1.526  S  technical-library/recalls/rc-2022-07.pdf#defect-description
       Defect Description: A subset of Heron 2.5 engines were assembled with ignition coils (part IC-2044-A) whose secondary winding insu
1.402  D  technical-library/recalls/rc-2021-01.pdf
       "Safety Recall – Ignition Coil Connector Overheating":
1.198  D  technical-library/recalls/rc-2021-02.pdf
       Safety Recall – Ignition Coil Internal Arcing and Stall:
1.149  S  technical-library/recalls/rc-2022-07.pdf#remedy
       Remedy: Dealers will replace all ignition coils carrying part number IC-2044-A with the superseding coil at no cost to
1.149  S  technical-library/recalls/rc-2021-02.pdf#remedy
       Remedy: Dealers will replace all affected IC-2044-A ignition coils with a redesigned coil featuring improved secondary
@zach-blumenfeld ➜ /workspaces/genai-workshop-lakehouse (main) $

Interface text:
Build the Themes · theme-format.md · Codespaces · .env — genai-work · workshop-credent… · Home - Neo4j San… · Neo4j Browser · AI Engineer World · neo4j-skills/neo4j… · Ask Gemini
glowing-funicular-945p95g9p7g3vqw.github.dev
genai-workshop-lakehouse [Codespaces: glowing funicular]
EXPLORER
GENAI-WORKSHOP-LAKEHOUSE [CODESPACES…
.claude
settings.local.json
.devcontainer
devcontainer.json
post-create-local.sh
post-create.sh
bigquery
corpus
bulletins
manuals
recalls
databricks
docs
outline-format.md
theme-format.md
load
build_connections.py
load_documents.py
parse_corpus.py
OUTLINE
TIMELINE
[Preview] README.md · .env · search.py M · theme-format.md
PROBLEMS · OUTPUT · DEBUG CONSOLE · TERMINAL · PORTS 2
claude · Coun… · bash
Codespaces: glowing funicular · main* · Ln 1, Col 1 · Spaces: 2 · UTF-8 · LF · Markdown · Sign In · Layout: U.S.

## Full transcript

[0:00](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=0s) [upbeat music] Hello, everyone. Welcome today to get started on AIE.

[0:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=18s) Um, so right here, uh, there's some steps for getting started. I went over this around ten minutes ago. Um, but basically, our workshop that we're gonna take today is driven by a website called GraphAcademy.

[0:32](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=32s) And if you go to that first QR code there to the left, uh, that'll take you there. You have to enroll with your email. Um, and then if you go down to set up your environment, uh, there's a Codespaces there with everything set up, and you can get that rolling now.

[0:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=47s) It'll take maybe about five minutes or so. Um, and you can grab your credentials then as well. There's, um, an Anthropic API key. Uh, if you have your own Claude Code key, um, or your, your own subscription, please feel free to use that.

[1:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=62s) Uh, otherwise, we provided one for you. Um, and then there's another key there for reading from BigQuery tables. So with that in mind, uh, we'll go ahead and get started with AI on your lakehouse, um, which is about context coming in shapes and not necessarily queries.

[1:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=81s) So today, your team, um, will be myself. My name is Zach Blumenfeld. I am an AI Research Engineer at Neo4j. Uh, we also have Ben Squire over there in the back, um, who is our Senior Developer Advocate, um, as well as Ryan here in the center, um, who is our Partner Architect, helping customers get this stuff up

[1:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=103s) and running. So as you have questions, um, I don't have a mic set up for you, but, uh, go ahead and raise your hand. I'll sort of... I'll take time around every ten, fifteen minutes during natural breaks, um, and, and I can take some questions, but also flag Ben and Ryan as well if you're going through some

[2:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=122s) steps and you're having some trouble setting things up, um, and they can help get you unblocked.

[2:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=128s) Um, so what we're gonna talk about today, um, is really about when you start using, uh, a lakehouse, right? There's sort of two sides, obviously. There's a warehouse, which is your structured data and your tables, um, and then there's the data lake part, which is all of your unstructured documents.

[2:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=147s) And oftentimes, what can happen is you're given these tools like Text-to-SQL and vector search, and nowadays, we don't really have trouble accessing that data, um, but sometimes there are still some challenges around how you give your agent the right type of context, whether or not they can see all the data in the way that they need, um,

[2:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=167s) and really take a slice to answer the right type of question. And so what we've put together today inside of our course is an agnostic data model, um, that basically creates a graph representation, uh, both from the structured side, your warehouse, um, but then also provide structure for some of your documents, um, and then allows you to

[3:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=188s) do a lot of useful stuff with that. Uh, we do have a scenario here that we're gonna go over. Um, we've created a fictional Autofix group, uh, which is, think of it like a, a Pep Boys Auto or something.

[3:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=201s) It's a national auto repair chain, and they have all of these bays. They have these libraries of manuals on vehicles, um, as well as safety bulletins and recalls, and then a warehouse of all of the repairs that they've logged.

[3:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=216s) Um, and you have sort of your floor technicians, right? These are the Dannys that are listed here, where they have the cars inside of the bay, and they're gonna need to ask some questions.

[3:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=227s) Um, you have leadership at this organization that wants to create a co-pilot, uh, to be able to assist these technicians on the floor. Um, and then obviously, you have people like Sam, who are us, the AI engineers, who actually have to build the thing.

[4:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=242s) Um, and so if they already have their data in a warehouse, what we're gonna be looking at today is the example on BigQuery. So imagine, right, you have your documents inside of cloud storage, and then, um, you have BigQuery as your, uh, data warehouse, but these patterns are also extendable to Databricks as well as Snowflake.

[4:24](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=264s) Um, and essentially, if they just create a co-pilot on top of, uh, BigQuery and the data that they have, um, it can pull the data correctly, um, but sometimes it can be confidently wrong.

[4:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=277s) Um, basically, it could get-- it could pull stuff with vector search from documents, and it can do Text-to-SQL, um, on just a few tables. But when those tables become massive and you get hundreds of tables or when you have large document stores where you have hundreds and thousands and millions of documents, um, things can get lost and

[4:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=297s) fall through the cracks. And really, where Graph can come in to help with these shapes, um, is not only in these single questions that someone might have about how, you know, I have a broken part that I need to replace and how do I repair this vehicle?

[5:12](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=312s) But oftentimes, it's gonna be on these estate-level questions. So things like, for example, what are we missing? Like, what documentation maybe don't we not have to cover all of the different cars that are coming in?

[5:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=325s) Or what documentation maybe are we not leveraging at all? So this is sort of like proving a negative, which can be very hard with something like semantic search, uh, which can only match similar things, right?

[5:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=336s) It can't really find a negative example. Um, there's also questions that we have where we want patterns across everything. So say, you know, you might wanna ask, "Well, you know, what are, you know, the common types of patterns that we see?

[5:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=349s) Is there anything that we're failing to fix over and over again?" Or, you know, are there, you know, specific groupings of different types of recalls that are popping up, and, and things of this nature, where you really need to traverse the entire dataset.

[6:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=363s) Um, and then the other one is just asking how records relate inside of a large SQL store. Um, and when you have one big lookalike schema where you have lots of similar tables, um, how do you understand how to join those together correctly?

[6:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=377s) And so there's gonna be three concrete shapes that we'll introduce you to today. Uh, so the first one, um, we're gonna call Table of Contents, and it's somewhat like a tree structure, uh, but with also different types of links between them.

[6:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=393s) You'll see how it works. Um, this is gonna be used on the unstructured data. Uh, on the unstructured data side as well, we have something called Themes. Um, and what Themes does is it surfers- surfaces global patterns inside of your data that might not be apparent, um, or you might not know about beforehand, so uncovering unknown patterns

[6:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=414s) and groupings. Um, and then the third one, which when we take the course, we're actually gonna go over first, is this Connection Shape, which is essentially a semantic layer on top of your data warehouse.

[7:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=429s) And so how many people, just a show of hands here, are familiar with, like, graph and GraphRAG?

[7:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=438s) Okay, so a, a fair number of you then. Uh, how many people in here have used Neo4j before?

[7:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=445s) Okay, also, like, uh, maybe thirty, forty percent, something like that. So this room seems like, you know, fairly, fairly, uh, fairly familiar with graph basics, but I'll, I'll go over it just here really quick.

[7:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=457s) So Neo4j, we're, we're a graph intelligence platform. We have a graph database at, at, at the heart of it, right? Um, and basically, when we talk about what a graph is, we're talking about it from a property graph data model perspective.

[7:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=473s) And so everything inside of our database and our analytics platform is modeled to these three different types of elements, which is nodes that represent people, places, and things;

[8:04](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=484s) relationships, which are the verbs or associations between those things, so, like, person owns car, person drives car, person is related to other person;

[8:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=495s) and then we have properties that go on these, both, uh, relationships or sometimes we'll call those edges and the nodes, which are sometimes called vertices. And those can be anything from strings, they can be numbers, they can be dates, they can be vectors, um, um, all sorts of things.

[8:35](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=515s) And so the basic idea with a graph, right, is as you start adding data to it, it's almost like a bunch of pre-joined tables, and everything's already interconnected, and you can hop between all the nodes very easily.

[8:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=526s) Um, and that provides a ton of benefits that we'll see, we'll see later in the course.

[8:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=532s) So with that in mind, why don't you go ahead and get started and go over into these if, um, if you haven't heard before, and I'll actually just show you here really quick, um, inside of the Workshop Lakehouse.

[9:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=548s) I have it open here. I should probably restart this guy really quick.

[9:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=556s) Start my Codespace. Um, but basically, it should take you to this page. Um, this shouldn't-- wouldn't say Continue Course for you. If you-- if it's your first time going to it, it would say Enroll to Take the Course.

[9:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=570s) So you have to go through that path to enroll to take the course. Um, and then basically the whole thing from here on out is just gonna be me guiding you through this course and kind of showing you all the different, uh, things that we can do.

[9:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=584s) Um, there's about seventeen lessons in here. We've broken it up by the different shapes. Um, but basically, if you go into the course,

[9:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=594s) you can kinda click through here. I go over this scenario a little bit. But while I'll do that, I think what you should do is jump to this Environment section.

[10:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=603s) So basically, you go down to the bottom here. You go to your environment.

[10:12](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=612s) And, um, from here, you can say Open Codespace.

[10:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=617s) And in fact, I will open a new codespace here just so that I can, I can walk through it, uh, live with you as well. [background noise]

[10:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=627s) Yeah, of course. Um, go back to here.

[10:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=639s) Why don't you all go ahead and capture that?

[10:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=656s) Yeah. So it'll create a codespace in your own GitHub. If you want, there's a link to the repository. You could run it locally. Um, just keep in mind that we've set it up so that it will auto-start in the codespace.

[11:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=669s) So there's another, um, Bash script that you can run to run it locally, and it will, it will set everything up for you. Um, but it, it depends a little bit on your local environment.

[11:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=679s) You have to make sure you have Claude Code installed, um, and, and all these sorts of things. So we'd recommend that you use the codespace one. If, if you're comfortable doing it locally, I, I, I've run it locally all the time before too.

[11:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=698s) All right. Everyone's, everyone's copied the, uh, QR code?

[11:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=706s) I'll give it another ten seconds or so, and then I'll, I'll move back over. [background noise]

[11:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=712s) Yep, Ben and Ryan can help you out if, if you don't have this one.

[12:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=722s) Okay. So when you start up as well, there's some workshop credentials here. Uh, I'll, I'll be, uh, blasting these keys, uh, af-after the course. I'll leave them up for just a little bit.

[12:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=736s) Um, but basically, give it some time to start up. Uh, o-once it, once it does, what you'll do is you'll go into your environment file here,

[12:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=748s) and maybe I can make this just a little bit bigger so it's easier to see. You will place your Anthropic key in here, as well as the, uh, the BigQuery key.

[12:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=758s) The other thing that you're gonna wanna place in, uh, if you go back to the course,

[12:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=764s) um, is down here, it should give you

[12:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=772s) a, um, Neo4j credential. So this is accessing our graph database.

[12:59](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=779s) They've been all pre-provisioned, so it's gonna be different for each one of you. You're gonna have a different one, so copy the one from your screen, um, and you're gonna put it inside of this environment file as well.

[13:11](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=791s) Um, and it, it will take a while. Like you see with mine, the environment file hasn't, uh, quite popped up yet, so it can take a few minutes for that all to go through.

[13:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=807s) Um, so we'll, we'll give that a little bit of time to start up.

[13:32](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=812s) Hopefully, it won't take too long here. While that's running, I'll, I'll give a little bit of an overview here of what we're gonna do in the next step. So basically, once we get this set up and, and we get Claude up and running, um, we're gonna walk through, um, the first shape for our day, which is this

[13:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=835s) connection semantic layer shape. And then after that, we'll follow it with the table of contents one that we were talking about and the communities one. So we'll be working with the structured data first, and then moving into the, into the unstructured bits.

[14:13](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=853s) Um, once for those of you who do have it loaded, basically, you have your terminal down here. You can go ahead and call Claude. Um, we'll be working through Claude Code as our agent.

[14:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=870s) Um, the internet is a little bit slow too, which is expected.

[14:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=891s) As you go through with this as well, I should, um-- Oh, here we go. We'll set up.

[15:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=902s) Show you what this looks like here in a second.

[15:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=905s) The Anthropic key, as well as BigQuery one.

[15:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=938s) All right, perfect. Um, and if it does this, where it's trying to get you to do a subscription, basically what you do is you'll go plus, open up a new terminal window,

[15:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=955s) and then it should here give you the option to select the Anthropic key.

[16:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=961s) Just go through and press Enter. Yes. And then you wanna use the MCP server here, so you're gonna select yes, and I'm gonna skip the tour.

[16:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=974s) And then I can ask that question just to make sure it can connect to Neo4j.

[16:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1003s) And for me, I already have some nodes in because I'm working with, um, a graph that's been cached already from my previous course.

[16:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1011s) Um, for you, it will, it will come up as an empty sandbox. So one thing that, uh, I want to note here too in terms of how we're gonna be working with Neo4j and with Cypher, which is our graph query language.

[17:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1027s) I, I think we've come to a point now where a lot of us aren't ha-hand typing, uh, our own code line by line anymore. We're using agents to help us build things, obviously.

[17:20](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1040s) And so in this workshop, it's going to be the same thing. Basically, we're going to be using something called the Neo4j CLI, um, which is, uh, a CLI tool that will allow your agent to run, uh, queries directly against the database.

[17:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1056s) When I was working inside of Claude, you saw it running there for a second, and that also ships with a Neo4j Cypher skill, um, that allows the agent to sort of leverage how to put queries together in the most modern way, as well as something called a GDS skill, which stands for Graph Data Science.

[17:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1074s) And we'll use that for some of the algorithms when we get to the theme section, um, in terms of just helping us do agentic coding. So I'm not gonna have you guys handwrite really any, um, Cypher query code.

[18:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1087s) I'm going to show you throughout this workshop how to use an agent to help you write that with, with specs that are created. Um, and hopefully that'll be useful for you more on, on the day-to-day if you ever decide to use, uh, Cypher, um, whether it be with Neo4j or not.

[18:29](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1109s) Um, so how is everybody doing now? I, I'll take a second here to stop. Go ahead and raise your hand if you have any questions here. Or are we all good?

[18:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1123s) I have one question over there.

[18:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1128s) Can I use this in my Cursor, or like can I just put that link in my Cursor document now inside of [inaudible] for a follow link?

[18:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1135s) Yeah. So the question is, "Can I use Cursor?" And so the answer to that is you should be able to, yes. I haven't personally tested it, but if you go up to the top, there should be a link to the repository that we're using.

[19:11](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1151s) Um, if not, when you go to open code... Yeah, you can clone the repository here. So if you follow that link, you could clone it, you can get it locally.

[19:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1161s) Um, and then you'll see here there's directions. There's a shell file that you can run that will set everything up. You can take a look at what's in there.

[19:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1168s) It's a, it's a bunch of basic stuff. Okay, awesome. Yeah, and just a little bit about the structure of this thing I forgot to mention. Um, there's a lot of stuff inside of the, uh,

[19:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1181s) inside of, uh, the, the workspace here. Um, obviously inside of .claude, we have our skills file. Um, so I've gone ahead and pre-written a skill here, um, that covers basically a lot of what we're going to do.

[20:00](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1200s) Uh, inside of here as well, there's, uh, a file called, uh, outline and search and themes. So these are gonna be the shapes that we'll be working with. Um, to query BigQuery, there's just a very small, um...

[20:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1214s) Well, that, this is actually a, a database query file. But the run SQL one, this one for querying BigQuery, just a very simple, um, um, basically way of, uh, reaching out to BigQuery and just calling it with a, uh, with, um, a simple, uh, SQL command.

[20:32](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1232s) Um, and inside of these files, as we'll see later, there's, there's places to fill in a spec, so that's gonna be where we do some of our agentic coding.

[20:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1240s) Um, but everything in here, if you wanted to like load this into BigQuery yourself, there's something for that. Um, if you wanna go ahead and run this on Databricks, there's directions here to like load it into, into Databricks and use Genie and AI Search there if you wanna compare.

[20:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1257s) Um, and then this is just a, a symlink over back to the, uh, .claude files. Um, there's solution scripts as well. So if you get stuck, for example, where for some reason the coding agent can't, you know, create the right query, um, you, you can copy stuff from there.

[21:13](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1273s) Um, and then all of the source data too for creating both the structured and unstructured data is in here, um, including our, uh, PDF sources. So if you go in here, we have our bulletins and our manuals and everything.

[21:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1290s) Um, this is showing you the, uh, the raw PDF, but if I go to the corpus, which is the, the markdown version, uh, you'd be able to, to actual re- actually read the contents inside of them.

[21:42](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1302s) Um, so anyway, with all of that, we'll go ahead and hop along to our first section here, which is going to be connections.

[21:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1317s) So we have basically our data inside of BigQuery right now, and in fact, if I go back to my overview just to show you what it looks like really quick, this is the, uh, schema.

[22:11](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1331s) Uh, so we've kept it very simple for this course. Obviously, in, in the real world you'd have a much bigger schema than this. Um, but we have our work orders, uh, which is sort of like the middle or the star of the schema.

[22:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1343s) You have your vehicles, um, the different, uh, DTC codes, procedures, uh, work order parts, and then the parts themselves. Um, and they join together in a very simple primary foreign key pattern.

[22:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1356s) Um, we'll get into our documents later in the course. It's just basically a, a big bucket of, of PDF files.

[22:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1364s) And so what we wanna do here is to help our agent be able to effectively join everything together. We're gonna give it a graph representation of a semantic layer.

[22:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1378s) And to build that semantic layer, we're gonna use something called NeoCarta. Um, NeoCarta is a labs project. So at Neo4j, we have our core engineering, which is like our, you know, cloud SaaS platform, um, our graph analytics stuff and all of that.

[23:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1394s) But we also have labs projects where we move a little bit faster. Um, and inside of that labs project, we created this NeoCarta, um, and what it does is it creates a metadata graph.

[23:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1405s) Um, also, although we're not leveraging it here, you can add business terminology and business processes and a whole bunch of other things into a graph structure. Um, and if you think about it, people, you know, these words ontology and semantic layer and all these things are kinda, they're thrown around a lot lately, especially this summer.

[23:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1423s) And so the, the basic, uh, mental paradigm that I use to kind of think through these is an ontology helps something interpret and reason about the data. A semantic layer helps something understand what the consistent agreed-upon terms are, so it can query the data accurately.

[24:00](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1440s) Um, and then at Neo4j, we have other things like virtual graph. There's a link i-in here in the course which will actually give you a sort of a, a federated graph view of your SQL data that you can, um, that you can query over directly with Cypher.

[24:12](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1452s) We won't be using that here, but there, there is, um, sometimes, uh, value in, in giving sort of a, a graph view of the schema. Um, but anyway, what NeoCarta will do is it, it we'll see in this next section- Um, uh, it, it has an MCP server basically that will sort of suck all the data in

[24:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1473s) from the, um, from BigQuery, just the metadata, and it will create this graph. Um, and that's what you do in the next section. So I'll go ahead and jump to there.

[24:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1485s) And I've already run this script, but basically when you see this script, what you're gonna do is you're gonna copy it, and then you're gonna go over to here, and then, um, you're gonna bring up your terminal, which...

[24:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1498s) Oh, pull up our terminal. And then, uh, I like to just open a new window that's not my Claude window. Um, and you're gonna allow paste, and then you're gonna go ahead and run that.

[25:12](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1512s) I've, I've already run it, um, so I won't do it again, although it should be idempotent. Um, but basically once you run that, uh, it'll create this, um, these six tables with these five reference join paths.

[25:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1528s) And once you've done that, you can come back in here and even just run this to check. And it's kinda cool what it looks like. Um, this is, this is what it looks like inside of the graph database.

[25:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1538s) And basically what you'll see is you have this, uh, this node in the middle that represents your, um, your, well, first your database, and then your schema. And then if you follow that out, it'll be Has Table, and then you'll have your table, and then you have your columns.

[25:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1558s) Um, and those can also optionally have these representative values on top of them.

[26:04](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1564s) Um, and so these join paths, basically, when the agent reads that, um, if I was to go down here and try. So I have, um,

[26:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1575s) a little example prompt down here, and I can go into my window and go back to Claude,

[26:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1585s) and I can ask a question like this: "Which vehicle received part IC2042?" And I, in this case, I'm gonna tell it to use the, the MCP server and the warehouse schema and all this to, to go and grab everything, then use the Python script.

[26:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1600s) Um, in- inside of this skill itself, it, it has directions for this as well. Um, but what you'll see it'll do...

[26:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1609s) Get rid of this. Yeah. It'll call that MCP server for NeoCarta, and it's, it's gonna read that metadata semantic layer graph that we have in Neo4j. And so if you notice, the thing that we're doing here really is we're not using the graph to copy the data over.

[27:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1630s) There's not like an ETL into graph. What we're doing is we're using the graph as a semantic layer. So we're just pulling metadata about the columns and the rows and all of these things, and we're gonna use that to inform a Text2SQL query.

[27:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1647s) Once it, uh, decides to go ahead and run here.

[27:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1663s) How is Claude working for everyone right now? Are we seeing a lot of, uh, slow Claudes, fast? Yeah, slow? Okay. Interesting.

[28:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1683s) I'll give it a little bit of time here. I don't know why it would be go- moving...

[28:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1688s) Huh, a little bit slow. Does anybody need the QR codes?

[28:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1708s) Does everyone have a copy too? Zach and I both have QR codes if you want one. Yeah. Does everyone have-

[28:34](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1714s) I have the other one too. Yes. Is it possible to just, like, for some people who are a little bit further behind, just,

[28:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1725s) like, recount how you got to here just while we're waiting? Yeah, sure. So the question is, can I recount how I got to here while we're, we're waiting for Claude to come along.

[28:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1735s) So basically, um, the QR codes that are being passed around, there's two here. Uh, one of them is, uh, to enroll in the course. So the course that I'm going through is online.

[29:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1748s) It's freely available, so anyone can go. Uh, once, once you show up inside of the course, basically you'll see a button here to enroll. Um, so you have to sign in and enroll in the course.

[29:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1759s) Uh, once you do that, you'll, you'll click through, and you'll make it to this environment section. Um, and when you do that, uh, it'll take about five minutes or so for everything to, to populate and come up.

[29:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1773s) Um, if you're using Codespaces, you have the option to run it locally. Um, but then once you do that, essentially, um, you're gonna come in here, and you're gonna get your credentials, and you're gonna grab the Anthropic API key, um, as well as the, uh, the BigQuery key here.

[29:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1792s) Um, and then on- once you do that, you'll be able to, um, access all the things in, in the workshop basically. Um, you're using, as you saw here, um...

[30:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1803s) Oh, cool. We're actually making progress. Um, we're using Claude Code inside of, inside of Codespaces here.

[30:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1815s) Sorry. Can you just run us through Sorry. Can you repeat that really quick?

[30:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1819s) Can you just run us through when you go through the Neo4j sandbox? Yes. The Neo4j sandbox

[30:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1825s) is, um, if I, if I were to go back to that environment section

[30:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1833s) Um, you see the button here to open Codespace, and then you should have down here, um, your own sandbox credentials. So it won't be these exact ones because it's, it's different for every participant.

[30:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1848s) But you'll go ahead and copy those, and then once you're in here, uh, you'll be able to, um... You have your .env file, and inside of your .env file where you put your Anthropic and your BigQuery key, you'll also put the Neo4j ones.

[31:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1865s) If you don't see a .env file, and you see nothing, or you see the .env.example, it just takes it a while to populate that. There's other things that are running.

[31:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1874s) It's running another shell script to kinda create everything for you and, um, populate all the different files.

[31:22](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1882s) So my, my hunch, and someone can scream at me if I'm wrong, is that if Anthropic's this slow right now, it's more of a Anthropic problem and not necessarily a one person's account problem.

[31:35](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1895s) Um, if it continues being bad, I might try to provide another key from a different account, but I don't actually think that that's what the problem is. I think it's, it's- [laughs] Yeah.

[31:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1906s) Well, yeah, but this should also be running remotely, right?

[31:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1909s) I think it's the Wi-Fi.

[31:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1910s) Is it the, is it just the Wi-Fi? Okay.

[31:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1914s) The Wi-Fi might be bigger bandwidth.

[31:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1915s) Yeah. Well, the funny thing is, I would think in c- Well, never mind. I don't know. It's running in Codespaces, but maybe Codespaces is still using the, the local, uh, the local Wi-Fi.

[32:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1925s) Um, okay. So, so we're back here. Basically, what- basically, what it did to grab this list is, um, it, it went through and it, it basically read, um, that metadata graph, um, and then it created the, um, the different, uh, queries that it wanted to run.

[32:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1948s) And you can see it ran that, the small SQL, uh, Python file. Um, and it, it did all of its selects and its joins to be able to, to pull together, uh, these tables that list out, um, the, the answer to that question essentially.

[32:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1963s) Um, if I was to go back, the question here was, uh... What, what would we ask?

[32:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1970s) Go back to the connection shape. Um, yeah, which vehicles received, you know, these, these separate parts over here. Um, and, and it, it provided that list, um, down here by, by make apparently, um, and then vehicle count.

[33:13](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=1993s) Um, so that's the basics of this first shape, and you can imagine that as your data starts to grow and you start to get more and more tables, it's very useful to have this semantic layer, um, to then help essentially guide how everything joins together.

[33:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2008s) Uh, in this case, it used a primary foreign key, uh, relationship inside of the information schema inside of BigQuery. Uh, to enforce that, NeoCarta can also use things like query logs, or you can sort of manually put together how you want things to join with other terminology and metrics.

[33:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2025s) Um, are there any, uh, questions about this first sort of connections, graph, semantic layer shape that we're working with?

[33:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2037s) Yes, I have one over here.

[34:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2041s) Um, yeah. I guess just the question is, is, um,

[34:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2045s) can you give us a little bit more context around the Neo4j and the NeoCarta concept and how those two connect together?

[34:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2054s) Neo4j. So the question was, can you give us more context into the Neo4j NeoCarta, like what we're doing strategically? Um, so what Neo4j is doing strategically and this right here.

[34:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2066s) So Neo4j as a company, right, we're-- For a long time, you've been able to import data into our database and represent things as a graph, which is very useful if you're trying to run these graph-specific queries, right?

[34:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2079s) So if you have a supply chain, for example, and you need to find an optimal route from point A to B, and you have to do these like variable length, you know, shortest path type of calculations, a graph database is very good for that.

[34:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2092s) But the other thing that a graph database is really great at doing is not just making those complicated queries run faster, but actually providing a view or a representation of the data that allows an agent to understand how tables might interrelate, so that even though the end join, it might only be a three or four-hop join, you

[35:12](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2112s) might have had to understand hundreds of tables or something to be able to arrive at that conclusion. And so we're focused a lot right now on these concepts of ontologies and semantic layers in what we're calling virtual graph, where the focus isn't only just ETL-ing data in, but about, okay, maybe you wanna keep your data where it

[35:32](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2132s) is, and you still wanna use SQL and, and access it in, in the ways you have been, but you need some way to sort of guide the agent to be able to do that correctly.

[35:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2141s) Um, so NeoCarta is a labs project. We also, uh, just recently, um, in preview have released, uh, something called virtual graph, which will be similar, but it essentially gives you, um, almost like these push down Cypher queries.

[35:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2156s) So it gives you a graph schema of your, of your database, and then it allows you to run Cypher directly, um, which can be useful because then you sort of have the view down at the query interface level and not just at the metadata semantic layer level.

[36:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2169s) So there was a table in there, um, in the previous section that was kind of explaining that. But hopefully that's helpful, so you understand kind of where, where we're going with things.

[36:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2179s) Yeah, that's great. I use that in the DT and the whole [inaudible].

[36:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2183s) Yeah, yep, graph and yes. And we're doing a ton of stuff inside of context graphs as well and, and, and, um-

[36:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2190s) Very interesting

[36:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2190s) ... in all of those layers. Yep. All right. Awesome. Um, any other questions?

[36:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2196s) No.

[36:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2197s) Okay. I have one question over here.[inaudible]

[36:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2203s) on top of the actual existing table. If the

[36:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2209s) existing data like that OLTP, whatever, transaction data, why won't we just put that into the graph itself? Would it be maybe like dependent on the graph or would it not necessarily be protocol dependent?

[37:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2221s) Or also like engaging with that one-

[37:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2223s) Yeah. So, so the question is, um, why would we not take our OLTP data, which right now is this warehouse of vehicle orders and, and stuff, and just migr- and just push that into the graph, right?

[37:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2236s) Or some subset of it.

[37:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2236s) Or s- or some subset of it. Why wouldn't we just push that into the graph? Well, I think in a lot of cases, that's easier said than done, right?

[37:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2245s) Because if you think about a lot of production use cases, you can have terabytes of data which gets updated continuously. And so, right, if you had to put that into the graph, you would then have to find some way of syncing that while you move everything in.

[37:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2260s) Um, and there's might be a lot of extra properties or things that you might need to build a custom ETL, where there's some things that you might not want in the graph.

[37:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2269s) And then a lot of times what people end up not really understanding until they get in those situations is the security posture. Because if you have sensitive data that's in one system, it might not a- even though you could physically move it in, you, you might not actually be able to, for security reasons, take that data and

[38:06](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2286s) just move it into another database. Um, so, so there's various reasons why you might not necessarily want to move your data over. Uh, where you do get an advantage of the ETL over just this metadata and semantic layer graph is if you have graph queries that need that performance, right?

[38:24](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2304s) So, like that supply chain case that I had earlier. Like, if you're doing these really large recursive joins, and you need them to happen very quickly, or you need to run graph algorithms, maybe you need to produce graph embeddings, or you need to do some sort of clustering or something.

[38:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2318s) In that case, it makes a lot more sense to, to bring the data in. Um, and then as we'll see in the next sections, when your data's already unstructured, uh, there's benefits to bringing it into a graph 'cause then you can give it a graph structure.

[38:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2331s) All right. I saw one other question over here. Yes.

[38:59](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2339s) Yeah. The semantic graph, every time you initialize, you talked about it, does it rebuild the graph?

[39:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2343s) So the, the question is, does it rebuild the graph every time you initialize NeoCarta? And the answer to that question is no. What it will do is, basically, when, uh, we, we ran that build connection script, um, it went ahead and ran an MCP server function that ports the data in.

[39:22](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2362s) Um, and then once it's ingested into NeoCarta, just the metadata, then that stays there. So, like, for me, I didn't re-run that build connections, right? 'Cause I just took the course last night.

[39:34](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2374s) Um, so, like, for me, I-- And I probably could have 'cause it, I think it's an idempotent load, but I just didn't wanna deal with it. So it was there for me, and then I just ran the queries on top of it.

[39:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2384s) Yep. All right. Any, any other questions? I have one over here at the end.

[39:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2393s) Yes. Yes, here.

[39:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2396s) Can you go over the skills for, uh, writing Cypher, just the structure of it and how you landed on

[40:04](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2404s) that skill?

[40:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2405s) The, the Neo4j Cypher skill?

[40:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2407s) Yeah.

[40:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2407s) Um, so I don't know if... I might be able to, uh, grab it really quick. Um, I can show you Neo4j Cypher skill. Um.

[40:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2426s) But this is-- These are all developed internally by our team. So if you go here, actually, through the Neo4j CLI, you can download a ton of these skills. So if you see here, there's, there's the Cypher one, but then we also have the different ones for, like, agent memory, for cloud infrastructure, for different drivers, like if you're

[40:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2444s) using Java or Go or, or all those things. Um, so I can't necessarily say exactly how we landed on this particular SKILL.md, but what I will tell you is it was created by us and our internal teams who are, like, always up to date on the latest, you know, Cypher '25, '26.

[41:04](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2464s) Um, so, so it, it, it has all of the latest. So I'd, I'd encourage you to use the Neo4j CLI because it will basically-- You can load one or multiple of these skills, and you'll get the latest knowledge there.

[41:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2479s) 'Cause it-- Otherwise, what the, what it's gonna do is it's gonna go on the internet and get the Stack Overflow questions from, like, four or five or six years ago, and it's gonna give you outdated bad Cypher, basically.

[41:29](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2489s) Is the, uh, Neo4j's, uh, CLI loaded into the skill?

[41:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2493s) It is, yes. It is loaded in and, uh, if you look at the, um, the install file that we use inside of dev container, uh, you'll see, um,

[41:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2505s) where we installed the Neo4j agent CLI in the skill. So the command's in there, um, that, that basically allowed you to do that.

[41:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2514s) All righty. I'll take one more question, and then I think I'll have to move on to the next section unless there isn't another one. Yes, right there.

[42:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2522s) Um, when you talked about, uh, using specifically semantic layer on datasets that are more, maybe that can have, uh, let's say like bronze, silver, gold, maybe that can have more structure.

[42:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2537s) Um, bronze. So, so, like, del-- the different, like, Delta tables where. Um, so I mean, we're, we're exploring basically, um, the traditional, like, SQL structured data warehouse. And then NeoCarta Is really customer-driven for us.

[42:35](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2555s) It was made by our field team. So as customers come in and they have questions about less un- less structured data, um, and even things like documents have been considered for NeoCarta.

[42:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2567s) And so there's this backlog that's forming right now. Everything's moving very fast. I think-- I mean, Ryan is, is somewhere in the room, and he might be able to, to speak on sort of what's, what's the next thing for NeoCarta.

[43:00](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2580s) But I know we're working right now on a Databricks connector, um, and maybe some metric views and other things. [background noise]

[43:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2596s) All right. Awesome. So meet with, uh, with Ryan afterwards. And then everyone watching on YouTube, I guess, can meet with Ryan afterwards. But by then, we'll, we'll have-- we'll have an amazing NeoCarta, so.

[43:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2608s) All righty. So, so let me go ahead and move on for sake of time because we're coming fifteen minutes on the halfway point, and it might be nice to take a few minute break at some point in the middle just to, just to give ourselves a little bit of a breather.

[43:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2620s) So now I'm gonna jump to the next section. So that was working with structured data, and now I wanna work with unstructured data. And this is going to be the documents part.

[43:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2633s) So we have-- If we go over to our little, uh, our little, uh, source data stuff here. So in this case, I'm, I'm gonna read them locally to load them.

[44:06](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2646s) Um, but, you know, it's reading from cloud storage, reading locally. Um, there, there often isn't a huge difference. Um, basically, we have these bulletins in PDFs. Um, we have manuals which are considerably longer, um, and then we have recalls as well.

[44:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2665s) And 'cause it's giving me this funky view, I also have them in, uh, in this raw corpus where they're, they're inside of, um, these, uh, markdown files here. Um, but basically, uh, they're these multi-section documents, um, and you'll see here that they have, uh, this linking that they do to go over to different sections in other documents.

[44:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2690s) So they're all kinda interlinked together as they refer to one another. Um, so for example, right inside of, inside of this manual, um, it talks about these different platform codes for this ABS system, um, and, and diagnostic troubleshooting, right?

[45:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2705s) And it, it will link over to there. All of this data, by the way, is I simulated. It's not sensitive or, or anything like that. Um, but it's meant to simulate kind of a real-world, uh, system where you have these manuals and bulletins and recalls of all these different vehicles.

[45:20](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2720s) And, um, what we want to accomplish with this basically is we wanna give our agent a table of contents. And, and by a table of contents, I mean something that looks like this thing.

[45:32](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2732s) So what I want to allow the agent to do here is not just search for key terms. We'll give it search too. But I don't know if-- Has anyone heard of, like, PageIndex?

[45:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2744s) Anyone know who they are? Maybe some of you. So there's, so there's this idea of navigation through your documents, where basically almost like a human, if you look at the table of contents inside of a book.

[45:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2758s) It's not-- I'll call this outline. I'll call it table of contents. It-- I need to get the terminology a little bit better aligned. But the idea here is that you can sort of read the indentations, and you can see how you have a document, or in this case, a library.

[46:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2774s) You have the bulletins, which is a subfolder, then you have these documents underneath. And so you have this containment tree. And then in addition to that containment tree that gives you all these sections, you also have these links that'll take you to a different document.

[46:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2790s) So it's more than just a table of contents. It's a tree that goes down. It also has these links, right? And these links and basically everything in here have a URI.

[46:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2803s) And the idea is that if we can come up with a good graph representation, we can basically make it so that these URIs, which are hierarchical, so you see like you have your technical library here, right?

[46:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2816s) And then slash bulletins. So, like, if I'm down in, you know, this TSB recall notice, I know that it's part of bulletins, um, or no, safety bulletin rather, not a recall, and it's part of a technical library.

[47:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2829s) Just like it's also linking to a manual here. Um, and you can see the name of the manual, and then it's inside of the manuals subfolder. Um, and if I have that, I can take that, and I can plug it in, and I can actually grab a node from the graph that has, like, the raw text or,

[47:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2848s) or even a link to the raw text. And so-- And then if I, uh, plug this in, and we'll see how this works with outline later, I can actually get these subtrees, so I can, like, dig in and drill down on these different pieces of content.

[47:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2861s) Um, that's the general idea with this. And so what this gives the agent to do is not just search, like vector search or, or lexical search, but actually kinda traverse through the documents in a sense.

[47:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2873s) Um, and the data model that we're gonna use, in this case, we're actually gonna import the data into graph. Um,

[48:04](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2884s) I have a, I have a quick question for everyone. So everyone here is, is-- knows GraphRAG. When they think of GraphRAG, and maybe someone can raise your hand, and I'll, I'll pick on someone for a second.

[48:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2896s) I want you to tell me what you think, like, the pro-- the, the building the graph process is for GraphRAG. So I don't know if anyone wants to, wants to volunteer if they're familiar.

[48:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2907s) You know, raise your hand. Yes. [background noise]Yeah, and they build, they build the, they build the links from,

[48:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2930s) from the... So there's different ways, right? And, and you said, I think in, in your-- when you talk about creating, um, a graph, you build the links from the data that's inside of the documents, right?

[49:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2943s) And so GraphRAG, oftentimes we have a, an entity extraction piece to it as well. So we can use an LLM to kinda say, "Hey," like, "extract the different parts, the different vehicles from the, from the graph, and have them be separate entities, and then have those links to, like, the original document chunks and all, all of this

[49:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2965s) stuff." And that's great, and that works really well. Um, here, what I'm gonna show you is something that's much more lightweight. And I think maybe to the point that you were making, um,

[49:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2978s) we're actually going to use a deterministic loading. So if you see here, the way that this graph structure is gonna work is you're gonna have your library, and then you're gonna have this containment tree that reflects what we saw.

[49:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=2990s) And all this is really doing is it's breaking down. You see the folders, the document, and then these documents have sections. And they can have multiple sections that will also nest underneath each other at different section levels.

[50:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3005s) Um, and then you'll have, um, next section links, so you get a concept of ordering, as well as linked to, which uses-- In this case, we're using, um, these named links, uh, sort of like you might see inside of an Obsidian vault a little bit.

[50:22](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3022s) Um, but you can also do it with hyperlinking and oth-other things depending on, depending on what your data looks like. So it's all deterministic with the containment tree for one, and then the ordering of the sections, as well as the links.

[50:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3036s) Um, and the benefits of having a deterministic load like this is, number one, it's gonna be idempotent. Um, so, like, you're not relying on an LLM in the beginning.

[50:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3046s) Um, it's often going to be a little bit faster. Um, and so if you already have documents which have a lot of inherent structure to them and a lot of interlinking, um, sometimes just to get something up and running quickly, it can be very beneficial to approach a graph like this where I'd say it's more of a

[51:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3062s) lexical or a document structure graph rather than, like, a full, like, entity, uh, you know, extraction type of pipeline to create a graph.

[51:11](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3071s) Um, and it, it explains here what I just said with the different link types. Um, and then-- And, and like I said before, these URLs carry a hierarchy. So every node, for example, that, that head node at the top here for the library will get its URI.

[51:29](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3089s) Manuals will have that plus backslash manuals, um, going down sort of the containment tree to the, to the file name and then the different sections.

[51:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3101s) And, um, this is the script you're gonna copy to load it in. So basically, again, you'll take this, you'll go over to here, um, you'll open your bash terminal.

[51:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3112s) Now, I believe I've already done this, so I'm not gonna run it again, even though I-I'm pretty sure it's idempotent. Um, but basically, once you run that,

[52:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3122s) um, what you'll end up getting is-- And I'll just make sure it's actually in the graph. It is for me. You'll end up getting that, uh, that model that we just saw.

[52:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3130s) It should only take... Well, now it might take a while if the Wi-Fi is slow, although I don't think it should. Um, but it'll create this graph for you where you have, again, your top of the folder structure, which is our technical library, our folders, and then you can see we have our documents followed up here into

[52:29](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3149s) our different sections. And if I were to zoom in

[52:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3153s) here, um, you'll see we have the Has Links and the next section, but then we also have the Links To. And I can go in and click on these, and you can see kinda how the thing grows and everything kinda links to each other.

[52:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3168s) And now that we have that, um, basically in the next section, we can go ahead and build our tool, um, basically the script that goes along with our, our custom Autofix skill that we've created, uh, to, um,

[53:06](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3186s) to actually, like, create that outline shape that we saw in the beginning, the, the table of contents to hand our agent. And so if I go ahead to that next section,

[53:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3197s) um, and again, like I said, what I want to do here is I'm gonna go ahead and copy this. This is the, this is the prompt, um, where if I go ahead and feed that to my Claude,

[53:35](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3215s) it should go ahead and get to work.

[53:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3219s) Nope. And because I know that's gonna take a while, I'm gonna have it start running here. But basically, what it's doing as it goes through, um,

[53:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3234s) is if I go under and I look at my skills, um, you'll see that I left

[54:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3241s) inside of here, um, this wired query. This is, this is the query that it needs to create, um, to be able to create that outline shape. And I've instructed it inside of the language here to use the Neo4j Cypher skill and reference a spec in doc outline format.md.

[54:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3263s) So this is generally how I like to write more complicated Cypher queries, more complicated graph queries, is I'll write a spec for it. Um, and so if I go to my docs folder, you'll see I'll have my specs in here.

[54:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3277s) And I have my outline format spec where I say, "This is the shape," right? And then it, it talks here about how it needs to be human-readable. And, uh, yes, we will allow you to edit.

[54:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3291s) And, um, and some of the optional arguments it needs to have. So,

[54:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3297s) right, 'cause oftentimes it's like pseudocode that we'll have, and we don't know exactly how to put it together. Um,

[55:06](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3306s) yeah. Um, in this case, I did give it the hint about having a variable length path query, uh, to basically put everything, uh, put everything in order.

[55:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3328s) And then it will say when it's done. And so basically what you'll get out of that is if I go back here, and it will talk a little bit about, like, one of the keys to this you'll see in the file is it does this thing here, which it goes HAS, which is that containment edge.

[55:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3348s) And you see this *0..25. It's actually parameterized inside of the, inside of the query. So, like, uh, if I go back to my, um... Let me go to where it changed.

[56:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3365s) Um, if I go back to here, it's, it's parameterized inside of this string, so it gives you, like, an option to how deep you wanna traverse. Um, and that, that's sort of the graph-y nature of kinda putting together this table of contents.

[56:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3378s) There's, there's a second query in here too, which is much simpler, which just grabs the links after that. Um, so that basically,

[56:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3387s) uh, once that's done, I can go ahead and, um, copy.

[56:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3393s) I'll run the whole thing. If you, if you just run the script without any depth parameter or without a specific URI, it's gonna return everything in the graph, so it wouldn't be the way that I'd, I'd run it with, with a larger graph here.

[56:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3407s) We'd end up with... 'Cause we only have a couple hundred documents, um, so I can get away with it here. Um, but if I ran that,

[56:59](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3419s) um, it'll, it'll bring back a ton in...

[57:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3423s) But you can kinda see here. Um, zoom out. So you can see, right, that it gave me this, and you can see how a lot of these documents, again, it's not just the structure, it's like the links between all of them that it's, that it's providing.

[57:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3439s) Um, and then what you can do with this, right, is you can give it an optional depth parameter. So for example, if I just said depth equals one. Uh, there we go.

[57:32](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3452s) You don't like clearing. Down to the bottom, and I say depth one. Then if I did that, um,

[57:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3463s) it'll go ahead and just go one down. And then the other cool thing with this too is you can provide it a specific URI. So if I saw, you know, a, a specific URI here

[57:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3476s) that I wanted to give it, um, then it will basically start at that Falcon 2.0 document. Just like, you know, if I, um, if I was here now and say, "Okay, well, actually, you know, I wanna do, um, I wanna do it, but I wanna do it for something that I see up above," like maybe this links

[58:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3498s) to, you know, this, um, coil identification thing. I can go ahead and snag that URI, and then I can put it in the script,

[58:29](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3509s) and then it will basically give me, you know, that document and everything it links to. So you can say-- see here how an agent can go through now and sorta grab these URIs and kinda traverse all through the g- through the graph and give itself these hierarchical and, and linked views.

[58:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3530s) Um, the-

[58:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3531s) Questions.

[58:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3532s) Okay. Yeah. Why don't we, why don't we take a second for questions? So we have one in the middle here.

[58:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3537s) Yeah. From Claude Code, looking at the agent responses that we've actually been using the connection throughout to create these results, like-

[59:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3549s) Um, well-

[59:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3550s) Like citations or how do we know from the response that it actually used the connection throughout?

[59:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3557s) So later in the course I'll, i- in the workshop here for some of the last questions, I do instruct the agent to say exactly what steps it followed. Um, are you seeing it use something different?

[59:31](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3571s) No. I asked a question. I got a response.

[59:34](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3574s) Right.

[59:34](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3574s) How do I know that you're actually-

[59:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3577s) Yeah. So it should be apparent in the tool call history. Um, so you can, you can look at the different tool calls that it made 'cause in the case, for example, of connections, it's calling an MCP server, so those should be inside of your tool calls.

[59:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3592s) And the question, by the way, for everyone if, if you couldn't hear, um, it was just how do you know that the agent's actually calling the things that it's supposed to, right?

[1:00:00](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3600s) Yeah. So, so for Claude Code, for example, you can look at the tool call history. Um, and informally what I'll do in this course is for some of the prompts I'll, I'll just direct it to say, like, "Hey," like, "clearly," like, "say," like, "what steps and logic you used to be able to, to answer this question."

[1:00:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3619s) Um, you know, obviously for a production system you would, you would wanna look at the logs and everything, but for here and for learning, I, I, that's how I'll, I'll expose it.

[1:00:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3628s) Um, is there-- are there any other questions about, uh, trees, outlines?

[1:00:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3637s) Yes, one right here.

[1:00:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3638s) Just the, the build process in, in the beginning. Is that, is that just using, like, NL, like NLP library or-

[1:00:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3645s) Yeah

[1:00:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3646s) ... processing?

[1:00:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3647s) Yeah. So, um, it is, uh, all the code for that should be in

[1:00:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3657s) Uh-oh. Yeah, we'll need to restart. Um, oh, that was a different code base. Thank, thank goodness. All right. Um,

[1:01:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3667s) yes. Inside of load, if you go to load documents,

[1:01:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3674s) this is, this is just the Cypher queries, and then you have your parse corpus, um, down here. So yeah, it's basically just using regex in here to try to find, you know, the, the right stuff.

[1:01:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3687s) In this case, the documents are already fairly well-structured, right? So mileage may vary depending on what your document sources are. In this case, right, if you have a lot of manuals that are often structured the same way, then you can get away with sort of regex and plain NLP techniques.

[1:01:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3704s) If it's messier, you might have to use like, um, you know, GLiNER or different types of LLMs, and there, there are stages of complexity for that.

[1:01:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3713s) All right. Yes, we have a question over here. [muffled voice]

[1:02:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3721s) Yes. [muffled voice] Yep. Can be done with any model. The only thing that NeoCarta and any of these tools-- Well, Neo-- I'll start with NeoCarta.

[1:02:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3737s) NeoCarta is really responsible for creating that semantic layer metadata graph and providing you an MCP endpoint, right? So as long as you can access that MCP server, it's model agnostic.

[1:02:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3750s) Okay. So that was, that was a question about basically we're using Opus "Four point eight" here, and can we use any other type of model? And the answer to that question is, is yes.

[1:02:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3758s) We're using a skills framework, and then in-- and then for NeoCarta, we have MCP. Um, NeoCarta also has a CLI, so there's, there's other ways to access it too.

[1:02:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3769s) Um, I have one question all the way in back there. [muffled voice]

[1:03:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3789s) So the question is around, if, if I'm understanding correctly, how do you decide what to name your relationships and how descriptive they have to be and who is naming the relationships.

[1:03:24](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3804s) So in this case, we have a very deterministic load, so we're deciding ahead of time what to name our relationships. Um, and that is a bit of a, a taste or a judgment call, right?

[1:03:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3817s) Uh, I tried to keep it simple here, where has is very simple. It's a containment relationship. I also know that I'm going to want containment going multiple levels, so I probably want a common name for all of those.

[1:03:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3828s) 'Cause if I have like has folder, has document, has section, it makes the Cypher more complicated. Um, so that's sort of why I kept that naming very short. Same with links to.

[1:04:00](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3840s) Um, the more granular you get with naming, um, basically the more sort of detailed, uh, your agent can be and your end tools can be in putting a query together, but then the more complicated your data model becomes.

[1:04:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3854s) And so it's a little bit of a thing that you have to manage because if you end up in like a production scenario where you have hundreds of different types of relationships, um, that might get hard to manage inside of a context window, and it can lead to models maybe not having the easiest time putting those Cypher

[1:04:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3870s) queries together with something like the Neo4j CLI or humans, right? When we're trying to, you know, write our own, uh, sort of parameterized queries. [muffled voice]

[1:04:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3885s) Yep. Yeah, that's true too. Um, and we see that, right? The- that's the whole semantic layer argument too with-- on the warehouse side is you can have these metrics, views, and other things which use business terminology that might be different from sort of the, the physical data model that you have, right?

[1:05:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3901s) Um, and so that's, that's part of the reason why we have these metadata graphs too, to help manage that sort of different or that delta between them. I'll take a couple more questions, and then we'll-- we can move on.

[1:05:12](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3912s) Yep. [muffled voice] Yeah, that's an interesting question.

[1:05:29](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3929s) Um, I think it would be naive to call it a replacement because most of the people that I see using this will eventually incorporate some sort of hybrid vector retrieval or full-text search.

[1:05:42](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3942s) Work I do, I use at least full-text search, um, with this sort of stuff. So I don't think it's a replacement. But as we'll see later in the course when we get to some of the estate-level questions, um, there's certain things that semantic search is not as great at answering that really require document navigation to understand

[1:06:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3961s) how, like, everything connects together, right? Um, so big examples would be like proving something doesn't exist. Like, how do you do that with semantic search, right? Or understanding, um, even like how, you know, there might be different parts related to a specific document or something.

[1:06:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3981s) You know, you could, in theory, do like a bunch of vector queries to keep pulling passages, but there's more, you know, chance for error that you might not catch the right document and all this sort of stuff.

[1:06:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=3993s) Um, so yeah, not-- So the question was if it's a replacement for semantic search, and I don't think it is, no.

[1:06:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4001s) Um, one more in, uh, back there. [muffled voice]

[1:06:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4017s) So, um, the question is, do we redefine the types of nodes, um, as well as the relationships? And so is this a question of like- How do you decide what the node labels should be?

[1:07:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4029s) Um, yeah, and again, it's, it's again, it's a data modeling question, right? Like, there could be a situation where we're looking at the data model that we just went over.

[1:07:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4039s) Um, well, it's not, it's not here. It's in my, uh, it's in my last one, um,

[1:07:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4047s) when I look at the, uh, when I look at the shape. Um,

[1:07:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4053s) zoom out of that a little bit. Where it might be useful to have, like, an actual parts node or something, right? Um, here, for this type of use case, I'm letting the agent kind of infer parts and extract them from the documents and then use that to go find something from the metadata graph and query SQL or

[1:07:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4073s) the, or the other way around. Um, and that works for me. So, you know, it's still, as, as things evolve, like two years ago, I probably would have said you would need that, like, additional node.

[1:08:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4085s) Now it seems like agents are smart enough where maybe you don't all the time anymore. Um, and but for some use cases, you know, especially like we see this in life sciences, for example, where there's these really specific ontologies around, you know, different types of molecules and medications and things, and you, you really do need that represented

[1:08:24](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4104s) in the graph or else it just-- your inference isn't gonna make sense. So

[1:08:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4110s) I hate to say it depends on use case. I don't wanna be that person, but it, it sort of does in a sense. All right. Um, uh, okay. Yeah, let me, let me move on only because we're a little bit over halfway through, and I have some more course to, to get through here.

[1:08:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4127s) So, um, let's go to... So we went over all of this stuff. Um, oh, yes. So we need to build our, uh, search query. So similar to what we just did and to the, the question that the gentleman had over here, uh, we wanna be able to,

[1:09:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4150s) um, we wanna be able to build, um, some sort of search as well to help augment this. So, um, this is going to be much more simple, this search, uh, file.

[1:09:24](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4164s) Um, basically, I'm just gonna use a Lucene-style index, uh, and I, I'm not gonna use vector search here. It's the-- Neo4j itself has, um, a Lucene full-text search index.

[1:09:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4178s) But-- And I'm gonna do-- And if you see here, it's basically doing the same thing where I have the, uh, this, this wire that it has to fill in.

[1:09:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4188s) Um, and it's gonna use a Cypher skill to do that. While it's, it's running there,

[1:09:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4194s) you're allowed to edit. Um, well, we might as well wait till it's done.

[1:10:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4201s) All right, it's gonna finish up. So basically, uh, if you see here, it's the way this works is you call the full text index. Um, it's indexed on the document and the section nodes.

[1:10:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4215s) Uh, so every node that has text in it, like we don't put folders through, through this Lucene index. Um, and then it will give you a score. Um, and then because we have everything in those URI format, um, we can also scope it by subtree, um, basically with like a starts with filter.

[1:10:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4233s) Uh, so if I was to go back into this, uh, file, um, and I looked at the way that this query run, I can have this where statement. Um, so I'm applying this as a post filter, basically, where I'm doing the full text search first, um, and then I am filtering it by the URI.

[1:10:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4250s) So I can say, like, only search under, like, this section of the manuals, basically. Um, so it just helps me refine my, my search a little bit with those hierarchical URIs.

[1:11:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4262s) Uh, and the other thing that I do, um, and, and I do this because I work, like in, in the, the research role that I have, I work with so many different AI models and on so many different platforms.

[1:11:13](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4273s) Like, you wouldn't think about it, but like choosing, like, and wiring, like, options to be-- like work with like six different vendors for vectors is kinda tough. So what, what I often do for this is I'll use something called semantic expansion.

[1:11:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4287s) Um, and basically what it does is it, it instructs the AI model to say, "Hey, like, use your world knowledge. If someone asks for, you know, an engine shuddering, that it might be a misfire or something else too."

[1:11:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4300s) So it-- And because this is a Lucene index, um, I can do that. So for example, right, I can say like, "Misfire or rough idle," and it will search for either of those.

[1:11:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4312s) Um, if I go back to my Bash script,

[1:11:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4317s) and I run that now that the, uh, thing's filled it in, and then you'll see it'll give me my, uh, documents with, with the scoring for, for the different ones.

[1:12:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4327s) And it-- I can also search, as I was saying before, under recall. So I can search for coils under just the recall, uh, subfolder library. So that would be the idea here.

[1:12:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4338s) Um, so you can see it's still leveraging, uh, that hierarchical containment shape, the tree, um, but it's doing it through the URI, um, ID structure so that it doesn't have to do as much graph traversal.

[1:12:32](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4352s) It just sort of filters down to, to things underneath.

[1:12:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4357s) Um, all righty. Awesome. Um, are there any quick questions just around the search piece there, the Lucene search?

[1:12:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4374s) We have one question over here. [muffled voice]Yeah.

[1:13:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4389s) So the question is, we just loaded the documents into the, into Lucene, and then we're searching underneath different sections.

[1:13:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4399s) Appropriate sections.

[1:13:20](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4400s) Appropriate sections. Yeah. So basically, if it might make more sense if we actually took a look at the, the query that we're running here. So

[1:13:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4408s) here's the-- Oh, make it this big. Um, so here's, here's a Cypher query, and I'll, I'll create a space here so you can, you can kinda see it better.

[1:13:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4418s) So we have a Lucene index that we've set, and we set it when we loaded the graph. We set it on the document and the section nodes. So every time that I call this index.fulltext.queryNodes, it's going to hit the name of the index, which is content search, and then this parameter, Lucene, is whatever

[1:14:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4441s) I've fed it in with that script. And so that's gonna do our initial filter to just a bunch of documents and sections. Because we've structured our IDs for every node as a URI that's hierarchical, if I know that I only want to search underneath recall notices, I can hand it that URI, and it will say, "Only nodes

[1:14:24](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4464s) whose URI starts with that thing." So it's applying that post filter afterwards, if that makes sense.

[1:14:31](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4471s) Yeah. And that's what it-

[1:14:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4476s) Yeah. Yep. You have a question?

[1:14:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4478s) Yeah. Just a quick follow on. Um, if you're creating this, uh, this search script with Lucene in code, can you just say, "Take a look at the data, create an index for me," and just like that?

[1:14:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4492s) Or do you have to go into the data?

[1:14:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4495s) So the question is, if, say, you already have a graph and you haven't created your Lucene index yet, could you prompt Claude Code to basically do this for you?

[1:15:04](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4504s) Um, so I th- I think yes. To be honest, I might have done that with this course initially. [laughs] I might have actually had Claude do it. I can't remember 'cause I've, I've rebuilt this in a, in a, in a few different ways.

[1:15:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4518s) Um, but I think the answer to that is yes. Your mileage may vary depending on your data. A lot of times with when you set up a Lucene index too, it's like, especially it's very flexible inside a Neo4j.

[1:15:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4530s) So it's like you can set it up on top of one node or multiple nodes, multiple properties on one node, multiple properties between multiple nodes. So, you know, there, there's a lot-- That flexibility can also be a little bit of an Achilles heel too when you ask a, an AI model to do, 'cause it might not understand,

[1:15:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4547s) you know, all those various options. But in general, yes. You, you can have, um, you can use agentic coding to, to help put this together for you.

[1:15:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4557s) Um, yes, in the middle there.

[1:16:00](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4560s) So the, the tree we build, how, how important is it that the, the names of the documents are, like, accurate and relevant to the effectiveness of the agent being able to traverse it?

[1:16:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4574s) So the question is: How important is it that the name of the documents be accurate for the agent to traverse it effectively? Um-

[1:16:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4586s) Like, does the, does the performance go down? Uh-

[1:16:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4590s) I think it, I think it would. Yeah. If, if your documents weren't named very well and you had that outline view, then the agent doesn't have as much to go off of when it's trying to traverse.

[1:16:42](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4602s) So it might misinterpret what a document means, right? Um, that's why having the search augmentation alongside of it is useful 'cause it can actually look in the document. Um, with these things, what I've seen, 'cause I use this actually for my own knowledge base management.

[1:16:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4618s) I have my own open source library that I use, which has these tools. Um, the biggest problem is when you have documents that are outdated or drifted, right? Um, so I've thought about, well, maybe we need to add like a last updated date or something, or like, um, whether or not a document should be authoritative, right?

[1:17:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4638s) Um, and the document naming is actually important, um, and link naming as well is important too. Um, so if you're, if you have links that, um, don't have like, um, I forget what it's called, but like in Obsidian, you can sort of, you can name the, you can give it, um, uh, a synonym or something, right, where

[1:17:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4657s) you name the link. Like that can be very helpful if that's there, right? 'Cause then it's like, "Oh, I know exactly why I'm linking out to this other thing."

[1:17:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4665s) Um, so that's when sometimes using like a language model inside of the ingest, if you don't have that, could, could be beneficial.

[1:17:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4674s) Um, yes, over here. [faintly speaking] So when you get a new document that comes in. Um, so the nice thing about this is that it's all an idempotent load, which means that, say, your entire graph went away tomorrow.

[1:18:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4697s) As long as your documents didn't change, if you load them, you'll get the same graph. So worst case scenario, yes, you would have to reload the whole graph, but this load is deterministic.

[1:18:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4708s) Now, you can also, and I've experimented with this again in some of my other, um, work, is you can say, "Oh, well, I have one document that changed, and I just wanna update that one node."

[1:18:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4720s) You just have to be aware of the fact that it links to other things. So it's like, well, if you got, if you changed, if you updated this document over here, um, if you changed its name, then you have to, you know, did your other documents also update like the URL reference to that document?

[1:18:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4737s) So there's little edge cases there that you have to work through. But, um, if you're able to do that, maybe we can talk after, and I can show you.

[1:19:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4745s) You can have like just add to like this part of the tree and clean up this part of the tree. So you can do like partial syncing. Yep.

[1:19:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4754s) All righty. Um- Is-- Do you have your hand up back there?

[1:19:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4761s) No, you're just scratching your head? All right. Um, then I guess we have one question here.

[1:19:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4768s) Sure. So I asked the agent if we have, uh, two thousand things to be made on. We have fifteen thousand files, right? So it says, "Out of that, five hundred files could be done by a PageIndex."

[1:19:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4781s) It's saying that only the detailed documents in. So PageIndex seems to be one of the ways to retrieve data, right?

[1:19:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4788s) Right.

[1:19:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4789s) So, so only it's saying the detailed documents will do that. And after that, so I already started doing that. It's saying once you PageIndex, the default that I should do search index.

[1:20:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4801s) Is, is the next step, right? For-

[1:20:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4805s) The default after something like PageIndex would be search?

[1:20:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4809s) Or search index could be search.

[1:20:11](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4811s) Yeah. Yeah, so I think the quest-- Is the question basically like, um, for the way that the agent reasons about it, that it would call outline first and then use like search?

[1:20:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4823s) Yeah. Yeah, it, it could work that way. It could work some way where it will search first to find a relevant document and then try to find everything it links to, in which case it would be the opposite way around.

[1:20:35](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4835s) All right. Awesome. Let's, um, let's move on here then to make sure that we have time for all of our questions. Um, so what are we doing here?

[1:20:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4848s) Um, this is just more asking, more full text search.

[1:20:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4857s) Why don't we-- 'cause we're at [REDACTED:age]. Well, I think we already effectively went over a lot of this material here.

[1:21:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4868s) Um, some of this is designed so that if you come back to it later, it kinda over-documents what I'm talking about. So these steps and some of the optional work will help you understand Cypher a little bit better, um, and kind of exactly how, exactly how all the pieces fit together.

[1:21:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4887s) But given, given the pace that we're moving at, why don't I go ahead and jump over to our, um, theme section. So I'm gonna skip over the optional practice lesson here, and I'm gonna go right into, into themes.

[1:21:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4903s) So this is going to be our third and, uh, last shape of today.

[1:21:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4911s) The, the idea with this is if you run-- So this Cypher query is basically going to be loading a graph, and I'm just looking at that links to relationship between the documents.

[1:22:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4922s) So when you have documents that have a lot of interlinking, um, it's always nice to think about using a graph because this structure, basically, what it's saying here is, right, we have these manuals, these recalls, and these bulletins, and they link, and they refer to each other.

[1:22:20](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4940s) Um, so you can see like this, you know, um,

[1:22:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4945s) uh, this, this g- the, this guide here that's a, um, that's a manual is being linked to by all of these different repair procedures. And if, if you zoom out, what ends up happening is you'll get natural clusters of things.

[1:22:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4961s) And this also happens a lot like in these Karpathy style, like, knowledge bases, where you'll see that, like, concepts will naturally start grouping together. And a graph can help you surface those themes, those things that you didn't actually know existed before.

[1:23:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4981s) And so we're going to use something called Leiden community detection. By a show of hands, how many people in this room are familiar with what graph data science is?

[1:23:12](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=4992s) Oh, yes. Okay. So, so not too many of you. Do-- How many people know what community detection in a graph is?

[1:23:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5001s) Okay. So we, we have some of you. So, so community detection is this idea where, right, if, if I have this graph, um, and you can kinda see if I zoom out.

[1:23:32](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5012s) If you r- If you-- Especially if you're running it locally, you can see that there's these like natural little like clusters, right, of nodes that are highly interlinked together.

[1:23:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5021s) And so the idea is like, well, what if we can like try to label these clusters such that within a cluster, things are highly interconnected. So like this little globule of nodes becomes a cluster, then this globule becomes a cluster, and this one down here.

[1:23:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5036s) Um, and, and if, if we can do that, we can start understanding our data at a global scale really well. So this is like that initial like Microsoft GraphRAG idea too of like, of global versus local search, right?

[1:24:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5050s) We're doing something very similar here, but we're doing it in a very lightweight way where we're not really doing any LLM extraction. We're just going off of literally the, the structure of the documents.

[1:24:20](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5060s) The algorithm that we're gonna use for that labeling is called Leiden, which is similar to Louvain, which if you talk about community detection, Louvain will come up a lot.

[1:24:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5070s) Leiden is-- I almost see it as like a, um, the, the sorta next step. It's a little bit more efficient, um, in, in the way that it runs. And, um, it, it's using Neo4j's Graph Data Science library.

[1:24:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5085s) So basically, what we have to do or what, what we do to make it very performant is, in addition to just having the database, we have this other, um, projection where we'll take, uh, a part of the graph into memory, um, and then we'll run these high concurrency algorithms on top of that so that if you have

[1:25:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5105s) a graph that has, say, millions or billions of nodes, we can start performing this clustering and then identifying, um, basically the different interconnected communities.

[1:25:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5117s) And, um, these clusters, if I- Um, keep going down. Basically, the, the format that we're gonna go for here, sort of the view that we're gonna show our agent is this one.

[1:25:30](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5130s) And so it's a little bit hard to see 'cause it's a sliding window. But basically, we'll have what are, what we're calling themes, and these themes are gonna be these buckets of documents.

[1:25:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5141s) We'll have a sense of how tightly or loosely they're interlinked using, um, this conductance metric. It's basically, like, gonna be this metric around, like, how interconnected the nodes inside a cluster are versus how much they, um, are connected outside of the cluster.

[1:25:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5156s) So we can say something's tightly interlinked or loosely interlinked. Um, we'll use the labels on the links as the top shared targets, and we will, um, talk about the most linked docs and so, so the highest centrality docs inside each of them.

[1:26:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5175s) And so what we end up with, without any sort of, um, tagging or labeling by AI or a language model, um, is just simply from the document structure, we can tell that this first one, um, is about, you know, um, BCM, um, and, and bus and all these sorts of things.

[1:26:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5196s) And then if, if I go down, like this next cluster is gonna be about brakes, um, and rotor pads and, um, hydraulic lines, so it's all braking stuff. So you end up with these very natural sort of clusters that come up.

[1:26:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5211s) And so this is very useful from sort of a, a whole estate-wide question because you can start to understand in your data kind of how everything kinda groups and clusters together.

[1:27:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5222s) Uh, and so the way that we build that is similar to what we were doing before, where we have that script

[1:27:11](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5231s) and we have our, um, our spec as well. Um, so collapsing the sections. And it talks a little bit here, and I, I kinda wanna give us the last half an hour to go through the final questions, so I'll, I'll speed run this a little bit.

[1:27:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5248s) But basically, when we create this projection, and I'll just go ahead and, and copy this thing. Maybe I'll, I'll do this script thing first. So like we did before, um, inside of here, uh, if I look inside of my themes.py file, I, I have this wire for this, for the projection, that piece where we take the graph

[1:27:48](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5268s) and put it into memory. Um, the actual Leiden algorithm here, I'm just calling it with that graph data science library. Um, so in theory, you can, you can kinda agentic code all of these, um, but I'll just do this one.

[1:28:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5282s) Um, [lips smack] oh, that's not good. I wanna be able to make sure I copy the right thing. So similar to before,

[1:28:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5290s) um, we have our, um... Make sure that it's the right thing that I just gave it. Yeah. So we're telling it to use a Cypher skill and the GDS skill to put this together.

[1:28:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5301s) If you went inside of our docs, you have the, uh, theme format one, which will give it the, the shape that we want and everything, um, and it will tell us kinda how we want all the headers and things to be formatted.

[1:28:35](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5315s) Um, and so we'll go ahead and, um,

[1:28:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5319s) create that. Create that for me. And while it's creating that, I'll talk a little bit about the projection. So there's some graph manipulation that we do to bring it efficiently into a projection.

[1:28:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5334s) Um, basically, we collapse the relationships on the URI so that the, if we have sections that interlink with each other, like one section of a doc interlinks to another subsection, we just aggregate that all to the document level.

[1:29:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5349s) Um, and that creates kind of a cleaner interpretation to how our documents link together rather than just all our individual sections, and we can get out the communities, um, easier that way as well.

[1:29:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5367s) All righty. So that looks like it's coming along.

[1:29:34](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5374s) Yes. Okay. So it went ahead and created that for us. Um, and then once we get that back, uh, I can go ahead and copy the script that it just helped me create, and I can run that inside of my terminal.

[1:29:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5394s) Let me just make a new terminal. So I can go ahead and, and call that theme script that we just edited, and then it will do that community detection, and you'll see here that I'll get that view.

[1:30:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5408s) Um, so I have my... You can see, like, the, this, you know, circuit, you know, one. I have all my different, uh, documents that are, that are grouped together, and I kinda understand now, um, all the different areas of the vehicles that, that my documents go over.

[1:30:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5425s) Um, there's other parameters that you can feed this. Um, so for example, we have a gamma parameter, and basically what that will do is, uh, it, it basically tells it kind of, um, how, how much to split everything into.

[1:30:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5441s) So it's one parameter that we're exposing. If you went to the graph data science documentation, there's many others. Um, but, like, I got 13 groups running it here. If I, if I try to make it more refined with gamma equal to two, so it's one by default, it will give me, I think in this case, 14.

[1:31:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5461s) Um, so it basically splits it up into more groups. A lot of these community detection algorithms, also because they're hierarchical, you can choose to sort of, you can choose your level of granularity that you want, basically.

[1:31:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5476s) Um, as you go along and you start to understand your corpus better, you may wanna tune some of these hyperparameters. Um, and this gave us 14 different themes.

[1:31:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5486s) Um, let's see what we have here. Yeah, and it never really names a theme is sort of the point of this. So, like, everything that you get, um, is just directly from the document.

[1:31:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5505s) If I was to look at, like, the, the theme here, the names of the links, the, the names of the, the top files that were read, all of those things are just directly from the data.

[1:31:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5518s) And then because you get the URIs, the IDs of some of these documents, you can start doing the thing where you can go back to the outline shape or the search shape, and you can, like, search underneath all of these things.

[1:32:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5530s) So it gives the agent the ability to kinda jump between multiple tools like this.

[1:32:18](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5538s) Um, and so this-- these sorts of things are good for these estate-level questions. So things like where are issues concentrated, where is documentation thing-- thin, where does new documentation possibly belong on some other different type of subject that just came in if you're adding more data?

[1:32:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5556s) Um, all righty. So that brings us to the end of themes, and we have twenty-seven minutes about left. Are there any questions around this themes, uh, algorithm that anyone has really quickly?

[1:32:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5574s) Uh, yes. [speaking faintly] Yep. [speaking faintly]

[1:33:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5603s) Yeah. So that's a good question. And the question is, once you get the themes, so all the Leiden algorithm will do is assign an ID to these different groups, right?

[1:33:36](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5616s) And then what you do after that is sort of your choice. And your question is, are you just sort of taking from the data and just putting that there, or are you making some sort of inference afterward to label the different groups?

[1:33:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5631s) Here, we're doing the former. We're just taking what's in the data, and we're just showing it to you. Um, and the advantage of that is every time I run it, it'll be the same as long as the data stays the same.

[1:34:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5643s) If the data changes, it will change to reflect the data, so it's very stable. The disadvantage to it would be if your links in your documents, um, and the titles and things that are being scooped up, 'cause this is really only looking basically at document metadata and link metadata.

[1:34:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5661s) Um, if those things aren't already well labeled, this view might not be super informative right off the bat. That's why you have, you know, with, like, the, the GraphRAG, uh, methods that Microsoft came up with, they do a lot of heavy entity extraction because then what that will give is this sort of-- and they'll do, um, hierarchical

[1:34:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5680s) level summaries, right? So after they get the Leiden community, which is the same algorithm that they use, they'll do a summary that'll be LLM-driven on top of each theme.

[1:34:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5690s) Um, so that's possible to do, um, but then obviously it costs more money, it's slower, and the-- you-- if you run it twice, it might not return the same thing.

[1:35:00](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5700s) So there's trade-offs between, uh, each way of doing it. I'm showing you the sort of lighter way of doing it and the easier way, um, just 'cause it's faster.

[1:35:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5709s) And if you're just getting started with this, it might be easier to, to start there.

[1:35:13](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5713s) Um, may we have time for maybe one, maybe two more questions? Um, yes.

[1:35:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5721s) Um, how would you apply this to, like, uh, temporal data?

[1:35:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5725s) Temporal data?

[1:35:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5726s) Like, if it-- it's getting updated every day or every-

[1:35:31](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5731s) Yeah. I mean, um, I've seen customers, you know, basically every time-- 'Cause, 'cause the way Leiden works is it's, it really is this algorithm that where it will suck everything into a projection.

[1:35:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5746s) It will create its labels, and it'll, it'll die down, and then it will go away. So if your data is being updated constantly, um, you can recreate your IDs, um, and you can have almost like a time series of, of different theme IDs, for example.

[1:36:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5763s) Um, and then you can, if you want, create summaries sort of at the snapshot of when they existed, or you can see how they evolved over time. Um, but it's a very common use case.

[1:36:13](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5773s) A lot of people will actually use this even before AI. They'll use this for things like, um, fraud detection. So, like, looking at, um, like, credit chargeback fraud or, like, some of these other, like, um, anti-money laundering type of stuff to look at clusters.

[1:36:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5787s) And for that, they'll have to do it temporally. Like, they have to keep running it, um, to kinda see how things change over time and then predict the future, right?

[1:36:34](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5794s) So they're used in those scenarios too. Yep.

[1:36:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5798s) All right. One more maybe, and then I'm gonna have to move on.

[1:36:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5805s) All right. I th-- okay. Yes. [speaking faintly] So the question is, if you have a really big graph, how do you take this view that's being sent to the AI model and make it manageable? [speaking faintly]

[1:37:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5834s) Yeah. Um, so for this view, I think it's showing, like, fourteen themes. So there are cutoffs that you can make. So for example, you can say, like, for communities that are smaller than X threshold, like you don't necessarily need to highlight them.

[1:37:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5853s) Sometimes small communities are very important, which is why that conductance metric, which is powering that tightly interlinked, loosely interli-interlinked, you can also make that a cutoff. So if I'm interpreting your question correctly, it's sort of filtering down kind of the amount of information to what's most important to show the AI model on a larger graph.

[1:37:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5872s) Do I understand that correctly? [background noise] To make sure that the relationships on the nodes are correct. And, um, so- [background noise]

[1:38:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5896s) Y- [background noise] Yeah. Yeah. I mean, so i-in this scenario, we're really sort of trusting-- We're taking the source data kind of at its word, right?

[1:38:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5913s) If something links to a document, it links to the document, right? If it's erroneously linking to a document or there's a section that's malformatted, um, we wouldn't necessarily catch that.

[1:38:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5925s) But I suppose a good thing about something like an outline shape is that you can have your agent traverse it, um, automatically without you necessarily seeing it in small pieces, and it might be able to catch some of those things, right?

[1:38:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5938s) Um, so I suppose it does give you that navigation, would give you a way to kind of s-have an agent supervise the graph and understand, like, malformed data or data that's been a problem.

[1:39:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5950s) Um, but it's a good question. I don't know if we have a perfect solution to it. It's cleaning and cleaning messy data has always been a, you know, a thing.

[1:39:19](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5959s) Yep. All righty. Let's go ahead and, and move on 'cause I only have twenty minutes left, and I anticipate this last, uh, section will be-- this is the, the meatiest one, so hopefully we'll have enough time to go through it, especially because Claude has been slow.

[1:39:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5973s) So cross your fingers, um, 'cause this one is probably the heavier use of Claude. So there is a section here around just using the Neo4j CLI. Because we're just twenty minutes in, um, I'm probably and I'm not gonna go through it all the way, but I'll talk about it for a couple minutes.

[1:39:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=5990s) The Neo4j CLI is a, a CLI tool, so I can run it in the command line. So here, for example, um, if I opened up a terminal window, I can go ahead and copy it in, and it will run a, uh, query for me.

[1:40:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6007s) And it also has the ability here to grab the graph schema. Um, and what that enables me to do is, um,

[1:40:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6017s) basically understand what's in the graph and then write a query based on that. And if you have your agent, which in this case, 'cause we're using a coding agent, it can access the Neo4j CLI.

[1:40:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6027s) It gives it the ability to do this graph reasoning, read the schema, and then do flexible queries. So this is very useful if you have a question that you didn't anticipate and the agent's sort of gluing things together between the shapes, right?

[1:40:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6039s) It has to write its own custom Cypher query. It can do that very efficiently. Um, and I've seen a lot of improvements using this along with the skills, so much better than the text to Cypher experience that we've had, like, even as, as soon as a year ago or six months ago.

[1:40:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6055s) Um, if you are going to be writing Cypher or doing anything text to Cypher with an agent, I'd highly recommend using the Neo4j CLI, um, as well as the Cypher and GDS skills that we were just going over.

[1:41:08](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6068s) So with that in mind, I'll go ahead and take it on to our,

[1:41:13](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6073s) in this case, our last section. If you were to take this offline, there's other informational sections that come after it.

[1:41:21](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6081s) Um, but basically, we're gonna start asking, um, some questions. Uh, the first question that we're going to ask it, and I will jump right to it because a lot of this documentation we've already went over, is if you remember from the beginning, we had our different personas, right?

[1:41:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6100s) We had sort of our, our, who we called our Danny, which is the floor technician, right? And they might have a question like, "Hey, for this VIN, with this specific code that I'm getting, what fix-- what have we done that has fixed this on similar vehicles?"

[1:41:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6117s) Um, and the motivation behind this, right, and behind a lot of the data is, as an auto repair shop, you want to minimize your, um, comeback ratio, which is basically how many times a customer has to come back because, you know, the fix didn't work, right?

[1:42:16](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6136s) Um, and so the data, especially on the warehouse side, will show, um, some of that history combined with the documentation on the actual parts and the recalls and the bulletins.

[1:42:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6147s) So what I've done here is similar to the gentleman's questions before. I, uh, I have it explain the steps that it used for the different shapes. You can see it's loading, um, the auto server skill.

[1:42:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6160s) So that was the, um, the skill that we have here, where we've also contains all of the scripts we've been working with. So the outline search and theme script it has access to, as well as the run SQL and the MCP server.

[1:42:55](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6175s) Um, that was this skill that we were talking about before. So I've pre-written that for this. You can read it, um, if you like. But it basically gives it, um, some general guidance on how to access the warehouse and also deal with the different shapes in the command interface.

[1:43:13](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6193s) And you'll see what it will do here. Um, it will look for the document code. It'll do a full-text search. It'll use the tree shape here to find cross-links and causes.

[1:43:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6206s) Um, and then it will get the join paths. Um, and it will actually do the query for the VIN. Um, and when it does that- It will go ahead and come back with basically the part number that needs to be replaced.

[1:43:42](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6222s) And in this case, there was, like, an old ignition coil that got revised that it had to replace it with. Um, so if you look, right, it, it first it did full-text search to sort of ground, uh, the document with the right code.

[1:43:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6236s) Um, so it looked for the code and also the misfire or rough idle. Um, it found, uh, the, uh, top hit, which is this engine type. It confirmed its grounding, so it shows the links from that, uh, manual over to these different procedures.

[1:44:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6255s) And then from there, it did a join path. Um, it basically looked at all the work orders, um, with that DTC code, uh, for that part. Um, and then it was able to bring back the ultimate question, which is, like, you know, from all the parts that were replaced and how we dealt with that code, it went

[1:44:33](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6273s) to the warehouse to grab that information. So you can see, like, this is a simple question. So

[1:44:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6279s) if you were using vector search and, like, Genie, like their AI search in Genie, and, like, Databricks, you could do this. But what often happens is to f- to basically find what it would need to do that linking on that tree shape, it would have to do much more vector hits, which each of those is, is, uh,

[1:44:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6297s) a chance, right, for this to run into an issue and not find the right document or potentially hallucinate and get something on a misfire that maybe wasn't related to that specific part.

[1:45:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6309s) But because it grounded in the tree shape and everything, it was able to, to get the right information and then link that back to the table.

[1:45:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6317s) Um, and so it can be helpful, and it can help with efficiency in these smaller questions. Um, but then what can happen is when you get to the estate-level questions is where it can get really interesting.

[1:45:31](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6331s) So I'm gonna go ahead and just copy this question, and I'm gonna let it start running. Um, it seems like Claude is moving faster now, which is good. But I'll talk about it as it's running.

[1:45:41](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6341s) So this is a question around, "Hey, are there mismatches between our documented procedures and problems that we're seeing in the field? And what documents are missing?" Or sort of on the other half, like, what documentations are we not leveraging at all inside of our warehouse data, right?

[1:45:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6358s) 'Cause we have our documents which tell us about, like, the recalls and the bulletins and, and all this sort of stuff, um, and the manuals, but then we also have the work order history from our warehouse.

[1:46:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6369s) And so this is a question that someone in a supervisor role or someone in an analytics role might be interested in, right? Because it's sor-- and it's sort of like, uh, proving a negative or a mismatch because you're, you're sort of saying, like, you know, "I, I don't know what I'm looking for.

[1:46:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6387s) I'm looking for a gap though." And you can imagine that with a tool like vector search, this would be a very hard question 'cause sim-- or any type of similarity or lexical search.

[1:46:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6397s) Because, like, if you're just doing that alone, by definition, your search, you, you can't really search for a negative. You have to search for things that are there. Um, and so what I've found, and I think what we've found as a company, is that graphs can be very useful when you start having these more global types of

[1:46:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6413s) questions, these estate-level questions that you wanna ask, particularly if they're around patterns where you might not even know what you're looking for yet.

[1:47:03](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6423s) Yes. And so this one takes a while to run because it, at, at a certain point here, it does have to take a large number of codes and join some data together.

[1:47:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6435s) Um, but you'll see at the end here, it'll, it'll kind of, uh, trickle in, and it, it'll, it'll tell you how it went about finding everything.

[1:47:25](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6445s) Um, and oftentimes, what you see with these, it'll, it'll use some of the document data with some shapes, and then it will go back, and it will go back to the warehouse and query from there.

[1:47:35](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6455s) Um, it'll run through a lot of different things here.

[1:47:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6466s) Um, another thing just as, just while it's working on that, I did tell it here for sake of clarity to not use the Neo4j CLI. And the reason that I, I have that here is so that you can see it using the, um, the different shapes.

[1:48:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6481s) Um, just for sake of understanding how the different shapes kinda fit together and work, once you hand your agent the Neo4j CLI, um, it becomes very powerful because it can start writing custom Cypher queries, and there will be instances where it'll prefer doing that over some prefixed shape.

[1:48:20](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6500s) Um, and I'd say that as our sort of text-to-Cypher capabilities and as we k- keep building up more skills, it'll start to prefer more free-form Neo4j CLI stuff, uh, much more frequently.

[1:48:34](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6514s) And this one can sometimes take a little while to work through, so I'll give it, uh, another twenty seconds or so. Or maybe I can even open it up to a few questions while we're here, while we're waiting for this one.

[1:48:46](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6526s) Yep.

[1:48:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6527s) So, um, can you explain how

[1:48:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6531s) So basically, what it's going to do is it's going to look at the, um-- Well, I actually have to have it come back and remind me exactly how it, um, how it goes through.

[1:49:14](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6554s) Um, but basically, what you can see from the work orders, what has been worked on, right? And then you can sort of take the codes from there and see what documents it has been using.

[1:49:26](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6566s) Um, and then there's also going to be high comeback history on using some of the wrong parts. And so you sort of get this view of, okay, well, there's a bunch of documentation that we're maybe not hitting because we're getting all these comebacks from, like, using potentially the wrong part.

[1:49:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6584s) Um, so there, there's that half of it, and then there's some, um, documentation that we have, um, that's just not covered because we're using DTC codes that's just not covered inside of the warehouse.

[1:49:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6594s) So we'll know that, like, certain documentation hasn't really been hit at all. Um,

[1:50:01](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6601s) so here it actually took a little bit more of a, um... And it will do this sometimes. Here it used semantic expansion, um, so it actually went in and did a much more comprehensive, um, semantic expansion search.

[1:50:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6617s) It's supposed to, in this case, use the outline. I think because it had the hierarchical URI, um, it was probably able to do this a little bit better. Um, but if I go back up,

[1:50:31](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6631s) um... Where did it show? Um, yeah, it, well, it was able to show here the field codes essentially, um, that it was missing. So, um, there was basically two field codes, um, that got a, um...

[1:50:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6650s) Yeah, the headline mismatch or diagnosed with the wrong procedure. Um, so there's basically this mismatch, um, where a couple of these parts among or something, typical in the library document.

[1:51:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6665s) Yeah, so there, where the correct repair isn't, um, referred to basically, um, and that's causing a problem for the warehouse. Um, and then there's other, um, DTC codes that occur in the field exactly that have two, um, that have no code-level documentation, which are these two.

[1:51:24](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6684s) So there's codes that are occurring which basically aren't documented that it was able to find. Um, it's a shame here that it didn't use the outline. It's supposed to do that because basically, when it uses the outline, it's able to traverse through and find all of the links a little bit more efficiently.

[1:51:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6700s) It wouldn't have done the comprehensive full text search call that took it a while in this case. Um, but even here, because it had the hierarchical URIs and that semantic expansion on the, um, full text search, it, it was able to eventually find it.

[1:51:54](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6714s) Um, but it's a good lesson that when it does use the outline, it can come back faster because it can traverse out on the different links. Um, the other thing here, the last one that I'll run 'cause we only have eight minutes left, um, I'll go ahead and copy it here.

[1:52:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6730s) Um, and this one is primarily, let me go ahead and copy it, um, is gonna leverage our, uh, themes shape. So, um, this is asking for common patterns across all our bulletins and recalls, um, and how many of the cars sort of each affects.

[1:52:29](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6749s) So we have our work order history inside of our, uh, warehouse and, uh, basically what we wanna find out is like which, you know, how does the sort of themes that we have correlate with, um, our work order history.

[1:52:47](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6767s) Um, and to do that, it runs that theme pattern to be able to pull out the high-level themes and then correlate it back to our work orders.

[1:52:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6777s) And so it's a coverage question, so you can see it'll run the themes.

[1:53:05](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6785s) And then, um, eventually here, we'll go ahead and bring back,

[1:53:10](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6790s) um, all of the, all of the relevant information. But basically what it's doing is it's, it's finding the themes. It's going to go and pull sort of the types of fixes that it has in the work order history, and then it's gonna do a join of sorts to kind of group them under the right theme.

[1:53:31](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6811s) While this is running, are there any other questions about this theme shape or anything? I know other people there's... Yes, go ahead.

[1:53:40](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6820s) How are we identifying this shape? At what point-

[1:53:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6823s) Sorry, say that one more time.

[1:53:45](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6825s) At what point are we identifying this shape? Is it during the graph construction?

[1:53:50](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6830s) How-- During the graph construction...

[1:53:53](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6833s) When we are defining this shape.

[1:53:56](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6836s) Yeah.

[1:53:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6837s) Shape.

[1:53:57](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6837s) Yeah. So the shapes that I, that I-- are actually defined inside of our specs. So if you go here, our outline shape, um, our specs and our docs. So this was defined more thinking through like, "Here's what we want it to look like for the agent."

[1:54:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6855s) And then once we define that, we come up with sort of the query structure that we want, and we use that to inform our data model. Um, and all of the documents were loaded essentially, um, into that data model that I went over about an hour and a half before, towards the beginning of the course.

[1:54:31](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6871s) Um, it's that Has Containment tree with all of the links between it.

[1:54:37](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6877s) Um, and you can see here it'll, um, it'll bring back the, um, that it used themes, it didn't use search, um, and then it, uh,

[1:54:51](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6891s) also, uh, queried the, um... If I go up here, you can see the cars affected. So it basically went to the warehouse with that connection shape, um, and it was able to sort of group them under, um, the different, uh, theme types.

[1:55:07](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6907s) So it's like that theme, the different theme types, um, that we have, and then kind of with the cars and the work orders grouped underneath by percentages.

[1:55:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6915s) Um, yes. Over here. [background noise] Yeah, either one.

[1:55:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6923s) So do you have a sense of accuracy of how, how, how well these, or how much

[1:55:34](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6934s) Do you, do I have a sense of accuracy of how well these do when the knowledge bases expand? Um, for this, I haven't run any specific benchmarking on like exactly what I've shown you today.

[1:55:43](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6943s) But I will say that when we do have customers that run these, they'll often come up with their own custom ontologies, and then they will run benchmarks, uh, that will basically say like, "How effective is this query pattern against, you know, ordinary vector search," for example.

[1:55:58](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6958s) And you would use that to prove it out on, on a specific type of dataset. Here for this course, this is more conceptual to understand kind of like the different shapes that you would use to help ground your data.

[1:56:09](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6969s) Um, and it can always use work in terms of how you were to, like, build the skill, right? To make sure it guides through the right thing so it uses, you know, each step efficiently essentially.

[1:56:23](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6983s) Are there any other questions? Yep.

[1:56:28](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6988s) In, in this case, the, the sparks code for, uh, your page between graphs and documents and tables and data-

[1:56:38](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6998s) Yeah

[1:56:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=6999s) ... are there cases where that is a bit clustered? For example, like, say you have data and you have like a good connection between the data points and, and you want to make analysis like that?

[1:56:52](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7012s) Well, maybe I don't fully understand. So you have, um, a connection point between,

[1:56:59](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7019s) between what? Between...

[1:57:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7022s) Let's say, let's say hopefully you have the financial statements. You have like a database structure data.

[1:57:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7035s) Yeah.

[1:57:15](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7035s) Also you have like the CEO comments about the basic info.

[1:57:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7037s) Yeah.

[1:57:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7037s) You, you will have a very-

[1:57:17](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7037s) Yeah. No, we do not have... So right now, so now that I understand your question, the question is, is there a link between the connection structured data graph and the unstructured one, and there is not in this case.

[1:57:29](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7049s) It's, they're completely unlinked, and the agent is sort of extracting, you know, a part from one or it's extracting like, um, yeah, basically like a code, right? And then referring back to the document.

[1:57:44](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7064s) So it's, it's using that to kind of do its own join in real time. You could create that linking, and that could be valuable for more deterministic mapping. Um, here we didn't do it, and the main reason I didn't do it was really for speed of getting started 'cause I did kinda wanna provide code that would be

[1:58:02](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7082s) easy and also model agnostic. Um, and this is that. And then if you wanted to later create those connections or those links, you could.

[1:58:12](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7092s) Um, any other questions? All righty. Well, thank you everyone. Hopefully, that was, uh, informative. [clapping] If you wanna go back, the, the workshop's available for you to take. It's, it's online.

[1:58:27](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7107s) I am gonna have to take the Anthropic key and the BigQuery key down eventually. I'll, I'll leave the BigQuery one up, uh, for a while, but the Anthropic one I'll have to, uh, eventually take down.

[1:58:39](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7119s) So unfortunately, you will need to provide your own key. But other than that, you should be able to take it just fine. Um, and I think that's all I have, so I'll leave it for my next guest.

[1:58:49](https://www.youtube.com/watch?v=kRkcNOsRyYg&t=7129s) Hopefully, you guys have enough time to jump into your next session. [outro music]
