AI Engineer World's Fair 2026
Beyond RAG: A Relational Context Engine That Cuts Token Burn — Unblocked
Slide chapters
20 slide and demo groups · Timestamps are approximate. 27 additional groups were omitted from this pass.
Read the talk
Beyond RAG: A Relational Context Engine That Cuts Token Burn
Peter Werry explains how Unblocked connects code with organizational history, then builds a document query engine that turns natural-language questions into validated, time-scoped database queries.
From a talk by Peter Werry
At a glance
Ideas worth remembering
Task-specific context connects code with architecture documents, PRs and conversations, reducing the discovery work an agent must perform.
Identity, time, state and aggregation questions require structured queries alongside semantic retrieval.
Schema discovery and identity matching can use procedural methods, reserving the model for query synthesis or ambiguous matches.
Validate model-generated operations and inject tenant constraints outside the model’s control.
Retries can produce an allowed query that misses the intended meaning. The authentication example needed full-text search as well as error feedback.
The local context engine simulator tests task-specific bundles through A/B comparisons on the same task.
The new employee problem follows agents into the codebase
A new employee can read the code and still miss why it works that way. Architecture documents explain some decisions; pull requests and conversations explain others. Incident history carries the “battle scars” that make an apparently straightforward change risky. Peter Werry, founding engineer at Unblocked, starts with this familiar discovery problem: before coding agents, people assembled that context themselves.
A context engine pulls that scattered institutional knowledge together. In Werry’s definition, the output is grounded context: information connected across sources, selected for the task, personalized to the user and aware of permissions. The useful unit is therefore more than a relevant document. It is a set of related evidence that helps explain the work in front of the agent.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give Claude the architecture and the discussion behind the code
The first comparison asks Claude to plan an optimization of an internal component. Without a context engine, Claude starts by searching the codebase and assembling its own understanding. It eventually produces a recommendation, although Werry adds the familiar qualification: it at least “confidently projects” that it has figured things out.
With Unblocked attached, Claude calls the context engine and receives a task-specific bundle. The bundle connects source code with Notion architecture documents, pull requests and Slack conversations about optimizing the component. Those links also give Claude places to investigate further. Instead of discovering every relationship through successive searches, the agent starts with related material already assembled.
Werry reports $1.29 and about a minute and thirty seconds for the session with context, compared with up to $2.60 and three minutes without it—approximately a 50% cost reduction. These are results from the displayed sessions, rather than a general benchmark. They illustrate the proposed economy: useful context can reduce the exploration an agent performs before it can produce a plan.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A regression needs history as well as code
The next example starts with a change in product behavior. Richie, a colleague working on Unblocked’s code review product, notices that it is surfacing fewer issues. Unblocked narrows the investigation to a likely cause: a model switch changed the review behavior. That remains a diagnosis presented in the demonstration; the recording does not establish a controlled causal test.
Richie then asks Unblocked to fix it. The system creates a pull request whose description connects the earlier PR associated with the regression, a Slack discussion and relevant data. The important relationship runs across systems: a code change records what changed, a conversation supplies the surrounding interpretation, and product data helps explain the symptom. Together, those sources inform the proposed fix.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
“My PRs last week” requires a query
The build portion begins with a distinction inside the same organizational data. PRs, reviews, comments, tickets, documents and conversations contain searchable content, but they also have structure. RAG can find code, the PR that changed it and the thread debating it. An agent loop with additional file-search tools can carry that investigation further.
Other questions require selecting and computing over records:
- Identity and time: Which PRs did I merge last week?
- Counts and rankings: Who reviewed payments the most?
- State: Which PRs are open?
- Time series: Show a weekly ratio chart since January 1st.
The time-series request was how Richie’s investigation began. Retrieving semantically similar text does not itself compute a weekly series over the underlying records. That requires a query engine. Werry describes retrieval as having two halves: semantic retrieval for relevant content, and structured retrieval for filters, temporal conditions and aggregations.
Generating the query is the easy part of this design. The surrounding system must tell the model which fields exist, resolve a person’s name to the identifier used by the source system, validate the proposed operations and recover from errors. A fluent query can still be dangerous or cross tenant boundaries. Natural language makes the interface convenient; it does not make the resulting database operation trustworthy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover the schema, resolve the person, then synthesize
The open-source Document Query Engine starts as an unwired demo: a question produces nothing. Werry adds the surrounding capabilities in stages. The first is automatic schema discovery. For the Mongo document store, the engine samples documents and derives a schema at runtime, giving the model a description of the data it can query.
Inferring enums—fields with a limited set of possible values—is one tricky part of that discovery. An LLM can help, but Werry’s preference is to get as far as possible with procedural methods. Work that looks uncertain at first can still admit a deterministic solution. The resulting demo exposes the schema of a GitHub pull-request collection along with indexes that can help query performance. Because discovery uses samples, it describes the sampled documents rather than guaranteeing that every possible document shape has been seen.
Identity resolution follows the same preference for simple machinery. A human name in a question may differ from a GitHub username or an identity in another system. Procedural matching, including fuzzy matching, connects those identities. If several candidates remain, the model can act as a tiebreaker over a supplied list rather than inventing an identifier.
Query synthesis then receives the discovered schema, the current date and time, and potentially information about the requesting user. Each input removes an ambiguity: the schema constrains field choices, the clock grounds relative periods, and user information lets “my PRs” refer to the right person. With these pieces connected, the demo generates a query and pulls documents.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep database permissions outside the model’s proposal
Validation is the most important stage in Werry’s account. The proposed query is untrusted even when synthesis has the right schema. Two kinds of control matter:
- Operator restrictions: Reject dangerous operations, such as a function operator that executes raw JavaScript on the database server.
- Protected fields and tenant constraints: Keep hidden metadata out of the model’s choices. If a tenant ID must constrain every query, inject that constraint behind the scenes rather than asking the model to manage it.
The next question makes validation observable: find PRs with authentication in the title. The model produces a regular-expression approach, and the demo’s validator rejects it because regular expressions are disallowed. This is an application policy demonstrated by this engine. Rejection preserves the restriction, but it also leaves the user’s useful question unanswered.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A safe retry can still answer the wrong question
The retry mechanism feeds the error back to the LLM and asks for a revised query. Technically, it is a loop with a maximum retry count. Validation errors—and, in the broader design, database execution errors—become information the model can use to correct its next attempt.
The authentication-title question now gets through after a couple of attempts, but the correction introduces a different problem. The query uses an exact title match, and no PR has that exact title. The model has found an allowed operation while losing the intended search behavior. Passing validation establishes that a query is permitted; it does not establish that the query expresses the user’s intent.
Adding full-text search supplies the missing capability, and the next attempt succeeds in the demonstration. The same request—find PRs with authentication-related text in their titles—produces three different outcomes:
| Proposed query behavior | Validation or result | What changes next |
|---|---|---|
| Regular-expression title search | Rejected by the demo’s validation policy | Return the error to the model |
| Exact title equality | Allowed, but finds no matching PRs | Recognize that a permitted query can still miss the intent |
| Full-text title search | The next demonstration succeeds | Supply a permitted search capability, rather than retrying exact equality |
The relationship to notice is that feedback and capability do different jobs. Error feedback helps the model stop using a forbidden operation. Full-text search gives it an operation capable of answering the original question. The completed engine combines structured retrieval with a text-search capability rather than expecting retries alone to solve the problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn retrieved records into an answer—and test context locally
The final agent demonstration combines the capabilities in a broader question: “What authentication-related things have we been working on last week?” The engine supports filters, aggregations and time-scoped queries; the chat interface shows the queries it creates, then reasons over the retrieved data. Structured retrieval supplies the records that the answer needs, while the agent turns those records into an explanation.
The ending returns to the practical question raised by the first cost comparison: does pre-gathered context help your agent on your tasks? Werry introduces an open-source context engine simulator for environments where connecting Unblocked immediately is impractical. It runs locally, creates a context bundle for each task, and uses that bundle in an A/B test against the same task. That offers a way to investigate the value of context in a particular workflow before wiring in the full product.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Unblocked’s open-source tool for comparing coding-agent runs with and without gathered context.
Read the complete timestamped transcript
11 slide groups · 9 demo milestones. Extracted from the recording; expand builds to follow changes. Timestamps are approximate. 27 additional groups were omitted from this pass.
- 0:12
Are we up? Okay. Can you hear me just like this just fine? Everyone good? Okay. Um, so we're, we're gonna be doing a bit of a speed run here 'cause, uh, uh, we've only got twenty minutes. But, um, I'm c- I'm Peter from Unblocked, and at Unblocked, we build a context engine. So I'm gonna show you a couple of things. The talk's gonna be split into two parts. Um, I'm gonna show you what a context engine is and how it can add value to your organization, and then we're
Beyond RAG
Peter Werry introduces a relational context engine and plans to build one of its components after explaining its value.
Text in image
BEYOND RAG See a relational context engine reduce token burn Peter Werry Unblocked
- 0:42
gonna actually go and build a component of a context engine, and I'm gonna show you a couple of open source projects that you can, uh, go and play with after this.
Agenda
Peter Werry will explain context engines and their value, then build a component and share open-source projects to explore.
Text in image
Agenda • What is a Context Engine? • How does it deliver value? • Let’s build it! Unblocked
- 0:54
So there's the agenda. What's a context engine? How does it deliver value? And then we'll build. Um, so let, let's just hop back in time maybe a few years to the times before AI agents. And in those days, um, you were the context layer.
AI-generated code and team context
AI-generated code should reflect the knowledge of a longtime teammate, but the slide says it falls short today. That context includes code, documentation, and an organization’s shared history.
Text in image
AI-generated code should feel like it was written by someone who's been on your team for years. But it doesn’t today. Unblocked
You have been the context layer
Before AI agents, people built context by searching code and documentation and learning an organization’s shared history.
Text in image
For years, you have been the context layer. Unblocked
Remember how you built context?
A new employee trying to understand auth validation must piece together scattered records, including conflicting claims about whether internal traffic checks tokens.
Text in image
Remember how you built context? Outdated change GitHub — 14d ago Refactor auth flow - #3243 Replaced legacy JWT flow. Unknown status Jira — 5d ago UNB - 1337 Rejecting valid sessions/ Outdated docs Notion — Yesterday System Architecture Tokens validated by Auth Service. Tribal knowledge Slack — 3d ago #eng-discussion Internal traffic skips the check. Symptom, not cause Datadog — 4 hrs ago Auth.latency.p95 Spikes correlated with deploys. Conflicting info Confluence — 1mo ago Auth Service Overview All requests check tokens. How does auth validation work? Unblocked
- 1:15
Imagine that you're a new employee to an organization, and you're just trying to get to grips with what's, what's happening. What you usually end up having to do is look at the... look at a whole bunch of code, rip around documentation systems, just trying to find the information that you need to get your work done. Um, and not to mention, like, all of the, the sort of shared history that's built up over time through an organization dealing with incidents and the battle scars that come with it. Um, so what a
How the context engine works
The context engine brings scattered knowledge from code, docs, conversations and other sources into interrelated, personalized, permissions-aware context for agents and other workflows.
Text in image
How the context engine works Sources Code + PRs Planning Tools Docs Conversations Incident Management Workflows Coding Agents Code Review Messaging Apps MCP CLI API Unified system context Merges signals from all sources before delivery to the agent. Targeted retrieval Retrieve only the information agents need from the graph. Conflict resolution Recency + authority signals resolve contradictions automatically. Personalized relevance Scoped to your repos, teammates, and work history. Token optimization Ranked, compressed context — no waste in the prompt window. Permission enforcement Permissions and policies enforced automatically across systems. Unblocked
- 1:45
context engine does is it takes all of that institutional knowledge, and it pulls it in. And what you get, the scattered context that goes into the context engine, and out the other end, you get grounded context, which means that the context is all interrelated to each other. It's intent-specific. It's personalized. And it's permissions-aware.
Demo
The speaker moves from the context engine overview to a demonstration.
Text in image
DEMO Unblocked
- 2:12
And so I'm gonna show you exactly what I mean by that now. Let's imagine
Terminal prompt
The presenter prepares to show a Claude session planning SourceMark engine optimization without a context engine. The terminal is still idle.
Text in image
Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code ~/Dev/document-query-engine main *2 10:48:26
2:18Demo milestone
- 2:21
that I'm gonna go into here into Claude. I'm gonna show you a couple of interesting sessions that I had. So, um, I asked Claude here, this is without a context engine attached, to create a plan to optimize a component in our layer called the Source Mark engine. And, uh, I'm sure you're all familiar with what Claude does when it starts a new task. It's like a new employee. It doesn't understand anything about your, your code base or your organization. So the first thing it does
Planning without a context engine
Without Unblocked, Claude searches the repository, reads hot-path files, and follows the GitRunner import while building an optimization plan.
Text in image
create a plan to optimize the sourcemark engine. do not use unblocked • Caveman mode + no unblocked. Find sourcemark engine first. Searched for 2 patterns, ran 1 shell command • SourceMark lives in frontend (node + ide). Explore structure. Searched for 1 pattern, ran 1 shell command • Read core hot-path files. Read 2 files Loaded ../CLAUDE.md Loaded ../AGENTS.md Loaded ../.agents/VIEWS.md Loaded ../.agents/STYLES.md Loaded ../.agents/REMOTION.md • Read remaining hot files + git layer. Read 4 files • Now git layer — the real cost center. Check GitRunner API + caching. Ran 6 shell commands • GitRunner imported from @shared/node/git/GitRunner but not in frontend/shared. Check path alias. Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code Jump to bottom (ctrl+End) ↓ [CAVEMAN] ← for agents
2:34Demo milestone
- 2:51
is it greps around your code base and, and tries to figure out what's going on. So you can kind of see that here. And, you know, it, it figures it out eventually, or at least, you know, it, it confidently projects that it's figured it out. Um, and so it provides kind of like a, a recommendation for me there. But now I'm gonna show you what it looks like when you attach a context engine to the back end. So now what it does is it calls the Unblocked context engine, and Unblocked now delivers a very
Unassisted optimization plan
Without a context engine, Claude proposes measuring Git operations before optimizing and keeping regression fixtures green. It also reports that the engine is orphaned and won't compile, leaving revival as an open question.
Text in image
9. SnippetFinder.findSegment O(fileLines·snippetLen). SnippetFinder.ts:71 slices+ArrayUtils.equals per line, allocating a slice each iteration. Index candidate lines by first-line hash, compare only on match. Matters for large files. 10. computeOtherDiffs caching. SourceMarkCalculator.ts:443 rebuilds fileContentLines + grep pattern per edge. Cache grep pattern per (mark, fromCommit). Heuristics (:482 large-file prune, :486 grep gate) already good — leave them. 11. getOtherDiffs lazy already (RangeAdjust.ts:110) — only fires on range-fail. Fine. The fuzzy findIntersectionIndex nested loop has early-exit thresholds; leave unless profiled hot. Cross-cutting 12. Instrument first. git.commandLog already records commands. Before touching anything, dump per-op counts/durations for fullRecalculation and a hot-file getSourceMarksForFile on a real large repo (the hardcoded clio/themis in SourceMarkScheduler.ts:62). Establishes baseline + confirms ordering above. 13. Regression safety: the engine/fixtures/*.diff corpus is the golden set — every change must keep fixtures green. Add a marks/sec benchmark harness alongside. Suggested sequencing Instrument (12) → P0 (1–4, pure wins, fixture-safe) → measure → P1 (5–8) → measure → P2 only if needed. --- Want me to: - (a) dig into the revive question — trace what #29884 removed and what CES/hybridSourcemarkEngine expects — before planning further, or - (b) write this up as a doc / start on P0 items? Also flag: are you optimizing the client engine (this TS, per-IDE) or a server-side reimplementation? Changes whether parallelism/caching lives in-process or needs a shared cache. Say the word and I'll refine. ✻ Crunched for 3m 10s ※ recap: Goal: plan to optimize the SourceMark engine. I found it's orphaned dead code (deps removed in #29884, won't compile) and produced a prioritized optimization plan. Next: you pick whether to investigate reviving it or write up/start the P0 wins. (disable recaps in /config) Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code [CAVEMAN] ← for agents
2:58Demo milestone
SourceMark optimization plan with Unblocked
Claude Code calls Unblocked, then combines local code with PRs, a Notion architecture proposal, Slack threads and Linear tickets to identify serial resolution and a repeated linear scan.
Text in image
/ask-unblocked help me create a plan to optimize the sourcemark calculator Called unblocked • Good coverage. Missing: hydrated PR + sourcemarkengine.ts source. Grab local files + PR #970 in parallel. Read 1 file, called unblocked • Read hot-path sections to anchor concrete targets. Searched for 1 pattern, read 1 file, ran 3 shell commands Loaded ../../../../CLAUDE.md Loaded ../../../../AGENTS.md Loaded ../../../../.agents/VIEWS.md Loaded ../../../../.agents/STYLES.md Loaded ../../../../.agents/REMOTION.md • Have everything: 6 PRs, Notion arch proposal, 4 Slack threads, 4 Linear tickets, both source files read locally. Plan below. (Plan = multi-step sequence; writing clear, not caveman.) Optimize the SourceMark Calculator — Plan Bottom line: the calculator's stated limitation — "not yet o[illegible]ution across many marks/commits" (Slack deep-dive) — is real Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code Claude Code v2.1.198 Opus 4.8 (1M context) with high effort · API Usage Billing ~/Developer/unblocked 1 MCP server needs authentication · run /mcp Jump to bottom (ctrl+End) ↓ [CAVEMAN] ← for agents
3:11Demo milestone
Show 2 builds
- 3:11
A coding assistant calls Unblocked and begins a SourceMark optimization plan using several sources.
Text in image
/ask-unblocked help me create a plan to optimize the sourcemark calculator Called unblocked • Good coverage. Missing: hydrated PR + sourcemarkengine.ts source. Grab local files + PR #970 in parallel. Read 1 file, called unblocked • Read hot-path sections to anchor concrete targets. Searched for 1 pattern, read 1 file, ran 3 shell commands Loaded ../../../../CLAUDE.md Loaded ../../../../AGENTS.md Loaded ../../../../.agents/VIEWS.md Loaded ../../../../.agents/STYLES.md Loaded ../../../../.agents/REMOTION.md • Have everything: 6 PRs, Notion arch proposal, 4 Slack threads, 4 Linear tickets, both source files read locally. Plan below. (Plan = multi-step sequence; writing clear, not caveman.) Optimize the SourceMark Calculator — Plan Bottom line: the calculator's stated limitation — "not yet o[illegible]ution across many marks/commits" (Slack deep-dive) — is real Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code Claude Code v2.1.198 Opus 4.8 (1M context) with high effort · API Usage Billing ~/Developer/unblocked 1 MCP server needs authentication · run /mcp Jump to bottom (ctrl+End) ↓ [CAVEMAN] ← for agents
- 3:15
A terminal displays a SourceMark optimization plan and its first identified hotspot.
Text in image
Loaded ../../../../CLAUDE.md Loaded ../../../../AGENTS.md Loaded ../../../../.agents/VIEWS.md Loaded ../../../../.agents/STYLES.md Loaded ../../../../.agents/REMOTION.md • Have everything: 6 PRs, Notion arch proposal, 4 Slack threads, 4 Linear tickets, both source files read locally. Plan below. (Plan = multi-step sequence; writing clear, not caveman.) Optimize the SourceMark Calculator — Plan Bottom line: the calculator's stated limitation — "not yet optimized for concurrent resolution across many marks/commits" (Slack deep-dive) — is real and visible in code. Resolution is serial at three levels: per-mark, per-chunk, and per-edge. Plus there's a hot O(n) scan on every edge and no git-call memoization across marks that share files. Below, ranked by ROI. Confirmed hotspots (grounded in current code) #: 1 Optimization: Map.has() not Array.from(keys).includes() Where: SourceMarkCalculator.ts:379 Problem: O(n) linear scan over pointCache keys, per edge, per mark. Pure waste. Impact: Low-Med Effort: Trivial Risk: None #: 2 Optimization: Parallelize per-mark resolution Where: SourceMarkScheduler.ts:348 for (const mark of sourceMarks) Problem: Each mark's calc fully awaited before next. Marks a[illegible] Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code Jump to bottom (ctrl+End) ↓ [CAVEMAN] ← for agents
- 3:11
- 3:20
comprehensive but task-specific context bundle for this task. And what you can see is that it, it's referencing not only the source code, but also, you know, Notion documents that talk about the architecture, um, pull requests, Slack conversations that were had about optimizing this particular component. Um, and the, the, the net out is that you get this really tight context bundle with all these links. So now, uh, if, if, um, you know,
Concurrency recommendations
The plan proposes bounded concurrency for mark and chunk loops, with cache and startup constraints. It calls for benchmarks and regression tests; high-risk edge pipelining is only worthwhile for repos with heavy merge topology.
Text in image
Phase 2 — concurrency (the actual limitation): - #2 + #3: replace serial for/chunk loops with bounded-concurrency fan-out (e.g. p-limit style, cap = N). Must respect: SourceMarkStore cache writes (putSourcePoint/putFilePoint) and the LRU git cache. The 2022 arch proposal explicitly designed the server-side pipeline (K8s ephemeral jobs) around the fact that calculation is CPU-bound, needs no SCM tokens, and can run in parallel (Notion) — the client calculator never got that treatment. - Watch the startup-ordering constraints from PR #2854: store must finish downloading marks before per-file lookup, and partial-recalc must complete before full-recalc — don't let parallelism re-introduce the "event loop filled with full-recalc jobs" bug it fixed. Phase 3 — deeper (optional, high-risk): - #5 edge pipelining. Only worth it for repos with heavy merge topology. The merge-reconciliation logic makes this the riskiest; gate behind benchmarks. Before you start - Benchmark first. No perf numbers exist in any source. Instrument recalculateForCommit with the existing git.commandLog timing (Scheduler.ts:478) to find whether time is in git I/O (→ #4 wins) or CPU traversal (→ #2/#3/#5 win). This decides Phase 2 vs Phase 4 priority. - Note the hybrid engine. ISourceMarkProvider has a hybridSourcemarkEngine flag for server-side resolution (Slack). Check whether the calculator hot path even runs client-side for target repos before optimizing the client loop. - Tests. Merge-commit + file-rename resolution are the fragile paths (see PR #2564 / UNB-507); any concurrency change needs regression coverage there. --- Sources - SourceMarkCalculator.ts — resolution core (read locally: 691 lines) - SourceMarkEngine.ts — singleton entry point (92 lines) - SourceMarkScheduler.ts (local) — serial mark/chunk loops at :213, :348 - SourceMarks Service Arch Proposal (Notion, 2022) — fetch/calc decoupling; calc is parallelizable - PR #970 — incremental / tree-SHA caching - PR #1236 — optimize initial calculation - PR #2854 — startup ordering - PR #3026 — compress LRU cache keys (UNB-665) Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code Jump to bottom (ctrl+End) ↓ [CAVEMAN] ← for agents
3:30Demo milestone
Plan sources and next steps
The plan draws on code, architecture notes, pull requests and Slack. It recommends benchmarking first to distinguish Git I/O from CPU traversal before choosing an optimization.
Text in image
Phase 3 — deeper (optional, high-risk): - #5 edge pipelining. Only worth it for repos with heavy merge topology. The merge-reconciliation logic makes this the riskiest; gate behind benchmarks. Before you start - Benchmark first. No perf numbers exist in any source. Instrument recalculateForCommit with the existing git.commandLog timing (Scheduler.ts:478) to find whether time is in git I/O (→ #4 wins) or CPU traversal (→ #2/#3/#5 win). This decides Phase 2 vs Phase 4 priority. - Note the hybrid engine. ISourceMarkProvider has a hybridSourcemarkEngine flag for server-side resolution (Slack). Check whether the calculator hot path even runs client-side for target repos before optimizing the client loop. - Tests. Merge-commit + file-rename resolution are the fragile paths (see PR #2564 / UNB-507); any concurrency change needs regression coverage there. --- Sources - SourceMarkCalculator.ts — resolution core (read locally: 691 lines) - SourceMarkEngine.ts — singleton entry point (92 lines) - SourceMarkScheduler.ts (local) — serial mark/chunk loops at :213, :348 - SourceMarks Service Arch Proposal (Notion, 2022) — fetch/calc decoupling; calc is parallelizable - PR #970 — incremental / tree-SHA caching - PR #1236 — optimize initial calculation - PR #2854 — startup ordering - PR #3026 — compress LRU cache keys (UNB-665) - PR #2564 — SourceMark thread caching (UNB-507) - Slack: verbose engine deep-dive — hybrid flag, known limitation - Linear UNB-1248 — improve topic/expert accuracy with source git Want me to run the benchmark instrumentation, or start on Phase 1 (fix #1 + #4)? Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code Cooked for 1m 48s Jump to bottom (ctrl+End) ↓ [CAVEMAN] ← for agents
3:37Demo milestone
- 3:50
Claude wants to go back and fetch more information, it has all these great jump-off points. But the really interesting thing is, is this. So here I spent, um, you know, $1.29 on this, and it took about a minute and thirty seconds. Um, that was with the context engine. But without, I'm spending up to, you know, $2.60 and three minutes. So what you see essentially is a fifty percent reduction in cost, which is pretty incredible.
Context-engine session usage
The context-engine session costs $1.29 and takes 1m 33s of API time. The speaker reports up to $2.60 and three minutes without the context engine.
Text in image
the fact that calculation is CPU-bound, needs no SCM tokens, and can run in parallel (Notion) — the client calculator never got that treatment. - Watch the startup-ordering constraints from PR #2854: store must finish downloading marks before per-file lookup, and partial-recalc must complete before full-recalc — don't let parallelism re-introduce the "event loop filled with full-recalc jobs" bug it fixed. Phase 3 — deeper (optional, high-risk): - #5 edge pipelining. Only worth it for repos with heavy merge topology. The merge-reconciliation logic makes this the riskiest; gate behind benchmarks. Before you start - Benchmark first. No perf numbers exist in any source. Instrument recalculateForCommit with the existing git.commandLog timing (Scheduler.ts:478) to find whether time is in git I/O (→ #4 wins) or CPU traversal (→ #2/#3/#5 win). This decides Phase 2 vs Phase 4 priority. - Note the hybrid engine. ISourceMarkProvider has a hybridSourcemarkEngine flag for server-side resolution (Slack). Check whether the calculator hot path even runs client-side for target repos before optimizing the client loop. - Tests. Merge-commit + file-rename resolution are the fragile paths (see PR #2564 / UNB-507); any concurrency change needs regression coverage there. --- Sources - SourceMarkCalculator.ts — resolution core (read locally: 691 lines) - SourceMarkEngine.ts — singleton entry point (92 lines) - SourceMarkScheduler.ts (local) — serial mark/chunk loops at :213, :348 - SourceMarks Service Arch Proposal (Notion, 2022) — fetch/calc decoupling; calc is parallelizable - PR #970 — incremental / tree-SHA caching Session Total cost: $1.29 Total duration (API): 1m 33s Total duration (wall): 1h 11m 50s Total code changes: 0 lines added, 0 lines removed Usage by model: claude-opus-4-8: 25.1k input, 6.1k output, 429.5k cache read, 128.1k cache write ($1.29) Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code Settings Status Config Usage Stats Esc to cancel
3:57Demo milestone
Unassisted optimization plan
Without a context engine, Claude proposes measuring Git operations before optimizing and keeping regression fixtures green. It also reports that the engine is orphaned and won't compile, leaving revival as an open question.
Text in image
9. SnippetFinder.findSegment O(fileLines·snippetLen). SnippetFinder.ts:71 slices+ArrayUtils.equals per line, allocating a slice each iteration. Index candidate lines by first-line hash, compare only on match. Matters for large files. 10. computeOtherDiffs caching. SourceMarkCalculator.ts:443 rebuilds fileContentLines + grep pattern per edge. Cache grep pattern per (mark, fromCommit). Heuristics (:482 large-file prune, :486 grep gate) already good — leave them. 11. getOtherDiffs lazy already (RangeAdjust.ts:110) — only fires on range-fail. Fine. The fuzzy findIntersectionIndex nested loop has early-exit thresholds; leave unless profiled hot. Cross-cutting 12. Instrument first. git.commandLog already records commands. Before touching anything, dump per-op counts/durations for fullRecalculation and a hot-file getSourceMarksForFile on a real large repo (the hardcoded clio/themis in SourceMarkScheduler.ts:62). Establishes baseline + confirms ordering above. 13. Regression safety: the engine/fixtures/*.diff corpus is the golden set — every change must keep fixtures green. Add a marks/sec benchmark harness alongside. Suggested sequencing Instrument (12) → P0 (1–4, pure wins, fixture-safe) → measure → P1 (5–8) → measure → P2 only if needed. --- Want me to: - (a) dig into the revive question — trace what #29884 removed and what CES/hybridSourcemarkEngine expects — before planning further, or - (b) write this up as a doc / start on P0 items? Also flag: are you optimizing the client engine (this TS, per-IDE) or a server-side reimplementation? Changes whether parallelism/caching lives in-process or needs a shared cache. Say the word and I'll refine. ✻ Crunched for 3m 10s ※ recap: Goal: plan to optimize the SourceMark engine. I found it's orphaned dead code (deps removed in #29884, won't compile) and produced a prioritized optimization plan. Next: you pick whether to investigate reviving it or write up/start the P0 wins. (disable recaps in /config) Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code [CAVEMAN] ← for agents
4:07Demo milestone
Comparison session usage
Without the context engine, the session cost $2.59 and took 3m 6s of API time. The speaker compares it with $1.29 and about 1m 30s using the context engine.
Text in image
Map<file, Set<markId>> in SourceMarkStore, maintained on updateCachedSourceMarks. Turns O(marks·filesPerMark) per keystroke into O(files). 8. Share topological traversal. Marks on the same file/commit range each call topologicalOrdering + diffCommits independently. Batch by (originalCommit→target, file) and reuse the edge list + diffs across marks in that group. P2 — algorithmic (do only if profiling still shows hotspots) 9. SnippetFinder.findSegment O(fileLines·snippetLen). SnippetFinder.ts:71 slices+ArrayUtils.equals per line, allocating a slice each iteration. Index candidate lines by first-line hash, compare only on match. Matters for large files. 10. computeOtherDiffs caching. SourceMarkCalculator.ts:443 rebuilds fileContentLines + grep pattern per edge. Cache grep pattern per (mark, fromCommit). Heuristics (:482 large-file prune, :486 grep gate) already good — leave them. 11. getOtherDiffs lazy already (RangeAdjust.ts:110) — only fires on range-fail. Fine. The fuzzy findIntersectionIndex nested loop has early-exit thresholds; leave unless profiled hot. Cross-cutting 12. Instrument first. git.commandLog already records commands. Before touching anything, dump per-op counts/durations for fullRecalculation and a hot-file getSourceMarksForFile on a real large repo (the hardcoded clio/themis in SourceMarkScheduler.ts:62). Establishes baseline + confirms ordering above. 13. Regression safety: the engine/fixtures/*.diff corpus is the golden set — every change must keep fixtures green. Add a marks/sec benchmark harness alongside. Suggested sequencing Session Total cost: $2.59 Total duration (API): 3m 6s Total duration (wall): 1h 9m 23s Total code changes: 0 lines added, 0 lines removed Usage by model: claude-haiku-4-5: 530 input, 13 output, 0 cache read, 0 cache write ($0.0006) claude-opus-4-8: 25.8k input, 12.0k output, 2.0m cache read, 187.4k cache write ($2.59) Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code Settings Status Config Usage Stats Esc to cancel
4:11Demo milestone
- 4:21
Um, now I wanna show you another interesting demo, um, which is, uh, an interesting session that my colleague... Whoops, sorry. Bring this out here. It's an interesting, um, session that my colleague had with Unblocked the other week. So what he... We also have a code review product, and Richie works on that. And he discovered that the number of issues that the code review product was surfacing was, was starting to drop. So he just asked Unblocked, and
- 4:51
Unblocked was able to pull this information, um, and, and, you know, narrow it down very quickly to, uh, a likely cause, which was that we switched from Opus four-eight... or Opus four-six to Opus four-eight, and the behavior characteristics behind the scenes, um, you know, caused all those issues to drop. And, uh, so what he did is he asked Unblocked to just fix it. And now Unblocked, um, was able to just
Requesting a fix
After Richie says “Fix it please,” Unblocked reports that job UNB-128 completed and created PR #31716.
Text in image
Weekly Ratio of Code Review Issues Since 2026 And two test assertions to update: 1. CodeReviewerTest.kt — eq(MLInferenceEngine.Claude46Opus) 2. OrgSettingsStoreTest.kt — test name back to `getCodeReviewEngine defaults to Claude46Opus` That's it — 4 lines across 4 files. The pricing consolidation in EnginePrice.kt is cosmetic (Opus 46/47/48 are all the same price) so you can leave that as-is. Alternative: Rather than reverting globally, you could use the per-org engine override from #31072 to A/B test — set a few orgs back to Claude46Opus via OrgSettingsStore.setCodeReviewEngine() and compare certainty distributions before doing a full rollback. References: Code Reviewer uses Opus48 by default; ReviewBlueprint.kt richiebres 6d ago Fix it please Unblocked 6d ago Started work on job UNB-128. I'll update this thread when it completes with a PR. Unblocked 6d ago Implementation job UNB-128 has completed and created PR #31716. Interface text: getunblocked.com Weekly Ratio of Code Review Issues Since 2026 - Next Chapter Or... Switch default review engine to Claude46Opus by richiebres · Pull R... unblocked/document-query-engine: Natural language → validated... Document Query Engine Unblocked Ask a New Question Questions Coding Code Reviews Recently Archived Settings Analytics Invite Members Download Unblocked Documentation Contact Support This was: Helpful; Not Helpful Continue the discussion Answer Preferences Next Chapter Org Peter Werry
5:01Demo milestone
- 5:21
create a PR behind the scenes, and it has all that context, and it knows what to do. But the really cool part is you can see that it created this description, and it's, it's just remarkable. Like, it, it, um, understood the PR that caused the regression in the first place, and then it found a Slack conversation where the convers-- where, where this was being discussed and pulled on some of the data here to help inform, um, the fix, which, which is just nuts.
- 5:53
So I'm gonna go back to... Whoop, I think I lost my keynote presentation here. Let's see. There we go.
The cost compounds
The slide says agents face the same problem and costs compound. The speaker then says this is the wrong keynote.
Text in image
Now your agents have the same problem. Only the cost compounds. Unblocked
- 6:03
So you have the same problem with agents, right? Cost... Oh, sorry, this is the wrong it's the wrong keynote.
- 6:17
There we go. So now I think what we're gonna do is we're just gonna get right into the building part 'cause we're, we're quite short on time today. Um, and so I- I'm just gonna summarize this. You've got context scattered all over the place. You've got context in PRs, reviews, comments, issues and tickets, docs, conversations. Um, and, and all of that is, uh, semantically queryable, but it's also structured.
Context is everywhere
Context is scattered across PRs, reviews, issues, docs and conversations. The diagram connects these sources to a central code window: the context is both semantically queryable and structured.
Text in image
Context is everywhere Semantically queryable — but also structured Ping aws Unblocked
- 6:45
And RAG nails the content part. Um, it, it can be, it can handle things like content questions, find the code, the PR that changed it, the Slack thread that's debating it. And if you wrap that stuff in an agent loop, it can go really, really far. You give an agent loop tools to do file search after this, and it, it's very powerful. But it doesn't handle some of the other type of, uh, structural questions, like PRs that I merged last week.
RAG nails content
For an authentication question, semantic retrieval can find relevant code, pull requests, and discussion threads. The speaker says an agent loop with file-search tools can take this further.
Text in image
Rag nails content Semantic retrieval is the right tool — wrap it in an agent loop, it goes far. How does our auth handle expired tokens? Code Pull request Discussion thread Unblocked
But the real questions
Questions about who reviewed payments most or which PRs have no reviews call for structural queries, which the speaker distinguishes from semantic searches.
Text in image
But the real questions... • PRs I merged last week • Who reviewed payments the most • Open PRs with no reviews • Issues closed in Q1, by label Unblocked
- 7:15
Who reviewed payments the most, open PRs, et cetera. Um, and these are queries, they're not semantic searches. So I wanna just show you again, um, going back to Richie's example here, that he started the, the conversation with Unblocked by asking it a query-based question 'cause there's a temporal element. Can you show me a chart of ratio weekly since January 1st? Okay. So in order to do this,
- 7:46
it's not gonna be able to just pull this information from a vector store 'cause that's not where that information's encoded. So it's, it's gonna have to do that, um, through some sort of query engine.
- 7:59
And so, you know, retrieval has two halves essentially. We've got semantic retrieval, and that can handle those types of semantic-based questions. But today we're gonna work on this, the structured part. We're gonna turn structured queries or natural language queries into structured queries.
- 8:20
But generating the queries is actually the easy part. The hard part is the scaffolding that sits around it. So you need to do things like understand the schema, so you can pass it to the LLM, so it generates the, a tightly bound query. Uh, you need to also be able to map identities. If I ask a question like, "Open PRs by Rasheen," um, it's not gonna know who Rasheen is because his identity in GitHub might be completely different. So we have to be able to map that somehow. And
- 8:50
then when you get the query out of the LLM, it's untrusted. It could have, uh, operations that are dangerous. It could cross tenant boundaries. And so you have to have some way of validating that the query is, is, uh, correct and safe. And then finally, uh, even though you do that validation, there could be errors. Um, and when you actually do the query against the database, there could be errors there as well. And so we can actually get the LLM to try to correct itself if it gets into that
- 9:19
situation. So this is what we're aiming for here. We wanna go to, uh, to this. We wanna be able to say something like, "Open PRs by Rasheen," and then we wanna get this kind of query on the back of that. So just take a sec here. Uh, this is the open source repo that we're gonna be building against, and I'm going to jump right into what that looks like actually.
- 9:48
So this is the repo here. It's called the Document Query Engine, and I wanna show you what it looks like when we just boot it up the first time. So I can say, you know, "PRs by Peter at Ground Zero." Oh, actually, we're actually not running it, so I'll go back.
- 10:07
Let's run it first.
Terminal prompt
The presenter prepares to show a Claude session planning SourceMark engine optimization without a context engine. The terminal is still idle.
Text in image
Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code ~/Dev/document-query-engine main *2 10:48:26
10:10Demo milestone
- 10:15
Now I can ask my question.
- 10:19
So when you start, it's gonna have... It's not gonna be wired up at all. It's gonna produce nothing. So we're gonna just gradually build this thing up here, okay?
- 10:29
First thing we're gonna talk about is automatic schema discovery. This is completely dynamic. Mongo is schemaless, not... A- and not all databases are schemaless, but the same kind of principle can apply. You can dynamically discover the schema at runtime. Um, and so what, what the technique involves is sampling a whole bunch of documents in the document store and then deriving a schema from that. And, uh, you know, I just wanna show you, like, one tricky part of that, which is
- 10:59
inferring enums. Um, now the, the, the outcome from this is that you can use an LLM as a Thor hammer, um, and it can do a pretty good job. But actually, you get, uh, really, really far just using traditional procedural methods. So you can... Y- you know, don't always fall back to an LLM to do this kind of what looks like non-deterministic work. This is actually quite deterministic, this kind of thing. So
- 11:29
now that we've discovered schema, I'm just gonna show you what that looks like in our demo. Whoops. First, we have to level up to the next...
Terminal prompt
The presenter prepares to show a Claude session planning SourceMark engine optimization without a context engine. The terminal is still idle.
Text in image
Interface text: ~/Developer/document-query-engine Plan sourcemark engine optimization Claude Code ~/Dev/document-query-engine main *2 10:48:26
11:34Demo milestone
- 11:41
And then run. So now we've discovered the schema from our, um, GitHub collection, and this is just pull requests, okay? So it has not only the schema here, but also, um, indexes that will help improve the performance of queries.
- 12:05
The next step is identity resolution. It's that thing I talked about, about mapping, you know, users to, uh, real identities. And, um, again, this is- this can be solved just using simple procedural tricks. So, um, that- that's actually more or less how we do it, uh, in underneath the hood at Unblocked. When we take usernames or identities from different systems and map them across, um, you can do this kind of fuzzy matching, and then in the end, if you have to, you
- 12:35
can pass a list of names to an LLM and have it act as a tiebreaker.
- 12:45
And finally, uh, the last step before we can actually show the cool stuff is this. It's the synthesis part. And roughly speaking, this is actually the easiest part. We just take the schema that we dynamically resolved from earlier, we take the current date and time, uh, maybe some information about the user that's asking the question, so it can easily resolve questions like my PRs and not just Peter's PRs. Um, and then we pass it to the LLM and ask it
- 13:15
to synthesize a query for us. And the outcome, uh, will look something like this.
- 13:24
Let's get to this one.
- 13:29
So now I can go back to my engine and try to run that query again.
- 13:37
And it's going to pull the documents for me now and generate this query.
- 13:45
Now, the next step I- I was talking about was, was validation. This is actually the most important thing. Um, your, your, uh, query could contain really dangerous operators, like the function operator, for example, which would execute raw JavaScript, um, on the DB server, which is definitely not ideal. Um, but there could be other things that you wanna do with this. Like, for example, you might have hidden fields that you don't want, uh, to be surfaced either to the LLM or for the LLM to, to derive and execute.
- 14:15
If you have metadata fields, for example, that contain, um, tenant IDs and you wanna isolate each query to a specific tenant, you're better off making sure that the LLM doesn't interact with those fields at all, and then just constraining it behind the scenes. You can inject those fields underneath.
- 14:37
And that's kind of what that looks like. So again, we're gonna just quickly show the payoff for that and what that looks like.
- 14:47
Let's go up to our validation step. Oops. Run it again. So if I now ask a question like this, PRs with authentication in the title, that is probably gonna derive some kind of like regular expression thing. And there we go. Okay. So, uh, our validation engine said, "Sorry, regular expressions aren't allowed. You're getting bounced." And now we're not allowed to do that.
- 15:17
So validation is cool, um, but what do we do with that? You know, we've got an error, um, but what we can do instead is just take that error and just loop it back in to the LLM and have the LLM kinda self-correct.
- 15:34
And that's kind of what it... That's all it is. That's all it looks like from a technical standpoint. It's just a loop with some sort of maximum, uh, retry.
- 15:45
And so if I try this again...
- 15:52
Bounce up to my retries and run that.
- 15:57
Now, if I try that query again... Oops, that's the wrong one. Let's try this one.
- 16:12
Now what you can see is that it actually took a couple attempts . Um, but it got there in the end. However, um, it constrained it down to use match, and unfortunately, this is an exact title match. We don't have any PRs with that exact title, so we need one more thing,
- 16:32
which is this, full text search. So now if we add full text search to our engine, um, hopefully the outcome that we get is something that looks a little bit like this. So let's try it one more time.
- 17:05
Awesome. So there you go. We got... We went from start to finish there. This is all open source, so you can take a look at how it's implemented and play around with it yourself.
- 17:17
And this is what we built, something that can effectively handle all of the, uh, other half of RAG. So anything that has to do with, uh, structural based data.
- 17:29
You can have filters, aggregations, it's time scoped, and it self-corrects. But what's really cool is that it also powers a chat agent. And so if I go over here and I say something like, um, "What authentication-related things have we been working on last week?"
- 18:00
Now you get this whole thing that will show you exactly what it's doing, all the queries that it creates.
- 18:08
And now it has the data, and it's just kind of reasoning about it behind the scenes, and there we go.
- 18:14
Very cool.
- 18:19
So there you are, structured, queryable context for agents and humans. Now, there's just one last thing. Um, on the description you probably saw that we were promising a context engine simulator. If you don't want to wire up Unblocked, uh, right away because you've got some constraints in your, in your environment, um, we have an open source project that will allow you to do this entirely locally yourself. And so the idea is, uh, it will create
- 18:49
a context bundle for you on a task by task basis, and then take that context bundle and run an AB test against the same task and show you the benefits of having all of that context at your fingertips for your agents. And that's the, uh, PR code for that, if you want that.
- 19:16
All right. And one last thing . If you want some coconuts, come by the booth
- 19:25
. Thank you.