# Benchmarking semantic code retrieval on Claude Code

Kuba Rogut, turbopuffer

AI Engineer Europe 2026

[Talk page](https://ai.engineer/talks/zKk7sDMGDEQ-benchmarking-semantic-code-retrieval-on-claude) · [Video](https://www.youtube.com/watch?v=zKk7sDMGDEQ)

## Benchmarking semantic code retrieval on Claude Code

Adding semantic search makes Claude Code’s context discovery more selective, but recall depends on the task—and on whether the agent knows which retrieval tool to use.

Familiarity with command-line code search and coding agents is helpful; embeddings, precision, and recall are explained as they appear.

### Why index code when an agent can grep?

[0:16](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=16s)

Should a coding agent discover a repository by repeatedly searching and reading files, or should it start with a searchable index of the code’s meaning? That is the practical difference behind this experiment. Kuba Rogut, a deployed engineer at turbopuffer—a serverless full-text and vector search database built on object storage—starts with Claude Code’s choice. At the time of the recording, it relied on agentic filesystem search by default. Rogut cites Boris Cherny’s account that early versions used a local vector database, but iterative search proved simpler and worked better for Claude Code.

![Slide titled “claude doesn’t use semantic code search” showing Ethan Lipnik’s question and Boris Cherny’s reply about agentic search.](https://ai.engineer/api/talk-essay-media/3634518b720c0b3ffa8f2c4d896aaa1aa9ef0623b95c7748a79d087f239c99b3)

Boris Cherny describes Claude Code’s move from RAG and a local vector database to agentic search.

Cursor made a different investment: it indexed codebases into turbopuffer to support semantic retrieval. In [Cursor’s semantic-search evaluation](https://cursor.com/blog/semsearch), Rogut highlights roughly 24% relative answer-accuracy improvement for **Composer**, with about 12.5% averaged across models. These results predate **Composer 2**; they do not establish its performance. He also recommends Cursor’s account of [indexing large codebases](https://cursor.com/blog/secure-codebase-indexing).

Cursor’s separate online A/B test measured user outcomes rather than benchmark answer accuracy. Its report gives 2.6% higher code retention with semantic search for codebases containing at least 1,000 files. It also reports 2.2% more dissatisfied follow-ups when semantic search was disabled. The direction of that second comparison matters: it is not exactly the same percentage as a reduction measured from the disabled condition. Aggregate effects also include requests that have little use for semantic search, so a modest overall change can conceal larger gains on the requests that need it.

### Embeddings amortize repeated discovery

[2:44](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=164s)

Consider the question: how does metadata filtering work in this codebase? An agent using filesystem search starts with candidate terms, reads matching files, and searches again when those files do not answer the question. A shell search can begin broadly and then request nearby context:

```bash
rg -n -i 'metadata|filter' .
rg -n -i -C 12 'metadata.*filter|filter.*metadata' .
```

The agent still has to interpret the matches and decide where to look next. Rogut’s illustrative trace spends about 6,000 tokens on this discovery process. That is one example, not a benchmark-wide average. The same work can recur in another session or in another agent working on the same repository.

The indexed route moves some of that work earlier:

1. Split the codebase into chunks.
2. Embed those chunks and store them in a searchable index.
3. At runtime, use the metadata-filtering question to retrieve relevant chunks.

**Embeddings are cached compute** in this sense: they preserve a reusable representation of meaning that can shorten later discovery. They do not cache a complete answer. The agent still needs to read and reason about the returned code. Nor does indexing eliminate upkeep; changed code requires the index to be maintained. The complete slide compares repeated grep-and-read discovery with the upfront indexing work and subsequent retrieval.

![Two panels compare per-session grep and file reads totaling 6,314 tokens with indexed retrieval returning ranked chunks totaling 424 tokens.](https://ai.engineer/api/talk-essay-media/de911419c46008f73cc10044215f4b29784de300ad2feaf844476fba33e4351d)

“Embeddings are cached compute”: repeated discovery versus indexing once and retrieving at runtime.

The economic argument becomes stronger across repeated sessions and concurrent agents. A small saving on one question may be unremarkable, but paying to rediscover the same area of a repository repeatedly can accumulate. Whether indexing pays off depends on how often the representation is reused relative to the cost of building and maintaining it.

### Give Claude Code a semantic retrieval tool

[4:12](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=252s)

[TurboGrep](https://github.com/turbopuffer/turbogrep) implements a small version of that pipeline. It walks the filesystem, uses `tree-sitter` to parse and chunk code, generates embeddings with a Voyage code model, and uploads them to turbopuffer. Claude Code then gets a search tool backed by that index. At recording time, Rogut described V1 as open source and V2 as awaiting release; the public repository should not be assumed to reproduce the exact demonstrated integration.

The demonstration uses the Django repository. Claude Code invokes the `TPuff` search tool with a query for a password reset token generator, receives the relevant content, and uses it to explain the implementation to the user. The useful handoff is from a description of the desired behavior to code the agent can inspect. A successful trace shows that the integration works in that example, but does not yet establish whether it improves retrieval across tasks.

### Measure the context found along the way

[5:02](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=302s)

To move beyond a convincing demonstration, the evaluation asks what context the agent discovers while attempting a task. Rogut distinguishes Cursor’s internal context benchmark from the public [ContextBench](https://arxiv.org/abs/2602.05892) work. The latter uses human annotations identifying files, lines, and symbols that an agent should inspect to complete a task well. This evaluates the retrieval trajectory, rather than simply whether the final code solves the issue. **Retrieval precision and recall are not answer accuracy or task completion.**

The experiment compares three conditions:

| Condition | Available retrieval behavior |
| --- | --- |
| Stock Claude Code | Default search and file reads |
| Windowed reads | Reads limited to 50 lines at a time |
| Windowed reads + semantic search | Same read limit, plus `TPuff` Search |

The read limit matters because reading an entire thousand-line file can sweep in annotated context without showing whether the agent located it precisely. Large reads made the measurements noisy and obscured differences between retrieval strategies. The windowed condition provides a more informative baseline for assessing the added semantic tool.

### Precision improves with more selective retrieval

[6:24](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=384s)

**Precision** asks how much of the retrieved context belongs to the annotated target set:

```math
precision = relevant retrieved items / all retrieved items
```

If the agent reads ten files and eight are required, file precision is 80%. The same idea applies to lines and symbols. Rogut reports these approximate results:

| Condition | File precision | Line precision | Symbol precision |
| --- | --- | --- | --- |
| Stock Claude Code | 65% | 33% | 43% |
| Windowed reads + semantic search | 87% | — | — |

The intermediate windowed condition also improves precision. Consequently, the entire increase from stock Claude Code to the combined condition cannot be attributed to semantic search alone; restricting reads contributes too. These are Rogut’s reported experimental results, not a general guarantee for other models or repositories.

Rogut describes default Claude Code as highly exploratory: it tries to read broadly before settling on the relevant implementation. His rounded interpretation is that files outside the labeled target set account for about one in three baseline reads, one in five windowed reads, and one in eight reads with semantic search added. That makes the selectivity improvement tangible. Calling those reads wasted is shorthand for being outside the benchmark’s annotations, not proof that every exploratory read was useless.

### Recall reveals what selective search can miss

[7:30](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=450s)

**Recall** changes the denominator: how much of the required context did the agent find?

```math
recall = relevant retrieved items / all required relevant items
```

Suppose a task requires ten files. The agent reads five files, but only three belong to the required set. Its file recall is 30%. Reading fewer irrelevant files can improve precision without improving this coverage measure.

Stock Claude Code has the highest file recall in the reported comparison. Broad exploration helps it encounter more target files. Its line recall is weaker, however: reaching the right file does not necessarily mean inspecting the right parts of it. Windowed reads and windowed reads plus semantic search have roughly similar aggregate recall, with some decreases when semantic search is added. The aggregate covers 50 tasks in Rogut’s experiment, not the full public benchmark. Higher precision therefore coexists with a mixed coverage result.

### Behavioral similarity and import tracing find different code

[8:38](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=518s)

Splitting the tasks by which condition performs better reveals differences that the aggregate hides:

- **Semantic search:** Useful when relevant files implement related behavior but use different keywords. Rogut recalls an example involving related handling across several ORMs or libraries: lexical search missed some implementations, while semantic search found their behavioral relationship.
- **Windowed grep:** Useful when an early search finds a good identifier and the remaining work follows explicit connections such as imports. Once the agent has the right names, keyword search can trace the relevant files directly.

These are different ways to navigate a repository. One follows similarity of behavior; the other follows names and explicit code relationships. The task breakdown supports choosing tools according to that structure, rather than declaring one search method universally better.

![Recall deep-dive slide with two grouped bar charts and boxes describing when semantic search won and when grep won.](https://ai.engineer/api/talk-essay-media/040e391403ce4a8d1bea586db34d983daaf77dce7801833f95a72c92052ab360)

Semantic search finds behavior-adjacent files; grep excels at tracing imports and finding function names near bugs.

### A useful tool still needs a tool-selection policy

[9:43](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=583s)

The experiment improves precision, but the agent also has to decide when and how to use the new tool. Rogut’s interpretation is that Claude Code is oriented toward grep: attaching a semantic-search tool does not automatically give it a reliable policy for choosing that tool, phrasing a query, or deciding when to return to lexical search. He contrasts this with Composer’s integrated retrieval behavior.

That is a plausible explanation for why an added tool might fall short of an integrated system, rather than a causal result isolated by this experiment. Cursor’s answer-accuracy evaluation and Rogut’s context-retrieval evaluation also measure different outcomes, so their gains cannot be compared directly. His closing design principle is to provide lightweight, complementary ways of finding context. Even very large context windows still benefit from selecting the relevant information before asking a model to reason over it.

### What semantic search means here

[11:07](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=667s)

The audience first asks for an exact definition of semantic search. In this implementation it means vector similarity search: embed the code, embed the query text, and search the stored vectors in turbopuffer to retrieve code chunks. The embedding model is **[voyage-code-3](https://blog.voyageai.com/2024/12/04/voyage-code-3/)**. This is a concrete, relatively simple retrieval setup, not a claim that every system described as semantic search uses the same architecture.

Rogut names Cursor, Anthropic, and Notion as turbopuffer customers. A company’s use of the database does not identify the retrieval architecture of each of its products; in particular, that customer relationship does not establish semantic code search inside Claude Code.

### Comments help a chunk explain itself

[11:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=718s)

What happens on poorly written code? Rogut does not give a controlled benchmark answer. Instead, he describes a pattern from reviewing trajectories, including using Claude to help examine them: examples with useful function comments and inline documentation tended to retrieve better. Those comments supply context about behavior that both the embedding model and the coding agent can use.

The difficult part is not merely producing a vector. It is making the chunk’s meaning available to the representation. A function’s syntax may show what operations it performs without clearly expressing why it exists or which user-facing behavior it supports. Good documentation can help connect those levels. This remains a qualitative observation from the trajectories, not a measured rule that comments always improve retrieval.

### Bridge the gap between a question and raw code

[12:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=778s)

A follow-up question points to a representation mismatch: a natural-language query and raw code may express the same behavior in very different forms. The audience proposes a parent–child arrangement in which a query-like description points to a child containing the actual code. Rogut clarifies that this experiment did not apply that preprocessing; it embedded the raw code chunks.

Authentication flow is the audience’s example of a human-level concept that may not appear directly in implementation text. A description at that level could provide a useful retrieval target, while the linked code remains the evidence the agent needs to inspect. Rogut mentions Cursor’s own embedding model as another way to address the translation between code and human queries, but says he cannot explain its internal implementation.

Another audience member suggests that Cursor generates comments and embeds them with code. That explanation is unverified in the discussion. Rogut agrees that comment enrichment could work; he does not establish that it is Cursor’s actual method. The concrete distinction is between the raw-code experiment presented here and possible enrichment methods that would need their own evaluation.

### Where shared vector retrieval earns its place

[14:22](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=862s)

The final question raises long-term memory and reducing database size. Rogut answers in terms of workload placement, without specifying when to encode memories or how to compact them. If all relevant information can be downloaded to a local filesystem, grep is convenient and requires little setup. That simplicity is valuable, even though the agent’s repeated searching and reading still consumes compute.

Shared workloads change the calculation. A Notion-like knowledge base contains relationships across information that may be awkward to gather and search as local files. A shared vector index can make those semantic relationships available to multiple agents. Multimodal data extends the case further: grep does not directly recover the semantic content of raw video, audio, or images. Matching a filename is different from finding what the media contains.

The choice therefore depends on the data, the workload, and the amount of repeated discovery. Local lexical search remains useful when the relevant material and identifiers are readily available. As agents share larger or more varied information spaces, storing reusable semantic representations can offload work they would otherwise repeat. That is where the cached-compute argument reaches beyond code search.

## Resources

- [Improving agent with semantic search](https://cursor.com/blog/semsearch) — Cursor's offline answer-accuracy evaluation and online tests of semantic search, plus its approach to training a custom embedding model.
- [Securely indexing large codebases](https://cursor.com/blog/secure-codebase-indexing) — How Cursor synchronizes code changes, caches chunk embeddings and securely reuses indexes across teammates.
- [TurboGrep repository](https://github.com/turbopuffer/turbogrep) — Public code-search CLI with installation instructions and a semantic-query example using Voyage and turbopuffer.
- [ContextBench: A Benchmark for Context Retrieval in Coding Agents](https://arxiv.org/abs/2602.05892) — A benchmark using human-annotated code contexts to evaluate retrieval precision, recall and efficiency during issue resolution.
- [Voyage code embedding model announcement](https://blog.voyageai.com/2024/12/04/voyage-code-3/) — Technical background on voyage-code-3, including code-retrieval training, embedding dimensions and quantization.

## Slides

### Benchmarking semantic code retrieval on Claude Code

Kuba Rogut introduces the benchmark and turbopuffer, a serverless full-text and vector search database built on object storage.

[0:30](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=30s) [Talk title slide naming Kuba Rogut of turbopuffer.](https://assets.ai.engineer/sites/aiecode2025/sha256/56c02fc875a76f6077a4b5bf72f648d5865800fbb8f5bfbff88b2c6c9f444f0c.webp)

Benchmarking semantic code retrieval on Claude Code

turbopuffer

Kuba Rogut
Deployed Engineer

April 10, 2026

AIE
Google DeepMind

### Claude doesn’t use semantic code search

Boris Cherny says early Claude Code used RAG and a local vector database, but the team found agentic search generally worked better and was simpler. Rogut describes this as searching the filesystem with tools such as grep.

[0:43](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=43s) [A social media exchange describes early Claude Code use of a local vector database and the preference for agentic search.](https://assets.ai.engineer/sites/aiecode2025/sha256/c4c0d5c8cec808f4784c4a4582a6e17df7418bc59a2d69b7216c500ea4418154.webp)

claude doesn’t use semantic code search

turbopuffer

Ethan Lipnik @EthanLipnik · Jan 30
Does anyone know why Codex and Claude doesn't use cloud-based embeddings like Cursor to quickly search through the codebase?

Boris Cherny
@bcherny

👋 Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search generally works better. It is also simpler and doesn’t have the same issues around security, privacy, staleness, and reliability.

9:56 PM · Jan 31, 2026 · 1.1M Views

Interface text:
68 replies · 8 reposts · 658 likes · 266K views
AIE
Engineering the future of AI

[1:00](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=60s) [A slide shows a discussion in which Boris Cherny says early Claude Code used a local vector database but agentic search generally worked better.](https://assets.ai.engineer/sites/aiecode2025/sha256/6659de6090b3b225da5fea0d139dccd45cc3e524c3ac883d218dc82f9300a77d.webp)

claude doesn’t use semantic code search

turbopuffer

Ethan Lipnik @EthanLipnik · Jan 30
Does anyone know why Codex and Claude doesn't use cloud-based embeddings like Cursor to quickly search through the codebase?

Boris Cherny
@bcherny

👋 Early versions of Claude Code used RAG + a local vector db, but we found pretty quickly that agentic search generally works better. It is also simpler and doesn’t have the same issues around security, privacy, staleness, and reliability.

9:56 PM · Jan 31, 2026 · 1.1M Views

Interface text:
68 replies · 8 reposts · 658 likes · 266K views
AIE
Braintrust · WorkOS · OpenAI

### Cursor uses semantic code search

Compared with the same models without semantic search, Cursor reports 6.5%–23.5% higher answer accuracy, 2.6% higher code retention in large codebases, and 2.2% fewer dissatisfied user requests.

[1:03](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=63s) [A Cursor article and charts report improvements in answer accuracy and code retention with semantic search.](https://assets.ai.engineer/sites/aiecode2025/sha256/fcdb3d6063b4b64512a2acb1bceb8e9a6d13ace407a05a1e57cc7d89412874cb.webp)

cursor does use semantic code search

turbopuffer

CURSOR
Improving agent with semantic search
Nov 6, 2025 by Stefan Heule, Emily Jia & Naman Jain in Research

Table of Contents
Offline evals
Online A/B tests
Custom retrieval models
Conclusion

When coding agents receive a prompt, returning the right answer requires building an understanding of the codebase by reading files and searching for relevant information.

One tool Cursor’s agent uses is semantic search, which retrieves segments of code matching natural language queries, such as “where do we handle authentication?”, in addition to the regex-based searching provided by a tool like grep.

To support semantic search, we’ve trained our own embedding model and built indexing pipelines for fast retrieval. While you could rely exclusively on grep and similar command-line tools for search, we’ve found that semantic search significantly improves agent performance, especially over large codebases:

- Achieving on average 12.5% higher accuracy in answering questions (6.5%–23.5% depending on the model).
- Producing code changes that are more likely to be retained in codebases.

All models improve with semantic search
Model (alphabetical) | Relative improvement (Cursor Context Bench)
Composer | 23.5%
Gemini 2.5 Pro | 8.7%
GPT-5 | 6.5%
Grok Code | 11.9%
Sonnet 4.5 | 14.7%

Semantic search improves code retention and reduces dissatisfied user requests
Code Retention | +0.3%
Code Retention (large codebases) | +2.6%
Dissatisfied User Requests | −2.2%

Improvement of having semantic search (vs same model without semantic search).
For “Code Retention” higher is better, and for “Dissatisfied User Requests” lower is better.

Interface text:
AIE
Braintrust
WorkOS
OpenAI

[2:27](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=147s) [A Cursor article and charts show model accuracy improvements, code retention, and dissatisfied user requests with semantic search.](https://assets.ai.engineer/sites/aiecode2025/sha256/fd3c5dd80fd3b178d2702b6f6a325a98d3d478f5acc0266ec457fe9b98f6bdf3.webp)

cursor does use semantic code search

turbopuffer

CURSOR
Improving agent with semantic search
Nov 6, 2025 by Stefan Heule, Emily Jia & Naman Jain in Research

Table of Contents
Offline evals
Online A/B tests
Custom retrieval models
Conclusion

When coding agents receive a prompt, returning the right answer requires building an understanding of the codebase by reading files and searching for relevant information.

One tool Cursor’s agent uses is semantic search, which retrieves segments of code matching natural language queries, such as “where do we handle authentication?”, in addition to the regex-based searching provided by a tool like grep.

To support semantic search, we’ve trained our own embedding model and built indexing pipelines for fast retrieval. While you could rely exclusively on grep and similar command-line tools for search, we’ve found that semantic search significantly improves agent performance, especially over large codebases:

- Achieving on average 12.5% higher accuracy in answering questions (6.5%–23.5% depending on the model).
- Producing code changes that are more likely to be retained in codebases.

All models improve with semantic search
Model (alphabetical) | Relative improvement (Cursor Context Bench)
Composer | 23.5%
Gemini 2.5 Pro | 8.7%
GPT-5 | 6.5%
Grok Code | 11.9%
Sonnet 4.5 | 14.7%

Semantic search improves code retention and reduces dissatisfied user requests
Code Retention | +0.3%
Code Retention (large codebases) | +2.6%
Dissatisfied User Requests | −2.2%

Improvement of having semantic search (vs same model without semantic search).
For “Code Retention” higher is better, and for “Dissatisfied User Requests” lower is better.

Interface text:
AIE
Engineering the future of AI

### Embeddings are cached compute

In an illustrative metadata-filtering search, repeated discovery uses 6,314 tokens; retrieving indexed chunks uses 424. Indexing is an upfront cost shared across sessions.

[2:31](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=151s) [A comparison illustrates 6,314 tokens for repeated code discovery versus 424 tokens for retrieved chunks after indexing.](https://assets.ai.engineer/sites/aiecode2025/sha256/008f323ab9cc26f6c7a8a3c86d52eed69d1749eabe341bd75fb3357f9e73a848.webp)

embeddings are cached compute

per-session discovery
Agent greps, reads, assesses, repeats

grep “metadata filter” src/          +820 tokens
read indexing.py (wrong section)    +2,106 tokens
grep “ingest pipeline” src/          +640 tokens
read api_client.ts:280-400         +1,450 tokens
read types.ts (for context)        +1,298 tokens

Repeated every session by every agent, across every task

→ 6,314 tokens ⚠️

amortized understanding
index once, retrieve at runtime

Index time:
- Codebase chunked, embedded, indexed
- Semantic meaning encoded upfront
- Model has already “read” every file
- One-time cost, amortized across all sessions

Runtime:
Agent query: “How is metadata filtered?”

Results(ranked chunks):
- indexing.py       +142 tokens
- api_client.ts     +188 tokens
- types.ts           +94 tokens

→ 424 tokens ✅

Interface text:
AIE
Engineering the future of AI

[4:04](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=244s) [A comparison shows 6,314 tokens for repeated discovery and 424 tokens for retrieved code chunks.](https://assets.ai.engineer/sites/aiecode2025/sha256/d4e910294318ad5d0eaf8fcd4c08a1d65d23b903c3b49fd7bcb5ca89a5cc6011.webp)

per-session discovery
Agent greps, reads, assesses, repeats

grep “metadata filter” src/          +820 tokens
read indexing.py (wrong section)    +2,106 tokens
grep “ingest pipeline” src/          +640 tokens
read api_client.ts:280-400         +1,450 tokens
read types.ts (for context)        +1,298 tokens

Repeated every session by every agent, across every task

→ 6,314 tokens ⚠️

amortized understanding
index once, retrieve at runtime

Index time:
- Codebase chunked, embedded, indexed
- Semantic meaning encoded upfront
- Model has already “read” every file
- One-time cost, amortized across all sessions

Runtime:
Agent query: “How is metadata filtered?”

Results(ranked chunks):
- indexing.py       +142 tokens
- api_client.ts     +188 tokens
- types.ts           +94 tokens

→ 424 tokens ✅

Interface text:
AIE
AI Engineer
EUROPE

### TurboGrep project structure

TurboGrep parses and chunks a codebase with tree-sitter, embeds it with Voyage, and uploads it to turbopuffer. The project also includes search, branch handling, and a terminal interface.

[4:07](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=247s) [A file tree outlines the TurboGrep CLI, indexing, search, filters, branch handling, and terminal interface modules.](https://assets.ai.engineer/sites/aiecode2025/sha256/ca0e442013de537da5bbcef104529a289d181581784337d8711bd03380e9588e.webp)

turbogrep-v2/
├── src/
│   ├── main.rs          # CLI entry point (clap commands)
│   ├── chunker.rs       # tree-sitter code chunking
│   ├── embedding.rs     # Voyage AI HTTP client
│   ├── turbopuffer.rs   # turbopuffer v2 HTTP client
│   ├── indexer.rs       # index pipeline: walk → chunk → embed → upsert
│   ├── search.rs        # search: embed query → tpuf query → format results
│   ├── filters.rs       # attribute filter builder
│   ├── project.rs       # project root detection + git helpers
│   ├── branches.rs      # branch overlay logic
│   └── tui/
│       ├── mod.rs       # TUI app state + event loop
│       ├── search.rs    # interactive search view
│       ├── index.rs     # indexing progress view
│       └── widgets.rs   # code preview, status bar
├── Cargo.toml
├── CLAUDE.md
└── README.md

Interface text:
AIE
AI Engineer
EUROPE

### Claude semantic search demo

Claude Code searches the Django repository for password-reset token handling with tpuf, then reads django/contrib/auth/tokens.py.

[4:49](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=289s) [Claude Code searches for Django password-reset token handling and begins reading a source file.](https://assets.ai.engineer/sites/aiecode2025/sha256/7f85966fa7da84dfe36c1dc171e749848b925dc1cf4d0fc488809189a5fa9665.webp)

let’s give Claude semantic search!
turbopuffer

How does Django handle password reset token generation and validation?

Bash(tpuf search "password reset token generation validation" --hybrid --json 2>/dev/null | head -80)
[
  {
    "content": "def test_PasswordResetConfirmView_invalid_token(self):\n        # PasswordResetConfirmView invalid tok[illegible]
  … +64 lines (ctrl+o to expand)

Reading 1 file… (ctrl+o to expand)
  django/contrib/auth/tokens.py

Crunching… (thinking)

Interface text:
Claude Code v2.1.96
Opus 4.6 (1M context) · Claude Max
~/conductor/workspaces/learning/sacramento/ai-council-workshop/eval/repos/django__django
esc to interrupt
AIE
Engineering the future of AI

### How we benchmark

ContextBench checks whether agents find labeled files and lines. On 50 tasks that don’t name files or functions, the test compares raw Claude Code with reads capped at 50 lines, with and without semantic search.

[5:00](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=300s) [ContextBench evaluates whether agents read relevant files and lines. The slide compares baseline, windowed reads, and windowed reads with semantic search.](https://assets.ai.engineer/sites/aiecode2025/sha256/68fa819cca46b9de1b4b343dfada87c73109e2a621833eab03c2922dd76c20f4.webp)

how we benchmark

ContextBench
- Tests whether agents agents read specific “golden files and lines” (labeled context that the agent should find to solve the task well)
- *Used subset of 50 tasks that don’t mention files or functions by name

Three conditions:
- Baseline: raw claude code
- Windowed: claude code w/ max 50 line reads at a time
- Semantic: claude code w/ max 50 line reads + tpuf search cli tool

Interface text:
turbopuffer
ContextBench
GITHUB  DATASET  PAPER
ContextBench
A scientific benchmark evaluating the dynamics of multi-file context retrieval in LLM agents.
FOUNDATION MODELS: 4
BEST PASS@1: 53.0%
AVG EFFICIENCY: 0.599
AVG. LINE F1: 0.325
Benchmark Rankings
Sorting by: Pass@1 Rate
PRIMARY METRIC
SYSTEM VIEW
Backbone Only  Agent + Backbone
Main Board  Detailed Analysis
Showing Raw Backbone Capabilities
Search models...
Rank  MODEL / SYSTEM  PASS@1  CONTEXT F1  EFFICIENCY  AVG. COST
1  Claude Sonnet 4.5  53.0%  0.344  0.658  $0.76
2  GPT-5  47.2%  0.312  0.591  $0.45
AIE
Engineering the future of AI

### Benchmark: precision

File precision rises from 65% for baseline Claude Code to 87% with windowed grep and semantic search. Line and symbol precision also improve.

[6:27](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=387s) [Bar chart comparing file, line, and symbol precision for baseline, grep, and grep plus semantic search. File precision rises from 65% to 81% to 87%.](https://assets.ai.engineer/sites/aiecode2025/sha256/e7d44d321f7800df14fa9b964688ca8e5ad14d9c59f3a8c5e803af9b60f628cb.webp)

benchmark: precision
turbopuffer

precision (did it only read files it needed?)
Legend: baseline; grep; grep + semantic
Y-axis ticks: 0%, 25%, 50%, 75%, 100%
File precision: 65%, 81%, 87%
Line precision: 33%, 42%, 47%
Symbol precision: 43%, 44%, 51%

Claude reads substantially less irrelevant code with semantic search
- Raw claude code: 1 in 3 file reads is wasted on irrelevant code
- Windowed grep: 1 in 5 reads is wasted
- Semantic search: only 1 in 8 reads is wasted

Interface text:
AIE
Engineering the future of AI

[7:28](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=448s) [Bar chart compares file, line and symbol precision for baseline, grep and grep plus semantic search. File precision rises from 65% to 87%.](https://assets.ai.engineer/sites/aiecode2025/sha256/63199f9314e34b7a4fe111e10043537168441aba65e786a05d106b28d6b30539.webp)

benchmark: precision
turbopuffer

precision (did it only read files it needed?)
Legend: baseline; grep; grep + semantic
Y-axis ticks: 0%, 25%, 50%, 75%, 100%
File precision: 65%, 81%, 87%
Line precision: 33%, 42%, 47%
Symbol precision: 43%, 44%, 51%

Claude reads substantially less irrelevant code with semantic search
- Raw claude code: 1 in 3 file reads is wasted on irrelevant code
- Windowed grep: 1 in 5 reads is wasted
- Semantic search: only 1 in 8 reads is wasted

Interface text:
AIE
Google DeepMind

### Benchmark: recall

Baseline Claude Code reaches 49% file recall while exploring more files. Grep reaches 40% line recall and 43% symbol recall; adding semantic search yields similar or slightly lower recall.

[7:44](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=464s) [Bar chart compares file, line and symbol recall for baseline, grep and grep plus semantic search. Baseline has the highest file recall at 49%.](https://assets.ai.engineer/sites/aiecode2025/sha256/d2f5905f6e272e3546a82e174293d76bf4cd1b7bdbfa9bfbee70d8ef5b2a4fd6.webp)

benchmark: recall

turbopuffer

recall (how many needed files did it find?)
Legend: baseline; grep; grep + semantic
Y-axis ticks: 0%, 10%, 20%, 30%, 40%, 50%
File recall: 49%, 39%, 39%
Line recall: 29%, 40%, 39%
Symbol recall: 36%, 43%, 40%

Claude Code finds more files with grep, but it is less efficient
- Baseline opens more files so finds more by volume, but wastes reads
- grep - opens fewer files but more right functions
- grep + semantic explores differently - similar recall to grep, but slightly worse

AIE
Engineering the future of AI

### Benchmark: recall

Across 50 tasks, the baseline found more files, while grep found more relevant lines and symbols. Adding semantic search produced similar or slightly lower recall than grep alone.

[8:13](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=493s) [Bar chart compares file, line and symbol recall for baseline, grep and grep plus semantic search. Baseline has the highest file recall; grep leads on line and symbol recall.](https://assets.ai.engineer/sites/aiecode2025/sha256/725135f10ae691295ba13393f399a8051b08eb09ee179eeb8ff173830933ac9e.webp)

recall (how many needed files did it find?)

Legend: baseline; grep; grep + semantic
Y-axis ticks: 0%, 10%, 20%, 30%, 40%, 50%
File recall: baseline 49%; grep 39%; grep + semantic 39%
Line recall: baseline 29%; grep 40%; grep + semantic 39%
Symbol recall: baseline 36%; grep 43%; grep + semantic 40%

Claude Code finds more files with grep, but it is less efficient
- Baseline opens more files so finds more by volume, but wastes reads
- grep - opens fewer files but more right functions
- grep + semantic explores differently - similar recall to grep, but slightly worse

Other visible text: AIE; AI Engineer; EUROPE

### Recall deep dive: semantic search and grep

Grep + semantic led on all three recall measures in the 8 tasks where semantic search won; grep led in its 7 winning tasks. Semantic search found behavior-adjacent files, while grep helped trace imports.

[8:37](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=517s) [Two charts compare recall on tasks where semantic search or grep won. Notes associate semantic search with finding behavior-adjacent files and grep with tracing imports and finding function names near bugs.](https://assets.ai.engineer/sites/aiecode2025/sha256/3334080c9388f14e92c8c23c083e1b3d02ce189844606b81cf8293fe2c5a4d75.webp)

recall when semantic search "wins" (8 tasks)
Legend: grep; grep + semantic
Y-axis ticks: 0%, 20%, 40%, 60%, 80%
File recall: 39%, 61%
Line recall: 29%, 45%
Symbol recall: 33%, 47%

recall when grep "wins" (7 tasks)
Legend: grep; grep + semantic
Y-axis ticks: 0%, 20%, 40%, 60%
File recall: 59%, 32%
Line recall: 46%, 25%
Symbol recall: 51%, 22%

When semantic search won:
- finding lots of behavior-adjacent files and modules

When grep won:
- tracing through imports
- finding function names near bugs

Interface text: AIE; AI Engineer EUROPE

### Summary

Semantic search boosted precision, while grep found different code. The speaker says Claude Code struggled to know when and how to use the added semantic search tool.

[9:48](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=588s) [Summary: semantic search boosts precision; grep and semantic search find different code; effective tool use depends on reinforcement learning and embeddings. The closing message calls for lightweight tools to find relevant context.](https://assets.ai.engineer/sites/aiecode2025/sha256/6d58d77c72e6a2cf3b94c045235e612f075aea5d7fef61141932aa9ed9f4d9ed.webp)

summary

turbopuffer

- Semantic search boosts precision
- grep and semantic search find different code
- effective tool use depends on RL and embeddings (Cursor vs Claude)

Long term winners will provide lightweight tools to find the right context in various ways

AIE
Engineering the future of AI

[10:57](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=657s) [Summary of semantic search, grep, and lightweight tools for finding relevant context.](https://assets.ai.engineer/sites/aiecode2025/sha256/4d47ff63a60671eae1766fddea107627800a124d529686b127f14dab5e566e7b.webp)

- Semantic search boosts precision
- grep and semantic search find different code
- effective tool use depends on RL and embeddings (Cursor vs Claude)

Long term winners will provide lightweight tools to find the right context in various ways

AIE
AI Engineer
EUROPE

### Thank you

[11:01](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=661s) [Closing slide with a thank-you message and two pixel-art animals.](https://assets.ai.engineer/sites/aiecode2025/sha256/22eae1638bf9411c9c8df34533357a223df9783dd5cc09c570d99d382d2cc3ef.webp)

thank you

@turbopuffer
@rogutkuba

AIE
AI Engineer EUROPE

## Full transcript

[0:00](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=0s) [upbeat music] Hi, everyone.

[0:16](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=16s) Uh, welcome to Benchmarking Semantic Sear- or Semantic Code Retrieval on Claude Code. Uh, my name is Kuba. I- I'm here from turbopuffer. For those unfamiliar with turbopuffer, we are a serverless full text and vector search database built from first principles on top of object storage.

[0:29](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=29s) Uh, we serve some of the fastest growing AI companies in the world. And if you'd like to know more about the talk or any question about turbopuffer or anything, just find-- feel free to find me after.

[0:38](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=38s) Uh, so let's get started. So for those of you who are unaware, uh, Claude Code by default, or I mean, yeah, by default doesn't use semantic code search. Uh, there's actually a tweet from Boris.

[0:47](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=47s) Uh, those unfamiliar with Boris, he's essentially like the founding father of Claude Code, and he talks about how Claude-- the early versions of Claude Code did actually use semantic search with a local vector DB, but they just kinda found agentic search, essentially, you know, grepping through your file system kinda worked better and, you know, seems to work

[1:02](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=62s) simpler for, for Claude Code. However, uh, one of turbopuffer's customers, actually, fun fact, one of our very first customers, Cursor, uh, does use semantic code search and does in-index codebases into turbopuffer.

[1:14](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=74s) And you may think, like, this seems like a lot of work, and it kind of is. But the reason they do this is because they see r-real performance gains because of this.

[1:23](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=83s) Uh, they have this amazing blog post, uh, about their semantic search. They also have one about indexing codebases, which I highly recommend as well. Uh, but you can see on the right, the kind of performance gains they see with a Composer model is like twenty-four percent, uh, increase in an- or relative improvement in answer accuracy.

[1:38](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=98s) And I think it's like twelve and a half or thirteen percent across, uh, all models. And, you know, I, I-- This benchmark came up, uh, before Composer 2, but you can probably imagine there's some sort of similar performance gains with that, that new model as well.

[1:49](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=109s) And in the bottom right, you can see, uh, this is from-- So this same blog post, uh, they talk about how in-- they performed an online A/B test where they found that adding semantic-- either allowing or disallowing semantic, uh, code search led to, like, a two point six percent increase in code retention in large codebases and an

[2:05](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=125s) n-two point two percent decrease in dissatisfied user requests. And you may also think, like, these numbers seem kinda small, like what, two point six percent, two point two percent?

[2:14](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=134s) Um, well, keep in mind that, you know, out of a hundred queries, not all queries will ever really need semantic search or, like, benefit from it. Uh, so these-- They even state in the blog post that these numbers look small because not every-- You know, you could imagine a very simple tool call or very simple query would

[2:29](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=149s) not really use semantic search. Um, so what we think about here at turbopuffer and why we kind of really, uh, think vector search and why Cursor probably sees this, like, real performance gain is we think about how embeddings are cached compute.

[2:44](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=164s) And you could also, you know, what does even-- what do you mean cached compute? This sounds like a bunch of baloney you're throwing at me. Um, you know, if we would, like, walk through, like, a Claude Code trace and, like, a Cursor s-type trace of kind of the same thing, you can imagine that, uh, on the left

[2:56](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=176s) side, there's kind of like a, you know, grepping through the file system or what they call agentic search, um, how it just has, like, grep through. Uh, this is for finding metadata filtering and understanding it in your codebase.

[3:05](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=185s) It essentially grep through, read the files, grep through more if it doesn't find the right things. And again, this is repeated on every session across, you know, every agent, um, running in, you know, uh, for the same codebase so that even if you're asking the same question or trying to do the same thing multiple times, you always

[3:21](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=201s) have to do this compute again. And you can see, like, in this case, six thousand tokens isn't a lot in one time, but across every session and every agent, this really starts to add up.

[3:29](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=209s) And if we-- You know, on the right, if this is more of a, like, Cursor style or Cursor level trace, where you have this upfront cost, where we have to, like, chunk, embed, and index the codebase.

[3:37](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=217s) But then you essentially have this cache of the semantic meaning so that when a f-when an agent wants to understand, you know, where or how does metadata filtering work, uh, it can simply query, you know, how is metadata filtered and get a lot, uh, get the chunks it needs and kinda the understanding it needs a lot faster

[3:54](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=234s) and, uh, with token savings. And again, this doesn't-- It's not a lot of savings in one time, uh, but we all, you know, we're not really running one agent anymore.

[4:03](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=243s) Like, I'm running, like, three at one time. So, um, there's definitely some l-long-term savings. So, uh, what we did, what we did is we built a simple CLI tool for Claude Code.

[4:12](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=252s) Is-- We call it TurboGrep. Um, it's essentially, you can imagine, it's just a simple way of, um, using a tree-sitter library to essentially parse through your codebase, uh, chunk it, embedding using the Voyage code model, and then upload it to turbopuffer.

[4:26](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=266s) Um, this is just a simple, um, file system walkthrough. Uh, there is an open source library for the V1 version, and then soon the V2 version will be, uh, open sourced.

[4:37](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=277s) Um, here there's a little video of example a, a tool call trace for a Claude Code where you'll see, uh, it will call the TPuff tool. Now, it's called TPuff Search Password Reset Token Generator.

[4:49](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=289s) This is in the, uh, Django repo. This is one of the, the repos I was testing it on. And you can see that, uh, it gets the content and then is able to kind of give the full, um, explanation to the user.

[5:02](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=302s) And, you know, it's very easy to say, like, "Oh, this works better," but the important thing is obviously benchmark. Let's see for real, what, what-- how much better does it work?

[5:10](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=310s) Uh, Cursor has their own internal, uh, context bench, and there's this paper, this public paper called Context Bench, um, where essentially the benchmark is not really testing whether or not the coding agent solved or didn't solve a problem.

[5:22](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=322s) It tests when it-- In the process of solving the problem, did it find certain files? Did it s-find certain lines and did it find certain symbols? Because they, you know, they kind of have this, uh, this thesis that, like, it, you know, it's also important how you get there, not just, um, like, the end goal.

[5:39](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=339s) The process really matters for understanding, like, are agents actually looking for the right files? So essentially, it's a human-labeled data set of, like, in order to complete this task, the, the agent should have looked at this file, these few lines, and these few symbols in order to, like, actually complete the task well.

[5:54](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=354s) Uh, and I tested it with three conditions. Uh, essentially raw Claude Code out of the box. Uh, then I tested it with Claude Code with a fa- uh, the max of fifty line reads at a time.

[6:03](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=363s) And then, uh, the same thing with the windowed reads, uh, with the TPuff Search tool. Uh, you may ask me, like, why this, like, fifty line read, uh, thing.

[6:10](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=370s) ... or limit. It's because it's, it became really noisy really fast. If it's just reading, like, long files, it's really hard to, like, understand, uh, and get a difference because it all ...

[6:19](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=379s) If it just reads a whole f- like, a thousand line file, it doesn't really make a lot of difference in the numbers.

[6:24](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=384s) So we'll start with the first result of it. Uh, this is precision. Precision is essentially the, [coughs] the measure of, of how many files did it read, like, in its total process, how many of them were actually golden files.

[6:37](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=397s) So if there was ... If the, if Claude read 10 files and eight of them were needed, it hits 80% precision. Uh, you can see the baseline, it hit, like, 65% precision, 33% on line precision, and 43% on symbol, and it kind of goes up as we add windowed grep.

[6:52](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=412s) And then windowed grep plus semantic search got it to, like, 87% file precision, for example. Uh, and this is also due to the fact, like, Claude Code by default, like, is really exploratory.

[7:00](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=420s) It loves to read as much as it can and, like, try to read everything, for example, and it kind of shows up in the 65% precision. Um, and, like, it's hard to, like, translate these numbers, but a more, like, English or, like, human version of it is that, you know, Claude Code, one in every three file reads

[7:15](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=435s) is actually just a completely wasted file. Um, and with windowed grep, it was one in five reads was a irrelevant file. And then with semantic search, it was only one in eight files was a, you know, quote, unquote, "wasted or irrelevant file."

[7:27](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=447s) And it kind of, like, scales up as you kind of add these tools.

[7:30](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=450s) And, you know, obviously this, this is already, like, pretty good. Uh, then we have recall. Recall is essentially how many of the needed files did it find. So for example, if there was 10 files in the task that it should have found, if Claude found five of them, five files in its total trajectory, and only three of

[7:46](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=466s) them were actually these golden files, it would hit 30% recall. Um, by default, again, uh, Claude Code actually does win the file recall. And again, part of this is because it just loves to explore every single file it can.

[7:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=478s) Um, then we have line recall, where it drops a lot more. This is kind of a ... Because it loved to read a lot of files, but also read to le- read a lot of files that didn't have a lot of, uh, golden context lines.

[8:10](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=490s) So you can kind of see that even though it did explore a lot, it also kind of explored the wrong things a lot. And then with grep and grep plus semantics, or windowed grep and windowed grep plus semantic search, kind of about the same recall.

[8:20](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=500s) Uh, you can kind of see on the right, like, kind of the, what happened with the behavior between these three conditions. Uh, and you may be thinking like, "Well, you know, semantic search didn't really, like, add improvements here.

[8:29](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=509s) Like, what, what, what went wrong?" Like, actually some decreases. And we can, like, dig in a little further into the recall numbers for certain tasks. This was across, uh, 50 tasks by default.

[8:38](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=518s) Uh, but if we break it down into where semantic search won, you know, quote, unquote, "won," and where just, like, windowed grep won, we see some stark differences between, like, certain tasks performed a lot better with semantic search, uh, allowed, and then certain tasks performed a lot better with, with no semantic search.

[8:54](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=534s) And this kind of proves, um, you know, certain tasks require s- different types of tools. For example, when semantic search won, uh, it was really good at finding a lot of behavior adjacent files that didn't have the same kind of keywords.

[9:06](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=546s) For example, it was like ... I think one example off the top of my head was with trying to, like ... It was, like, multi... Like, first handle lots of different ORMs that have to, like, handle across, uh, different libraries, and it didn't by default, the keyword search didn't by default find all of them, but behaviors, uh,

[9:19](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=559s) semantic search was able to kind of, like, understand these are all related files. Uh, and then grep won when it was really good at just ... The task was a lot about, like, tracing through imports, and if it, like, was able to find the keywords in, like, the first or second, uh, tool call, it was able to,

[9:34](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=574s) like, just keyword search through that and find the relevant files. So again, it's like two different types of, uh, how to find files, but they kind of lead to, like, really different results.

[9:43](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=583s) So in summary, like, what does this mean? Well, we, we saw semantic search, like, did boost precision quite a bit. Uh, and we, like, kind of understand, like, grep and semantic search kind of find different code different ways.

[9:55](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=595s) Um, and an interesting thing to note as well was, um, these numbers weren't as great as Cursor's because part of, like, part of Claude Code is it's built for just grepping.

[10:05](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=605s) Like, that's, that's what Anthropic kind of focuses it on. Like, it's not built to under- like, to really understand, like, when to call semantic search or how to call it.

[10:14](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=614s) Uh, we kind of, like, add it as an extra tool, and it's like, "Hey," like, "here's this cool tool. You probably should use it sometimes," but it's very hard for it to have a true understanding of when to use it, why to use it, uh, versus, like, like, for example, Cursor's Composer model, they understand this is a

[10:27](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=627s) built-in tool that, um, it knows when and how to use it, and that's why they saw this, like, 23 and a half perfor- uh, percent, uh, performance gain. Uh, so, like, in summary, uh, you know, we think long-term winners will find con- kind of provide these lightweight tools to find the right context in various different ways, and

[10:43](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=643s) I think it's something important to think about. Uh, you can't just, like, grep through everything, unfortunately, in a file system. Uh, we think there's a lot of different ways to access lots of different types of information, uh, and the people that provide these, like, easy tools to provide, to shrink down these billion context windows into the right

[10:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=658s) million, uh, will win in the long term. Um, that's the general talk. Thank you. If you have any questions, feel free to come up. [audience applauding]

[11:07](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=667s) I have my own idea of what semantic search is, but could you define what you, what your definition is of semantic search?

[11:13](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=673s) Sure. Uh, so this was just, uh, just doing vector search. It was just performing vector searching, uh, embedding it using Voyage's code model, and then just embedding the th- the, the query, um, query sentence or, uh, query tokens, and just sending them back to Tur- to turbopuffer.

[11:28](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=688s) Yeah.

[11:29](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=689s) Uh, sorry. But which, uh, embedding model are you using?

[11:32](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=692s) Uh, voyage-code-3.

[11:34](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=694s) Okay. And which vector DB?

[11:36](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=696s) turbopuffer.

[11:37](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=697s) Ah, okay. [laughs]

[11:38](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=698s) Yeah.

[11:38](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=698s) Sorry. [laughs]

[11:39](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=699s) Oh, no worries. Yeah, yeah. Yeah, for those, for those who are unfamiliar with turbopuffer-

[11:43](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=703s) That's all right

[11:43](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=703s) ... we are the, we are the vector database that powers companies like Cursor, Anthropic, Notion. So when you use something like Cursor, uh, you have by proxy used turbopuffer.

[11:53](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=713s) Um, so you may know us just not by name, I guess.

[11:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=718s) So...

[11:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=718s) I just wanted to ask, you may not have a benchmark for this, but how does it perform on shit code?

[12:04](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=724s) Oh, that's, that's tough. I mean, it's hard to say.

[12:08](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=728s) Is grepping better or is semantic better for code that- That sort it out

[12:12](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=732s) Uh, so I think it works best when there's a lot of d- like, comments-

[12:17](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=737s) Mm-hmm

[12:17](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=737s) ... on code, uh, because it kind of provides that semantic meaning

[12:20](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=740s) So working alongside, like, documentation that's kind of inline?

[12:22](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=742s) Yeah, if it's like inline documentation, that was like a big, um, boost. I believe, uh, one of the repos I remember, like, looking through some trajectories and, like, asking Claude, be like, "Why did it perform so well here versus not?"

[12:32](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=752s) And it was when it, it kind of explains to me like, um, when I was looking through as well, like, those with, like, really good comments, for example, just, like, comments about the function, it's able to, like, really understand a lot more 'cause you, you kind of give this context to the model and the embedding model so

[12:45](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=765s) then it can, like, actually search better. Um, 'cause that's, that's part of it. Like, the, the embedding i- and, uh, is not the hard part. It's, like, figuring out what meaning really is of that chunk.

[12:56](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=776s) Yeah.

[12:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=778s) Okay.

[12:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=778s) Yeah, you may have, you have, you've mentioned some of it now, but of course semantic search is just similarity search, so if your query doesn't really match the format of what you're querying against, then you get some kind of innate distance.

[13:12](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=792s) Do you do any kind of pre-processing of what your, uh, of the target data before you... Like, do you do any parent-child where the parent is query-ish and the child is the real code?

[13:23](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=803s) Or how do you-

[13:24](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=804s) In, in this case, it was just simple, just-

[13:26](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=806s) Yeah

[13:26](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=806s) ... just the code.

[13:26](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=806s) Just the raw code?

[13:27](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=807s) Yeah, just the raw code, just as a thing. But, um, uh, I can't speak for, like, how these more complicated and sophisticated customers use this, but I can imagine it's definitely something of providing not just code level meaning, but, you know, as you said, at least a parent-child relationship of like-

[13:40](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=820s) Yeah, like the authentication flow. Like, that could be a good query to do a similarity search against, but the code itself is more, like, raw code, so.

[13:47](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=827s) Yeah. Yeah. Like, Cursor has their own embedding model, which I think kind of helps with this, of like how do you translate code into more of, like, a human level-

[13:54](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=834s) Yeah

[13:55](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=835s) ... like, query. And I mean, they're kind of been experts on that. I can't speak for how they do it, but I just know they do do that.

[14:01](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=841s) Yeah.

[14:01](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=841s) Yeah.

[14:01](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=841s) I think they actually do what you said, like they create fake comments on top your, of your code, and they embed the code with the comments.

[14:09](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=849s) Yeah.

[14:09](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=849s) So that's how they can have, like, the higher recall when, uh-

[14:14](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=854s) Oh, okay. Yeah

[14:15](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=855s) ... they search all the code.

[14:15](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=855s) So they add, they kind of inject comments?

[14:17](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=857s) Yeah. Yeah.

[14:18](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=858s) Yeah, something like that I think definitely could work, yeah.

[14:21](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=861s) Interesting.

[14:21](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=861s) In the back?

[14:22](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=862s) Yeah. I was gonna ask, like, how do you see, uh, I guess, the vector database, uh, kind of working with the Rennis and, and how, like, when do you encode, like, long term memory to reduce the size of the, uh, database?

[14:38](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=878s) I mean, I think it depends. Like, obviously the easiest way, like, people love grepping because it's zero cost. Like, if you're able to, like, download everything to your local file system and just, like, grep through it, like, yeah, that works.

[14:48](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=888s) Um, I think vector DBs are built for essentially, like, multiplayer and, like, this, like, super [cough] maybe in a sense, like, hard to understand or, uh, complicated relationships between lots of data.

[14:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=898s) For example, like a knowledge base, like a Notion. Like, you can imagine kind of hard to, like, really grep through that really easily on your local machine. Like, it's in a sense best to, like, have that vectorized, uh, for the agents.

[15:09](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=909s) Um, and even stuff, like we have customers doing stuff like with multimodal data. Like, you can't really grep through a video file. You can't grep through an audio file.

[15:15](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=915s) You can't grep through an, an image file. Like, maybe you can glob on the, on the filename, um, but to, like, get a true understanding the kind of multimodal datas as well is, that's something that we find a lot of customers are doing.

[15:26](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=926s) Um, so and it just kind of depends on the workload and, um, I, I, yeah, if you're, if you're hitting qu- like at some sort of even, like, miniature scale, like a vector DB kind of like helps offload a lot of this, this work into, like, you know, cached compute- cache this semantic meaning.

[15:42](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=942s) Um, any other questions? Perfect. Thank you all. [audience applauding] [upbeat music]
