AI Engineer World's Fair 2024
Embeddings are Stunting Agents: How Codeium Breaks Through the Ceiling for Retrieval
Loading slides…
Read the talk
Embeddings are Stunting Agents: How Codeium Breaks Through the Ceiling for Retrieval
A React contact form exposes the limits of similarity search: useful generation needs several complementary sources, an evaluation that measures their recovery, and affordable compute to rank them.
From a talk by Kevin Hou
Before you start: Familiarity with React components and the basic idea of supplying retrieved context to a language model is helpful.
A contact form that belongs in the repository
How do you generate a React contact form that actually belongs in an existing codebase? For Codeium, this is a product problem spanning autocomplete, chat, and search inside an IDE. Kevin Hou describes a plugin with over 1.5 million downloads, supporting 70 languages and 40 IDEs at the time of the talk. He also cites strong marketplace ratings and a leading developer-tool rating in the Stack Overflow survey. Enterprise customers add requirements beyond plausible code: security, licensing, attribution, and output that can make it into production. These constraints lead to the central question of context awareness: what must the model know before it generates?
The basic retrieval-augmented generation flow is straightforward: receive a query, gather context from several sources, supply it to the language model, and generate a response. The contact form shows why the gathering step is difficult. A generic answer might produce a valid React component while missing the repository’s existing buttons and inputs. Even if it finds those components, it still needs examples of nearby forms to understand how the project uses them.
Styling introduces another dependency: a project using Tailwind needs a form that follows its configuration and visual conventions. Local documentation and external dependency documentation supply further constraints. The slide makes this concrete with internal components, other forms, styling guidance, and documentation for packages such as react-hook-form and zod. The retrieval target is a useful set of complementary context, not one similar document. The engineering problem is to collect and rank that set quickly enough for interactive code generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three ways to supply repository knowledge
Long context makes the interface to this problem easy: put more material in the prompt. But input capacity does not remove the cost of processing that input. Hou cites a historical Gemini example taking 36 seconds to ingest 325K tokens, without specifying the model version or test conditions. He estimates roughly one million tokens for 100K lines of repository code and describes enterprise code estates exceeding a billion tokens. His practical concern is the mismatch between the amount of available code and the latency and cost an IDE interaction can tolerate.
Fine-tuning moves customer-specific information into model weights. In the per-customer setup Hou considers, keeping those weights aligned with changing code requires continuing updates and expensive computation. Embeddings offer a cheaper way to compute and store searchable representations, but introduce a different question: whether their representation captures all the distinctions needed to select several relevant items.
| Approach | Mechanism | Constraint emphasized in the talk |
|---|---|---|
| Long context | Supply more input at generation time | Input latency and cost |
| Fine-tuning | Adapt weights to customer data | Updates and per-customer computation |
| Embeddings | Store compact searchable vectors | Relevance across multiple items |
These approaches place the work in different parts of the system. Enlarging the prompt, updating the weights, and retrieving compact representations each change the economics of supplying context; none makes the contact form’s collection problem disappear.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Similarity is not the same as task relevance
Embedding search starts by choosing searchable units: functions, documents, or other chunks of code. An embedding model converts each unit into a fixed-dimensional array of numbers. Those vectors make it possible to search a large collection efficiently by similarity, without reading every original item through a language model for each request.
Hou observes that the embedding results displayed in the talk cluster within approximately plus or minus five score points. He interprets this convergence as a ceiling and offers a hypothesis: a fixed vector cannot preserve every distinction that might matter across all possible queries. That is Codeium’s explanation for the observed pattern, not a demonstrated universal bound on embeddings. The product question is narrower and more actionable: does semantic distance between vectors reliably identify the functions needed for a particular coding change?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure recovery of the whole relevant set
Hou characterizes the evaluation mismatch as finding one needle when the task requires several. For the contact form, retrieving a relevant form example does not compensate for missing the design-system inputs or the styling rules. His criticism concerns what a coding product needs to measure: successful recovery of multiple complementary documents, rather than success at locating a single relevant instance.
Codeium therefore uses Recall@50: the fraction of the ground-truth relevant set present among the first 50 retrieved items. Let G be the set of relevant items and R₅₀ the set of the first 50 results:
The denominator is the number of relevant items, not 50. Dividing by the number of returned results would measure precision instead. A retrieval system can return many individually plausible files and still have poor recall if it misses the dependencies the task needs.
A small TypeScript implementation makes the distinction explicit. Here the contact form’s teaching set contains component files, an existing form, styling configuration, and dependency documentation. Retrieving three of those five items yields 0.6, even when the returned list also contains an unrelated file.
typescript
function recallAt50(
rankedIds: readonly string[],
relevantIds: ReadonlySet<string>,
): number {
if (relevantIds.size === 0) {
throw new Error("Recall requires a nonempty relevant set");
}
const retrieved = new Set(rankedIds.slice(0, 50));
let hits = 0;
for (const id of relevantIds) {
if (retrieved.has(id)) hits += 1;
}
return hits / relevantIds.size;
}
const relevant = new Set([
"src/ui/button.tsx",
"src/ui/input.tsx",
"src/forms/signup.tsx",
"tailwind.config.ts",
"docs/form-validation.md",
]);
const ranked = [
"src/forms/signup.tsx",
"src/ui/button.tsx",
"src/pages/home.tsx",
"src/ui/input.tsx",
];
const recall = recallAt50(ranked, relevant); // 0.6
This measures coverage of the labeled set. Whether that set accurately represents what generation needs is the next problem.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn commits into retrieval tasks
A useful metric needs queries and relevance labels that resemble the product’s workload. Codeium builds these from pull requests:
- Break a pull request into its commits.
- Extract each commit’s English description.
- Pair that description with the files modified by the commit.
- Use the description as a retrieval query and the modified-file set as relevance labels.
This creates a scalable mapping from an expressed change to a set of files involved in implementing it. It approximates the situation in which a developer asks a coding assistant to make a change across a repository.
The labels are product-shaped, but their meaning matters: modified files identify where a change landed, not necessarily every unchanged file or document needed to understand it. For the contact form, an existing button implementation might be necessary context without needing modification. The commit construction is therefore an approximation to useful generation context, consistent with Hou’s goal of getting closer to the end-user distribution.
Hou reports that publicly available models performed worse on this commit-based retrieval task and that Codeium’s approach improved retrieval for its chat and autocomplete workflows. The talk does not supply numerical Recall@50 results or a reproducible evaluation protocol for that comparison. The substantive shift is the evaluation target: measure how well retrieval supports an English request over a real codebase, then optimize against that target.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make more retrieval computation affordable
Codeium’s response is to spend more compute on retrieval. That immediately raises a serving question: how can an interactive product afford richer judgments on every request? Hou connects the answer to vertical integration, with three parts:
- Custom models: train models around the workflows the product actually serves.
- Custom infrastructure: control the serving stack, drawing on Codeium’s origins as the ML infrastructure company Exafunction. Hou credits that control with lower costs and higher serving efficiency, including competitive advantages he asserts in the talk.
- Product feedback: judge shipped features through end-user behavior, rather than treating a local benchmark gain as the final result.
The infrastructure investment matters because it changes how much model computation the product can spend on deciding what to retrieve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
M-Query judges candidate items in parallel
M-Query takes a query and a collection of N codebase items, then makes parallel language-model calls to reason about individual items. Hou illustrates the operation as asking a model whether an item is relevant and receiving a yes-or-no answer. That explains the kind of judgment involved; it is not a disclosure of an exact production prompt or output schema. The key change is that relevance can be assessed with the actual query and item together, rather than inferred only from their positions in a fixed vector space.
Those judgments feed a ranking that can also account for the active files, neighboring directories, recent commits, and the ticket currently being worked on. The system selects the top-ranked documents and supplies them to generation, which can then stream code or a chat response. For the contact form, this means evaluating whether a component, form example, or style document helps satisfy the request—not merely whether it resembles the words in the request. The description establishes a model-based relevance stage; it does not establish that embeddings have been removed from every part of the retrieval system.
Hou claims Codeium’s computation costs one-hundredth as much as competitors’ computation, allowing 100 times as much compute per user. He attributes that economics to owned models and infrastructure rather than external API calls; no comparison workload or measurement setup is disclosed. The design commitment is nevertheless clear: spend the resulting compute budget on more thorough context selection because it can improve the generated result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test retrieval in the interaction it serves
The product still has to satisfy three requirements: speed, correct generations, and an understandable interaction. Hou reports that M-Query runs thousands of LLM executions in parallel and can begin streaming code within seconds, sometimes milliseconds. The talk does not give a latency distribution or define the timing boundary. Parallel execution addresses the waiting time associated with many judgments, while the infrastructure economics address their aggregate cost. Usability adds another requirement: developers should be able to understand what context the tool is using.
Codeium rolled the system out to a small percentage of users. Hou describes a download base exceeding one million, but that is not the experiment’s participant count. He reports selecting relevant context across thousands of files in monorepos and remote repositories. The demonstration asks for usage of an internally modified shadcn/ui Alert Dialog, then retrieves the relevant source. That local modification is the point: generic knowledge of a public component cannot establish how a repository’s own version should be used. The linked documentation now describes the Base UI variant, not Hou’s modified component from the recording.
Hou reports more thumbs-up on chat messages, more accepted generations, and more code written for users after the experiment. No effect sizes are supplied. These outcomes connect retrieval quality to the behavior the product ultimately cares about: whether developers find the answer useful enough to accept and use it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A context engine and a production feedback loop
Once context selection becomes a shared engine, it can support more than one generation surface. At the time of the talk, Hou names autocomplete, chat, and search as current uses. He proposes documentation, commit messages, code reviews, and code scanning as future applications. Figma-to-UI generation extends the contact-form problem: converting a design should produce an interface built from the repository’s own components. These are proposed uses of the retrieval foundation, not a claim that every workflow had already shipped.
The development loop starts with the user problem and returns to it through production:
- Build the evaluation from the task. Construct datasets and local metrics that reflect what users ask the product to do.
- Apply infrastructure and compute. Spend additional computation on improving results for paying and nonpaying users.
- Deploy the change. Hou describes pushing changes overnight to obtain a production pulse check.
- Use the feedback in the next iteration. Thumbs-up and thumbs-down signals help determine what to revisit.
M-Query is the example of this loop in action: define a retrieval problem, improve the mechanism, deploy it, and examine whether the resulting interactions are better.
The context engine also depends on less visible work: modeling, serving infrastructure, retrieval, AST parsing, indexing large collections of repositories, knowledge graphs, documentation parsing, and gathering website context. Hou presents these as pieces developed through the same iteration cycle, rather than a single retrieval algorithm that solves the entire problem. Their value compounds when they improve what the product can understand and deliver.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let shipped products test the compute argument
Hou closes by drawing on his experience in autonomous driving. He recalls optimistic predictions in 2015 and contrasts them with the field as it stood in 2024. Earlier systems used sensor fusion, lower polling rates, and off-board models to work around limited onboard compute. In his account, more available computation made larger models, more sensors, and higher-frequency processing practical; Waymo rides in San Francisco provide his concrete contemporary example. The analogy motivates a design direction rather than proving a retrieval result.
Applied to coding tools, the argument is that embedding-based retrieval may be a heuristic shaped by compute constraints. If richer model judgments become affordable, a product can spend more computation on the relevance decision itself. Hou’s final standard is practical: ideas about better retrieval must survive implementation, deployment, and use. The case for more compute is made through the working product it enables.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Current installation and usage examples for the alert-dialog component family used in Hou's repository-context example; this page documents the Base UI variant.
Further reading
- Kevin Hou explains M-QueryArticle
Hou's companion explanation of parallel relevance judgments, custom infrastructure, and the retrieval argument behind the talk.
The original benchmark paper evaluates embeddings across eight task categories, including retrieval, clustering, and reranking.
Updates since the talk
A December 2024 interview with Hou about vertical integration, MQuery, internal product testing, and the move to the Windsurf editor.
Documentation covering indexed repository context, context pinning, remote repositories, and M-Query under the current Devin Desktop branding.
Full transcript
Loading transcript…