AI Engineer Summit 2023
Retrieval Augmented Generation in the Wild
Read the talk
Retrieval Augmented Generation in the Wild
Vector search starts the RAG loop, but useful AI memory also needs feedback, updates, task-aware chunking, and a way to recognize when retrieved results are irrelevant.
From a talk by Anton Troynikov
Before you start: Familiarity with embeddings, nearest-neighbor search, and an LLM’s context window will help you follow the retrieval design choices.
Start with the retrieval loop
Take a corpus of documents, embed it, and store the vectors alongside their source documents. When a query arrives, embed that too. Find its nearest neighbors, retrieve the associated documents, and place those documents and the query into an LLM’s context window. The model then generates a response. This is the familiar starting point for retrieval augmented generation, or RAG.
Anton Troynikov, Chroma’s co-founder, calls this open-loop retrieval. Information flows from the corpus through retrieval into generation, but the basic diagram contains no path for learning from the result. Vector search supplies candidates; building more capable applications requires a memory system that can also change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Feedback turns retrieval into mutable memory
The first addition is human feedback. An embedding model trained for general use does not automatically encode what matters for a particular dataset, task, generation model, or user. Relevance judgments provide a signal for adapting the data and embedding model to that application. The memory layer therefore needs somewhere for those judgments to go and a way for them to influence future retrieval.
Agents add another update path: they need to write to memory themselves. Instead of repeatedly searching a fixed corpus, an agent can retain information produced during its work. That makes the dataset continuously mutable. Troynikov’s objection to a search-index-only architecture is that nearest-neighbor lookup alone does not provide the feedback and update behavior these applications need.
An agent with a world model takes this further. It must store its interactions with the environment and revise its working data based on what happened. Human feedback, agent-written memory, and observations of the world then become parts of the same system. At the time of the talk, Troynikov describes this combination as research-grade, while pointing to early applications already using parts of it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Remembering skills in Minecraft
Voyager, the Minecraft agent shown in the talk, illustrates why memory can be more than a collection of reference documents. The agent learns skills in its environment and recalls them when it encounters a suitable context again. Retrieval lets a previously acquired capability become available for later work instead of requiring the agent to rediscover it.
Troynikov also characterizes some complex skills as learned through human demonstration and retained in Chroma. That characterization differs from the original Voyager project’s account, which describes autonomous skill acquisition without human intervention. Chroma’s role in retrieval is supported by the public Voyager skill manager, though that source is not a snapshot of the conference demonstration. The useful architectural point is the connection between acquiring a skill, retaining it, and retrieving it in a later context.
The basic RAG loop remains useful for many applications, but this kind of reusable memory asks more of the retrieval layer. Search and recommendation systems provide a long history of information-retrieval techniques. Production AI applications introduce a different consumer of the results: a model that will use the retrieved material as part of its computation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve the relevant information—and exclude distractors
Useful context requires both inclusion and exclusion: retrieve the information the task needs, while keeping irrelevant material out. Troynikov emphasizes that distractors can sharply degrade an AI application’s performance. The talk provides no measured degradation rate, so this is a warning about context quality rather than a universal numerical result.
That requirement exposes three connected decisions:
| Decision | What it determines |
|---|---|
| Embedding model | Which documents appear close to the query |
| Chunking | Which units of information are available to retrieve |
| Relevance assessment | Whether a result helps this task and user |
A provider’s claim that an embedding model is strong on code, English, or multilingual data does not settle its suitability for your corpus. Chunking changes the candidates themselves, and even a close candidate still needs to be relevant to the intended use.
There is no universal recipe for these choices in the talk. The opportunity is to use production data to investigate them: real queries, real documents, and examples of what users actually find useful can expose requirements that academic benchmarks miss.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose embeddings with application evidence
Troynikov describes existing academic embedding benchmarks as largely saturated and insufficiently representative of AI retrieval workloads. He does not identify a benchmark or supply a saturation measurement. His practical response is to bring the evaluation machinery to the application’s own data.
The proposed workflow is straightforward:
- Apply existing open-source benchmark tooling to your datasets and use cases.
- Collect human relevance judgments through a feedback endpoint.
- Build evaluation datasets from the queries and outcomes observed in production.
- Compare candidate embedding models using evaluation tools against those application-specific examples.
Chroma’s relevance-feedback endpoint and support for evaluation vendors are presented here as planned work, not as demonstrated features. The important dependency is that model comparison needs a definition of usefulness grounded in the application.
There is also a more speculative possibility: the initial embedding-model choice might become less binding if representations can be mapped between models. Troynikov’s claim is conditional on models having the same training objective and roughly the same training data. Under those conditions, he says, learned representations tend to be similar up to an affine transformation. Written explicitly, the proposed mapping has the form:
Here A is a learned matrix and b is an offset. The suggestion is to determine such a mapping from your own dataset, potentially projecting one model’s embeddings into another model’s space. This is a research possibility, not a guarantee that arbitrary embedding models are interchangeable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Chunk boundaries determine what the model can see
Chunking begins with a capacity constraint: retrieved material must fit within the LLM’s bounded context. But fitting is only one requirement. A chunk should preserve semantic content, and specifically the semantic content relevant to the task. Document structure helps because text written for people already contains cues to meaning: its natural boundaries can guide where to split.
NLTK, LangChain, and LlamaIndex are named as existing sources of chunking tools. Troynikov then turns to an experimental approach using a lightweight language model: slide a window over the preceding tokens and ask how probable the next actual token in the document is. A token that the model assigns very low probability may indicate that the text has crossed a semantic boundary.
A compact way to express that signal is token surprise. Given next-token probabilities supplied by a model, this Python function returns candidate boundary positions:
python
from math import log
def candidate_boundaries(
next_token_probabilities: list[float],
minimum_surprise: float,
) -> list[int]:
boundaries = []
for position, probability in enumerate(next_token_probabilities):
if not 0 < probability <= 1:
raise ValueError("Probabilities must be in (0, 1].")
if -log(probability) >= minimum_surprise:
boundaries.append(position)
return boundaries
Each probability refers to the actual token at that position, conditioned on its preceding window. Lower probability produces higher surprise. The threshold produces candidates; it does not establish that every surprising token is a useful semantic split. Because a model supplies the boundary signal, Troynikov proposes fine-tuning it so that its boundaries better match the application.
Other approaches complement or replace that signal:
- Information hierarchies: Organize content at multiple levels; Troynikov points to LlamaIndex’s support for this approach.
- Multiple retrieval signals: Bring multiple data sources and signals into reranking rather than relying on one similarity value.
- Embedding continuity: Embed successive sliding windows of a document and look for discontinuities in the resulting sequence of vectors.
The last approach asks where the representation changes abruptly; the token-prediction approach asks where the text becomes unexpectedly hard to predict. Both boundary-detection ideas are described as experiments, without an effectiveness result in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Five bird pages cannot answer every fish question
Imagine a corpus containing every English-language Wikipedia page about birds. For each query, the application retrieves the five nearest neighbors and puts them into the model’s context. Then a user asks about fish. In Troynikov’s hypothetical, the system still returns five neighbors, but none is relevant to the query. Being nearest is a relative property; being useful to the task is a separate judgment.
How can the application make that judgment?
- Human feedback: Gather explicit judgments about whether retrieved material was relevant.
- Auxiliary reranking: Use another model and additional signals, including what the user was looking at and what they previously found useful.
- Scoped retrieval: When useful constraints are known beforehand, narrow the search through keywords or metadata filters. Troynikov describes these as supported by Chroma at the time; that statement does not establish a ranked lexical or hybrid-search implementation.
These approaches introduce evidence beyond the fact that a document occupies a position in the nearest-neighbor list.
The more ambitious proposal is to calculate conditional relevance directly. Given the available dataset and what the user is trying to accomplish, a relevance model would distinguish usefulness per user, per task, per generation model, and per individual task instance. That requires understanding both the query’s semantics and the content available in the corpus. Troynikov identifies lightweight open-source language models operating at the data layer as a promising experimental direction, not a demonstrated relevance guarantee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From vector storage to an AI data layer
The closing roadmap moves from retrieval research to infrastructure. Chroma’s next engineering step was a horizontally scalable cluster version, extending the existing single-node system across multiple nodes. Troynikov announced a December target for a Chroma Cloud database-as-a-service technical preview and a January target for hybrid deployments in enterprise clusters. These are targets from the 2023 recording, not statements of current availability.
The planned data layer also extended beyond text. Troynikov anticipated the arrival of the GPT Vision API, possibly at OpenAI’s developer day, and image and voice capabilities in Gemini. Those contemporaneous expectations motivated multimodal retrieval support: if a model can consume more kinds of data, its retrieval system must be able to supply them.
Multimodal data expands the definition of a useful result. Semantic relevance remains important, but qualities such as aesthetic suitability can also matter. A retrieval system may therefore need to evaluate several dimensions of usefulness rather than treating text similarity as the entire problem. Model selection appears as another item of work in progress, without implementation details.
The intended developer experience is analogous to using Postgres in a web application: the database handles the data layer so that developers can concentrate on application logic. Chroma’s ambition is to provide that boundary for AI applications, absorbing the work of storing, updating, and selecting useful information while leaving developers responsible for making the application behave correctly.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original project explanation of Minecraft exploration, reusable executable skills, and retrieval from a growing skill library.
Implementation of persistent skill-description embeddings and retrieval using Chroma.
Further reading
A 2021 study of conditions under which learned representations are identifiable up to a linear transformation.
Arithmetic-reasoning experiments measuring the effects of irrelevant context and ways to mitigate them.
Anton’s April 2023 interview on Chroma's application-memory goals and developer experience.
An implementation of topical text segmentation based on changes in lexical co-occurrence.
Practical examples of splitting documents with ordered separators, size limits, and overlap.
Updates since the talk
Current examples for nearest-neighbor queries, result counts, metadata filters, and document-content filtering.
Full transcript
Loading transcript…