AI Engineer Summit 2023

Retrieval Augmented Generation in the Wild

Anton Troynikov· CTO, Chroma12:20

Read the talk

Retrieval Augmented Generation in the Wild

Vector search starts the RAG loop, but useful AI memory also needs feedback, updates, task-aware chunking, and a way to recognize when retrieved results are irrelevant.

From a talk by Anton Troynikov

Before you start: Familiarity with embeddings, nearest-neighbor search, and an LLM’s context window will help you follow the retrieval design choices.

Start with the retrieval loop

Take a corpus of documents, embed it, and store the vectors alongside their source documents. When a query arrives, embed that too. Find its nearest neighbors, retrieve the associated documents, and place those documents and the query into an LLM’s context window. The model then generates a response. This is the familiar starting point for retrieval augmented generation, or RAG.

Anton Troynikov, Chroma’s co-founder, calls this open-loop retrieval. Information flows from the corpus through retrieval into generation, but the basic diagram contains no path for learning from the result. Vector search supplies candidates; building more capable applications requires a memory system that can also change.

0:270:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:27 · section reference included

Feedback turns retrieval into mutable memory

The first addition is human feedback. An embedding model trained for general use does not automatically encode what matters for a particular dataset, task, generation model, or user. Relevance judgments provide a signal for adapting the data and embedding model to that application. The memory layer therefore needs somewhere for those judgments to go and a way for them to influence future retrieval.

Diagram titled “human feedback” connects a user, the OpenAI logo, and Chroma, with red feedback and update arrows.Open full source frame
Human feedback adds an update path to Chroma.

Agents add another update path: they need to write to memory themselves. Instead of repeatedly searching a fixed corpus, an agent can retain information produced during its work. That makes the dataset continuously mutable. Troynikov’s objection to a search-index-only architecture is that nearest-neighbor lookup alone does not provide the feedback and update behavior these applications need.

An agent with a world model takes this further. It must store its interactions with the environment and revise its working data based on what happened. Human feedback, agent-written memory, and observations of the world then become parts of the same system. At the time of the talk, Troynikov describes this combination as research-grade, while pointing to early applications already using parts of it.

1:201:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:20 · section reference included

Remembering skills in Minecraft

Voyager, the Minecraft agent shown in the talk, illustrates why memory can be more than a collection of reference documents. The agent learns skills in its environment and recalls them when it encounters a suitable context again. Retrieval lets a previously acquired capability become available for later work instead of requiring the agent to rediscover it.

Troynikov also characterizes some complex skills as learned through human demonstration and retained in Chroma. That characterization differs from the original Voyager project’s account, which describes autonomous skill acquisition without human intervention. Chroma’s role in retrieval is supported by the public Voyager skill manager, though that source is not a snapshot of the conference demonstration. The useful architectural point is the connection between acquiring a skill, retaining it, and retrieving it in a later context.

The basic RAG loop remains useful for many applications, but this kind of reusable memory asks more of the retrieval layer. Search and recommendation systems provide a long history of information-retrieval techniques. Production AI applications introduce a different consumer of the results: a model that will use the retrieved material as part of its computation.

2:503:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:50 · section reference included

Retrieve the relevant information—and exclude distractors

Useful context requires both inclusion and exclusion: retrieve the information the task needs, while keeping irrelevant material out. Troynikov emphasizes that distractors can sharply degrade an AI application’s performance. The talk provides no measured degradation rate, so this is a warning about context quality rather than a universal numerical result.

That requirement exposes three connected decisions:

DecisionWhat it determines
Embedding modelWhich documents appear close to the query
ChunkingWhich units of information are available to retrieve
Relevance assessmentWhether a result helps this task and user

A provider’s claim that an embedding model is strong on code, English, or multilingual data does not settle its suitability for your corpus. Chunking changes the candidates themselves, and even a close candidate still needs to be relevant to the intended use.

“Challenges in Retrieval” slide states that models need relevant information without irrelevant information, then lists embedding selection, chunking, and task and user relevance.Open full source frame
Retrieval challenges: embedding choice, chunking, and relevance to the task and user.

There is no universal recipe for these choices in the talk. The opportunity is to use production data to investigate them: real queries, real documents, and examples of what users actually find useful can expose requirements that academic benchmarks miss.

4:004:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:00 · section reference included

Choose embeddings with application evidence

Troynikov describes existing academic embedding benchmarks as largely saturated and insufficiently representative of AI retrieval workloads. He does not identify a benchmark or supply a saturation measurement. His practical response is to bring the evaluation machinery to the application’s own data.

The proposed workflow is straightforward:

  1. Apply existing open-source benchmark tooling to your datasets and use cases.
  2. Collect human relevance judgments through a feedback endpoint.
  3. Build evaluation datasets from the queries and outcomes observed in production.
  4. Compare candidate embedding models using evaluation tools against those application-specific examples.

Chroma’s relevance-feedback endpoint and support for evaluation vendors are presented here as planned work, not as demonstrated features. The important dependency is that model comparison needs a definition of usefulness grounded in the application.

There is also a more speculative possibility: the initial embedding-model choice might become less binding if representations can be mapped between models. Troynikov’s claim is conditional on models having the same training objective and roughly the same training data. Under those conditions, he says, learned representations tend to be similar up to an affine transformation. Written explicitly, the proposed mapping has the form:

mappedembedding=A×sourceembedding+bmapped_embedding = A × source_embedding + b

Here A is a learned matrix and b is an offset. The suggestion is to determine such a mapping from your own dataset, potentially projecting one model’s embeddings into another model’s space. This is a research possibility, not a guarantee that arbitrary embedding models are interchangeable.

5:275:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:27 · section reference included

Chunk boundaries determine what the model can see

Chunking begins with a capacity constraint: retrieved material must fit within the LLM’s bounded context. But fitting is only one requirement. A chunk should preserve semantic content, and specifically the semantic content relevant to the task. Document structure helps because text written for people already contains cues to meaning: its natural boundaries can guide where to split.

NLTK, LangChain, and LlamaIndex are named as existing sources of chunking tools. Troynikov then turns to an experimental approach using a lightweight language model: slide a window over the preceding tokens and ask how probable the next actual token in the document is. A token that the model assigns very low probability may indicate that the text has crossed a semantic boundary.

A compact way to express that signal is token surprise. Given next-token probabilities supplied by a model, this Python function returns candidate boundary positions:

python

from math import log


def candidate_boundaries(
    next_token_probabilities: list[float],
    minimum_surprise: float,
) -> list[int]:
    boundaries = []
    for position, probability in enumerate(next_token_probabilities):
        if not 0 < probability <= 1:
            raise ValueError("Probabilities must be in (0, 1].")
        if -log(probability) >= minimum_surprise:
            boundaries.append(position)
    return boundaries

Each probability refers to the actual token at that position, conditioned on its preceding window. Lower probability produces higher surprise. The threshold produces candidates; it does not establish that every surprising token is a useful semantic split. Because a model supplies the boundary signal, Troynikov proposes fine-tuning it so that its boundaries better match the application.

Other approaches complement or replace that signal:

  • Information hierarchies: Organize content at multiple levels; Troynikov points to LlamaIndex’s support for this approach.
  • Multiple retrieval signals: Bring multiple data sources and signals into reranking rather than relying on one similarity value.
  • Embedding continuity: Embed successive sliding windows of a document and look for discontinuities in the resulting sequence of vectors.

The last approach asks where the representation changes abruptly; the token-prediction approach asks where the text becomes unexpectedly hard to predict. Both boundary-detection ideas are described as experiments, without an effectiveness result in the talk.

6:577:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:57 · section reference included

Five bird pages cannot answer every fish question

Imagine a corpus containing every English-language Wikipedia page about birds. For each query, the application retrieves the five nearest neighbors and puts them into the model’s context. Then a user asks about fish. In Troynikov’s hypothetical, the system still returns five neighbors, but none is relevant to the query. Being nearest is a relative property; being useful to the task is a separate judgment.

How can the application make that judgment?

  • Human feedback: Gather explicit judgments about whether retrieved material was relevant.
  • Auxiliary reranking: Use another model and additional signals, including what the user was looking at and what they previously found useful.
  • Scoped retrieval: When useful constraints are known beforehand, narrow the search through keywords or metadata filters. Troynikov describes these as supported by Chroma at the time; that statement does not establish a ranked lexical or hybrid-search implementation.

These approaches introduce evidence beyond the fact that a document occupies a position in the nearest-neighbor list.

The more ambitious proposal is to calculate conditional relevance directly. Given the available dataset and what the user is trying to accomplish, a relevance model would distinguish usefulness per user, per task, per generation model, and per individual task instance. That requires understanding both the query’s semantics and the content available in the corpus. Troynikov identifies lightweight open-source language models operating at the data layer as a promising experimental direction, not a demonstrated relevance guarantee.

Slide titled “Is the retrieval result relevant?” lists re-ranking, human feedback, and augmented retrieval with keywords and metadata filtering, followed by “an algorithmic approach?” and “watch this space.”Open full source frame
Approaches to retrieval relevance include reranking, human feedback, keywords, and metadata filtering.
8:519:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:51 · section reference included

From vector storage to an AI data layer

The closing roadmap moves from retrieval research to infrastructure. Chroma’s next engineering step was a horizontally scalable cluster version, extending the existing single-node system across multiple nodes. Troynikov announced a December target for a Chroma Cloud database-as-a-service technical preview and a January target for hybrid deployments in enterprise clusters. These are targets from the 2023 recording, not statements of current availability.

The planned data layer also extended beyond text. Troynikov anticipated the arrival of the GPT Vision API, possibly at OpenAI’s developer day, and image and voice capabilities in Gemini. Those contemporaneous expectations motivated multimodal retrieval support: if a model can consume more kinds of data, its retrieval system must be able to supply them.

Multimodal data expands the definition of a useful result. Semantic relevance remains important, but qualities such as aesthetic suitability can also matter. A retrieval system may therefore need to evaluate several dimensions of usefulness rather than treating text similarity as the entire problem. Model selection appears as another item of work in progress, without implementation details.

The intended developer experience is analogous to using Postgres in a web application: the database handles the data layer so that developers can concentrate on application logic. Chroma’s ambition is to provide that boundary for AI applications, absorbing the work of storing, updating, and selecting useful information while leaving developers responsible for making the application behave correctly.

10:4510:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:45 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…