AI Engineer World's Fair 2024

RAG and the MongoDB Document Model

Ben Flast· Director of Product, MongoDB13:13

Loading slides…

Read the talk

RAG and the MongoDB Document Model

A useful AI application needs more than similar documents: MongoDB combines vector retrieval with the operational records, conversation history, and cache entries that supply application context.

From a talk by Ben Flast

Before you start: Familiarity with JSON objects, database queries, and LLM prompts is helpful; the article introduces the retrieval and vector-search concepts it uses.

Why an LLM cannot know your bank balance

Ask an unconnected LLM how much money is in your bank account, and it cannot answer. The missing ingredient is access to your private data. Ben Flast opens with this example because it exposes a basic application boundary: a model can generate language without possessing the information needed to make that language useful to a particular user.

Slide showing the prompt “How much money is in my bank account?” and response “Uh what?” on a dark green background.Open full source frame
A bank-account question receives the response “Uh what?”

Retrieval-augmented generation supplies context at prompting time. A model has a training cutoff and lacks your private, personalized information. An application can retrieve company records, product information, or order history and include that material in its prompt. The aim is more relevant, consistent, and accurate answers; supplying context is not itself a guarantee of correctness. MongoDB’s role in this architecture is to make data already powering an application available to its AI features.

1:211:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:21 · section reference included

The standard RAG request path

The baseline architecture turns a user’s question into a retrieval request before asking the LLM to answer:

  1. Receive the user’s question.
  2. Send it to an embedding model, which produces a vector representation.
  3. Use that vector to search MongoDB Atlas Vector Search for similar documents.
  4. Combine the retrieved documents with the original question in the LLM’s input.
  5. Return the generated answer to the user.

This is the familiar chatbot or copilot pattern: semantic retrieval selects context, and generation turns that context into a response.

2:472:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:47 · section reference included

Documents hold richer application context

More capable applications need context beyond a set of semantically similar passages. The MongoDB document model provides a way to keep richer application objects together. Developers work with JSON-like documents, stored internally as BSON, or binary JSON, containing fields such as a name and a nested profile.

Flast compares this with reconstructing a customer or contact object from relational tables. With a document, the application can store and retrieve the object in a form closer to the one it already uses.

RepresentationHow the application obtains an object
Multiple relational tablesAssemble fields from related records
A document containing the objectRetrieve the stored object structure

The benefit in this comparison is less reconstruction between the application’s data structures and its database representation.

Documents can represent tabular, key-value, geospatial, and graph-shaped data as well as conventional JSON objects. Flast associates this flexibility with greater efficiency and developer productivity, and points to sharding as MongoDB’s mechanism for horizontal scaling. These are architectural benefits he argues for, rather than measured comparisons presented in the talk.

“Documents are universal” slide listing JSON documents, tabular, key-value, text, geospatial, graph, and time series data beside More Efficient, More Productive, and More Scalable columns.Open full source frame
Documents support multiple data models, with efficiency, productivity, and scalability benefits.
3:263:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:26 · section reference included

Add embeddings to existing documents

Vector retrieval adds another access path to those documents. Atlas uses HNSW indexes to support approximate nearest-neighbor search over stored vectors. In Flast’s example, a document already contains symbol, quarter, and content; adding content_embedding puts the vector alongside the data it describes.

The embedding can represent the whole document, a selected part of it, or external data mapped back to that record. At the time of the talk, Flast states a maximum of 4,096 dimensions per vector. That historical limit has since increased: the linked current Vector Search overview lists 8,192 dimensions. The mechanism remains the same—store the embedding as a field and index that field.

The vector index definition identifies the field’s type and path, its dimension count, and the similarity function used to compare vectors. For example, an embedding model producing 1,536 values could use this definition for the content_embedding field:

json

{
  "fields": [
    {
      "type": "vector",
      "path": "content_embedding",
      "numDimensions": 1536,
      "similarity": "cosine"
    }
  ]
}

The dimension count must match the embedding’s length. The similarity function determines how the index judges closeness between the query vector and stored vectors.

5:065:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:06 · section reference included

Query, tune, and filter retrieval

Atlas builds the vector index and maintains it as database records change. That removes the need to manage a separate index-update path in the application, but should not be read as a promise that every write is immediately visible to search. The $vectorSearch aggregation stage names the index, supplies the query vector, and identifies the document field to search.

Two parameters control different parts of retrieval:

ParameterRole
limitMaximum number of returned results
numCandidatesNumber of nearest-neighbor candidates considered during search

Increasing candidate exploration gives approximate search more opportunity to find relevant neighbors. Flast describes this using HNSW graph entry points; the documented query terminology is candidates considered, rather than a literal count of graph entry points.

A pre-filter restricts which documents are eligible during retrieval. This differs from retrieving a final set of neighbors and then discarding unwanted results. The displayed query uses numCandidates: 100, limit: 10, and a filter requiring year to be greater than 1995.

The same structure can be expressed as a JavaScript function, with the query vector supplied by the application’s embedding step:

javascript

function vectorSearchPipeline(indexName, queryVector) {
  return [
    {
      $vectorSearch: {
        index: indexName,
        queryVector,
        path: "content_embedding",
        numCandidates: 100,
        limit: 10,
        filter: { year: { $gt: 1995 } }
      }
    }
  ];
}

Here content_embedding retains the field used in the preceding document example; the filter adds a structured eligibility condition to semantic retrieval.

Sample Query slide showing $vectorSearch with index, queryVector, path, numCandidates set to 100, limit set to 10, and a year filter greater than 1995.Open full source frame
A sample vector search query includes candidate count, result limit, and a year filter.
6:186:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:18 · section reference included

Scale search separately from transactions

A unified database interface does not require every workload to share the same resource allocation. Flast’s transactional topology has a primary and two secondaries, providing the replication structure needed for durability and availability. Search can put a different kind of pressure on infrastructure than transactional reads and writes.

Search Nodes separate search capacity from transactional capacity. They hold the vector indexes and can scale independently of the infrastructure storing transactional data. An application keeps a consistent document model and query interface while allocating resources according to each workload’s needs. The separation is in deployment and scaling, without requiring the application to abandon its shared data model.

7:107:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:10 · section reference included

Framework integrations beyond a vector store

The integration ecosystem connects these database capabilities to application frameworks. Flast names LlamaIndex, LangChain, Microsoft Semantic Kernel, AWS Bedrock, and Haystack among MongoDB’s AI integrations. Their supported primitives differ; the list does not imply feature parity.

For example, the LangChain integration includes both a vector-store abstraction and chat message history. These give an application different ways to use the same database: semantic retrieval supplies relevant material, while stored operational data supplies additional context for the prompt. This is the extension beyond the baseline RAG loop that the following examples illustrate.

8:168:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:16 · section reference included

Reuse answers with a semantic cache

A semantic cache can avoid a new model call when an existing answer matches the meaning of a request. MongoDB can serve as the cache backend for LangChain. In the flow shown here, the retriever first gathers additional context and combines it with the user’s prompt. That augmented prompt then goes to the cache for a semantic-similarity check.

The lookup creates two paths:

  • Cache hit: Fetch the cached answer and return it without calling the LLM again.
  • Cache miss: Send the prompt to the LLM and return its generated answer.

The saving comes from bypassing generation on a hit. Retrieval and caching can use the same MongoDB database, with LangChain connecting the pieces. Flast supplies no measured hit rate or quantified resource savings.

9:119:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:11 · section reference included

Persist conversation history for continuity

Conversation history solves a different problem: keeping a later request connected to what the user and assistant already discussed. LangChain’s chat message history abstraction can persist those messages in MongoDB, supporting the continuing conversation experience familiar from ChatGPT.

On a subsequent request, the application fetches prior messages, includes them alongside context from vector search, and sends the assembled prompt to the LLM. The returned answer continues that conversation. Semantic retrieval finds relevant data; message history preserves the interaction that gives the current question its meaning.

10:0610:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:06 · section reference included

4149 combines retrieval with operational memory

Flast presents 4149 as a customer example of an AI teammate beyond coding. In his account, the system listens to meetings, tracks activity, retrieves additional information, and surfaces context that could help a user complete a task, write an email, or schedule a project. This is a reported application use case, not a live demonstration of autonomous task completion.

The database holds more than embeddings. User records, meeting information, and conversation history sit alongside the data used for semantic search. That combination lets an application draw on both similarity-based retrieval and ordinary operational records. Flast’s broader point about agents follows from this combination: a transactional database gives agents more ways to store and interact with application state than vector retrieval alone.

10:4611:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:46 · section reference included

Deploying the application on Atlas

The closing layer is operational. Atlas supplies security and privacy controls around the database. Flast also emphasizes availability and performance automation, without presenting an SLA or performance benchmark.

Flast states that Atlas is deployable in more than 100 regions across major cloud providers. He also highlights Search Nodes as a way to optimize resource placement, but Atlas’s overall regional footprint should not be treated as identical to dedicated Search Node availability; the deployment documentation specifies supported regions for that offering.

For an entry point, Flast advertises an Atlas free tier that includes vector search. The path through the talk builds from that initial retrieval capability: keep embeddings with application documents, add the context needed for useful responses, and separate search resources from transactional resources as the workload grows.

12:0412:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:04 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…