AI Engineer World's Fair 2024
RAG and the MongoDB Document Model
Loading slides…
Read the talk
RAG and the MongoDB Document Model
A useful AI application needs more than similar documents: MongoDB combines vector retrieval with the operational records, conversation history, and cache entries that supply application context.
From a talk by Ben Flast
Before you start: Familiarity with JSON objects, database queries, and LLM prompts is helpful; the article introduces the retrieval and vector-search concepts it uses.
Why an LLM cannot know your bank balance
Ask an unconnected LLM how much money is in your bank account, and it cannot answer. The missing ingredient is access to your private data. Ben Flast opens with this example because it exposes a basic application boundary: a model can generate language without possessing the information needed to make that language useful to a particular user.
Retrieval-augmented generation supplies context at prompting time. A model has a training cutoff and lacks your private, personalized information. An application can retrieve company records, product information, or order history and include that material in its prompt. The aim is more relevant, consistent, and accurate answers; supplying context is not itself a guarantee of correctness. MongoDB’s role in this architecture is to make data already powering an application available to its AI features.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The standard RAG request path
The baseline architecture turns a user’s question into a retrieval request before asking the LLM to answer:
- Receive the user’s question.
- Send it to an embedding model, which produces a vector representation.
- Use that vector to search MongoDB Atlas Vector Search for similar documents.
- Combine the retrieved documents with the original question in the LLM’s input.
- Return the generated answer to the user.
This is the familiar chatbot or copilot pattern: semantic retrieval selects context, and generation turns that context into a response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Documents hold richer application context
More capable applications need context beyond a set of semantically similar passages. The MongoDB document model provides a way to keep richer application objects together. Developers work with JSON-like documents, stored internally as BSON, or binary JSON, containing fields such as a name and a nested profile.
Flast compares this with reconstructing a customer or contact object from relational tables. With a document, the application can store and retrieve the object in a form closer to the one it already uses.
| Representation | How the application obtains an object |
|---|---|
| Multiple relational tables | Assemble fields from related records |
| A document containing the object | Retrieve the stored object structure |
The benefit in this comparison is less reconstruction between the application’s data structures and its database representation.
Documents can represent tabular, key-value, geospatial, and graph-shaped data as well as conventional JSON objects. Flast associates this flexibility with greater efficiency and developer productivity, and points to sharding as MongoDB’s mechanism for horizontal scaling. These are architectural benefits he argues for, rather than measured comparisons presented in the talk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add embeddings to existing documents
Vector retrieval adds another access path to those documents. Atlas uses HNSW indexes to support approximate nearest-neighbor search over stored vectors. In Flast’s example, a document already contains symbol, quarter, and content; adding content_embedding puts the vector alongside the data it describes.
The embedding can represent the whole document, a selected part of it, or external data mapped back to that record. At the time of the talk, Flast states a maximum of 4,096 dimensions per vector. That historical limit has since increased: the linked current Vector Search overview lists 8,192 dimensions. The mechanism remains the same—store the embedding as a field and index that field.
The vector index definition identifies the field’s type and path, its dimension count, and the similarity function used to compare vectors. For example, an embedding model producing 1,536 values could use this definition for the content_embedding field:
json
{
"fields": [
{
"type": "vector",
"path": "content_embedding",
"numDimensions": 1536,
"similarity": "cosine"
}
]
}
The dimension count must match the embedding’s length. The similarity function determines how the index judges closeness between the query vector and stored vectors.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Query, tune, and filter retrieval
Atlas builds the vector index and maintains it as database records change. That removes the need to manage a separate index-update path in the application, but should not be read as a promise that every write is immediately visible to search. The $vectorSearch aggregation stage names the index, supplies the query vector, and identifies the document field to search.
Two parameters control different parts of retrieval:
| Parameter | Role |
|---|---|
limit | Maximum number of returned results |
numCandidates | Number of nearest-neighbor candidates considered during search |
Increasing candidate exploration gives approximate search more opportunity to find relevant neighbors. Flast describes this using HNSW graph entry points; the documented query terminology is candidates considered, rather than a literal count of graph entry points.
A pre-filter restricts which documents are eligible during retrieval. This differs from retrieving a final set of neighbors and then discarding unwanted results. The displayed query uses numCandidates: 100, limit: 10, and a filter requiring year to be greater than 1995.
The same structure can be expressed as a JavaScript function, with the query vector supplied by the application’s embedding step:
javascript
function vectorSearchPipeline(indexName, queryVector) {
return [
{
$vectorSearch: {
index: indexName,
queryVector,
path: "content_embedding",
numCandidates: 100,
limit: 10,
filter: { year: { $gt: 1995 } }
}
}
];
}
Here content_embedding retains the field used in the preceding document example; the filter adds a structured eligibility condition to semantic retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Scale search separately from transactions
A unified database interface does not require every workload to share the same resource allocation. Flast’s transactional topology has a primary and two secondaries, providing the replication structure needed for durability and availability. Search can put a different kind of pressure on infrastructure than transactional reads and writes.
Search Nodes separate search capacity from transactional capacity. They hold the vector indexes and can scale independently of the infrastructure storing transactional data. An application keeps a consistent document model and query interface while allocating resources according to each workload’s needs. The separation is in deployment and scaling, without requiring the application to abandon its shared data model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Framework integrations beyond a vector store
The integration ecosystem connects these database capabilities to application frameworks. Flast names LlamaIndex, LangChain, Microsoft Semantic Kernel, AWS Bedrock, and Haystack among MongoDB’s AI integrations. Their supported primitives differ; the list does not imply feature parity.
For example, the LangChain integration includes both a vector-store abstraction and chat message history. These give an application different ways to use the same database: semantic retrieval supplies relevant material, while stored operational data supplies additional context for the prompt. This is the extension beyond the baseline RAG loop that the following examples illustrate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse answers with a semantic cache
A semantic cache can avoid a new model call when an existing answer matches the meaning of a request. MongoDB can serve as the cache backend for LangChain. In the flow shown here, the retriever first gathers additional context and combines it with the user’s prompt. That augmented prompt then goes to the cache for a semantic-similarity check.
The lookup creates two paths:
- Cache hit: Fetch the cached answer and return it without calling the LLM again.
- Cache miss: Send the prompt to the LLM and return its generated answer.
The saving comes from bypassing generation on a hit. Retrieval and caching can use the same MongoDB database, with LangChain connecting the pieces. Flast supplies no measured hit rate or quantified resource savings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Persist conversation history for continuity
Conversation history solves a different problem: keeping a later request connected to what the user and assistant already discussed. LangChain’s chat message history abstraction can persist those messages in MongoDB, supporting the continuing conversation experience familiar from ChatGPT.
On a subsequent request, the application fetches prior messages, includes them alongside context from vector search, and sends the assembled prompt to the LLM. The returned answer continues that conversation. Semantic retrieval finds relevant data; message history preserves the interaction that gives the current question its meaning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
4149 combines retrieval with operational memory
Flast presents 4149 as a customer example of an AI teammate beyond coding. In his account, the system listens to meetings, tracks activity, retrieves additional information, and surfaces context that could help a user complete a task, write an email, or schedule a project. This is a reported application use case, not a live demonstration of autonomous task completion.
The database holds more than embeddings. User records, meeting information, and conversation history sit alongside the data used for semantic search. That combination lets an application draw on both similarity-based retrieval and ordinary operational records. Flast’s broader point about agents follows from this combination: a transactional database gives agents more ways to store and interact with application state than vector retrieval alone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deploying the application on Atlas
The closing layer is operational. Atlas supplies security and privacy controls around the database. Flast also emphasizes availability and performance automation, without presenting an SLA or performance benchmark.
Flast states that Atlas is deployable in more than 100 regions across major cloud providers. He also highlights Search Nodes as a way to optimize resource placement, but Atlas’s overall regional footprint should not be treated as identical to dedicated Search Node availability; the deployment documentation specifies supported regions for that offering.
For an entry point, Flast advertises an Atlas free tier that includes vector search. The path through the talk builds from that initial retrieval capability: keep embeddings with application documents, add the context needed for useful responses, and separate search resources from transactional resources as the workload grows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Reference for MongoDB documents and their field-and-value structure.
Further reading
Explains the package's vector store, conversation-history and semantic-cache components and links to examples.
MongoDB's account of AI assistants, including how 4149.AI combines summaries, conversation history and embeddings.
Updates since the talk
Current overview of embedding storage and vector retrieval, including today's dimensionality limit.
Deployment guidance for separating search resources from database nodes, with tier and regional requirements.
Implementation examples for vector retrieval, semantic caching and persistent chat history.
Full transcript
Loading transcript…