AI Engineer World's Fair 2025
The State of AI-Powered Search and Retrieval
Loading slides…
Read the talk
The State of AI-Powered Search and Retrieval
A shopping query like “My best friend is sick” should find get-well baskets. Making that work takes more than matching words: it takes retrieval that understands intent and respects constraints.
From a talk by Frank Liu
Before you start: Familiarity with LLMs and the idea of representing text as vectors will help; the article explains how those vectors fit into retrieval.
From matching words to understanding intent
Frank Liu approaches search from the embedding and reranking layer. His team at Voyage AI had joined MongoDB roughly three to four months before this talk. Voyage positioned its models around accuracy and cost for retrieval-augmented generation (RAG) and semantic search, with applications extending to classification and clustering. Liu listed the Voyage AI API, Azure, and AWS Marketplace as access points. Here, he moves beyond individual model evaluations to examine three application lessons and the capabilities search systems may need next.
AI-powered search finds related concepts even when the wording differs. Traditional methods such as TF-IDF and BM25 provide useful keyword retrieval, but conceptual relevance broadens what a system can find. The goal is not to discard keywords; it is to retrieve useful material when the query and the answer do not share them.
Consider the shopping query “My best friend is sick.” A useful result might be a get-well basket, even though the query never names that product. Merely matching “best friend” could produce generic gifts and miss the reason for buying one. Search needs to connect the situation to the user's likely intent. Liu adds limited reasoning and instruction following to this definition, returning to them later as directions for retrieval models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieval gives generation something to work with
RAG is the familiar application of this idea. Without relevant retrieved information, an LLM may hallucinate, refuse to answer, or produce a generic response. Retrieval supplies material from which it can construct a more grounded answer. Liu credits the RAG slides to Tengyu's talk and describes a short pipeline:
- Generate embeddings for the material to be searched.
- Search for information relevant to the prompt.
- Pass the retrieved information to the LLM as context for its response.
Embedding quality matters because it helps determine which information reaches the model. Liu estimates that 95–99% of the search systems he has encountered use embeddings in some form. That is an observation about his experience, with no sample size or survey method supplied. Text from PDFs, Word documents, Google Drive files, and PowerPoints can be represented in a shared vector space so that a query brings relevant documents toward the top of the results. A generator cannot make good use of a document that retrieval never supplies.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose models for the codebase, not the leaderboard
Chatting with a codebase makes the importance of domain fit concrete. Continue, whose code is largely open source, uses a RAG pipeline with reranking. Retrieval supplies candidate code and documentation; reranking helps order that material before it becomes context for an answer.
There is no single best embedding model or LLM for every application. Liu reports that Continue's evaluation found voyage-code-3 performed best for its application. He gives no numerical scores, dataset, or comparison protocol. The relevant requirement is clear, however: codebase chat depends on understanding code, documentation, and developer context. Evaluate model choices against those demands rather than assuming that a general-purpose result predicts performance on your repository.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Semantic relevance still needs structured constraints
In the second application, Liu draws attention to a blue box containing filtering and other structured inputs to search. The lesson is that embeddings alone are often insufficient. A document can be semantically relevant while failing a condition that determines whether the application may use it.
Take legal documents embedded with voyage-law-2. A query might concern the right legal topic but require documents from a particular state, only official documents, or documents containing a specific set of details. These requirements belong in a filtering stage. Liu says filtering typically happens inside the vector store, although some systems apply it afterward.
A small TypeScript example makes the distinction explicit. Suppose a semantic search has returned these three records, and the application requires official California documents:
typescript
type SearchHit = {
id: string;
state: string;
official: boolean;
score: number;
};
const hits: SearchHit[] = [
{ id: "doc-a", state: "NY", official: true, score: 0.94 },
{ id: "doc-b", state: "CA", official: false, score: 0.91 },
{ id: "doc-c", state: "CA", official: true, score: 0.87 },
];
const eligible = hits.filter(
(hit) => hit.state === "CA" && hit.official
);
console.log(eligible.map((hit) => hit.id)); // ["doc-c"]
The illustrative scores rank semantic matches; the metadata determines eligibility. The highest-scoring result does not satisfy the state requirement, and the next result is unofficial. Only doc-c survives. This shows the post-retrieval variant of Liu's filtering stage: structured data adds constraints that similarity scores do not enforce.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An agent can search, reconsider, and search again
The third application changes retrieval from a single input-output operation into a feedback loop. An LLM can issue a search, inspect what comes back, and then expand or decompose the query. Retrieval becomes part of an ongoing reasoning process rather than a one-time preparation step.
Liu illustrates decomposition with an X/Twitter earnings request. His spoken example moves from Q4 to full-year 2024; the full-year interpretation explains the resulting plan: query Q1, Q2, Q3, and Q4 separately, sending the four searches to vector stores or databases. The example describes how to break up a request, not a retrieved set of financial results. Each subquery creates an opportunity to gather a distinct part of the information needed for the larger answer.
Calling 2025—and possibly 2026—the era of agents, Liu emphasizes another requirement: search must work well with conversational data. The diagram connects a human and an agent to both a knowledge vectorstore and a conversation vectorstore, with retrieval and return paths. It separates stored knowledge from conversation history while allowing both to inform the interaction. Liu notes that the diagram omits implementation details; its useful architectural point is that future searches can depend on earlier exchanges and retrieved results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Text and images in the same search space
The first future direction is multimodality. Liu expects it to become foundational to search, just as it is becoming foundational to language models. He means joint understanding of text and images, or eventually text, images, and audio—not embodied combinations of sight, touch, and taste. The practical question is whether information in different media can participate in the same retrieval system.
voyage-multimodal-3 provides the concrete text-and-image example. It accepts text, images, or interleaved text and images, embedding them into one semantic space. Audio belongs to Liu's broader vision, not the model inputs described here. In the slide example, the query “strong LLMs” has a Claude 3.5 blog post as its nearest document. Although Liu points out that arrows are missing from the displayed diagram, the intended relationship is straightforward: a text query can retrieve a relevant document represented through multimodal content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tell the embedding what to emphasize
The next direction is instruction tuning, with reasoning as a secondary possibility. The baseline embedding interface takes a query or document and returns a vector. Liu envisions adding an instruction alongside that input to steer what the vector emphasizes. His forecast concerns richer steering and broader use of instruction-aware retrieval.
For example, a user might ask for documents that discuss one particular aspect of a topic in detail. Topic similarity alone can retrieve broad overviews; an additional instruction expresses the desired focus and depth. That differs from the earlier state or official-status filter: it changes the semantic preference the system should represent rather than testing a known metadata value. Liu presents this as a direction for search, without supplying a concrete reasoning implementation or a guarantee that an instruction will enforce every constraint.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring the retrieval pipeline into the database
The final direction is what Liu calls an agent-native database, acknowledging the term's buzzword quality. Even a simplified retrieval architecture requires several components to work together. Voyage joining MongoDB makes the prospect of integrating those components particularly relevant to his team.
The proposed shift is from assembling a pipeline to using one data platform that performs its core retrieval operations:
| Operation | Separate components | Proposed integrated platform |
|---|---|---|
| Embedding | Call an embedding model | Platform generates embeddings |
| Vector search | Query a vector database | Platform retrieves candidates |
| Reranking | Call a reranker | Platform reranks results |
| Query transformation | Coordinate additional logic | Possibly augment or decompose queries |
The slide contrasts an embedding model, vector database, and reranker with a single MongoDB + Voyage AI box between query and results. Embedding and reranking are central to the proposal; query augmentation and decomposition are possibilities Liu explicitly leaves open.
For this June 2025 talk, the integrated platform is a forecast, not a deployment recipe. Liu expects more consolidation during that year and the next. The substantive promise is to make the database responsible for more of the retrieval work—representing information, finding candidates, and ordering results—so application builders have fewer separate components to coordinate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Explains codebase indexing and shows how to configure Voyage and local embedding providers in Continue.
Describes code retrieval training, evaluation, variable embedding dimensions, and quantized storage formats.
Introduces the legal embedding model and reports evaluations across legal and long-document retrieval datasets.
Explains shared representations for text, screenshots, and interleaved images, with multimodal retrieval evaluations.
Further reading
Research on generating task- and domain-specific embeddings by supplying instructions alongside text.
Updates since the talk
Announces Atlas embedding and reranking API availability, the Voyage 4 family, and video retrieval with Multimodal 3.5.
Full transcript
Loading transcript…