AI Engineer World's Fair 2024

RAG at scale: production-ready GenAI apps with Azure AI Search

Loading slides…

Read the talk

RAG at scale: production-ready GenAI apps with Azure AI Search

Production RAG needs more than vector similarity: hybrid retrieval, careful candidate budgets, stronger ranking, compact indexes, and ingestion that keeps pace with changing data.

From a talk by Pablo Castro

Before you start: Basic familiarity with language models, embeddings, and Python will help; the article introduces the retrieval and ranking concepts as they appear.

Give the model access to facts it never learned

How do you get a language model to work with your application data, company documents, or information about your users—facts it never saw during training? Pablo Castro, from Microsoft’s Azure AI team, starts by separating three approaches that address different needs.

ApproachUseful when you need to…
Prompt engineeringSupply instructions and context directly, taking advantage of longer context windows.
Fine-tuningTeach recurring patterns or domain-specific jargon.
Retrieval-augmented generationAnswer using an external collection of facts.

Prompt engineering can go a long way. When the problem is access to a changing body of knowledge, however, RAG separates the model’s reasoning capabilities from the system that stores that knowledge.

The basic workflow is straightforward:

  1. An orchestrator receives a task, such as the next question in a chat.
  2. It searches a knowledge base for candidate pieces of evidence.
  3. It sends instructions, the question, and those candidates to the language model.
  4. The model produces an answer grounded in the retrieved material.

Real applications usually need multiple attempts, tuning, and more elaborate orchestration. But this division of responsibility remains: retrieval determines which evidence the model gets to reason over.

RAG recap slide showing an app orchestrator connected to a retrieval system and an LLM, with files and databases as data sources.Open full source frame
Retrieval-augmented generation combines reasoning with external knowledge.
0:161:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

A successful prototype creates several scaling problems

Castro describes 2023 as a year of prototypes and 2024 as the move into production. A convincing demo establishes that an interaction is useful; production adds simultaneous users who want more data and faster answers. Success creates pressure to put the organization’s entire collection into the system. Data volume grows, updates arrive more frequently, and query traffic increases.

The initial workflow—take a question, search, call the model—also tends to expand. A single user request may require several retrieval calls and several model calls, multiplying the work performed by both systems. Meanwhile, adding more organizational knowledge brings different file types and data sources. Scaling therefore means handling volume, change rate, query load, workflow complexity, and data diversity together.

RAG architecture diagram with a retrieval-system callout and bullets for data volume and diversity, sophisticated workflows, and usage.Open full source frame
Retrieval pressure points include data volume, data change rate, query load, and retrieval quality.
3:063:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:06 · section reference included

Treat vector search as part of a retrieval platform

Azure AI Search is designed to cover the broader retrieval problem rather than require applications to assemble every search component themselves. Vector search sits alongside Microsoft’s other retrieval capabilities, Azure data integration, and platform security and compliance features. The goal is to make these pieces work together as one service.

Vectors are useful because they capture soft conceptual similarity: a query can find related content without sharing its exact wording. Production applications also need control over how those matches are found and constrained.

  • Approximate and exhaustive search: Fast approximate nearest-neighbor search serves retrieval workloads; exhaustive search provides precise results for baselines and recall evaluation.
  • Filters and projections: Metadata conditions restrict eligible documents, while projections select the fields returned to the application.
  • Multiple vectors: A document can carry vectors for different content sections or data types, potentially using different embeddings. A query can also contain multiple vectors.

These capabilities let an application combine database-style selection with similarity search.

5:045:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:04 · section reference included

Build a small index, then add text and filters

The notebook demonstration starts with an already provisioned search service. Castro connects using a default Azure credential and creates an index containing categorical metadata, text, and a vector field. The vectors have just three dimensions so the example is easy to inspect; practical embeddings commonly have hundreds or thousands. He selects HNSW, a graph-based approximate nearest-neighbor algorithm, as the indexing strategy.

This Python example expresses the same index structure, using current SDK profile-style configuration rather than reproducing the historical notebook cell:

python

import os

from azure.identity import DefaultAzureCredential
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    HnswAlgorithmConfiguration,
    SearchField,
    SearchFieldDataType,
    SearchIndex,
    SearchableField,
    SimpleField,
    VectorSearch,
    VectorSearchProfile,
)

endpoint = os.environ["AZURE_SEARCH_ENDPOINT"]
credential = DefaultAzureCredential()
index_name = "toy-retrieval"

index = SearchIndex(
    name=index_name,
    fields=[
        SimpleField(name="id", type=SearchFieldDataType.String, key=True),
        SimpleField(
            name="category",
            type=SearchFieldDataType.String,
            filterable=True,
        ),
        SearchableField(name="text", type=SearchFieldDataType.String),
        SearchField(
            name="vector",
            type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
            searchable=True,
            vector_search_dimensions=3,
            vector_search_profile_name="toy-profile",
        ),
    ],
    vector_search=VectorSearch(
        algorithms=[HnswAlgorithmConfiguration(name="toy-hnsw")],
        profiles=[VectorSearchProfile(
            name="toy-profile",
            algorithm_configuration_name="toy-hnsw",
        )],
    ),
)

SearchIndexClient(endpoint, credential).create_index(index)

The categorical field must be filterable to support the metadata restriction added later. The vector field’s dimensions and search profile establish how its values will be indexed.

Once the index exists, Castro obtains a document client and explicitly pushes records containing vectors, text, and categories. Managed ingestion is another option, but direct uploads make the mechanics visible. He first searches with a vector and requests its two nearest candidates. Next, he adds the text hello: the system now retrieves through both the text and vector paths, then fuses and ranks their results. Finally, he restricts the query to category A, excluding records in category B.

Continuing the Python example, these teaching records make the same progression explicit:

python

from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery

client = SearchClient(endpoint, index_name, credential)
client.upload_documents([
    {"id": "1", "category": "A", "text": "hello world",
     "vector": [1.0, 0.0, 0.0]},
    {"id": "2", "category": "A", "text": "nearby concept",
     "vector": [0.9, 0.1, 0.0]},
    {"id": "3", "category": "B", "text": "hello again",
     "vector": [0.0, 1.0, 0.0]},
])

query = VectorizedQuery(
    vector=[1.0, 0.0, 0.0],
    fields="vector",
    k_nearest_neighbors=2,
)

vector_results = list(client.search(vector_queries=[query]))
hybrid_results = list(client.search(
    search_text="hello",
    vector_queries=[query],
))
filtered_results = list(client.search(
    search_text="hello",
    vector_queries=[query],
    filter="category eq 'A'",
    select=["id", "category", "text"],
))

Filters can also combine conditions with AND, OR, and ranges. Castro says filtering remains fast with hundreds of millions of documents, though he provides no latency measurements or workload configuration for that claim.

An audience member notices a detail in the live demonstration: why did a query with k = 2 return three results? Because k controls the vector branch’s candidate budget, not the size of the entire hybrid response. Two candidates came from vector retrieval; keyword retrieval contributed additional candidates; fusion selected three results.

Candidate count and returned-result count are separate controls. An application can ask vector retrieval for a larger pool, combine that pool with keyword matches, and still return only the top N results. In the Python call, k_nearest_neighbors controls the vector candidate count; adding top=2 to client.search(...) limits the final response instead.

7:337:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:33 · section reference included

Retrieve broadly, then rank a small candidate set

An answer can only use evidence the retrieval system actually finds. That makes retrieval quality a central production concern. Azure AI Search addresses it with two stages: a recall-oriented first stage gathers candidates using vectors and keywords, then a stronger second-stage model reranks them.

Consider Castro’s illustrative corpus of a hundred million vectors. A fast retriever can reduce that corpus to a small candidate set. Once the set is small, running a more sophisticated ranking model becomes affordable. The expensive model does not need to evaluate every document in the index.

The comparison progresses through four configurations:

ConfigurationRetrieval and ranking
KeywordsBM25 text scoring
VectorsOpenAI Ada embeddings
HybridFused keyword and vector results
Hybrid plus rerankingFusion followed by a stronger ranking model

Castro reports better retrieval results across the displayed evaluations when reranking is enabled. The related Microsoft hybrid retrieval evaluation measures ranking quality, not generated-answer accuracy: it uses NDCG@3 for customer datasets and NDCG@10 for academic datasets. The useful distinction is between finding plausible candidates and ordering those candidates well enough to supply the right evidence to the model.

Semantic reranking is not another cosine-similarity calculation. A bi-encoder encodes the document and query independently. That enables fast comparisons between their vectors, but the model never examines the query and document together. A cross-encoder takes both as input and predicts how well the document corresponds to the query.

Model patternWhat it evaluatesSuitable scope
Bi-encoderSeparately computed representationsLarge-scale candidate retrieval
Cross-encoderQuery and document togetherReranking a small candidate set

The cross-encoder performs inference at query time for the candidate pairs. Applying it to an entire large corpus would be impractical for an interactive application, which is why the fast first stage matters.

Castro puts Azure’s reranker latency at roughly 100 milliseconds, depending on the cross-encoder, as the service’s chosen speed–quality trade-off. He describes that overhead as largely hidden by the longer LLM call in the overall response time; he does not supply a percentile, concurrency level, or benchmark setup.

10:5411:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:54 · section reference included

Scope the search and preserve the original text

Before investing in more sophisticated ranking, use any known metadata that can narrow the relevant dataset. Discrete metadata gives the system a direct way to exclude irrelevant material. Apply the ranking machinery to that scoped set. The category filter in the notebook is a small example of this broader quality technique.

The next audience question concerns the keyword side of hybrid search. What input does it need? Castro returns to the notebook: the request can supply text and a vector, or it can supply text that the service also vectorizes. A vector alone does not recover the original keywords. Preserving the text gives the system an input for lexical retrieval as well as one from which it can obtain an embedding.

VS Code notebook showing a search cell with text and vector inputs, a category filter, and output rows for document IDs 1 and 2 in category A.Open full source frame
The notebook combines text, a category filter, and a vector query, with two results displayed below.

The audience member’s deeper question is how a chatbot should derive useful search terms dynamically from an arbitrary user prompt or conversation. That is a separate orchestration problem from accepting text and vector inputs. Castro mentions extracting a candidate search from the user’s input before calling the answer-generating LLM, but the exchange ends with the details deferred. No keyword-extraction algorithm or conversation-rewriting procedure is demonstrated.

14:0114:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:01 · section reference included

Larger collections change the capacity requirement

The next constraint is how much content fits in an index. As applications moved from tiny experimental collections to organizational datasets, Azure increased its service limits. Castro reports a historical increase of 10–12 times the vector density for most services on the same SKUs, without changing prices. Microsoft’s May 2024 announcement corroborates an increase of up to 12 times at no additional cost; the change was not uniform across every deployment.

Castro says the new limits enable multi-billion-vector applications through service provisioning and data upload. He contrasts this with the previous year, when a billion-vector dataset was primarily a benchmark curiosity. The talk does not specify vector dimensions, tier, partition layout, or performance conditions for that scale.

OpenAI provides his customer example. Castro identifies Azure AI Search as the backing system for vector stores used with ChatGPT files and the Assistants API, and connects Azure’s capacity increase to larger limits exposed by OpenAI. The concrete historical change in OpenAI’s April 2024 announcement was a 500-fold increase in files per assistant, from 20 to 10,000—not a speedup or a general increase in every ChatGPT limit.

16:0216:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:02 · section reference included

Trade vector precision for capacity and speed

Higher service limits are one way to fit more data. Another is quantization: represent each vector component with a narrower type. Moving from full floating-point values to int8 reduces storage at the cost of some precision. Single-bit encoding goes further, reducing each component to one bit.

Castro reports that Microsoft’s single-bit evaluations retained performance in the low-to-mid 90% range for some models, without naming the metric or models. He also cites a Cohere evaluation retaining almost 95% of the original precision with bit encoding, and says reranking can recover the remaining precision. Cohere’s int8 and binary embeddings work is the related approach; the talk does not establish the dataset or rescoring conditions behind that cited percentage.

Float32-to-one-bit encoding reduces the raw vector representation by a factor of 32. That arithmetic concerns the vector components, not the complete index, which also has indexing structures and may retain original vectors. Binary comparisons can use Hamming distance instead of computing cosine similarity over wider values, reducing the work involved in comparing representations. Castro presents support for these narrower vector types as available in AI Search at the time of the talk.

You can quantize vectors yourself or enable managed quantization. In the managed path Castro describes, the service keeps the original full-precision vectors alongside the compressed representation. Retrieval then proceeds in two steps:

  1. Search the compressed vectors and oversample candidates—retrieve more than the final answer needs.
  2. Rescore those candidates using their original full-precision vectors, then select the final results.

This full-precision vector rescoring is different from the earlier semantic cross-encoder: it revisits vector similarity rather than jointly reading query and document text.

That produces a practical set of choices:

ChoiceBenefitCost
Compressed retrievalSmaller vector representationSome quality loss
Compressed retrieval plus rescoringCapacity with quality recoveryAdditional query work and retained originals
Uncompressed retrievalOriginal vector precisionLarger vector representation

The controls let an application prioritize capacity, latency, and quality according to its workload. Oversampling helps only when relevant candidates survive into the pool that will be rescored.

17:5218:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:52 · section reference included

Keep the index following the source data

The final scaling problem is getting new and changing data into the index. Every source has its own connection mechanism and way to enumerate changes. Azure AI Search’s ingestion system incorporates integrated vectorization so applications do not have to rebuild that pipeline for each source. Castro names Blob Storage, OneLake, and Cosmos DB as examples, with source access and security handled through the configured integration.

Change tracking makes the pipeline incremental: it processes changed data rather than reprocessing the entire collection for every update. After source access, the pipeline proceeds through file extraction, chunking, vectorization, and indexing. PDFs, Office documents, images, and nested formats become inputs to the same sequence.

The intended operating model is to configure the pipeline once and keep the index following the source as it changes. In practice, this applies to supported sources with configured indexers and skills; scheduled runs collect changes and retry work, rather than providing instantaneous synchronization of arbitrary Azure data. This shifts application effort toward retrieval queries and RAG orchestration instead of repeated ingestion plumbing.

Five connected stages for data source access, file format cracking, chunking, vectorization, and indexing, alongside an Azure Portal import-and-vectorize instruction.Open full source frame
Integrated ingestion progresses from source access through file processing, chunking, vectorization, and indexing.

Castro closes by inviting developers to try AI Search, pointing to a starting page and a free search instance available through an Azure subscription. That is a way to explore indexing and retrieval; it is not a promise that every embedding, enrichment, or generation component in a complete RAG application is free.

20:0520:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:05 · section reference included

Resources

Updates since the talk

Transcript download
Full transcript

Loading transcript…