# RAG at scale: production-ready GenAI apps with Azure AI Search

Pablo Castro, Microsoft

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/_2tZaDs-w5s-rag-at-scale-production-ready-genai-apps) · [Video](https://www.youtube.com/watch?v=_2tZaDs-w5s)

## RAG at scale: production-ready GenAI apps with Azure AI Search

Production RAG needs more than vector similarity: hybrid retrieval, careful candidate budgets, stronger ranking, compact indexes, and ingestion that keeps pace with changing data.

Basic familiarity with language models, embeddings, and Python will help; the article introduces the retrieval and ranking concepts as they appear.

### Give the model access to facts it never learned

[0:16](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=16s)

How do you get a language model to work with your application data, company documents, or information about your users—facts it never saw during training? Pablo Castro, from Microsoft’s Azure AI team, starts by separating three approaches that address different needs.

| Approach | Useful when you need to… |
|---|---|
| Prompt engineering | Supply instructions and context directly, taking advantage of longer context windows. |
| Fine-tuning | Teach recurring patterns or domain-specific jargon. |
| Retrieval-augmented generation | Answer using an external collection of facts. |

Prompt engineering can go a long way. When the problem is access to a changing body of knowledge, however, RAG separates the model’s reasoning capabilities from the system that stores that knowledge.

The basic workflow is straightforward:

1. An orchestrator receives a task, such as the next question in a chat.
2. It searches a knowledge base for candidate pieces of evidence.
3. It sends instructions, the question, and those candidates to the language model.
4. The model produces an answer grounded in the retrieved material.

Real applications usually need multiple attempts, tuning, and more elaborate orchestration. But this division of responsibility remains: **retrieval determines which evidence the model gets to reason over.**

![RAG recap slide showing an app orchestrator connected to a retrieval system and an LLM, with files and databases as data sources.](https://ai.engineer/api/talk-essay-media/d5fe6a7d751df94e9bbf4d225e37d8a40811eb6de0a8a958ec39d243fbde4c09)

Retrieval-augmented generation combines reasoning with external knowledge.

### A successful prototype creates several scaling problems

[3:06](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=186s)

Castro describes 2023 as a year of prototypes and 2024 as the move into production. A convincing demo establishes that an interaction is useful; production adds simultaneous users who want more data and faster answers. Success creates pressure to put the organization’s entire collection into the system. Data volume grows, updates arrive more frequently, and query traffic increases.

The initial workflow—take a question, search, call the model—also tends to expand. A single user request may require several retrieval calls and several model calls, multiplying the work performed by both systems. Meanwhile, adding more organizational knowledge brings different file types and data sources. Scaling therefore means handling **volume, change rate, query load, workflow complexity, and data diversity** together.

![RAG architecture diagram with a retrieval-system callout and bullets for data volume and diversity, sophisticated workflows, and usage.](https://ai.engineer/api/talk-essay-media/02f441ecfd6a155af0f79db86b01c5998e72f7dcf8cfef82243dc25d8a8d8e92)

Retrieval pressure points include data volume, data change rate, query load, and retrieval quality.

### Treat vector search as part of a retrieval platform

[5:04](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=304s)

Azure AI Search is designed to cover the broader retrieval problem rather than require applications to assemble every search component themselves. Vector search sits alongside Microsoft’s other retrieval capabilities, Azure data integration, and platform security and compliance features. The goal is to make these pieces work together as one service.

Vectors are useful because they capture soft conceptual similarity: a query can find related content without sharing its exact wording. Production applications also need control over how those matches are found and constrained.

- **Approximate and exhaustive search:** Fast approximate nearest-neighbor search serves retrieval workloads; exhaustive search provides precise results for baselines and recall evaluation.
- **Filters and projections:** Metadata conditions restrict eligible documents, while projections select the fields returned to the application.
- **Multiple vectors:** A document can carry vectors for different content sections or data types, potentially using different embeddings. A query can also contain multiple vectors.

These capabilities let an application combine database-style selection with similarity search.

### Build a small index, then add text and filters

[7:33](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=453s)

The notebook demonstration starts with an already provisioned search service. Castro connects using a default Azure credential and creates an index containing categorical metadata, text, and a vector field. The vectors have just three dimensions so the example is easy to inspect; practical embeddings commonly have hundreds or thousands. He selects **HNSW**, a graph-based approximate nearest-neighbor algorithm, as the indexing strategy.

This Python example expresses the same index structure, using current SDK profile-style configuration rather than reproducing the historical notebook cell:

```python
import os

from azure.identity import DefaultAzureCredential
from azure.search.documents.indexes import SearchIndexClient
from azure.search.documents.indexes.models import (
    HnswAlgorithmConfiguration,
    SearchField,
    SearchFieldDataType,
    SearchIndex,
    SearchableField,
    SimpleField,
    VectorSearch,
    VectorSearchProfile,
)

endpoint = os.environ["AZURE_SEARCH_ENDPOINT"]
credential = DefaultAzureCredential()
index_name = "toy-retrieval"

index = SearchIndex(
    name=index_name,
    fields=[
        SimpleField(name="id", type=SearchFieldDataType.String, key=True),
        SimpleField(
            name="category",
            type=SearchFieldDataType.String,
            filterable=True,
        ),
        SearchableField(name="text", type=SearchFieldDataType.String),
        SearchField(
            name="vector",
            type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
            searchable=True,
            vector_search_dimensions=3,
            vector_search_profile_name="toy-profile",
        ),
    ],
    vector_search=VectorSearch(
        algorithms=[HnswAlgorithmConfiguration(name="toy-hnsw")],
        profiles=[VectorSearchProfile(
            name="toy-profile",
            algorithm_configuration_name="toy-hnsw",
        )],
    ),
)

SearchIndexClient(endpoint, credential).create_index(index)
```

The categorical field must be filterable to support the metadata restriction added later. The vector field’s dimensions and search profile establish how its values will be indexed.

Once the index exists, Castro obtains a document client and explicitly pushes records containing vectors, text, and categories. Managed ingestion is another option, but direct uploads make the mechanics visible. He first searches with a vector and requests its two nearest candidates. Next, he adds the text `hello`: the system now retrieves through both the text and vector paths, then fuses and ranks their results. Finally, he restricts the query to category `A`, excluding records in category `B`.

Continuing the Python example, these teaching records make the same progression explicit:

```python
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizedQuery

client = SearchClient(endpoint, index_name, credential)
client.upload_documents([
    {"id": "1", "category": "A", "text": "hello world",
     "vector": [1.0, 0.0, 0.0]},
    {"id": "2", "category": "A", "text": "nearby concept",
     "vector": [0.9, 0.1, 0.0]},
    {"id": "3", "category": "B", "text": "hello again",
     "vector": [0.0, 1.0, 0.0]},
])

query = VectorizedQuery(
    vector=[1.0, 0.0, 0.0],
    fields="vector",
    k_nearest_neighbors=2,
)

vector_results = list(client.search(vector_queries=[query]))
hybrid_results = list(client.search(
    search_text="hello",
    vector_queries=[query],
))
filtered_results = list(client.search(
    search_text="hello",
    vector_queries=[query],
    filter="category eq 'A'",
    select=["id", "category", "text"],
))
```

Filters can also combine conditions with AND, OR, and ranges. Castro says filtering remains fast with hundreds of millions of documents, though he provides no latency measurements or workload configuration for that claim.

An audience member notices a detail in the live demonstration: why did a query with `k = 2` return three results? Because `k` controls the vector branch’s candidate budget, not the size of the entire hybrid response. Two candidates came from vector retrieval; keyword retrieval contributed additional candidates; fusion selected three results.

**Candidate count and returned-result count are separate controls.** An application can ask vector retrieval for a larger pool, combine that pool with keyword matches, and still return only the top N results. In the Python call, `k_nearest_neighbors` controls the vector candidate count; adding `top=2` to `client.search(...)` limits the final response instead.

### Retrieve broadly, then rank a small candidate set

[10:54](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=654s)

An answer can only use evidence the retrieval system actually finds. That makes retrieval quality a central production concern. Azure AI Search addresses it with two stages: a recall-oriented first stage gathers candidates using vectors and keywords, then a stronger second-stage model reranks them.

Consider Castro’s illustrative corpus of a hundred million vectors. A fast retriever can reduce that corpus to a small candidate set. Once the set is small, running a more sophisticated ranking model becomes affordable. The expensive model does not need to evaluate every document in the index.

The comparison progresses through four configurations:

| Configuration | Retrieval and ranking |
|---|---|
| Keywords | BM25 text scoring |
| Vectors | **OpenAI Ada** embeddings |
| Hybrid | Fused keyword and vector results |
| Hybrid plus reranking | Fusion followed by a stronger ranking model |

Castro reports better retrieval results across the displayed evaluations when reranking is enabled. The related [Microsoft hybrid retrieval evaluation](https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/azure-ai-search-outperforming-vector-search-with-hybrid-retrieval-and-reranking/3929167) measures ranking quality, not generated-answer accuracy: it uses NDCG@3 for customer datasets and NDCG@10 for academic datasets. The useful distinction is between finding plausible candidates and ordering those candidates well enough to supply the right evidence to the model.

Semantic reranking is not another cosine-similarity calculation. A **bi-encoder** encodes the document and query independently. That enables fast comparisons between their vectors, but the model never examines the query and document together. A **cross-encoder** takes both as input and predicts how well the document corresponds to the query.

| Model pattern | What it evaluates | Suitable scope |
|---|---|---|
| Bi-encoder | Separately computed representations | Large-scale candidate retrieval |
| Cross-encoder | Query and document together | Reranking a small candidate set |

The cross-encoder performs inference at query time for the candidate pairs. Applying it to an entire large corpus would be impractical for an interactive application, which is why the fast first stage matters.

Castro puts Azure’s reranker latency at roughly 100 milliseconds, depending on the cross-encoder, as the service’s chosen speed–quality trade-off. He describes that overhead as largely hidden by the longer LLM call in the overall response time; he does not supply a percentile, concurrency level, or benchmark setup.

### Scope the search and preserve the original text

[14:01](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=841s)

Before investing in more sophisticated ranking, use any known metadata that can narrow the relevant dataset. Discrete metadata gives the system a direct way to exclude irrelevant material. Apply the ranking machinery to that scoped set. The category filter in the notebook is a small example of this broader quality technique.

The next audience question concerns the keyword side of hybrid search. What input does it need? Castro returns to the notebook: the request can supply text and a vector, or it can supply text that the service also vectorizes. **A vector alone does not recover the original keywords.** Preserving the text gives the system an input for lexical retrieval as well as one from which it can obtain an embedding.

![VS Code notebook showing a search cell with text and vector inputs, a category filter, and output rows for document IDs 1 and 2 in category A.](https://ai.engineer/api/talk-essay-media/f65ca556f54b020cffc828786de8d6a78d0a871203c52b1a8c9d4ea9822cb707)

The notebook combines text, a category filter, and a vector query, with two results displayed below.

The audience member’s deeper question is how a chatbot should derive useful search terms dynamically from an arbitrary user prompt or conversation. That is a separate orchestration problem from accepting text and vector inputs. Castro mentions extracting a candidate search from the user’s input before calling the answer-generating LLM, but the exchange ends with the details deferred. No keyword-extraction algorithm or conversation-rewriting procedure is demonstrated.

### Larger collections change the capacity requirement

[16:02](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=962s)

The next constraint is how much content fits in an index. As applications moved from tiny experimental collections to organizational datasets, Azure increased its service limits. Castro reports a historical increase of 10–12 times the vector density for most services on the same SKUs, without changing prices. Microsoft’s [May 2024 announcement](https://azure.microsoft.com/en-us/blog/from-code-to-production-new-ways-azure-helps-you-build-transformational-ai-experiences/) corroborates an increase of up to 12 times at no additional cost; the change was not uniform across every deployment.

Castro says the new limits enable multi-billion-vector applications through service provisioning and data upload. He contrasts this with the previous year, when a billion-vector dataset was primarily a benchmark curiosity. The talk does not specify vector dimensions, tier, partition layout, or performance conditions for that scale.

OpenAI provides his customer example. Castro identifies Azure AI Search as the backing system for vector stores used with ChatGPT files and the Assistants API, and connects Azure’s capacity increase to larger limits exposed by OpenAI. The concrete historical change in [OpenAI’s April 2024 announcement](https://openai.com/index/more-enterprise-grade-features-for-api-customers/) was a 500-fold increase in files per assistant, from 20 to 10,000—not a speedup or a general increase in every ChatGPT limit.

### Trade vector precision for capacity and speed

[17:52](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1072s)

Higher service limits are one way to fit more data. Another is **quantization**: represent each vector component with a narrower type. Moving from full floating-point values to `int8` reduces storage at the cost of some precision. Single-bit encoding goes further, reducing each component to one bit.

Castro reports that Microsoft’s single-bit evaluations retained performance in the low-to-mid 90% range for some models, without naming the metric or models. He also cites a Cohere evaluation retaining almost 95% of the original precision with bit encoding, and says reranking can recover the remaining precision. [Cohere’s int8 and binary embeddings work](https://cohere.com/blog/int8-binary-embeddings) is the related approach; the talk does not establish the dataset or rescoring conditions behind that cited percentage.

Float32-to-one-bit encoding reduces the raw vector representation by a factor of 32. That arithmetic concerns the vector components, not the complete index, which also has indexing structures and may retain original vectors. Binary comparisons can use Hamming distance instead of computing cosine similarity over wider values, reducing the work involved in comparing representations. Castro presents support for these narrower vector types as available in AI Search at the time of the talk.

You can quantize vectors yourself or enable managed quantization. In the managed path Castro describes, the service keeps the original full-precision vectors alongside the compressed representation. Retrieval then proceeds in two steps:

1. Search the compressed vectors and oversample candidates—retrieve more than the final answer needs.
2. Rescore those candidates using their original full-precision vectors, then select the final results.

This full-precision vector rescoring is different from the earlier semantic cross-encoder: it revisits vector similarity rather than jointly reading query and document text.

That produces a practical set of choices:

| Choice | Benefit | Cost |
|---|---|---|
| Compressed retrieval | Smaller vector representation | Some quality loss |
| Compressed retrieval plus rescoring | Capacity with quality recovery | Additional query work and retained originals |
| Uncompressed retrieval | Original vector precision | Larger vector representation |

The controls let an application prioritize capacity, latency, and quality according to its workload. Oversampling helps only when relevant candidates survive into the pool that will be rescored.

### Keep the index following the source data

[20:05](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1205s)

The final scaling problem is getting new and changing data into the index. Every source has its own connection mechanism and way to enumerate changes. Azure AI Search’s ingestion system incorporates integrated vectorization so applications do not have to rebuild that pipeline for each source. Castro names Blob Storage, OneLake, and Cosmos DB as examples, with source access and security handled through the configured integration.

Change tracking makes the pipeline incremental: it processes changed data rather than reprocessing the entire collection for every update. After source access, the pipeline proceeds through file extraction, chunking, vectorization, and indexing. PDFs, Office documents, images, and nested formats become inputs to the same sequence.

The intended operating model is to configure the pipeline once and keep the index following the source as it changes. In practice, this applies to supported sources with configured indexers and skills; scheduled runs collect changes and retry work, rather than providing instantaneous synchronization of arbitrary Azure data. This shifts application effort toward retrieval queries and RAG orchestration instead of repeated ingestion plumbing.

![Five connected stages for data source access, file format cracking, chunking, vectorization, and indexing, alongside an Azure Portal import-and-vectorize instruction.](https://ai.engineer/api/talk-essay-media/49713425f56bcc125ce28e2564d32711ca64aa789f99cf786ff7c8c5ceb5dfd4)

Integrated ingestion progresses from source access through file processing, chunking, vectorization, and indexing.

Castro closes by inviting developers to try AI Search, pointing to a starting page and a free search instance available through an Azure subscription. That is a way to explore indexing and retrieval; it is not a promise that every embedding, enrichment, or generation component in a complete RAG application is free.

## Resources

- [Azure AI Search hybrid search](https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview) — Current guidance for combining text and vector queries with reciprocal rank fusion.
- [Microsoft's hybrid retrieval and reranking evaluation](https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/azure-ai-search-outperforming-vector-search-with-hybrid-retrieval-and-reranking/3929167) — A 2023 evaluation of keyword, Ada-002 vector, hybrid and semantic reranking configurations, with datasets and measurement methodology.
- [OpenAI's 2024 Assistants API improvements](https://openai.com/index/more-enterprise-grade-features-for-api-customers/) — The historical announcement increasing file-search capacity from 20 to 10,000 files per assistant.
- [Cohere int8 and binary embeddings](https://cohere.com/blog/int8-binary-embeddings) — Cohere's March 2024 introduction of native int8 and binary embedding representations.
- [Configure vector compression](https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-quantization) — Current configuration guidance for scalar and binary quantization, oversampling and rescoring.
- [Configure integrated vectorization](https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization) — How indexers, chunking skills, embedding models and query vectorizers form a managed ingestion pipeline.

## Slides

### Presenting sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=5s) [Microsoft presenting sponsor logo on a black background.](https://assets.ai.engineer/sites/aiecode2025/sha256/c076c9000767eb0f5ceab12ba4aadaf3aed64760b8ac98c6d351290985e0bd5c.webp)

PRESENTING SPONSOR

Microsoft

### Agenda

After a RAG recap, Pablo focuses on retrieval at scale: result quality, the AI Search vector database, and more data sources and types.

[0:29](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=29s) [Agenda covering a RAG recap, results quality at scale, scaling an AI Search vector database, and more data sources and types.](https://assets.ai.engineer/sites/aiecode2025/sha256/7ef2ac5a0ea68d302561af2abf90e316a616ae0631f8a5c4670fd6369d65d712.webp)

Agenda

1 RAG recap
2 Results Quality at Scale
3 Scaling your AI Search vector database
4 More data sources, more data types

### Bringing domain knowledge to LLMs

Prompt engineering uses in-context learning; fine-tuning can teach patterns or domain jargon. Retrieval augmentation lets models work with data they did not see during training.

[0:50](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=50s) [Three approaches: prompt engineering for in-context learning, fine tuning for new skills, and retrieval augmentation for new facts.](https://assets.ai.engineer/sites/aiecode2025/sha256/3e614804685880ae14a28c75e7a372a8e80f4d255230aa349992aca9c7716c21.webp)

Recap: Bringing domain knowledge to LLMs

Prompt engineering
In-context learning

Fine tuning
Learn new skills

Retrieval augmentation
Learn new facts

### Scaling Retrieval in RAG Applications: Pressure Points

The app orchestrator queries retrieval for knowledge and sends it with the prompt to the LLM. More data, changing sources, query load and multi-step workflows add pressure as applications scale.

[3:10](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=190s) [RAG architecture with pressure points for data volume and diversity, workflows, usage, query load, and retrieval quality.](https://assets.ai.engineer/sites/aiecode2025/sha256/45614c435c04f67a14f2ff0e1d8d1cb1a8a9082bb26f449022a8ba613215a4c6.webp)

Scaling Retrieval in RAG Applications: Pressure Points

+ Data volume & diversity
+ Sophisticated workflows
+ Usage

Data volume
Data change rate
Query load
Retrieval quality

Data Sources
(files, databases, etc.)

Retrieval System

Query → Knowledge
Prompt + Knowledge → Response

App UX
App Server, Orchestrator
LLM
SQL

[4:52](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=292s) [RAG architecture diagram and scaling pressures, including data diversity, workflows, usage, retrieval load, and data sources.](https://assets.ai.engineer/sites/aiecode2025/sha256/acd6f8b44dacc0d9413692a824417600a8f59022aeebe48b20e66cc8ac566cf5.webp)

Scaling Retrieval in RAG Applications: Pressure Points

+ Data volume & diversity
+ Sophisticated workflows
+ Usage

Data volume
Data change rate
Query load
Retrieval quality

Data Sources
(files, databases, etc.)

Data types
Data sources

Retrieval System

Query → Knowledge
Prompt + Knowledge → Response

App UX
App Server, Orchestrator
LLM
SQL

### Azure AI Search

Azure AI Search combines vector database capabilities with broader retrieval technology, data integrations, and an enterprise-ready foundation.

[5:09](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=309s) [Azure AI Search overview highlighting search technology, vector database features, integrations, and enterprise readiness.](https://assets.ai.engineer/sites/aiecode2025/sha256/fa92d012b1363a261a2dd8f6d31fe8b97143c85c8f9d3b6e1a8b8e6871c81d6f.webp)

Azure AI Search
Revolutionary retrieval

State-of-the-art search technology
Feature-rich vector database
Seamless data & platform integrations
Enterprise-ready foundation

### Comprehensive Vector Search

Azure AI Search pairs fast approximate search with exhaustive search for recall baselines. Filters constrain results, while multiple vectors can represent different content types or be used in queries.

[6:12](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=372s) [Three columns describe vector indexing options, filtered vector search, and multi-vector scenarios.](https://assets.ai.engineer/sites/aiecode2025/sha256/c981c79562e3493fdd2d589f299d15a9e6ffd7f73b04a3fe3e084d51a87882fe.webp)

Comprehensive Vector Search

Versatile vector search indexing options

ANN search: fast, always-in-memory or tiered vector search at scale

Exhaustive search: useful for recall baselines, highly fragmented datasets where sparse indexes are not effective

Filtered vector search

Scope to date ranges, categories, geographic distances, etc.

Rich filter expressions

Pre-/post-filtering

Multi-vector scenarios

Multi-vector documents, same or different embeddings, same or different data types (e.g. images, text, graphs, etc.)

Muti-vector queries, targeting one or more document vector fields

### Vector Index Configuration

The notebook creates an Azure Search index with category, text and three-dimensional vector fields. It configures HNSW with cosine similarity, then creates the index and gets a search client.

[7:48](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=468s) [A notebook defines metadata, text and vector fields and configures HNSW vector search.](https://assets.ai.engineer/sites/aiecode2025/sha256/86aa5b109590abafbac84c119849942496bdd87f1bd2fdccde2afcb3ca484885.webp)

creds = DefaultAzureCredential()
index_client = SearchIndexClient("https://scale.search.windows.net", creds)

Create a vector index

from azure.search.documents.indexes.models import *

index = SearchIndex(
    name="tinyvectors",
    fields=[
        SimpleField(name="id", type=SearchFieldDataType.String, key=True),
        SimpleField(name="category", type=SearchFieldDataType.String, filterable=True),
        SearchableField(name="text", analyzer_name="en.microsoft"),
        SearchField(name="myVector",
                    type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
                    searchable=True,
                    vector_search_dimensions=3,
                    vector_search_profile_name="vprofile")
    ],
    vector_search=VectorSearch(
        algorithms=[HnswAlgorithmConfiguration(name="algo",
                                               parameters=HnswParameters(metric="cosine"))],
        profiles=[VectorSearchProfile(name="vprofile", algorithm_configuration_name="algo")]

Interface text:
File  Edit  Selection  View  Go
Setup.ipynb  Scale.ipynb  Retrieval.ipynb  VectorSimple.ipynb
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
Python 3.12.3
[7]  0.0s  Python
Spaces: 4  Cell 3 of 8

[8:31](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=511s) [Notebook code configures a three-dimensional vector field and HNSW search, then creates the index.](https://assets.ai.engineer/sites/aiecode2025/sha256/a199d267ab286542aae2784d24d031912011cbd7f655ef6e8b78346f71a4db17.webp)

    fields=[
        SimpleField(name="id", type=SearchFieldDataType.String, key=True),
        SimpleField(name="category", type=SearchFieldDataType.String, filterable=True),
        SearchableField(name="text", analyzer_name="en.microsoft"),
        SearchField(name="myVector",
                    type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
                    searchable=True,
                    vector_search_dimensions=3,
                    vector_search_profile_name="vprofile")
    ],
    vector_search=VectorSearch(
        algorithms=[HnswAlgorithmConfiguration(name="algo",
                                               parameters=HnswParameters(metric="cosine"))],
        profiles=[VectorSearchProfile(name="vprofile", algorithm_configuration_name="algo")]
    )
)

index_client.create_index(index)

search_client = index_client.get_search_client("tinyvectors")

Insert a few documents with tiny vectors

Interface text:
File  Edit  Selection  View  Go
Setup.ipynb  Scale.ipynb  Retrieval.ipynb  VectorSimple.ipynb
Code  Markdown  Interrupt  Restart  Clear All Outputs  Go To  Variables  Outline
Python 3.12.3
[8]  4.5s  Python
[ ]  Python
Spaces: 4  Cell 3 of 8

### Upload example documents

The notebook uploads three documents to the index, each with a vector, text, and a category.

[8:54](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=534s) [Notebook uploads three example documents containing vectors, text, and categories.](https://assets.ai.engineer/sites/aiecode2025/sha256/0d6f219c4690565a27263601c704e881253abc604cddcb0dcc6669d2fd56ab7b.webp)

search_client = index_client.get_search_client("tinyvectors")

Insert a few documents with tiny vectors

search_client.upload_documents(documents=[
    {"id": "1", "myVector": [1, 2, 3], "text": "hello, world", "category": "A" },
    {"id": "2", "myVector": [1, 1, 3], "text": "hello there", "category": "A" },
    {"id": "3", "myVector": [4, 5, 6], "text": "good bye", "category": "B" }])

[<azure.search.documents._generated.models._models_py3.IndexingResult at 0x16f274db3e0>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at 0x16f274db7d0>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at 0x16f274db800>]

Search using vector similarity

from azure.search.documents.models import VectorizedQuery

q = [2, 2, 3]

Interface text:
File  Edit  Selection  View  Go
Setup.ipynb  Scale.ipynb  Retrieval.ipynb  VectorSimple.ipynb
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
Python 3.12.3
[9] 0.0s  Python
[10] 2.8s  Python
Spaces: 4  Cell 6 of 8

### Search using vector similarity

The notebook searches for the two vectors closest to [2, 2, 3]. It returns document 3 first, followed by document 1, with their scores and categories.

[8:59](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=539s) [Notebook queries for the two nearest vectors and prints each result's ID, score, and category.](https://assets.ai.engineer/sites/aiecode2025/sha256/139b7ec39e9629af7959978f155baf6d7b0b123b0a2f0f14476409162a1f4b76.webp)

Search using vector similarity

search_client.upload_documents(documents=[
    {"id": "1", "myVector": [1, 2, 3], "text": "hello, world", "category": "A" },
    {"id": "2", "myVector": [1, 1, 3], "text": "hello there", "category": "A" },
    {"id": "3", "myVector": [4, 5, 6], "text": "good bye", "category": "B" }])

[<azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>]

from azure.search.documents.models import VectorizedQuery

q = [2, 2, 3]
r = search_client.search(
    vector_queries=[VectorizedQuery(vector=q, k_nearest_neighbors=2, fields="myVector")])

for doc in r:
    print(f"id: {doc['id']}, score: {doc['@search.score']}, category: {doc['category']}")

Interface text:
File Edit Selection View Go
 aieng
Setup.ipynb  Scale.ipynb  Retrieval.ipynb  VectorSimple.ipynb
VectorSimple.ipynb > ...
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
Python 3.12.3
[10] 2.8s Python
[11] 0.0s Python
Spaces: 4  Cell 8 of 8

[9:12](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=552s) [Notebook runs a vector search and displays two matching documents.](https://assets.ai.engineer/sites/aiecode2025/sha256/43f6f6c12fd33c600a0356ac56aa1fcd6db55e51cebc86f247a8802e239a37b5.webp)

Search using vector similarity

{"id": "2", "myVector": [1, 1, 3], "text": "hello there", "category": "A" },
{"id": "3", "myVector": [4, 5, 6], "text": "good bye", "category": "B" }])

[<azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>]

from azure.search.documents.models import VectorizedQuery

q = [2, 2, 3]
r = search_client.search(
    vector_queries=[VectorizedQuery(vector=q, k_nearest_neighbors=2, fields="myVector")])

for doc in r:
    print(f"id: {doc['id']}, score: {doc['@search.score']}, category: {doc['category']}")

id: 3, score: 0.99504673, category: B
id: 1, score: 0.97305185, category: A

Interface text:
File Edit Selection View Go
aieng
Setup.ipynb  Scale.ipynb  Retrieval.ipynb  VectorSimple.ipynb
VectorSimple.ipynb > ...
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
Python 3.12.3
[10] 2.8s Python
[11] 0.1s Python
Spaces: 4  Cell 8 of 8

### Hybrid text and vector search

The notebook combines a “hello” text search with a two-neighbor vector query, then fuses and ranks the results. It returns three documents, with ID 1 first.

[9:20](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=560s) [Notebook combines text and vector queries and displays three results.](https://assets.ai.engineer/sites/aiecode2025/sha256/7215e1539ff9850c64a0712eb10e4ece670c7ddcbec370c9e88d373b412cafdf.webp)

{"id": "2", "myVector": [1, 1, 3], "text": "hello there", "category": "A" },
{"id": "3", "myVector": [4, 5, 6], "text": "good bye", "category": "B" }])

[<azure.search.documents._generated.models._models_py3.IndexingResult at 0x16f274db3e0>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at 0x16f274db7d0>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at 0x16f274db800>]

Search using vector similarity

from azure.search.documents.models import VectorizedQuery

q = [2, 2, 3]
r = search_client.search(
    search_text="hello",
    vector_queries=[VectorizedQuery(vector=q, k_nearest_neighbors=2, fields="myVector")])

for doc in r:
    print(f"id: {doc['id']}, score: {doc['@search.score']}, category: {doc['category']}")

id: 1, score: 0.032786883413791656, category: A
id: 2, score: 0.01666666753590107, category: A
id: 3, score: 0.01666666753590107, category: B

Interface text:
File  Edit  Selection  View  Go
Setup.ipynb  Scale.ipynb  Retrieval.ipynb  VectorSimple.ipynb
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
Python 3.12.3
[10]  2.8s  Python
[12]  0.1s  Python
Spaces: 4  Cell 8 of 8

### Filtered hybrid search

Adding a category filter to the text-and-vector query returns two category A documents. The vector limit of two applies to vector candidates; keyword candidates can add more before fusion.

[9:47](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=587s) [Notebook adds a category filter to a hybrid query and displays two matching results.](https://assets.ai.engineer/sites/aiecode2025/sha256/ea8d5be1b28602ef030679593703159d5e78cf483c04a829d01d34b93a204797.webp)

Search using vector similarity

{"id": "2", "myVector": [1, 1, 3], "text": "hello there", "category": "A" },
{"id": "3", "myVector": [4, 5, 6], "text": "good bye", "category": "B" }])

[<azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>,
 <azure.search.documents._generated.models._models_py3.IndexingResult at [illegible]>]

from azure.search.documents.models import VectorizedQuery

q = [2, 2, 3]
r = search_client.search(
    search_text="hello",
    filter="category eq 'A'",
    vector_queries=[VectorizedQuery(vector=q, k_nearest_neighbors=2, fields="myVector")])

for doc in r:
    print(f"id: {doc['id']}, score: {doc['@search.score']}, category: {doc['category']}")

id: 1, score: 0.03306011110544205, category: A
id: 2, score: 0.03306011110544205, category: A

Interface text:
File Edit Selection View Go
Setup.ipynb  Scale.ipynb  Retrieval.ipynb  VectorSimple.ipynb
VectorSimple.ipynb > ...
Code  Markdown  Run All  Restart  Clear All Outputs  Variables  Outline
Python 3.12.3
[10]  2.8s  Python
[13]  0.1s  Python
Spaces: 4  Cell 8 of 8

### Retrieval Quality

Azure AI Search fuses vector and keyword results, then reranks candidates. Hybrid search with semantic ranking scores highest in all three benchmark comparisons.

[10:59](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=659s) [Slide shows vector and keyword results fused and reranked, alongside a retrieval quality comparison chart.](https://assets.ai.engineer/sites/aiecode2025/sha256/598a2b7b6ba730f92e33db143f543ec84810da702e0193ebee080ad4fe613b6d.webp)

Retrieval Quality

Generative AI app response quality directly depends on grounding data quality

• Hybrid (vectors + keywords) L1 stage yields improved recall
• Semantic reranking in L2 stage boosts ranking precision
• Advanced: rich control of the query pipeline

Diagram:
Vector
Keywords
Fusion (RRF)
Reranking

Chart:
Y-axis ticks: 0, 10, 20, 30, 40, 50, 60, 70, 80
Categories: Customer datasets [NDCG@3]; Beir [NDCG@10]; Miracl [NDCG@10]
Legend: Keyword; Vector (ada-002); Hybrid (Keyword + Vector); Hybrid + Semantic ranker

Retrieval comparison using Azure AI Search in various retrieval modes on customer and academic benchmarks
Source: Outperforming vector search with hybrid + reranking

### Increased Vector, Storage, Compute Capacity

Newly created services get up to 12x vector density and 6x total storage, with more compute in most tiers. The chart shows the largest new vector limits in S3 and L2.

[16:08](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=968s) [Slide describes increased vector, storage and compute capacity and charts previous and new vector limits by tier.](https://assets.ai.engineer/sites/aiecode2025/sha256/32f01e256c2262f4c99c91c52233702a04f4882d35ecda7e76877bb148754ea1.webp)

Increased Vector, Storage, Compute Capacity

Announcing

Same workloads can be handled with smaller tiers
Build billion-vector search services
Includes all the power of a full retrieval system

Details
• Up to 12x increase in vector density increase
• Up to 6x total storage limit increase
• More compute capacity in most tiers
• Applies to newly created services

Y-axis: Total vector limit (GB/service)
Y-axis ticks: 0, 500, 1000, 1500, 2000, 2500, 3000, 3500
X-axis: Basic, S1, S2, S3, L1, L2
Legend: Previous Vector Limit; New Vector Limit

### New Expanded Capacity

The table compares old and new limits across six SKUs. For S2, vector capacity rises from 12 to 150 GB per partition, while maximum service size grows from 1.2 TB to 6 TB. Basic now supports up to three partitions.

[16:58](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1018s) [Table comparing previous and new vector, storage, and maximum service limits across Azure AI Search tiers.](https://assets.ai.engineer/sites/aiecode2025/sha256/b72147a76c8a9f39fce5904ec4c64fa2d2dfcdaaee91566e4a2b40db39c336c0.webp)

New Expanded Capacity

SKU | Previous vector limit | New vector limit | Previous storage limit | New storage limit | Previous max service size | New max Service size
Basic | 1 | 5 (5x) | 2 | 15 (7.5x) | 2 GB | 45 GB
S1 | 3 | 35 (11.5x) | 25 | 160 (6.4x) | 300 GB | 1.88 TB
S2 | 12 | 150 (12.5x) | 100 | 512 (5x) | 1.2 TB | 6 TB
S3 | 36 | 300 (8.3x) | 200 | 1024 (5x) | 2.4 TB | 12 TB
L1 | 12 | 150 (12.5x) | 1024 | 2048 (2x) | 12 TB | 24 TB
L2 | 36 | 300 (8.3x) | 2048 | 4096 (2x) | 24 TB | 48 TB

→ Basic SKU now supports up to 3 partitions, up from a single partition
→ Limits are GB/partition

### OpenAI on Azure AI Search Scalability

OpenAI says Azure AI Search’s scalability let it increase the knowledge users can upload by 500x. The speaker says AI Search backs vector stores for ChatGPT files and the Assistants API.

[17:08](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1028s) [OpenAI testimonial says Azure AI Search scalability enabled a 500-fold increase in the knowledge users can upload.](https://assets.ai.engineer/sites/aiecode2025/sha256/36136c32dc01965c2765bec9d7d8204b64aea96d0f3c62a877cb76c69a53124c.webp)

OpenAI

“As a direct result of Azure AI Search’s scalability, we increased the amount of the knowledge users can upload by 500x.”

### Binary Quantization

Binary quantization assigns each dimension one bit using a center threshold, reducing vector data size 32x from float32. On MIRACL semantic search in 18 languages, Cohere Embed v3’s nDCG@10 drops from 66.3 to 62.8 (94.7% retained).

[18:34](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1114s) [Binary quantization steps and size reduction, with a Cohere benchmark comparing full precision, int8, and binary embeddings.](https://assets.ai.engineer/sites/aiecode2025/sha256/bba7748ca317a48f030b28251fc0c1e1e91d85b1604fac157b7bc9acc075ac3c.webp)

Even *more*: Binary Quantization

Quantize to 1 bit per dimension
1. Pick a center value C based on a sample or offline knowledge
2. If >= C, it’s a 1, otherwise it’s a 0
3. Repeat for each dimension

32x reduction in net vector data size from float32

Other overheads start to show up, but > 22x improvements achievable

Model | Search Quality MIRACL (Semantic Search on 18 languages, nDCG@10, higher=better)
Cohere Embed v3
Embed v3 | 66.3
Embed v3 – int8 | 66.1
Embed v3 - binary | 62.8

Example:
Cohere v3 embeddings, 62.8 vs 66.3
→ 94.7% original performance preserved

Source: https://cohere.com/blog/int8-binary-embeddings

Diagram labels: 0, 1

### Narrow types in Azure AI Search

Azure AI Search supports vector types from 32-bit floats to packed 1-bit binary values, for client-side quantization or models and APIs that already return narrower numeric types.

[19:06](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1146s) [Table comparing supported numeric types, their sizes, and index definition data types.](https://assets.ai.engineer/sites/aiecode2025/sha256/92b78b8e8241ad63eb917cdf709959f57a99b76afa38689b56c8efc5eaa23676.webp)

Narrow types in Azure AI Search

Description | Size | Index definition data type
Single precision floating point | 32 bits | Edm.Single
Half precision floating point | 16 bits | Edm.Half
Short int | 16 bits | Edm.Int16
Signed byte | 8 bits | Edm.Sbyte
Packed binary | 1 bit, packed in bytes | Edm.Byte, packedBinary=true

Use with client-side quantization, or with an API or model that already returns narrower numeric types

### Built-in Oversampling

Oversampling retrieves extra quantized vectors, then reranks them using original precision before returning the top K. The example enables reranking and sets the oversampling factor to 10.

[19:28](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1168s) [Oversampling retrieves extra vectors and reranks them using original precision; a code example highlights reranking and the oversampling setting.](https://assets.ai.engineer/sites/aiecode2025/sha256/f0d7082d7e2f06405d3a815bd74e86ac9c8236ade39d4382b121e3ffff900810.webp)

Built-in Oversampling

Recovering quality: oversampling
• Retrieve K’ = K * oversampling_factor vectors
• Re-rank K’ vectors using original precision values
• Return top K
→ Composes with hybrid, semantic ranking

compressions=[
  ScalarQuantizationCompressionConfiguration(
        name="myScalarQuantization",
        rerank_with_original_vectors=True,
        default_oversampling=10,
        parameters=ScalarQuantizationParameters(quantized_data_type="int8")
    )
]

### Ingestion & Integrated Vectorization for Text, Images

Azure AI Search connects to data sources, extracts and chunks content, creates vectors, and indexes the results. The pipeline tracks changes and processes updates incrementally.

[20:08](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1208s) [Pipeline from data source access through file processing, chunking, vectorization, and indexing.](https://assets.ai.engineer/sites/aiecode2025/sha256/86ff84ea2cc0489d0bd7dcfd3b9147eccaba66b4808678d2ad51994684a1d0c5.webp)

Ingestion & Integrated Vectorization for Text, Images

End-to-end support for text and image data
• Connect to data sources, detect changes, deal with file formats
• Extract text and images from PDFs, Office docs
• Compute vectors during ingestion using type-specific or multi-modal embedding models
• Query directly with text, an image, or execute hybrid + multi-vector queries

Start in the Azure Portal: “import and vectorize data”

Data source access
• OneLake
• Blob Storage
• ADLSv2
• SQL DB
• CosmosDB
• ...
+ Incremental change tracking

File format cracking
• PDFs, Office docs
• Image files
• Nested images in PDFs, Office docs

Chunking
• Split text into passages (for text portions)
• Propagate document metadata

Vectorization
• Image → vector
• Text → vector
• Embeddings from OpenAI, AI Vision, or custom models

Indexing
• Document index
• Chunk index
• Both

## Full transcript

[0:00](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=0s) [upbeat music] All right, let's get started. Hi, everyone. Thanks for coming.

[0:16](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=16s) Uh, I'm Pablo. I work in the Azure AI team at Microsoft. Um, and in this session, we'll talk about, uh, RAG at scale. Uh, and in particular, I'll focus on the retrieval portion of, of the RAG pattern.

[0:29](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=29s) The plan for, for this session is, uh, we'll do a quick recap, just make sure that we all have the s- we're, uh, use the same terms for the same things.

[0:36](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=36s) Uh, and then I'll go through kind of different dimensions of scale, and I'll comment a little bit on how do we tackle this in the context of AI Search, what we've learned from doing this, and what are we doing to make it easier, uh, to scale these applications.

[0:52](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=52s) So kind of as a... By u-by means of a quick recap, when it comes to bringing your own domain knowledge, uh, to work together with, with language models, you effectively you have three options.

[1:03](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=63s) Either you, you can do, um, uh, prompt engineering. And, you know, while it's easy to dismiss, you can go a long way with, uh, prompt engineering alone, especially these days, you know, models have longer co-longer context and whatnot.

[1:16](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=76s) Um, if, if that's not enough, sometimes the challenge is more along the lines of I wanna teach, uh, particular patterns o-of, or, or I want the model to learn jargon of some, in some particular vertical domain and things like that.

[1:28](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=88s) And for that, fine-tuning is often a good option.

[1:32](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=92s) However, in many, many of these cases, what I want is to have the model work over a set of data that the model didn't see during training. This could be my application data, my company data, information about my users or anything like that.

[1:45](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=105s) And for that, the, the, uh, prevailing pattern right now is to use retrieval-augmented generation. Effectively, uh, what that means is if what-- if you want the model to know facts, then what you do is you kind of separate the reasoning piece of the picture from the knowledge piece of the picture.

[2:01](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=121s) You lean on the language model for the reasoning capabilities, and you lean on the use of an external knowledge base to model what you know about a particular domain, the data that you have.

[2:10](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=130s) And you-- And the way you glue them together is, like, in principle, mechanically very simple. Of course, then it gets complicated because, you know, life is never that easy.

[2:18](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=138s) But in principle, uh, you have some orchestration component that, uh, you know, takes the task, uh, at hand. Let's say you have a chat application, and then the user takes a turn and asks, asks the next, uh, question.

[2:29](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=149s) Um, uh, the orchestration component hits some knowledge base, uh, seeking for pieces of information that could be used to produce an answer, uh, to this question and then grabs a bunch of candidates, and then you go to the language model and then give it a bunch of instructions, your candidates, uh, and ask the model to produce, uh,

[2:46](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=166s) like an answer to the, to the user question. That's kind of the essence o-of the pattern. We all know that in practice, it usually takes multiple goes, and there is a lot of tuning in the middle and, and whatnot, but the fundamentals boil down to that.

[2:58](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=178s) Um, so, uh, just by, by way of context, like, how many of you are doing, are creating and working on RAG applications today?

[3:06](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=186s) Everyone. Okay. Excellent. Um, so with that backdrop, what I wanted to do is talk about what are the pressure points when it-when you scale these applications. One, one thing that has been fascinating to see is, you know, we had, we had the opportunity to be involved in this space, uh, from very early, like Azure OpenAI has been

[3:25](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=205s) there from, you know, the early days of scaled language models. And one thing we saw was all of, I'd say last year, twenty twenty-three, everybody built a prototype of something to kind of learn and figure out what could be done with this technology.

[3:38](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=218s) Um, and, uh, the interesting shift to this year, to twenty twenty-four, it's been these applications are going to production. And when you go to production, you go from, "Oh, this demo is really cool," to all these users are, are, you know, are using it at the same time, and they want more.

[3:51](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=231s) They want more data. They want, uh, more, uh, faster answers and, and whatnot. Um, so the result of that is that, you know, when before we could focus only on figuring out the interaction model and the applicability of this technology, now, you know, elements of scale also, also play a role.

[4:08](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=248s) Uh, and scale can-- takes multiple kind of, uh, flavors. Like you-- These thing-things tend to scale in volume because one of the things that happens is when your application works well, users come back or the leadership of the organization come back, comes back and says, "Let's put all the data there."

[4:22](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=262s) Uh, and then you have to deal with it. Um, uh, the, also the rate of change of the data kind of increases, uh, and the query load increases as well because, you know, more people are using the stuff.

[4:32](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=272s) Um, also workflows tend to get more complicated. At first you go like, "Oh, like how complicated it can be? I'll take the question, search the thing, and then send it to the model."

[4:41](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=281s) Well, it turns out that, that sometimes it works, but often it doesn't. So you end up doing these multi-step workflows that hit the retrieval system and the language model multiple times, um, and that taxes all the systems and they all have to scale.

[4:54](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=294s) Also in, in the kind of, in the spirit of now let's put all the data there, now you have to deal with more, uh, more data types, different kinds, different data sources and whatnot.

[5:04](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=304s) Um, so, um, let's cover each, each of these dimensions of scale, um, in detail. And what, what I'll do is I'll cover it in the context of Azure AI Search.

[5:13](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=313s) That-that's what I, what I work mostly, uh, and, uh, because it's, you know, the way we think about this in Azure is we want to produce a system that is, that encompasses entire retrieval problem.

[5:23](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=323s) Uh, it's of course it has this kind of vector database capabilities. The, uh, vector-based retrieval emerged as a very, uh, useful solution in many contexts. So, uh, we have full support for that.

[5:35](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=335s) But we also brought into it like, uh, years and years of Microsoft experience in, in retrieval systems in the more general sense. Um, and, uh, we integrate them together so you don't have to connect a bunch of parts, uh, to have a proper, uh, high-quality retrieval system.

[5:49](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=349s) It all comes integrated. And of course, you know, we integrate it into the rest of Azure, so it's easy to pull data in, to connect to the other data sources and whatnot.

[5:57](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=357s) Um, and, you know, Azure is, is a place that is used to build some of the largest applications out there. So all the kind of enterprise readiness comes pre-built in, you know, from security to compliance to all these things that you don't wanna deal with directly.

[6:10](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=370s) You wanna kinda build on a platform that is already taken care of.

[6:14](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=374s) So while there are multiple moving parts to retrieval systems, what we've seen the last, uh, I don't know, eighteen months or so since the emergence of, uh, kind of RAG, uh, patterns at scale is, uh, vector retrieval is an important part of the solution.

[6:29](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=389s) And, you know, you, you can see why. Like, the interesting part about vector retrieval is, like, it's incredibly effective at getting this kind of soft conceptual similarity, uh, and, uh, put it to work right away.

[6:39](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=399s) So in Azure AI Search, we built a system that has kind of complete, uh, a complete feature set when it comes to, uh, vector search, including, uh, a pr-fast approximate nearest neighbor search, uh, and a-as well as also exhaustive search.

[6:52](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=412s) Sometimes you want precise search either to create baselines, uh, to know how kind of your recall performance is looking and things like that. Um, also, often applications need to combine, uh, vector search with the rest of the queries.

[7:04](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=424s) You need to filter, uh, filter things. You need to, uh, slice and dice, like, you know, filter, select the columns you want to project and things like that. Like effectively treat it like a database that also does retrieval.

[7:13](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=433s) Um, and we also see multiple scenarios where the documents have multiple vectors, maybe for different parts of the content or for different data types that use different embeddings. Um, and, uh, and sometimes the queries need multiple vectors.

[7:25](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=445s) So we try to make it, uh, as these, all these specific needs come up as you're building an application, you'll find answers to all of them directly in, in Azure Search.

[7:33](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=453s) Let me show you this in action. Um, so let me start with a very simple example. So what I'll do is, uh, I'll-- I'm just here, I have a sm-small, uh, Jupyter notebook, uh, and what I'll do is I'll connect to, um,

[7:49](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=469s) uh, I can set up a default credential point at my Azure Search service, and I'll create an index from scratch. So this is all, like, all it takes to, to create an index once you have a service provisioned.

[7:59](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=479s) So in this case, I create a few fields. I create a categorical field, like this serves as metadata that you want sometimes to attach, to be attached to, to this.

[8:07](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=487s) I create a text field. Sometimes you want to mix text and vectors. Um, and I'll talk a little bit more about this later. And I create a vector field.

[8:15](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=495s) In, in this case, this is a toy example, so dimension is three. Of course, that's not very useful. You know, in practic- in practice, this is gonna be in the hundreds or maybe thousands of dimensions.

[8:24](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=504s) I'll also say what strategy you want the system to use for vector search. In this case, I'm using HNSW, which is a well-known, um, uh, graph-based algorithm for,

[8:34](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=514s) um, for indexing vectors. So when I run this, and I hope, uh, the, like the network was a little wonky, so, oh, there. Perfect. Um, so now I have an index.

[8:42](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=522s) I'm gonna get a, kind of a client to it, and then I'm gonna index data. And you can see here, this is a very simple case. These are my vectors.

[8:48](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=528s) Um, this is my full text, and these are my cat-- these are my categorical data bits. So you can let us do all the ingestion for you, or you can push data into the index.

[8:57](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=537s) In this case, I'm pushing data explicitly into the index. And once you have data, uh, if you look at here, like we have some vectors and we have some categories.

[9:05](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=545s) So you can, first you can just search. And because it's, in this case, I'm indexing vectors, when you search, you search with a vector. Uh, so I can, I can search for that and say these are the two closest to the reference vector that I gave.

[9:17](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=557s) And I can do, uh, um, a few, a few things, uh, kind of incrementally. Like for example, if I wanna combine text and keyword search, I can also, uh, search text right here.

[9:27](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=567s) Uh, I'm gonna go hello. Uh, and now I'm searching for both, uh, the, the text and the vectors, and then we fuse the result and rank appropriately. Um, and, and often in applications, you also need to filter stuff.

[9:39](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=579s) You can see here I have A's and B's in categories. So I can, I can, like, write filters. Uh, for example, here I'll say category equals A.

[9:49](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=589s) Uh, and then, you know, I only get the A's. Uh, and, uh, this is a trivial example, but of course, you can do full filter expressions and ands and ors and, and, uh, uh, ranges and, and whatnot.

[9:58](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=598s) And the filters are fast. So, so even if you have, um, hundreds of millions of documents, these are not a problem.

[10:05](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=605s) So in, in that one, if you had, uh, k-nearest neighbors, k-nearest neighbors equals two, how come they had three results?

[10:12](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=612s) Oh, because, uh, great question. Because, so what I, what I was telling this, the, the system here is from the vectors retrieve two candidates. But then I was like, I also told it, "Go to the keyword side and retrieve a bunch of candidates from there and then fuse them."

[10:26](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=626s) Uh, so only two of them were from vectors, but the fuse, the fusion of the two, uh, selected three. I actually, like if I, um, I can make this a larger number, and that sometimes is useful, um, to get more candidates.

[10:40](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=640s) And then still I can say, uh, as a result of that, get the keyword ones too and then rank the top N. So basically separate how many candidates you want from how many you wanna return.

[10:54](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=654s) So, so as, those are the basics of a vector engine, but there are a few, uh, key elements to, to consider when you're actually building an application in production.

[11:01](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=661s) The probably the most se- the most salient one is quality. Like in the end, the, your application works well for your users when they ask a question and they get the answer they're looking for.

[11:10](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=670s) And that is highly, highly influenced by whether your retrieval system actually found the, uh, the bits of information that you wanted, uh, that, that you wanted to produce that answer.

[11:19](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=679s) Um, in, in AI Search, the way we do this is we do, um, what kind of most, uh, kind of more sophisticated search engines do that where you use a two-stage retrieval system.

[11:28](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=688s) First stage is recall-oriented and uses vectors and keywords and kind of all these recall-oriented tricks to produce as many candidates as we can find. And then the second stage re-ranks those candidates using like a, um, a, like a larger model that you maybe, let's say you have a hundred million, uh, vectors in your database.

[11:46](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=706s) Uh, you, you wanna use something fast to go from a hundred million to a small set. But then once you have a small set, you can afford to lar- run a larger, more sophisticated language, um, sorry, um, ranking model, uh, to create better quality ranking.

[11:59](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=719s) So that's what we're doing in the L2 stage. Uh, so when you turn this on, you effectively get better results. And, uh, what you can see here, there is a link at the bottom, happy to share it later, uh, with more details on the numbers.

[12:09](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=729s) But you can see that, um, so from the left, this is what you get when you just do keyword search using BM25. It's a well-known scoring approach. Second one is only vectors using Ada, uh, OpenAI Ada vectors.

[12:21](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=741s) Third one is using fusion, combined vectors and keywords. And the fourth one is using fusion plus, uh, re-ranking, um, re-ranking step, where across the board we see better results just out of the box when, uh, when re-ranking is enabled.

[12:34](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=754s) Yes.

[12:35](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=755s) So the re-ranking is the semantic re-ranking. Is that Is that another cosine similarity style thing, or how does that actually work?

[12:43](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=763s) Oh, great, great question. The question is, is it a cosine similarity thing? No. So the, the thing, bi-encoders are useful when, uh... because you can do-- you can encode all your documents as vectors and then the query as a vector.

[12:54](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=774s) Um, and then you can evaluate them fast because you are only comparing similarity. But that means that at no point the model sees the, the vec-- the document and the query at the same time.

[13:02](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=782s) So the re-ranker is, uh, these type of rankers are often called cross-encoders. And, uh, these are transformer models that you feed them the document and the query, and you ask them to predict a label of the correspondence of the query to the document.

[13:16](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=796s) So they are much better positioned to produce a high quality result. Uh, but they are, they're, they're, they're... At, at inference time, or rather at query time, you're running an inference step.

[13:26](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=806s) So, um, you can do it only on a smaller set. You couldn't do it on the entire data set. It wouldn't be practical. Uh, at least not for interactive performance applications.

[13:34](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=814s) And speed?

[13:36](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=816s) Speed?

[13:37](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=817s) Yeah.

[13:37](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=817s) Uh, this is like a hundred milliseconds, give or take, for a model like this. Depends on the cross-encoder that you, that you use. Ours, our trade-off between, you know, make it fast enough but make it very high quality, we landed on a hundred milliseconds ballpark.

[13:49](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=829s) Um, uh, and we found that that works well in terms of interactive performance because the majority of the latency ends up hidden on the LLM call. Uh, uh, and you still get very high quality results.

[14:01](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=841s) Um, the other thing, I won't drill into this, but, uh, often, uh, one second. Often the other dimension of getting quality out of the system is to narrow the data set.

[14:10](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=850s) If you know discrete, um, metadata elements that can help you narrow the data set, that's the most effective way. Uh, then do all the ranking tricks on top of the resulting set.

[14:19](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=859s) But, uh, first, uh, scoping is a, is a very effective way to, uh, to get quality up. Yes, question.

[14:25](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=865s) So my question is, do you, uh, extract the keyword from the prompt, or like how do you set the keyword?

[14:31](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=871s) The, for the keyword part of keyword search?

[14:33](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=873s) Yeah.

[14:33](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=873s) Oh. So if you look at, um... Let me show you this example. So if you look at what I did here, is, uh, you give us keywords, you give us text-

[14:42](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=882s) Yeah

[14:42](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=882s) ... and then you give us a vector as well. Or you just give us the text, and we'll turn it into a vector too. Uh, so we need the original text.

[14:48](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=888s) The vector alone is not enough because we can't go back to a keyword.

[14:52](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=892s) No. So, I mean, if you have a RAG application, let's say you have a chat bot.

[14:55](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=895s) Yeah.

[14:55](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=895s) How can you know how to find what kind of keywords could be relevant, let's say, to the prompt that the user is giving?

[15:01](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=901s) Oh. So, um, u-usually in a RAG app, w- the question is, what of the conversation so far do you wanna use for search? Is that, is that the question?

[15:10](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=910s) No, like I understand, but my question is, so if you wanna use... Because in your method you showed that the fused search-

[15:16](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=916s) Yeah

[15:17](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=917s) ... plus

[15:18](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=918s) Yeah

[15:18](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=918s) ... uh, prompt. So how, how do you... Like for vector search it's easy, right? You vectorize the prompt and then search.

[15:25](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=925s) Yeah.

[15:26](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=926s) But if you wanna like search and vector fuse-

[15:28](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=928s) Yeah

[15:29](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=929s) ... is there a way you can extract like from the user's element to, uh, extract the potential keywords for you to search? Because you cannot pre fix those keywords for every potential chat or, or prompt that the user can have.

[15:45](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=945s) Yeah, I... Maybe we should talk after the talk because I'm not sure I understand the question. Like usually, like you, in the index you have the se- the text and the vectors.

[15:53](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=953s) Uh, and then from the user, you extract a candidate search for, for that. This is before you send it to the LLM. Uh, but maybe we should chat right after the talk on the details.

[16:02](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=962s) Um, so let me go back here. So the, the other dimension of scaling is, um, is, uh, just, just how many vectors and how much content you can fit in one of your indexes.

[16:16](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=976s) Uh, and, uh, for that, in, in AI Search, like one thing we learned last year, uh, or in the beginning of this year, is that it went from everybody had tiny indexes to everybody put all their data in these systems.

[16:26](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=986s) Uh, so we re- we, uh, accommodated for that by significantly increasing the limits. Like for most of the ser- services, uh, you'll get anywhere ten, between ten and twelve times the vector density on, on the same SKUs, and we didn't change prices or anything, just so you can build larger applications on the same setup.

[16:42](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1002s) With these new limits, you can build multi-billion vector apps by just provisioning a service and uploading the data. And it's surprisingly straightforward. Like, you know, a year ago, the billion-vector dataset was the thing...

[16:53](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1013s) It was a curiosity used for, uh, for benchmarking. And now you can just create an index and upload them, which is very impressive to, to see. Um, I'm not gonna drain the slide.

[17:01](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1021s) I put it here for reference. Uh, these are kind of all the new limits that, uh, that we have, are significantly higher than the, than the ones we had before.

[17:10](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1030s) Um, one of the things that has been exciting for us to watch, uh, to watch grow is, uh, you know, among our customers, one of them is OpenAI themselves, uh, where they have a lot of RAG workloads, like when you have files in ChatGPT or when you use the Assistants API.

[17:24](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1044s) All of those, you can create these vector stores inside their system and, uh, all of those are backed by AI Search. Uh, and when we increased these limits, um, one of the things they did is they increased the limits they give to their users by five hundred times.

[17:37](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1057s) Um, so, uh, it's been impressive to see how fast they grow, and it's been fun to kind of see, uh, kind of first, um, uh, from up close how a system can scale at, at like this big and, uh, and still be fast and responsive and whatnot.

[17:52](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1072s) Um, so finally, the, the other thing you can do is sometimes the limits, higher limits are enough. Sometimes you wanna push even more data into it. So the other thing we've been working on is quantization, where we, we...

[18:04](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1084s) you can use narrower types. Like instead of using full floats, you can use ints, uh, like int8s. Uh, and they just simply use less space at the trade-off for a little bit of quality.

[18:14](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1094s) Uh, and interestingly, you can even do single-bit quantization. And I confess that when people said, "Hey, like we're gonna do metrics for single bit," I felt people were wasting their time.

[18:23](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1103s) Uh, but it actually works. Uh, it works surprisingly well. Our evaluations show that they are still, for some models, in the mid, uh, low to mid ninety percents of the original performance.

[18:34](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1114s) And other companies have seen the same thing. Like for example, this is an evaluation from Cohere, but separate company. Um, and they also see like about almost ninety-five percent of the original precision is preserved.

[18:44](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1124s) When, uh, when using bit encoding. And you can get the precision, remaining precision back by re-ranking, uh, when you're done. So surprising that it works, but you go from float32 to one bit.

[18:54](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1134s) That's thirty-two x the vector density. Um, and it's faster because you just do Hamming distance mu-much faster than computing on a small number of... a smaller number of bits instead of cosine similarity or something like that on a wider, wider set.

[19:08](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1148s) So because of this, we now support all these types in AI Search as well.

[19:13](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1153s) Um, uh, I'm gonna skip this slide in the interest of time. Um, so and, uh, you can do quantization yourself, or you can just enable it, and we will do quantization, uh, for you.

[19:23](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1163s) And, uh, if we do quantization for you, we will also store the original, uh, precision data there, which means we can do oversampling, where we query at the quantized kind of compressed, uh, version of the vectors, but we have the full precision stored, stashed as, on the side.

[19:39](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1179s) Uh, so later, we can re-rank at full precision. Uh, and, uh, so you can effectively choose between you want a highly compressed, uh, index that is, uh, a little lower quality, but, but, but larger, or it's a little slower but larger.

[19:53](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1193s) Uh, or you just don't compress it, and then you get the quality up. So effectively, you can choose any of the three, uh, and it's effectively up to you, uh, what you wanna prioritize.

[20:01](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1201s) But we give you control for all three dimensions.

[20:05](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1205s) And then the last thing I wanted to, uh, touch on is the, the other challenge is you have to keep adding data sources that you bring into these RAG systems.

[20:14](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1214s) Um, and, uh, each of them, you connect to them differently, you enumerate changes differently, um, and, uh, that's, that's just not where you wanna spend your time. Um, so we have this ingestion system that includes integrated vectorization as part of AI Search, where if the data is in Azure, whether it's in Blob Storage or OneLake or Cosmos

[20:32](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1232s) DB, we will connect, deal with all the security and all of that. We will automatically track changes, so it's not a one-shot thing. But as, as the data changes, we'll pick up only the changes and will only process the changes, so the cost is also incremental.

[20:45](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1245s) You don't pay as you update the stuff for the entire set. Uh, and then we'll deal with all file formats, you know, PDFs, Office documents, images, un-unpack the nested formats and whatnot.

[20:55](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1255s) Do chunking, do vectorization, and land it on an index all in one go. And this is a, like, industrial strength pipeline that you set up once, and then it continuously runs, uh, af-after that.

[21:04](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1264s) As your data changes, your, it reflects the changes. So you can focus on the RAG stack, you know, the workflow, how you query the system, but you don't have to think about how the data makes it there.

[21:13](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1273s) If the data is anywhere in Azure, we'll index it and create, like, an index that follows the original data automatically.

[21:21](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1281s) All right. And, and with that, uh, I know I raced through this content in these twenty minutes. I'll be hanging out outside if anybody wants to chat or have questions.

[21:29](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1289s) Um, and I would encourage you to go try AI Search today. Here's a link to the starting point. Um, uh, and Azure subscriptions include a, include a free instance of AI Search, so you can even give it a shot in a minute without, uh, without having to pay for any of this stuff.

[21:43](https://www.youtube.com/watch?v=_2tZaDs-w5s&t=1303s) With that, thank you. [clapping] [upbeat music]
