AI Engineer Summit 2025
The Hidden Costs of Building Your Own RAG Stack — Ofer Mendelevitch, Vectara
Loading slides…
Read the talk
The Hidden Costs of Building Your Own RAG Stack
Retrieving facts before generation is only the beginning: enterprise RAG also requires continuous evaluation, coordinated services, access controls and a team that can maintain the whole system.
From a talk by Ofer Mendelevitch
Answering questions over your own data
How do you get an LLM to answer a question using your PDFs or Notion workspace? Rather than send the question directly to the model, first retrieve relevant facts from those sources, then ask the model to answer using that evidence. That is the basic operation of retrieval-augmented generation, or RAG. Ofer Mendelevitch, Vectara’s head of developer relations, approaches it as a software and machine learning engineer who has worked with LLMs since GPT-2 in 2019.
The difficulty is turning that basic operation into an enterprise platform. A prototype establishes that retrieval and generation can work together. Production requires the surrounding system to keep working as its data, traffic and requirements grow. The hidden cost is operating the complete stack, not merely assembling its first working version.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two flows, many components
The architecture starts with two flows. In the diagram, the blue arrow represents ingestion: moving information from documents, databases, Jira, Salesforce or other sources into a form the retrieval system can search. The steps are:
- Extract the content from its original format, such as a PDF.
- Chunk the text into smaller pieces.
- Embed each chunk with an embedding model.
- Store the resulting vectors in a vector database.
The original text must also be stored somewhere so it can later be supplied to the LLM. That storage is omitted from the simplified diagram, but it remains part of the implementation.
The green arrow represents the query flow. A query arrives, sometimes accompanied by filtering parameters. The system embeds the query text and searches the vector store for relevant material. Vector similarity may be only the first retrieval step: hybrid search combines vector and keyword search, while reranking can prioritize relevance, diversity or application-specific business rules.
Once retrieval produces the relevant chunks, the system sends them to the LLM together with the question and a grounding prompt. An optional hallucination detector then checks whether the generated answer used the supplied facts properly. Only after those steps does the response return to the user. What looks like one question-and-answer interaction is therefore a chain of extraction, storage, retrieval, ranking, generation and checking responsibilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where a managed platform draws the boundary
Vectara’s proposed alternative is to put those components inside a managed boundary: RAG as a service. Mendelevitch describes three deployment options—SaaS, where customers send their data to Vectara; deployment in a customer’s VPC; and deployment on the customer’s own hardware.
The application then interacts with two external interfaces. An indexing API loads the data the system should use, and a query API connects the application’s user interface to the retrieval-and-generation flow. Under this arrangement, Vectara takes responsibility for managing the internal components. The application developer still supplies the data and builds the user-facing experience, but does not assemble each service inside the boundary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Quality continues after the first deployment
Building the stack yourself means choosing and integrating the vector database, retrieval engine, embedding model and LLM. From community and customer conversations, Mendelevitch groups the difficulties of moving that stack into production into seven categories: response quality and hallucinations, latency, scaling cost, security and compliance, vendor coordination, upkeep, and support for additional languages. These are the operating responsibilities behind the build-versus-buy decision.
The first responsibility is retrieving the right evidence. Parsing and chunking determine what information becomes available to search. Tables add another layer: extracting their text is not enough if the resulting representation loses the relationships needed to understand it. Different chunking strategies, table parsing and understanding, and hybrid search all require work beyond the initial proof of concept. Better generation cannot substitute for getting the relevant facts into the context.
Evaluation is an ongoing operation. A deployment can change when new data arrives, existing data is refreshed or a component is replaced. Response quality needs to be evaluated after those changes, not only before the first release, so that the team can determine whether results remain as good or improve.
Even with relevant evidence, hallucination prevention and correction remain difficult. Explainability adds a separate requirement: users need a way to inspect the facts behind an answer. A clickable citation in the interface depends on the system preserving a path back to the source. That provenance or audit trail must survive the processing flow; it cannot be reconstructed reliably by adding a link at the very end.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Latency and cost accumulate across the stack
A simple vector lookup can be fast, yet the complete query path can still be slow. Hybrid search and additional rankers introduce more work, and poorly coordinated components add delay. A slow component on the response path holds up the answer even if the other services perform well. Production latency therefore depends on the orchestration of the whole flow, not just the speed of the vector database.
The same expansion appears in cost. Embedding models, LLMs and the vector database are only the beginning; some stacks also use external services for PDF parsing or table understanding. Maintaining both quality and low latency requires capable components, each with its own expense.
Underneath the services sit GPU, CPU and storage costs. Mendelevitch contrasts a prototype containing ten documents with collections containing thousands, hundreds of thousands or millions: the example is a change in operating scale, not a measured cost curve. Token consumption adds another growing expense. The relevant budget is the cost of the entire system at its intended scale, rather than the price of one model call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Document permissions must survive retrieval
Security begins with attribute-based access control. If a document is visible only to HR or the CEO, an employee must not gain access to its contents by asking the RAG application a question. The document system’s restrictions must carry into the RAG flow, including the evidence that can reach generation.
For a concrete implementation illustration, consider an employee asking, “What is our leave policy?” The candidate records include a general employee policy and an HR-only case note. This TypeScript filter keeps only records whose allowed roles intersect the user’s roles:
typescript
type Role = "employee" | "hr" | "ceo";
type Chunk = {
id: string;
text: string;
allowedRoles: Role[];
};
function authorizedChunks(chunks: Chunk[], roles: Role[]): Chunk[] {
return chunks.filter(chunk =>
chunk.allowedRoles.some(role => roles.includes(role))
);
}
const question = "What is our leave policy?";
const candidates: Chunk[] = [
{
id: "leave-policy",
text: "Employees request leave through the staff portal.",
allowedRoles: ["employee", "hr", "ceo"]
},
{
id: "hr-case-note",
text: "An individual employee's leave case is under review.",
allowedRoles: ["hr"]
}
];
const userRoles: Role[] = ["employee"];
const generationInput = {
question,
context: authorizedChunks(candidates, userRoles)
.map(({ id, text }) => ({ id, text }))
};
Here, generationInput includes leave-policy and excludes hr-case-note. The role values must come from authenticated identity, and the document attributes must come from trusted policy metadata. The example illustrates the permission boundary; an enterprise policy may depend on more attributes than roles alone.
Sensitive information requires protection beyond access checks. Protected health information (PHI) and personally identifiable information (PII) need appropriate handling during ingestion and throughout the stack. Failures in those controls can impose substantial organizational costs, making security and compliance part of the architecture rather than a final deployment task.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Who owns the incident?
A stack assembled from separate providers can have one vendor for the vector database, another for the LLM, another for embeddings and another for keyword search. Each relationship introduces contracts, onboarding and integration work. Those costs exist even when every component works as intended.
When something fails, the boundaries become harder to manage. Which component caused the problem? Who can diagnose it? Who coordinates support across providers? Mendelevitch’s concern about vendor chaos is especially an incident-resolution concern: the application owner needs a working answer while individual vendors may each locate the fault somewhere else. Integration ownership includes resolving failures that cross service boundaries.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Maintaining expertise and language coverage
The sixth cost is the team needed to keep the stack current and operational. RAG combines several specialties in a rapidly changing field:
| Area | Continuing responsibility |
|---|---|
| LLMs | Generation expertise |
| Embedding models | Representation expertise |
| Hybrid search | Retrieval expertise |
| Data engineering | Ingestion, ETL, document parsing and metadata |
| DevOps and SRE | Availability and continuous operation |
| Security | Protection of the running system |
LLM expertise and embedding-model expertise are distinct; knowing one does not automatically cover the other. The staffing challenge is recruiting and retaining the combination needed to support multiple generative AI use cases over time.
The seventh cost often appears after an English-only proof of concept. English enjoys broad component support, but an organization’s users may require other languages. Every component in the flow must support those languages. If one does not, replacing it can change the behavior and quality of the overall system. Language requirements therefore belong in the initial component selection, before the prototype’s choices become production dependencies.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the platform proposes to take over
These seven costs emerge as teams move from proofs of concept toward production. Vectara’s proposition is to take responsibility for extraction, encoding, indexing, the vector database and retrieval—including maintaining and upgrading those components. Application teams use APIs to upload data, index files and build query, chat or agentic RAG experiences.
Mendelevitch identifies three reasons enterprises choose that arrangement:
- Accuracy: retrieval of the right facts, response quality and reduced hallucination.
- Security: access controls and prevention of prompt attacks.
- Explainability: observability into the system and citations throughout the flow.
These are Vectara’s stated platform priorities. They connect the managed-service boundary to the operational responsibilities described earlier.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Checking groundedness and choosing a model
Hallucinations remain a barrier to adoption, particularly in regulated environments. Mendelevitch returns to that problem with an article from the preceding Thanksgiving, using its discussion of hallucinations to explain why retrieving evidence is not the end of the quality-control process. The generated answer still needs to be checked against that evidence.
Vectara integrates HHEM, the Hughes Hallucination Evaluation Model, into that flow. Its factual consistency score assesses whether generated text is supported by the supplied evidence; a higher score means greater consistency, not independent proof that the underlying evidence is true. Mendelevitch describes scoring as automatic with query responses. That historical behavior should be distinguished from later API documentation, which instructs callers to set enable_factual_consistency_score to true. The scoring claim concerns generated query answers, not indexing operations.
The open HHEM model is also available on Hugging Face. Mendelevitch reports over three million downloads since release. He also describes it as the most popular hallucination evaluator, although no comparative usage figures accompany that claim.
The Vectara Hallucination Leaderboard uses a commercial version of HHEM to compare LLM outputs. Its document-summarization task provides a view of factual consistency alongside a model’s other capabilities. Mendelevitch shows only the top of the leaderboard and reports that some models elsewhere in it have hallucination rates around 10% or 14%. Those are benchmark-specific examples: the talk does not identify the models or test conditions behind those percentages, and the current leaderboard is a different snapshot. They should not be read as general error rates for every use of those models.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Model choice and deployment constraints
The model-selection implication is practical: consider a model’s tendency to produce unsupported claims alongside its other capabilities. Deployment constraints matter as well. Mendelevitch reiterates that Vectara can run in a customer’s VPC or on premises, describing those options as requirements for many customers rather than incidental hosting preferences.
He closes with a QR-code invitation to try the platform and contact him. Mendelevitch offers a 30-day free trial, described in the talk as including all platform capabilities. For a team evaluating the managed approach, that invitation follows a specific question: which parts of ingestion, retrieval, generation and ongoing operation should the application team own?
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Open hallucination detector with examples for checking whether generated text is supported by supplied evidence.
LLM document-summarization benchmark with evaluation methodology and links to previous leaderboard versions.
Further reading
- HHEM 2.1 and its leaderboardArticle
The speaker's coauthored release article explains factual consistency scoring, open and commercial model variants, and detector benchmarks.
Ofer Mendelevitch's companion discussion of seven operational challenges in enterprise RAG.
Updates since the talk
Later API documentation explains how to enable scoring and interpret consistency with retrieved evidence.
Full transcript
Loading transcript…