AI Engineer World's Fair 2024
RAG for VPs of AI
Loading slides…
Read the talk
RAG for VPs of AI: The Data Work Between Prototype and Production
Reliable enterprise RAG depends on how documents become usable context, how pipelines are evaluated, and how retrieval fits into deployment, conversation and agent workflows.
From a talk by Jerry Liu
Before you start: Familiarity with LLM prompts and basic database concepts is helpful; the article explains the RAG pipeline as it goes.
From a directory of files to an answer
Point an LLM application at a directory of files and ask it questions. That is the familiar promise of retrieval-augmented generation, or RAG: make private information available to a model so it can answer from that information. Enterprise demand extends beyond question answering to document extraction, ongoing conversations and agents. Jerry Liu reports seeing more constrained agents in production than fully autonomous ones.
The query side needs an LLM for final synthesis, an embedding model, and storage suited to the information being retrieved. That storage need not be exclusively a vector database: document stores, graph stores and SQL databases also have roles. Before any of those components can produce useful answers, an ingestion pipeline must turn source material into accessible context. A PDF has to be parsed, divided into chunks, indexed and represented in appropriate storage forms. This is the data-processing layer that LlamaIndex focuses on, and its requirements differ from traditional ETL built for analytics.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why a working prototype is not enough
Liu describes building a roughly working RAG prototype with LlamaIndex tooling in ten minutes. The difficult transition comes afterward: document counts grow, layouts become more complex, additional sources enter the system, and the quality bar rises. Accuracy problems and a growing set of tuning parameters consume developer time. A proof of concept prepared for executives can stall before it delivers the value that justified the project.
Enterprise data silos compound the problem. Unstructured documents, structured records, semi-structured content and APIs all need to become usable by the application. A unified knowledge interface would let the model synthesize answers—and eventually take actions—across those sources. But a stronger model cannot compensate reliably for badly represented information. If ingestion loses the relationships in the original data, the resulting context can lead the application to hallucinate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build application expertise; manage the data plumbing
For an enterprise AI leader, Liu recommends investing somewhat more in building than in buying only out-of-the-box solutions. The reason is adaptability: procurement can address an immediate pain point, while developers can translate new models, techniques and workflows into applications specific to the business. That does not mean assigning developers endless ingestion troubleshooting. The tooling should reduce setup and maintenance work while improving response quality and supporting more sources.
The product split follows that division of responsibility. The open-source framework handles application orchestration; LlamaCloud manages the data infrastructure behind it.
| Layer | Responsibility |
|---|---|
| LlamaIndex framework | Retrieval, prompting, agent reasoning and tool use |
| LlamaCloud | Source integration, data processing and managed ingestion pipelines |
After a period focused on productionizing RAG, Liu expects more agentic use cases over the following six months. LlamaCloud initially targets unstructured sources, including complex PDFs, PowerPoints and spreadsheets, so developers can spend more time on custom retrieval and application logic. The overview includes upcoming capabilities as well as available ones.
LlamaParse, a component of LlamaCloud, addresses the document-quality problem directly. Financial reports and presentations can contain messy text layouts, tables, images and diagrams; parsing must preserve enough of their structure for an LLM to interpret them. Liu reports half a million monthly client SDK downloads and tens of millions of pages processed within a few months of LlamaParse’s release. These are reported adoption figures, not accuracy measurements. At the time of the session, LlamaCloud had a general waitlist for managed processing and indexing. The broader application direction was from question answering toward actions and automated decision-making for internal and external teams.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where sensitive documents go
Uploading sensitive enterprise documents raises an immediate deployment question. Liu describes two options in the session: a cloud service and a VPC deployment, with AWS and Azure supported and GCP coming later. He directs customers needing deployment details to the enterprise contact form. He also describes LlamaCloud as an orchestration layer that intentionally avoids storing customer data and integrates with existing storage systems. That statement describes the intended architecture; it does not establish a detailed zero-retention policy. These availability statements describe the offering at the time of the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What an ETL practitioner needs to learn
An audience member asks what additional skills an experienced ETL practitioner needs to build these pipelines. Beyond SQL or dbt transformations, the work introduces a sequence of decisions whose quality depends on the eventual answer:
- Parse the document. Use LlamaParse or another parser to produce text, or potentially a multimodal representation. Errors here propagate into later stages.
- Choose meaningful chunks. Liu’s naive example splits every 1,024 tokens. Such boundaries can cut a table in half or separate a section that spans pages.
- Preserve relationships and metadata. Join related content where necessary and attach metadata that helps the application interpret and retrieve it.
- Index the representation. Use a vector, graph or document store appropriate to the retrieval strategy.
The transformations are different from analytics ETL, but so is the feedback loop. A chunk size cannot be judged solely by inspecting the transformation locally. An evaluation dataset and end-to-end tests are needed to determine whether the resulting application answers well.
Audio introduces another representation choice. For an audience member building semantic chunking and paragraphing over transcripts, Liu suggests the simplest route: convert audio to text, then ingest that text. He tentatively recalls audio loaders in the framework. Representing audio itself as a chunk and feeding it directly to a multimodal model is the future alternative he sketches, rather than a capability he considers ready in this session.
The same end-to-end logic applies when choosing a parser. Asked for public datasets demonstrating LlamaParse’s superiority, Liu says a general benchmark is still being developed. The practical comparison is a customer bake-off: build RAG pipelines using LlamaParse and alternative parsers, then compare them on the enterprise’s own data. He describes notebooks supporting that comparison, but supplies no general superiority score.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Support is part of the production contract
A LlamaParse user reports failures on a client project two weeks earlier. Support requested job IDs, then stopped responding, leaving the user asking how to obtain help over a weekend. Liu apologizes and acknowledges a cluster of failures. He says the company had fifteen people, with founding engineers often handling support, and promises a more streamlined process. The operational distinction is between the less formal support described for self-serve APIs and dedicated SLAs offered through enterprise plans; no specific response-time commitment is given in the session.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the relationships inside a table
An audience team building hallucination detection and evaluation over thousands of PDFs asks how to combine OCR with other PDF processing. Should LlamaParse handle the documents, or should they build a custom system on LlamaIndex? Liu starts with the minimum requirement: the parser must faithfully represent the text and its spatial relationships. Extra outputs such as bounding boxes do not compensate for an extraction that scrambles the content.
Tables expose the mechanism. A formatted table keeps labels, headers and values associated, giving the LLM a readable structure. A naive extraction can collapse the same text and numbers into a sequence that no longer makes those associations clear. Liu presents spatially aligned table output as a LlamaParse strength and identifies collapsed structure as a source of downstream hallucinations.
For a small illustrative financial table, the representation should retain which year each revenue value belongs to. Once extraction has recovered the correct cells, Python can serialize them into a readable table:
python
headers = ("Year", "Revenue ($m)")
rows = [("2022", "80"), ("2023", "95")]
lines = [
"| " + " | ".join(headers) + " |",
"| " + " | ".join("---" for _ in headers) + " |",
]
lines.extend("| " + " | ".join(row) + " |" for row in rows)
context = "\n".join(lines)
print(context)
The useful property is the preserved relationship between 2023 and 95, under the Revenue ($m) header. Formatting cannot recover that relationship if the parser has already lost it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Long context changes the unit of retrieval
Larger context windows and stronger needle-in-a-haystack results raise a natural question: how much RAG machinery will remain necessary? Liu separates parsing from chunking. Good parsing remains essential because a larger window still receives whatever representation the pipeline produces. Chunking can become simpler: larger chunks reduce the need to split within pages, and an entire document can become an indexed unit. Documents are often self-contained, making them a natural retrieval boundary.
The enterprise corpus still exceeds what it is practical to supply on every request. Liu invokes collections on the scale of billions of documents to explain why even long-context systems need selection. He considers context caching expensive for that use and raises concerns about accountability into the data supplied to a model. No cache size, reuse frequency or retention period is specified, so the cost judgment remains workload-dependent.
One technical distinction matters here: Liu describes caching as storing transformer weights, but context caching reuses input context; it is not training new weights on the documents. Google’s contemporaneous announcement for Gemini 1.5 Pro and Gemini 1.5 Flash describes repeated-token reuse as a cost-reduction feature. That correction leaves the architectural question intact: external vector or graph storage provides a way to select the information for each request. Liu expects fine-grained chunking decisions to diminish, while retrieval from external storage remains useful.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Store more than one representation
Multimodality creates another option: retain different representations of the same source. A PowerPoint page can be stored as an image alongside its parsed text. Text extraction is inherently lossy; an image retains visual information that the textual representation may omit.
| Representation | What it retains |
|---|---|
| Parsed text | Extracted wording and encoded structure |
| Page image | The visible page, including layout and graphics |
Keeping both lets an application choose among representations as model capabilities improve, balancing cost, performance and latency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From retrieval tools to agent services
Retrieval accuracy does not automatically produce a natural conversation. An audience member describes a system that overemphasizes contextual data when the user merely wants to provide an FYI and continue talking. Liu’s proposed architecture makes the RAG pipeline a tool available to an outer agent. That agent reasons over conversation history and decides what response fits the current turn. The main tuning points are its reasoning prompt and memory. He describes memory modules at the time as primitive, with few strong approaches beyond placing conversation history in the prompt.
The final question asks about the most complex internal use of Llama Agents. The framework, introduced at the preceding keynote, moves agents from notebooks into independently running API services. Liu’s answer is modest: internal use is mostly constrained, simple RAG pipelines, and the framework is still alpha.
Known limitations include the need for more general communication protocols and interfaces, plus a richer message-queue system. These are extensions to the service architecture, not a claim that the original framework lacked communication infrastructure altogether. The enterprise rationale is concrete: encapsulate an agentic capability behind a service boundary so it can be deployed and reused. The closing example therefore returns to constrained RAG, now packaged as a component other applications can call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source code and getting-started material for the LlamaIndex framework.
Further reading
Jerry Liu's launch explanation of managed data pipelines, structured PDF parsing and financial-document RAG examples.
The 2024 service-oriented agent architecture, including its control plane, message queue and historical setup examples.
Google's contemporaneous announcement explains repeated-context reuse for Gemini 1.5 Pro and Flash.
Updates since the talk
Documentation for reusing cached input context with the Generate Content API.
Full transcript
Loading transcript…