AI Engineer World's Fair 2024

RAG for VPs of AI

Jerry Liu· CEO, LlamaIndex26:51

Loading slides…

Read the talk

RAG for VPs of AI: The Data Work Between Prototype and Production

Reliable enterprise RAG depends on how documents become usable context, how pipelines are evaluated, and how retrieval fits into deployment, conversation and agent workflows.

From a talk by Jerry Liu

Before you start: Familiarity with LLM prompts and basic database concepts is helpful; the article explains the RAG pipeline as it goes.

From a directory of files to an answer

Point an LLM application at a directory of files and ask it questions. That is the familiar promise of retrieval-augmented generation, or RAG: make private information available to a model so it can answer from that information. Enterprise demand extends beyond question answering to document extraction, ongoing conversations and agents. Jerry Liu reports seeing more constrained agents in production than fully autonomous ones.

The query side needs an LLM for final synthesis, an embedding model, and storage suited to the information being retrieved. That storage need not be exclusively a vector database: document stores, graph stores and SQL databases also have roles. Before any of those components can produce useful answers, an ingestion pipeline must turn source material into accessible context. A PDF has to be parsed, divided into chunks, indexed and represented in appropriate storage forms. This is the data-processing layer that LlamaIndex focuses on, and its requirements differ from traditional ETL built for analytics.

Diagram showing Data → Data Parsing + Ingestion → Index → Retrieval → LLM + Prompts → Response, grouped under Data Parsing & Ingestion and Data Querying.Open full source frame
The RAG pipeline separates data parsing and ingestion from data querying.
0:410:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:41 · section reference included

Why a working prototype is not enough

Liu describes building a roughly working RAG prototype with LlamaIndex tooling in ten minutes. The difficult transition comes afterward: document counts grow, layouts become more complex, additional sources enter the system, and the quality bar rises. Accuracy problems and a growing set of tuning parameters consume developer time. A proof of concept prepared for executives can stall before it delivers the value that justified the project.

Enterprise data silos compound the problem. Unstructured documents, structured records, semi-structured content and APIs all need to become usable by the application. A unified knowledge interface would let the model synthesize answers—and eventually take actions—across those sources. But a stronger model cannot compensate reliably for badly represented information. If ingestion loses the relationships in the original data, the resulting context can lead the application to hallucinate.

Slide titled “Any LLM App is only as Good as your Data,” with a flow from Raw Data through Data Processing and Clean Data to production applications: Q&A, Chat, Copilot, Autonomous Agents and Structured Extraction.Open full source frame
Good data quality is a necessary component of production LLM applications.
2:332:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:33 · section reference included

Build application expertise; manage the data plumbing

For an enterprise AI leader, Liu recommends investing somewhat more in building than in buying only out-of-the-box solutions. The reason is adaptability: procurement can address an immediate pain point, while developers can translate new models, techniques and workflows into applications specific to the business. That does not mean assigning developers endless ingestion troubleshooting. The tooling should reduce setup and maintenance work while improving response quality and supporting more sources.

The product split follows that division of responsibility. The open-source framework handles application orchestration; LlamaCloud manages the data infrastructure behind it.

LayerResponsibility
LlamaIndex frameworkRetrieval, prompting, agent reasoning and tool use
LlamaCloudSource integration, data processing and managed ingestion pipelines

After a period focused on productionizing RAG, Liu expects more agentic use cases over the following six months. LlamaCloud initially targets unstructured sources, including complex PDFs, PowerPoints and spreadsheets, so developers can spend more time on custom retrieval and application logic. The overview includes upcoming capabilities as well as available ones.

LlamaParse, a component of LlamaCloud, addresses the document-quality problem directly. Financial reports and presentations can contain messy text layouts, tables, images and diagrams; parsing must preserve enough of their structure for an LLM to interpret them. Liu reports half a million monthly client SDK downloads and tens of millions of pages processed within a few months of LlamaParse’s release. These are reported adoption figures, not accuracy measurements. At the time of the session, LlamaCloud had a general waitlist for managed processing and indexing. The broader application direction was from question answering toward actions and automated decision-making for internal and external teams.

5:055:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:05 · section reference included

Where sensitive documents go

Uploading sensitive enterprise documents raises an immediate deployment question. Liu describes two options in the session: a cloud service and a VPC deployment, with AWS and Azure supported and GCP coming later. He directs customers needing deployment details to the enterprise contact form. He also describes LlamaCloud as an orchestration layer that intentionally avoids storing customer data and integrates with existing storage systems. That statement describes the intended architecture; it does not establish a detailed zero-retention policy. These availability statements describe the offering at the time of the recording.

10:2810:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:28 · section reference included

What an ETL practitioner needs to learn

An audience member asks what additional skills an experienced ETL practitioner needs to build these pipelines. Beyond SQL or dbt transformations, the work introduces a sequence of decisions whose quality depends on the eventual answer:

  1. Parse the document. Use LlamaParse or another parser to produce text, or potentially a multimodal representation. Errors here propagate into later stages.
  2. Choose meaningful chunks. Liu’s naive example splits every 1,024 tokens. Such boundaries can cut a table in half or separate a section that spans pages.
  3. Preserve relationships and metadata. Join related content where necessary and attach metadata that helps the application interpret and retrieve it.
  4. Index the representation. Use a vector, graph or document store appropriate to the retrieval strategy.

The transformations are different from analytics ETL, but so is the feedback loop. A chunk size cannot be judged solely by inspecting the transformation locally. An evaluation dataset and end-to-end tests are needed to determine whether the resulting application answers well.

Audio introduces another representation choice. For an audience member building semantic chunking and paragraphing over transcripts, Liu suggests the simplest route: convert audio to text, then ingest that text. He tentatively recalls audio loaders in the framework. Representing audio itself as a chunk and feeding it directly to a multimodal model is the future alternative he sketches, rather than a capability he considers ready in this session.

The same end-to-end logic applies when choosing a parser. Asked for public datasets demonstrating LlamaParse’s superiority, Liu says a general benchmark is still being developed. The practical comparison is a customer bake-off: build RAG pipelines using LlamaParse and alternative parsers, then compare them on the enterprise’s own data. He describes notebooks supporting that comparison, but supplies no general superiority score.

11:2511:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:25 · section reference included

Pipeline promotion and a bank’s shared CMS

Production operation also requires control over pipeline changes. Asked about versioning and promotion between staging and production, Liu says pipeline-versioning features are requested by enterprise customers and under development, but not yet available. He identifies North America as the SaaS hosting region and describes on-prem deployments as an enterprise-plan option.

A bank’s servicing system makes the source-integration problem concrete. Its team maintains a chatbot, a WhatsApp chatbot, a help center and an IVR system. An existing CMS supplies Q&A pairs to those channels, but the RAG developer needs additional metadata and document groupings—for example, to distinguish use cases involving different credit cards. Can the same system remain the source of truth for all those experiences?

Liu does not settle whether that CMS is the right system. Instead, he separates the ingestion problem by source type:

  • Homogeneous sources: Financial reports with similar formats can generally share parsing parameters.
  • Heterogeneous sources: PDFs, semi-structured Jira records and Salesforce JSON typically need separate pipelines.
  • Combined retrieval: Retrieve across those pipelines, combine their results and apply a reranking layer at the top.

A shared answer interface can therefore sit above source-specific processing, without requiring every source to fit the same ingestion recipe.

15:2015:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:20 · section reference included

Support is part of the production contract

A LlamaParse user reports failures on a client project two weeks earlier. Support requested job IDs, then stopped responding, leaving the user asking how to obtain help over a weekend. Liu apologizes and acknowledges a cluster of failures. He says the company had fifteen people, with founding engineers often handling support, and promises a more streamlined process. The operational distinction is between the less formal support described for self-serve APIs and dedicated SLAs offered through enterprise plans; no specific response-time commitment is given in the session.

18:3318:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:33 · section reference included

Keep the relationships inside a table

An audience team building hallucination detection and evaluation over thousands of PDFs asks how to combine OCR with other PDF processing. Should LlamaParse handle the documents, or should they build a custom system on LlamaIndex? Liu starts with the minimum requirement: the parser must faithfully represent the text and its spatial relationships. Extra outputs such as bounding boxes do not compensate for an extraction that scrambles the content.

Tables expose the mechanism. A formatted table keeps labels, headers and values associated, giving the LLM a readable structure. A naive extraction can collapse the same text and numbers into a sequence that no longer makes those associations clear. Liu presents spatially aligned table output as a LlamaParse strength and identifies collapsed structure as a source of downstream hallucinations.

For a small illustrative financial table, the representation should retain which year each revenue value belongs to. Once extraction has recovered the correct cells, Python can serialize them into a readable table:

python

headers = ("Year", "Revenue ($m)")
rows = [("2022", "80"), ("2023", "95")]

lines = [
    "| " + " | ".join(headers) + " |",
    "| " + " | ".join("---" for _ in headers) + " |",
]
lines.extend("| " + " | ".join(row) + " |" for row in rows)
context = "\n".join(lines)
print(context)

The useful property is the preserved relationship between 2023 and 95, under the Revenue ($m) header. Formatting cannot recover that relationship if the parser has already lost it.

19:4219:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:42 · section reference included

Long context changes the unit of retrieval

Larger context windows and stronger needle-in-a-haystack results raise a natural question: how much RAG machinery will remain necessary? Liu separates parsing from chunking. Good parsing remains essential because a larger window still receives whatever representation the pipeline produces. Chunking can become simpler: larger chunks reduce the need to split within pages, and an entire document can become an indexed unit. Documents are often self-contained, making them a natural retrieval boundary.

The enterprise corpus still exceeds what it is practical to supply on every request. Liu invokes collections on the scale of billions of documents to explain why even long-context systems need selection. He considers context caching expensive for that use and raises concerns about accountability into the data supplied to a model. No cache size, reuse frequency or retention period is specified, so the cost judgment remains workload-dependent.

One technical distinction matters here: Liu describes caching as storing transformer weights, but context caching reuses input context; it is not training new weights on the documents. Google’s contemporaneous announcement for Gemini 1.5 Pro and Gemini 1.5 Flash describes repeated-token reuse as a cost-reduction feature. That correction leaves the architectural question intact: external vector or graph storage provides a way to select the information for each request. Liu expects fine-grained chunking decisions to diminish, while retrieval from external storage remains useful.

21:2921:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:29 · section reference included

Store more than one representation

Multimodality creates another option: retain different representations of the same source. A PowerPoint page can be stored as an image alongside its parsed text. Text extraction is inherently lossy; an image retains visual information that the textual representation may omit.

RepresentationWhat it retains
Parsed textExtracted wording and encoded structure
Page imageThe visible page, including layout and graphics

Keeping both lets an application choose among representations as model capabilities improve, balancing cost, performance and latency.

23:2123:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:21 · section reference included

From retrieval tools to agent services

Retrieval accuracy does not automatically produce a natural conversation. An audience member describes a system that overemphasizes contextual data when the user merely wants to provide an FYI and continue talking. Liu’s proposed architecture makes the RAG pipeline a tool available to an outer agent. That agent reasons over conversation history and decides what response fits the current turn. The main tuning points are its reasoning prompt and memory. He describes memory modules at the time as primitive, with few strong approaches beyond placing conversation history in the prompt.

The final question asks about the most complex internal use of Llama Agents. The framework, introduced at the preceding keynote, moves agents from notebooks into independently running API services. Liu’s answer is modest: internal use is mostly constrained, simple RAG pipelines, and the framework is still alpha.

Known limitations include the need for more general communication protocols and interfaces, plus a richer message-queue system. These are extensions to the service architecture, not a claim that the original framework lacked communication infrastructure altogether. The enterprise rationale is concrete: encapsulate an agentic capability behind a service boundary so it can be deployed and reused. The closing example therefore returns to constrained RAG, now packaged as a component other applications can call.

24:0624:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:06 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…