AI Engineer World's Fair 2025

Effective agent design patterns in production

Laurie Voss· VP of Developer Relations, LlamaIndex15:38

Loading slides…

Read the talk

Effective agent design patterns in production

Build agents around messy inputs, selective retrieval, and explicit control flow, then combine chaining, routing, parallel work, and feedback to handle more demanding tasks.

From a talk by Laurie Voss

Before you start: Familiarity with LLM prompts, Python functions, type annotations, and async calls will help with the workflow example.

Getting from documents to an application

How do you get from a collection of documents to a useful generative AI application without spending all your time building plumbing? Laurie Voss, VP of Developer Relations at LlamaIndex, opens with the pieces his team provides for that job. The framework supports Python and TypeScript, with agents as a particular focus; the surrounding services handle document parsing, retrieval, and integrations.

LlamaParse handles complicated formats such as PDFs, Word documents, and PowerPoints. Parsing matters because an agent can only reason effectively about information it can understand. Voss attributes better agent quality to LlamaParse making those documents more intelligible to an LLM than unspecified open source parsers, though he provides no measured comparison.

In the talk’s product lineup, LlamaCloud is the paid service for putting documents in and getting a retrieval endpoint out, available as SaaS or deployed into a customer’s private cloud. LlamaHub supplies the integration registry: adapters for sources such as Notion, Slack, and databases, connections to vector stores and LLMs, and prebuilt agent tools. Voss reports that LlamaIndex supported 400 models across 80 providers at the time of the talk, including local models such as Llama 3. The framework’s practical promise is to remove boilerplate so developers can spend their limited time on the business problem.

LlamaHub slide showing Data Loaders, LLMs, Embedding Models, Agent Tools, Vector Stores, and Pre-built strategies.Open full source frame
LlamaHub’s integration categories.
0:160:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Use agent flexibility where the inputs are messy

Retrieval-augmented generation and agents are the two application categories Voss highlights. His working definition of an agent is semi-autonomous software that uses tools to achieve a goal without requiring the programmer to prescribe every step. Tools can retrieve information or take action; the LLM decides which ones to use. That decision-making power is useful when the application must deal with something unexpected or unknown, especially unstructured inputs.

Voss’s design heuristic is to look for tasks that turn a large body of text into a smaller, useful output. Interpreting a contract, processing an invoice, applying regulations, and summarizing documents all fit. The result might be a calendar event, a decision, a report, or an answer. He favors this reduction of existing material over asking a model to expand a small prompt into a large body of text.

The application does not have to be a chatbot. An LLM can convert messy input into structured data, make a decision about it, and feed the result into ordinary software. Voss characterizes chat as a 2023 usage pattern and sees a larger opportunity in embedding these capabilities into existing applications. The useful boundary is between the unstructured material the model interprets and the structured information the rest of the program consumes.

2:553:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:55 · section reference included

Retrieve the context the model needs

Unless the application is extremely generic, the model needs context about the company, domain, or particular problem. Asking questions without supplying that information is not enough. But a company may have far more data than it is practical to send with every request. This is the motivation for RAG.

The retrieval path has three stages:

  1. Embed the corpus. Represent source content as vectors in a searchable vector space.
  2. Embed and retrieve for the query. Represent the question in the same space and search for nearby content.
  3. Generate with the retrieved context. Give the model the relevant material rather than the whole corpus, then ask it to answer.

The slide makes the last step concrete with a cat-location question and the answer “The Mat.” Retrieval selects the context that makes the answer possible; generation uses that context to respond.

A diagram titled “RAG = infinite context” shows Embed, Retrieve, and Query stages, vector-space plots, and a cat-location question answered with “The Mat.”Open full source frame
RAG illustrated through embedding, retrieval, and answering with context.

Voss argues that larger context windows do not remove the value of retrieval: supplying less data reduces the amount the model must process, while more specific context can help it answer the question. His cost, speed, and quality claims here are a design rationale rather than a measured comparison. An agent can expose this retrieval-and-answering operation as one of its tools.

5:365:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:36 · section reference included

Give retrieval a control loop

Naive top-K RAG follows a fixed path: receive a query, retrieve the most relevant context, and send it to the LLM. That path cannot itself decide that the question needs to be decomposed or that an extraction should be repeated. An agent adds those decisions around retrieval.

The useful checks are specific:

  • Decompose the question. Could a complicated question be answered more easily as several simpler questions?
  • Retry extraction. Does the extracted data make sense, or should the system try again?
  • Review the answer. Did the response answer the question sensibly, or does it need another pass?

Voss reports improvements in speed and, especially, accuracy from agent layers in production RAG systems, without supplying measurements. Those extra decisions create opportunities to recover from a poor first result; they also introduce additional work.

The control-flow vocabulary comes from Anthropic’s December 2024 article Building effective agents: chaining, routing, parallelization, orchestrator-workers, and evaluator-optimizer. Anthropic describes these as workflow patterns and distinguishes prescribed workflows from agents that dynamically direct their own execution. It also describes the cost and latency tradeoffs of adding agentic behavior. Voss uses the five patterns to explain how to structure an agent application.

7:127:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:12 · section reference included

Start with a chain, then add routing

A chain passes the output of one LLM call into the next. In LlamaIndex workflows, regular Python functions perform the work, and type annotations determine how events pass between steps. The event type is the connection: a step emits an event that another step accepts.

A compact Python example can follow the invoice use case through two calls: extract the relevant information, then turn that extraction into a concise summary. The workflow structure below uses the event-based API style described in the talk:

python

from llama_index.core.workflow import (
    Event, StartEvent, StopEvent, Workflow, step,
)


class InvoiceExtracted(Event):
    details: str


class InvoiceSummary(Workflow):
    def __init__(self, llm, **kwargs):
        super().__init__(**kwargs)
        self.llm = llm

    @step
    async def extract(self, ev: StartEvent) -> InvoiceExtracted:
        response = await self.llm.acomplete(
            "Extract the vendor, invoice date, amount, and payment terms. "
            "Mark missing fields as unknown.\n\n" + ev.text
        )
        return InvoiceExtracted(details=str(response))

    @step
    async def summarize(self, ev: InvoiceExtracted) -> StopEvent:
        response = await self.llm.acomplete(
            "Summarize this invoice in one sentence, preserving unknowns.\n\n"
            + ev.details
        )
        return StopEvent(result=str(response))


async def summarize_invoice(llm, invoice_text: str) -> str:
    workflow = InvoiceSummary(llm=llm, timeout=60)
    return await workflow.run(text=invoice_text)

InvoiceExtracted carries the first call’s output to the second step. The workflow visualizer shown in the talk makes this sequential structure visible as a chain.

Routing adds a model-selected branch. Give the LLM several tools that solve different kinds of problems, or solve the same problem in different ways, and let it choose which path to follow. The chosen branch can contain an entire chain of its own. Voss skips the routing code, but the structural change is clear: the first decision selects a sequence of work rather than forcing every input through the same sequence.

8:478:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:47 · section reference included

Run independent work in parallel

Parallelization runs multiple LLM paths concurrently and aggregates their results. Its first flavor, sectioning, applies different operations to the same input. Voss’s example is a guardrail: one track processes the request or drafts an answer while another checks whether the request is illegal or violates the application’s rules. The two questions can be answered independently. If the guardrail rejects the request, its result can prevent the processing track’s answer from being delivered.

Slide titled “Parallelization: flavor 1 — Sectioning” shows an input branching to LLM Call 1, LLM Call 2, and LLM Call 3, then joining at an Aggregator and flowing to Out.Open full source frame
Sectioning runs three LLM calls in parallel before aggregating their outputs.

The second flavor, voting, sends exactly the same query to three tracks. Those tracks can use repeated calls to one nondeterministic LLM, or different models with different capabilities and specialties. An aggregation rule then requires a majority or unanimous agreement.

PatternWhat the tracks doHow results combine
SectioningDifferent operations on one inputCombine results or gate delivery
VotingAnswer the same questionRequire majority or unanimous agreement

Voss recommends voting to reduce hallucination, reasoning that different calls are unlikely to invent the same answer. Agreement is useful evidence, but it is not independent verification: models can share errors. The later study Correlated Errors in Large Language Models documents that limitation across models; it does not directly test this three-track example.

Both flavors use the same workflow machinery: emit multiple events, perform work concurrently, then collect the results. The difference is in what each worker is asked to do and what the collector decides. A guardrail collector decides whether an answer may be released; a voting collector decides whether the answers satisfy its agreement rule.

10:2110:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:21 · section reference included

Let an orchestrator determine the subtasks

In orchestrator-workers, an LLM looks at a complex task and decides how to split it into simpler tasks. A multipart question becomes several smaller questions, each of which can be assigned to a worker. This moves decomposition into the model’s decision-making process instead of requiring every subtask to be known in advance.

Voss uses deep research to explain the complete path:

  1. An orchestrator examines the broad research question and generates related questions to investigate.
  2. Workers answer those questions in parallel.
  3. A final stage aggregates the worker outputs into one coherent answer.

The synthesis step matters: a set of individual answers is not yet a response to the original question. The workflow must bring them back together. This pattern uses the same parallel execution mechanism introduced for sectioning and voting, but the model determines the work to distribute.

12:2412:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:24 · section reference included

Evaluate against the original goal, then revise

The evaluator-optimizer pattern, also called self-reflection, closes a feedback loop around generation. An LLM evaluates whether an output did a good job. To make that judgment meaningful, it receives both the output and the original question—or the original input and the goal. The criterion is whether the requested goal was reached.

If the output falls short, the evaluator returns feedback to the generating step. Voss’s examples are specific defects: the answer hallucinated something, or it missed part of the question. The generator then gets another attempt informed by that feedback. In a workflow, this is a loop: an event sends execution back to the first step instead of proceeding directly to completion.

13:0813:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:08 · section reference included

Compose workflows and hand control between agents

The patterns become more useful when combined. A workflow can route a request into a chain, distribute part of that chain across workers, and use an evaluator to send incomplete work back for revision. Voss closes by moving from these control-flow patterns to the tools and agents they coordinate.

A tool begins as a Python function supplied to an agent. Voss describes wrapping it in a step wrapper, but two concepts need to be kept separate: @step marks an event-handling workflow method, while an agent tool is a callable the model can select. The current FunctionAgent guide passes ordinary annotated functions directly in the tools list; a workflow step decorator is not required for those functions.

FunctionAgent takes a system prompt, an LLM, and a set of tools. Multiple configured agents can then be supplied to an AgentWorkflow, which supports passing control between them. The closing slide shows research, writing, and review agents, a root agent, and initial state fields for research notes, report content, and review. Those fields make the shared work visible: the system has more to coordinate than a sequence of isolated model responses.

“Multi-agent system” slide with an AgentWorkflow code block listing three agents, a root agent, and initial state fields for research notes, report content, and review.Open full source frame
A multi-agent workflow configured with research, writing, and review agents.

Voss calls the multi-agent construction technically one line of code. That describes assembling already configured agents into the workflow, not defining the complete application or all its coordination behavior. The talk ends with a pointer to a fuller notebook tutorial for building a deep-research workflow—the next step after learning the individual patterns and seeing how they fit together.

13:5614:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:56 · section reference included

Resources

From the talk

  • Anthropic’s guide to chaining, routing, parallelization, orchestrator-workers, and evaluator-optimizer workflows. The live article includes later tooling updates.

  • Registry of LlamaIndex data loaders, agent tools, packs, and datasets.

Updates since the talk

Transcript download
Full transcript

Loading transcript…