AI Engineer World's Fair 2024

The A to Z of Building AI Agents

Loading slides…

Read the talk

The A to Z of Building AI Agents

Build a research assistant by choosing where an agent is useful, connecting paper-search tools, composing an execution loop, and persisting conversation history in MongoDB.

From a talk by Apoorva Joshi and Ben Perlmutter

Before you start: Basic Python familiarity, comfort with Jupyter notebooks, and an understanding of LLM prompts will help you follow the implementation sections.

What does an agent add to an LLM?

When does an LLM application need an agent, and what do you actually have to build? This workshop starts with that decision, then turns the components into a working research-assistant exercise. The format is a conceptual introduction followed by hands-on work, with Tom, Ben, and Fabian helping participants troubleshoot. Apoorva Joshi introduces herself as a MongoDB developer advocate, five months into the role after roughly six years applying data science to cybersecurity problems such as phishing, malware, and ransomware detection. Questions and group work are encouraged throughout.

The slides and AI Agents Lab are the companion materials. In the room, participants access them through the displayed link and QR code, also printed on postcards.

Lab Content slide displaying mdb.link/agents-workshop-aiewf above a large QR code, with the presenter inset at lower left.Open full source frame
Workshop lab content link and QR code.

The end-to-end deliverable is an AI research agent. Its defining capability is straightforward: an LLM reasons about a problem, creates a plan, and executes that plan through tools. The workshop develops those pieces before the exercises; a separate Q&A depends on the time remaining.

0:210:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:21 · section reference included

Choose prompting, retrieval, or an action loop

Simple prompting asks the model to answer from its pretrained, parametric knowledge. In that setup, even a carefully worded request cannot give it missing information or a way to execute an external task. A single response also provides no persistent mechanism for revising an answer or remembering a user's preferences. Retrieval-augmented generation, or RAG, adds information from a knowledge base, but retrieval alone does not supply an iterative execution or personalization loop.

An agent adds tools, access to past interactions, and repeated opportunities to decide what to do next. A tool result becomes input to the next model call: the agent can repeat a step, choose a follow-up tool, or finish. Persisted interactions can inform later responses. These capabilities introduce costs, too. Long-term planning remains difficult, and repeated calls add latency and expense in pursuit of better accuracy. Joshi regards agents as a way to get more from the models available at the time, rather than a reason to turn every question into an agent workflow.

The first audience exercise asks who was the first president of the United States. Joshi recommends simple prompting because most LLMs likely already contain that information. The next asks for a company's travel reimbursement policy. An audience member points out that the application must first establish the employee's location, because that determines which policy applies. Joshi acknowledges the qualification and still recommends RAG once the relevant context and knowledge base are available. A clarification step does not automatically require an autonomous agent.

The next request combines several capabilities: determine how average daily calorie intake among adults has changed over a decade, consider its possible impact on obesity rates, and graph the trend. Answering it requires data aggregation, visualization, and interpretation of the results. That makes it a useful agent task. A personalized learning assistant offers a different reason to use an agent: it must adjust its language, examples, and teaching methods as it observes a student's responses over time.

RequirementSuitable starting point
Answer a stable general-knowledge questionPrompt the model
Answer from company documentsRetrieve the relevant information
Combine research, analysis, and executionUse tools in an agent loop
Adapt across interactionsAdd persistent, relevant memory

The useful boundary is the work the system must perform: combining question answering, task execution, and analysis toward an outcome, or adapting its responses over time.

3:223:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:22 · section reference included

Plan, act, observe, and revise

Planning begins with prompting. The simplest approach asks the model to create a plan from its initial understanding and then follow it without revising the plan in response to new tool results. The workshop groups chain-of-thought and Tree of Thoughts under planning without feedback, and ReAct and reflection under planning with feedback. Here, the distinction concerns feedback from external actions: Tree of Thoughts still evaluates and revises its own search choices.

Chain-of-thought prompts the model to work through a problem step by step instead of immediately producing an answer. A zero-shot prompt requests that behavior directly; a few-shot prompt provides one or more worked examples. Tree of Thoughts expands the process into multiple candidate reasoning paths. The model generates and evaluates intermediate choices, while a search procedure supports lookahead and backtracking. That allows it to reconsider a branch rather than commit to the first path it generates.

ReAct brings external observations into the loop:

  1. The model identifies the next step toward solving the problem.
  2. It selects an action and supplies its inputs.
  3. The action runs, producing information the model can observe.
  4. The model uses that observation to choose another action or finish.

The loop ends when the model determines it has reached the answer, or a user intervenes. In the workshop example, the model selects a search tool and its arguments, examines the result, recognizes that the answer is still incomplete, and chooses a further step. The observation changes what happens next.

Reflection adds critique of past actions or generated answers before the next attempt. The same model can generate and critique, different models can take those roles, or separate agents can divide the work. In each arrangement, the generation–reflection cycle may repeat before producing a final answer. The tradeoff is additional computation for a chance at greater accuracy.

8:539:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:53 · section reference included

Decide what the agent should remember

Memory lets an agent store and recall interactions so that later responses can use earlier information. The workshop separates two scopes:

  • Short-term memory: information within a single conversation.
  • Long-term memory: information stored, updated, and retrieved across conversations, supporting personalization over time.

The distinction is about the scope of the remembered information, not simply whether it is stored in a database.

A short conversation can be retained as a message list. As it grows, passing the entire list to the model becomes less practical. Two options are to retrieve only the n most recent messages or summarize the conversation. Keeping recent messages drops older context; summarization condenses that context at the cost of some information loss.

Long-term memory demands more application design: which states matter, how should they be represented, and when should they change? Joshi describes this as a largely unexplored area at the time of the workshop and recommends narrowing the problem to an application-specific agent. A focused application has fewer relevant states to track and clearer rules for updating them.

12:4312:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:43 · section reference included

Give the model an explicit tool contract

Tools are the agent's interfaces to the external world. They can expose search or weather APIs, vector stores, or specialized machine-learning models. They are commonly represented as functions: the model selects a function and supplies arguments, and application code performs the call. Selecting an action and executing it are separate responsibilities.

LangChain can handle the calling machinery, but the model still needs a useful contract. Give each tool a descriptive name, connect it to the function that performs the work, describe what it does, and specify its argument types. The slide also includes whether the result should return directly to the user. These details help the model distinguish available capabilities and produce an appropriate request.

Tools slide showing a short Python function beside five items: tool name, description, argument schema, function to call, and whether to return the result directly to the user.Open full source frame
A tool function and the information that describes its contract.
14:3514:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:35 · section reference included

Assemble the research assistant

The research assistant has three objectives: suggest papers to read, summarize papers, and answer questions about research topics. Its model is FireFunction v1 from Fireworks. FireFunction v2 had just appeared, but the workshop retains v1 because the materials were prepared before that release. The workshop's description of v1 as free and open source is historical: Fireworks released open weights and offered free hosted inference during a limited beta, not permanently free service. The exercises explore chain-of-thought and ReAct with this model.

ComponentResponsibility
Paper-summary toolSummarize a research paper
Reading-list toolReturn relevant papers
Knowledge-base toolAnswer questions using MongoDB-backed retrieval
MongoDB chat storagePersist short-term conversation memory

These are distinct capabilities that the agent can combine as the request requires.

The notebook gives the hands-on work a consistent structure:

  1. Complete the open-hands exercises first; attempt superhero exercises if time permits.
  2. Fill in the Jupyter Notebook's code_block placeholders.
  3. Consult the documentation indicated by the books emoji before each relevant cell.
  4. Try the exercise before opening the supplied solution, then make sure you understand the solution's behavior.

The first hands-on interval allocates 15–20 minutes to prerequisites, beginning with the lab's MongoDB Atlas section and continuing through Dev Environment.

15:4615:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:46 · section reference included

Load papers and return the right representation

The lab remains self-paced and accessible after the workshop. Its next stage begins with two ways to obtain paper data. Hugging Face's Datasets library supplies load_dataset for downloading the arXiv embeddings dataset used to populate the knowledge base. LangChain's ArxivLoader loads papers directly from arXiv into document objects. Each document separates its raw text in page_content from extracted fields in metadata, including publication date, title, authors, and summary.

The loader serves both the prebuilt reading-list tool and the paper-summary tool. The @tool decorator turns a Python function into a LangChain tool. For ArxivLoader, query accepts a topic or paper ID, load_max_docs limits how many documents to download, and .load() produces the document objects. A reading list returns their metadata rather than their full text:

python

from langchain_community.document_loaders import ArxivLoader
from langchain_core.tools import tool

@tool
def get_relevant_papers(query: str, max_papers: int = 3) -> list[dict]:
    """Find arXiv papers by topic or paper ID and return their metadata."""
    documents = ArxivLoader(
        query=query,
        load_max_docs=max_papers,
    ).load()
    return [document.metadata for document in documents]

This keeps the tool's output aligned with its purpose: selecting papers to read. The summary tool can instead use the loaded paper content.

PyMongo handles direct database operations: connecting to databases and collections, deleting documents, and inserting documents to build the research knowledge base. The workshop then uses separate LangChain provider packages to keep integrations independently versioned and easier to manage and test.

IntegrationRole in the agent
LangChain MongoDB integrationAtlas vector storage and chat-history storage/retrieval
langchain-huggingfaceOpen-source embedding models
langchain-fireworksFireworks chat-completion models

Retrieval, model inference, and conversation storage are separate responsibilities even when they participate in one agent workflow.

18:5619:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:56 · section reference included

Compose workflows with runnables

LangChain Expression Language, or LCEL, connects prompts, processing steps, models, and tools declaratively. Each unit is a runnable. The pipe operator passes the output of the runnable on its left into the runnable on its right. A basic model chain therefore has the shape prompt | llm | output_parser; calling .invoke(...) runs it and returns the result. This is the same composition mechanism used to build and test the workshop's RAG and agent workflows.

RunnableLambda brings an ordinary Python function into that composition model. For example, the paper-metadata transformation can become one runnable, followed by a title-extraction step:

python

from langchain_core.runnables import RunnableLambda


def extract_metadata(documents):
    return [document.metadata for document in documents]


def extract_titles(records):
    return [record["Title"] for record in records]


def paper_titles(documents):
    chain = (
        RunnableLambda(extract_metadata)
        | RunnableLambda(extract_titles)
    )
    return chain.invoke(documents)

The first function's returned list becomes the second function's input. The workshop allocates the next 20-minute exercise interval to Create Agent Tools.

21:4522:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:45 · section reference included

Separate agent decisions from execution

With the tools in place, the next exercise builds the agent and revisits the reasoning patterns. The workshop uses LangChain's classic constructor-and-executor API. Start with create_tool_calling_agent, which assembles a runnable sequence containing a prompt template, a tool-aware LLM, and an output parser. The prompt includes an agent scratchpad placeholder for intermediate actions and observations. That scratchpad gives the next decision access to what has already happened during the task.

The alternative, create_react_agent, uses a ReAct prompt to guide successive reasoning and action steps. Its setup includes a stop sequence to control the model's generation and a parser for the ReAct-style structure: thought, action, action input, and observation. The two constructors share a broad structure, but differ in how the prompt and parser express the model's next action.

AgentExecutor is the runtime that turns those decisions into work. It calls the agent, executes the chosen action, returns the action's output to the agent, and repeats as necessary. The workshop's loop can be read as this procedure:

  1. Ask the agent for its next action or final answer.
  2. If it selects an action, execute the corresponding tool.
  3. Feed the tool result back into the agent's intermediate state.
  4. Continue until the agent finishes.

A constructor creates the decision-making sequence; the executor drives repeated decisions and tool calls. The next 20-minute exercise interval covers Create Agent and any unfinished earlier work.

23:0123:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

23:01 · section reference included

Persist the conversation by session

The final addition is short-term memory through access to chat-message history. RunnableWithMessageHistory wraps another runnable and manages its conversation history. The workshop applies this pattern to the agent workflow built with the tool-calling or ReAct constructor.

The wrapper accepts a function that supplies persistent message history; MongoDB is the storage layer here. A session ID, passed alongside the input query or prompt, identifies which conversation's history to use. Reusing a session ID lets subsequent requests draw on that conversation instead of starting without its earlier exchanges. This is persistent short-term memory: the stored object is still the history of a particular conversation.

RunnableWithMessageHistory slide with two bullets describing management of another Runnable's chat history and organization by session ID.Open full source frame
RunnableWithMessageHistory manages chat history by session ID.

The remaining exercise is to add that history layer and experiment with the research agent across turns, with help available for questions or blockers. The build ends with a model that can select research tools, a runtime that executes them and feeds back results, and a conversation history that survives beyond a single request.

25:3325:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

25:33 · section reference included

Resources

From the talk

  • Source repository for the lab site, with agent-building exercises and instructions for running its documentation locally.

  • Historical introduction to the workshop's Mixtral-based function-calling model and its limited-beta availability.

  • Introduces the successor released shortly before the workshop, including multi-turn conversation and parallel function calling.

  • Tree of ThoughtsPaper9:36

    Explains deliberate reasoning through branching search, evaluation of intermediate thoughts, and backtracking.

  • ReActPaper9:53

    Introduces a framework that interleaves reasoning with actions and incorporates observations from external environments.

Transcript download
Full transcript

Loading transcript…