AI Engineer World's Fair 2024
The A to Z of Building AI Agents
Loading slides…
Read the talk
The A to Z of Building AI Agents
Build a research assistant by choosing where an agent is useful, connecting paper-search tools, composing an execution loop, and persisting conversation history in MongoDB.
From a talk by Apoorva Joshi and Ben Perlmutter
Before you start: Basic Python familiarity, comfort with Jupyter notebooks, and an understanding of LLM prompts will help you follow the implementation sections.
What does an agent add to an LLM?
When does an LLM application need an agent, and what do you actually have to build? This workshop starts with that decision, then turns the components into a working research-assistant exercise. The format is a conceptual introduction followed by hands-on work, with Tom, Ben, and Fabian helping participants troubleshoot. Apoorva Joshi introduces herself as a MongoDB developer advocate, five months into the role after roughly six years applying data science to cybersecurity problems such as phishing, malware, and ransomware detection. Questions and group work are encouraged throughout.
The slides and AI Agents Lab are the companion materials. In the room, participants access them through the displayed link and QR code, also printed on postcards.
The end-to-end deliverable is an AI research agent. Its defining capability is straightforward: an LLM reasons about a problem, creates a plan, and executes that plan through tools. The workshop develops those pieces before the exercises; a separate Q&A depends on the time remaining.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose prompting, retrieval, or an action loop
Simple prompting asks the model to answer from its pretrained, parametric knowledge. In that setup, even a carefully worded request cannot give it missing information or a way to execute an external task. A single response also provides no persistent mechanism for revising an answer or remembering a user's preferences. Retrieval-augmented generation, or RAG, adds information from a knowledge base, but retrieval alone does not supply an iterative execution or personalization loop.
An agent adds tools, access to past interactions, and repeated opportunities to decide what to do next. A tool result becomes input to the next model call: the agent can repeat a step, choose a follow-up tool, or finish. Persisted interactions can inform later responses. These capabilities introduce costs, too. Long-term planning remains difficult, and repeated calls add latency and expense in pursuit of better accuracy. Joshi regards agents as a way to get more from the models available at the time, rather than a reason to turn every question into an agent workflow.
The first audience exercise asks who was the first president of the United States. Joshi recommends simple prompting because most LLMs likely already contain that information. The next asks for a company's travel reimbursement policy. An audience member points out that the application must first establish the employee's location, because that determines which policy applies. Joshi acknowledges the qualification and still recommends RAG once the relevant context and knowledge base are available. A clarification step does not automatically require an autonomous agent.
The next request combines several capabilities: determine how average daily calorie intake among adults has changed over a decade, consider its possible impact on obesity rates, and graph the trend. Answering it requires data aggregation, visualization, and interpretation of the results. That makes it a useful agent task. A personalized learning assistant offers a different reason to use an agent: it must adjust its language, examples, and teaching methods as it observes a student's responses over time.
| Requirement | Suitable starting point |
|---|---|
| Answer a stable general-knowledge question | Prompt the model |
| Answer from company documents | Retrieve the relevant information |
| Combine research, analysis, and execution | Use tools in an agent loop |
| Adapt across interactions | Add persistent, relevant memory |
The useful boundary is the work the system must perform: combining question answering, task execution, and analysis toward an outcome, or adapting its responses over time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Plan, act, observe, and revise
Planning begins with prompting. The simplest approach asks the model to create a plan from its initial understanding and then follow it without revising the plan in response to new tool results. The workshop groups chain-of-thought and Tree of Thoughts under planning without feedback, and ReAct and reflection under planning with feedback. Here, the distinction concerns feedback from external actions: Tree of Thoughts still evaluates and revises its own search choices.
Chain-of-thought prompts the model to work through a problem step by step instead of immediately producing an answer. A zero-shot prompt requests that behavior directly; a few-shot prompt provides one or more worked examples. Tree of Thoughts expands the process into multiple candidate reasoning paths. The model generates and evaluates intermediate choices, while a search procedure supports lookahead and backtracking. That allows it to reconsider a branch rather than commit to the first path it generates.
ReAct brings external observations into the loop:
- The model identifies the next step toward solving the problem.
- It selects an action and supplies its inputs.
- The action runs, producing information the model can observe.
- The model uses that observation to choose another action or finish.
The loop ends when the model determines it has reached the answer, or a user intervenes. In the workshop example, the model selects a search tool and its arguments, examines the result, recognizes that the answer is still incomplete, and chooses a further step. The observation changes what happens next.
Reflection adds critique of past actions or generated answers before the next attempt. The same model can generate and critique, different models can take those roles, or separate agents can divide the work. In each arrangement, the generation–reflection cycle may repeat before producing a final answer. The tradeoff is additional computation for a chance at greater accuracy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decide what the agent should remember
Memory lets an agent store and recall interactions so that later responses can use earlier information. The workshop separates two scopes:
- Short-term memory: information within a single conversation.
- Long-term memory: information stored, updated, and retrieved across conversations, supporting personalization over time.
The distinction is about the scope of the remembered information, not simply whether it is stored in a database.
A short conversation can be retained as a message list. As it grows, passing the entire list to the model becomes less practical. Two options are to retrieve only the n most recent messages or summarize the conversation. Keeping recent messages drops older context; summarization condenses that context at the cost of some information loss.
Long-term memory demands more application design: which states matter, how should they be represented, and when should they change? Joshi describes this as a largely unexplored area at the time of the workshop and recommends narrowing the problem to an application-specific agent. A focused application has fewer relevant states to track and clearer rules for updating them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the model an explicit tool contract
Tools are the agent's interfaces to the external world. They can expose search or weather APIs, vector stores, or specialized machine-learning models. They are commonly represented as functions: the model selects a function and supplies arguments, and application code performs the call. Selecting an action and executing it are separate responsibilities.
LangChain can handle the calling machinery, but the model still needs a useful contract. Give each tool a descriptive name, connect it to the function that performs the work, describe what it does, and specify its argument types. The slide also includes whether the result should return directly to the user. These details help the model distinguish available capabilities and produce an appropriate request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Assemble the research assistant
The research assistant has three objectives: suggest papers to read, summarize papers, and answer questions about research topics. Its model is FireFunction v1 from Fireworks. FireFunction v2 had just appeared, but the workshop retains v1 because the materials were prepared before that release. The workshop's description of v1 as free and open source is historical: Fireworks released open weights and offered free hosted inference during a limited beta, not permanently free service. The exercises explore chain-of-thought and ReAct with this model.
| Component | Responsibility |
|---|---|
| Paper-summary tool | Summarize a research paper |
| Reading-list tool | Return relevant papers |
| Knowledge-base tool | Answer questions using MongoDB-backed retrieval |
| MongoDB chat storage | Persist short-term conversation memory |
These are distinct capabilities that the agent can combine as the request requires.
The notebook gives the hands-on work a consistent structure:
- Complete the open-hands exercises first; attempt superhero exercises if time permits.
- Fill in the Jupyter Notebook's
code_blockplaceholders. - Consult the documentation indicated by the books emoji before each relevant cell.
- Try the exercise before opening the supplied solution, then make sure you understand the solution's behavior.
The first hands-on interval allocates 15–20 minutes to prerequisites, beginning with the lab's MongoDB Atlas section and continuing through Dev Environment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Load papers and return the right representation
The lab remains self-paced and accessible after the workshop. Its next stage begins with two ways to obtain paper data. Hugging Face's Datasets library supplies load_dataset for downloading the arXiv embeddings dataset used to populate the knowledge base. LangChain's ArxivLoader loads papers directly from arXiv into document objects. Each document separates its raw text in page_content from extracted fields in metadata, including publication date, title, authors, and summary.
The loader serves both the prebuilt reading-list tool and the paper-summary tool. The @tool decorator turns a Python function into a LangChain tool. For ArxivLoader, query accepts a topic or paper ID, load_max_docs limits how many documents to download, and .load() produces the document objects. A reading list returns their metadata rather than their full text:
python
from langchain_community.document_loaders import ArxivLoader
from langchain_core.tools import tool
@tool
def get_relevant_papers(query: str, max_papers: int = 3) -> list[dict]:
"""Find arXiv papers by topic or paper ID and return their metadata."""
documents = ArxivLoader(
query=query,
load_max_docs=max_papers,
).load()
return [document.metadata for document in documents]
This keeps the tool's output aligned with its purpose: selecting papers to read. The summary tool can instead use the loaded paper content.
PyMongo handles direct database operations: connecting to databases and collections, deleting documents, and inserting documents to build the research knowledge base. The workshop then uses separate LangChain provider packages to keep integrations independently versioned and easier to manage and test.
| Integration | Role in the agent |
|---|---|
| LangChain MongoDB integration | Atlas vector storage and chat-history storage/retrieval |
langchain-huggingface | Open-source embedding models |
langchain-fireworks | Fireworks chat-completion models |
Retrieval, model inference, and conversation storage are separate responsibilities even when they participate in one agent workflow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compose workflows with runnables
LangChain Expression Language, or LCEL, connects prompts, processing steps, models, and tools declaratively. Each unit is a runnable. The pipe operator passes the output of the runnable on its left into the runnable on its right. A basic model chain therefore has the shape prompt | llm | output_parser; calling .invoke(...) runs it and returns the result. This is the same composition mechanism used to build and test the workshop's RAG and agent workflows.
RunnableLambda brings an ordinary Python function into that composition model. For example, the paper-metadata transformation can become one runnable, followed by a title-extraction step:
python
from langchain_core.runnables import RunnableLambda
def extract_metadata(documents):
return [document.metadata for document in documents]
def extract_titles(records):
return [record["Title"] for record in records]
def paper_titles(documents):
chain = (
RunnableLambda(extract_metadata)
| RunnableLambda(extract_titles)
)
return chain.invoke(documents)
The first function's returned list becomes the second function's input. The workshop allocates the next 20-minute exercise interval to Create Agent Tools.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate agent decisions from execution
With the tools in place, the next exercise builds the agent and revisits the reasoning patterns. The workshop uses LangChain's classic constructor-and-executor API. Start with create_tool_calling_agent, which assembles a runnable sequence containing a prompt template, a tool-aware LLM, and an output parser. The prompt includes an agent scratchpad placeholder for intermediate actions and observations. That scratchpad gives the next decision access to what has already happened during the task.
The alternative, create_react_agent, uses a ReAct prompt to guide successive reasoning and action steps. Its setup includes a stop sequence to control the model's generation and a parser for the ReAct-style structure: thought, action, action input, and observation. The two constructors share a broad structure, but differ in how the prompt and parser express the model's next action.
AgentExecutor is the runtime that turns those decisions into work. It calls the agent, executes the chosen action, returns the action's output to the agent, and repeats as necessary. The workshop's loop can be read as this procedure:
- Ask the agent for its next action or final answer.
- If it selects an action, execute the corresponding tool.
- Feed the tool result back into the agent's intermediate state.
- Continue until the agent finishes.
A constructor creates the decision-making sequence; the executor drives repeated decisions and tool calls. The next 20-minute exercise interval covers Create Agent and any unfinished earlier work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Persist the conversation by session
The final addition is short-term memory through access to chat-message history. RunnableWithMessageHistory wraps another runnable and manages its conversation history. The workshop applies this pattern to the agent workflow built with the tool-calling or ReAct constructor.
The wrapper accepts a function that supplies persistent message history; MongoDB is the storage layer here. A session ID, passed alongside the input query or prompt, identifies which conversation's history to use. Reusing a session ID lets subsequent requests draw on that conversation instead of starting without its earlier exchanges. This is persistent short-term memory: the stored object is still the history of a particular conversation.
The remaining exercise is to add that history layer and experiment with the research agent across turns, with help available for questions or blockers. The build ends with a model that can select research tools, a runtime that executes them and feeds back results, and a conversation history that survives beyond a single request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Source repository for the lab site, with agent-building exercises and instructions for running its documentation locally.
Historical introduction to the workshop's Mixtral-based function-calling model and its limited-beta availability.
Introduces the successor released shortly before the workshop, including multi-turn conversation and parallel function calling.
Explains deliberate reasoning through branching search, evaluation of intermediate thoughts, and backtracking.
Introduces a framework that interleaves reasoning with actions and incorporates observations from external environments.
Further reading
A public collection of arXiv paper metadata, abstracts, and embedding vectors for retrieval experiments.
Full transcript
Loading transcript…