AI Engineer World's Fair 2025
Building Multimodal AI Agents (From Scratch)
Loading slides…
Read the talk
Building Multimodal AI Agents (From Scratch)
A document assistant needs more than a vision model: it needs a tool loop, retrieval that preserves visual context, and memory that carries the conversation forward.
From a talk by Apoorva Joshi
Before you start: Basic Python and familiarity with LLM prompts are sufficient; prior experience with retrieval-augmented generation will help with the retrieval sections.
What makes an assistant an agent?
What does it take to build an assistant that can reason about a request, use tools, and work with images as well as text? Apoorva Joshi’s workshop approaches that question in plain Python, starting with the components before assembling a multimodal agent. Joshi is an AI-focused developer advocate at MongoDB, with a background in cybersecurity data science. The workshop moves from agent fundamentals to multimodality, then into a hands-on implementation.
An agent uses an LLM to reason through a problem, create a plan, and execute and revise that plan with tools. The useful distinction is how much responsibility the application gives the model for deciding what happens next.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose how much autonomy the task needs
Joshi distinguishes three interaction patterns:
| Pattern | Information available | How work proceeds |
|---|---|---|
| Simple prompting | Pretrained, parametric knowledge | Answer the supplied prompt |
| Retrieval-augmented generation | External information added as context | Retrieve information, then answer |
| Agent | Tools, their results, and past interactions | Select actions and revise the next step |
In the simple prompting baseline, the model has no external source for missing information. RAG supplies that information and can support light personalization when the retrieved sources contain relevant user context. In Joshi’s comparison, neither baseline supplies the autonomous, multistep feedback loop that lets an agent change its approach after seeing a result. The agent makes those sequencing decisions itself.
Use agents when the task warrants their additional cost and latency. Planning, tool use, and correction can require repeated model calls. Good candidates include workflows whose steps are difficult to predict, tasks that tolerate slower responses and variable outputs, and applications that benefit from personalization over time. The same input need not produce the same result; Joshi notes that an agent’s sequence of decisions amplifies this variability. The workshop deliberately uses a simple problem to make the machinery understandable, even though a production version might need less autonomy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inputs, plans, and feedback
The agent has four components: perception gathers information, planning and reasoning determine an approach, tools act on the outside world, and memory retains past interactions. The resemblance to human problem-solving is intentional: these components give the software enough autonomy to carry out a task.
Perception begins with inputs. A person might submit a query, or an email or Slack message might trigger the application. Text is one input format; images, voice, and video expand what the agent can observe. This implementation limits its scope to text and images.
The LLM supplies planning and reasoning, guided by a prompt. Joshi first describes planning without feedback: the model forms a plan from its initial understanding without revising it in response to tool outcomes. She presents chain-of-thought prompting in this category. A zero-shot instruction asks the model to work step by step; a few-shot prompt supplies worked examples that guide its approach to the next problem.
Planning with feedback lets new observations change the next action. The workshop uses ReAct, which interleaves reasoning and acting:
- Determine an action that could advance the task.
- Execute the selected tool.
- Return the result as an observation.
- Use that observation to decide what to do next.
- Continue until the model can produce a final answer.
The operational difference is the return path: tool results become inputs to another decision, rather than merely outputs of a fixed plan.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model requests a tool; Python executes it
Tools can expose weather and search APIs, vector stores, or specialized machine-learning models. They are usually represented as functions. A tool-capable LLM can select a function and supply its arguments, but the application must execute the function. A generated tool call is a request for an action, not evidence that the action happened.
The model also needs a tool schema: the function’s name, a description of its purpose, and its parameters with types and descriptions. Joshi mentions JSON schemas and MCP as ways of exposing tool definitions. Her weather example accepts a city as a string. In Python, that contract can be represented directly:
python
weather_schema = {
"name": "get_weather",
"description": "Get today's weather for a city.",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string",
"description": "The city to get weather for.",
}
},
"required": ["city"],
},
}
This describes the available operation. The weather client and the code that dispatches a model’s request to it remain separate responsibilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow a weather question through the loop
Memory completes the component model. Short-term memory stores and retrieves information within one conversation. Long-term memory stores, updates, and retrieves information across conversations, supporting personalization over time. The lab implements short-term memory; cross-conversation learning is outside its scope.
Consider the workshop’s weather question: what is the weather in San Francisco today? The agent is the surrounding software system, which forwards that query to an LLM. The model has descriptions of weather and search tools, along with past interactions. It selects the weather tool and extracts San Francisco as the city argument.
Python then calls the weather API with the selected arguments and returns the API’s response to the model. At that point, the model can request another tool if it still needs information, or generate a natural-language answer from the returned temperature. This is the complete feedback loop: request, execution, observation, and a new decision. The model never has to be the component that performs the network request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Retrieve documents and interpret figures
Multimodality extends the inputs a model can process and, depending on the model, the outputs it can generate. Research papers, financial reports, organizational reports, and healthcare documents routinely mix prose with graphs, images, and tables. Understanding these documents requires preserving relationships across those forms of information.
Two model roles matter:
- Multimodal embedding models turn different input types into vectors for retrieval through methods such as vector search, hybrid search, or graph-based retrieval.
- Multimodal LLMs interpret those inputs and generate responses. Joshi cites DeepSeek, Claude, and ChatGPT voice mode as examples of the broader multimodal model landscape; supported input and output formats depend on the particular model.
Give a multimodal LLM tools for searching multimodal data, then let it use the results to solve a problem, and the tool loop becomes a multimodal agent.
The Python assistant has two objectives: answer questions over a large document corpus, and explain or analyze a chart or diagram supplied by the user. Joshi’s everyday example is taking a screenshot of an equation in a research paper and asking Claude to explain it. The additional challenge for this agent is searching a corpus where text, images, and tables are interleaved. Before the agent can reason about a relevant figure, the retrieval system has to find it without stripping away its context.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What extraction pipelines lose
A text-only RAG pipeline typically chunks documents, embeds the chunks, and retrieves relevant chunks as context. Mixed documents require additional handling. Joshi names LlamaParse and Unstructured as tools for extracting text, images, and tables, then describes two ways to make the extracted elements searchable.
| Pipeline | Text handling | Image and table handling |
|---|---|---|
| Convert to text | Chunk and embed text | Summarize, then embed summaries |
| Embed multiple modalities | Chunk and embed text | Embed visual elements directly |
The first path brings everything into the text domain, using a text embedding model for both prose and visual summaries. The second uses a multimodal embedding model for the extracted elements. Both still depend on separating the document into pieces.
That separation creates two costs. Context can disappear at boundaries: a caption, chart, and explanatory paragraph may no longer travel together. Parent-document retrieval and metadata pre-filtering can help reintroduce relevant context during retrieval or generation. Preprocessing gains more stages: element detection and extraction come before chunking and embedding, and the summary-based path may add another LLM call. Each stage becomes another part of the pipeline to maintain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Embed the page with its layout intact
There is also a representation problem. CLIP uses separate networks to encode text and images. Joshi describes a modality gap in which unrelated items of the same modality can be closer in vector space than related items across modalities. An irrelevant text item may therefore be closer to a text query than a relevant image is.
The alternative is to embed a screenshot containing the text and visual elements together. Joshi motivates this with VLM-based embeddings and a more unified representation. The chosen model, Voyage Multimodal 3, uses shared-transformer vectorization. That model-specific architecture should not be read as a guarantee that every VLM eliminates modality separation. The practical benefit here is concrete: the embedding input retains the page’s text, figures, tables, and their visible arrangement. Joshi argues that this simplifies preprocessing and improves retrieval, but supplies no workshop-specific retrieval measurement.
The ingestion pipeline becomes:
- Render each document page as a separate screenshot.
- Save the screenshot locally for the lab; use blob storage such as S3 or Google Cloud Storage for a production deployment.
- Generate a multimodal embedding of the screenshot.
- Store that embedding and the image’s path or reference in MongoDB.
The vector database stores the embedding and the reference, not the raw screenshot. Rendering a PDF page means capturing the complete page, rather than extracting only the figures embedded within it. The file remains in local or blob storage, ready to be fetched after retrieval.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A page preserves local context, not the whole document
Why take a screenshot of every page? The purpose is to preserve continuity between prose and visual elements so they can be retrieved together, rather than to save storage space. An audience question exposes the remaining boundary: a page is still only part of a document. Joshi favors a complete page over a small paragraph chunk, while recommending metadata pre-filtering and other familiar retrieval techniques when more context is needed. The lab uses distinct pages; overlapping captures are a possible extension.
Why not pass the entire PDF to the model? Joshi invokes the lost-in-the-middle problem: a large context window does not ensure that the model will find and use relevant information buried inside it. That motivates selective retrieval without establishing that page screenshots always outperform whole-document input. To preserve continuity across retrieved pages, store page numbers and document metadata, then expand a match to nearby pages—for example, the previous two and next two pages.
The choice of Voyage Multimodal 3 is not essential to the agent loop. Asked why that model was selected, Joshi points to the VLM-based embedding approach and avoiding the CLIP-style modality gap she described. Another suitable model could fill the retrieval role. Audio and video require their own representations and processing; screenshots fit this workshop because text and figures naturally coexist on document pages.
A second question asks whether VLM embeddings must be less accurate than embeddings from Ada 002 because of presumed model-size differences. No such comparison is established. Joshi notes that VLMs can also be large and offers to point participants toward benchmarks, without giving a measured result. Running a model locally has the same practical constraint as running an LLM locally: select one that fits the machine’s hardware.
Finally, an audience member asks about combining text, images, and a time series that is weakly aligned with the other modalities. A shared vector space may not make those records meaningfully close. Joshi suggests treating the time series as features and using a different retrieval strategy. This is a design direction for disparate data, not an implemented time-series solution or a claim that time series cannot be embedded.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Search for references, then fetch the images
At runtime, the agent sends the query to a multimodal LLM. The embedding model finds relevant pages; this second model interprets their text and images. Joshi tentatively names Gemini 2.0 Flash Experimental for this role. The historical experimental endpoint is now shut down, so reproducing the architecture requires an available model with image input and tool-calling support.
The assistant exposes one tool: vector search. Its retrieval branch follows a specific sequence:
- The LLM requests vector search and supplies arguments.
- Application code executes the search.
- The search returns references to matching screenshots.
- Application code loads those screenshots from local or blob storage.
- The next model call includes the images, original query, and conversation history.
- The model produces an answer or continues the tool loop.
The extra image-loading step is essential. Returning a file reference does not give the model the image’s contents. Images already available to the conversation accompany subsequent calls, whether the model is choosing another tool or generating its answer.
Retrieval is optional. If the user supplies an image and asks for a summary, the model may already have everything needed to respond. It can answer directly without calling vector search. The same assistant therefore supports both corpus questions that require retrieval and image questions that do not.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Persist the conversation around the tool loop
Short-term memory makes the assistant coherent across turns. Each query carries a session ID that identifies its conversation. Before asking the model what to do, the application retrieves previous turns for that session and includes them alongside the current query and any retrieved context. History can change the tool decision itself: the answer may already be available from an earlier exchange.
After generating an answer, the application appends the current user query and response to the same session. The essential database operations can be expressed with a MongoDB collection passed into two Python functions:
python
def load_history(collection, session_id: str) -> list[dict]:
session = collection.find_one({"_id": session_id})
return session["turns"] if session else []
def save_turn(
collection,
session_id: str,
query: str,
response: str,
) -> None:
collection.update_one(
{"_id": session_id},
{
"$push": {
"turns": {
"$each": [
{"role": "user", "content": query},
{"role": "assistant", "content": response},
]
}
}
},
upsert=True,
)
Call load_history before the model loop and save_turn after an answer has been generated. User queries and model responses are the minimum persisted history. Tool calls and their outcomes can also be retained; Joshi additionally mentions recording reasoning traces where available.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build the loop in the notebook
The hands-on portion begins with the lab’s GitHub README: open the lab on a laptop and follow its setup instructions. Then choose how to work through the implementation:
lab.ipynb: fill in the code exercises, using the inline reference documentation marked with a books emoji.solutions.ipynb: work through the prefilled code and explanatory comments.
The solutions are also available to consult while completing the exercises. The final slide shows the lab entry point and code-placeholder guidance.
Joshi closes by encouraging participants to ask the instructor team for help and use the solutions when stuck. The next step is to connect the pieces in Python: page ingestion, tool dispatch, image loading, and session persistence. The recording hands that implementation work to the reader at the notebooks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The model’s original announcement explains interleaved text-image embeddings, document screenshots, and retrieval evaluations.
The original reasoning-and-acting pattern combines action selection, external observations, and updated plans.
Documentation for extracting structured content from PDFs, scans, tables, and charts.
Further reading
Python workshop materials with exercise and solution notebooks, screenshot assets, and a Codespaces setup guide.
- Mind the GapPaper
Research on why image and text representations can occupy separated regions within a shared embedding space.
- Lost in the MiddlePaper
Experiments showing how relevant information’s position affects question answering and retrieval from long contexts.
Updates since the talk
- Gemini 2.0 Flash lifecycle and capabilitiesDocumentation
Official model documentation records the shutdown of the Gemini 2.0 Flash variants.
Full transcript
Loading transcript…