AI Engineer World's Fair 2025

Building Multimodal AI Agents (From Scratch)

Apoorva Joshi· Senior AI Developer Advocate, MongoDB36:58

Loading slides…

Read the talk

Building Multimodal AI Agents (From Scratch)

A document assistant needs more than a vision model: it needs a tool loop, retrieval that preserves visual context, and memory that carries the conversation forward.

From a talk by Apoorva Joshi

Before you start: Basic Python and familiarity with LLM prompts are sufficient; prior experience with retrieval-augmented generation will help with the retrieval sections.

What makes an assistant an agent?

What does it take to build an assistant that can reason about a request, use tools, and work with images as well as text? Apoorva Joshi’s workshop approaches that question in plain Python, starting with the components before assembling a multimodal agent. Joshi is an AI-focused developer advocate at MongoDB, with a background in cybersecurity data science. The workshop moves from agent fundamentals to multimodality, then into a hands-on implementation.

Agenda slide listing AI agents, components of an AI agent, multimodality, building a multimodal AI agent, and Q&A.Open full source frame
Workshop agenda: AI agents, their components, multimodality, and building a multimodal agent.

An agent uses an LLM to reason through a problem, create a plan, and execute and revise that plan with tools. The useful distinction is how much responsibility the application gives the model for deciding what happens next.

0:161:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Choose how much autonomy the task needs

Joshi distinguishes three interaction patterns:

PatternInformation availableHow work proceeds
Simple promptingPretrained, parametric knowledgeAnswer the supplied prompt
Retrieval-augmented generationExternal information added as contextRetrieve information, then answer
AgentTools, their results, and past interactionsSelect actions and revise the next step

In the simple prompting baseline, the model has no external source for missing information. RAG supplies that information and can support light personalization when the retrieved sources contain relevant user context. In Joshi’s comparison, neither baseline supplies the autonomous, multistep feedback loop that lets an agent change its approach after seeing a result. The agent makes those sequencing decisions itself.

Use agents when the task warrants their additional cost and latency. Planning, tool use, and correction can require repeated model calls. Good candidates include workflows whose steps are difficult to predict, tasks that tolerate slower responses and variable outputs, and applications that benefit from personalization over time. The same input need not produce the same result; Joshi notes that an agent’s sequence of decisions amplifies this variability. The workshop deliberately uses a simple problem to make the machinery understandable, even though a production version might need less autonomy.

AI agents slide showing tools and memory connected to an LLM, an action-result feedback loop, three benefits, and higher cost and latency.Open full source frame
AI agents combine tools, memory, and an action-result loop, with higher cost and latency.
3:253:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:25 · section reference included

Inputs, plans, and feedback

The agent has four components: perception gathers information, planning and reasoning determine an approach, tools act on the outside world, and memory retains past interactions. The resemblance to human problem-solving is intentional: these components give the software enough autonomy to carry out a task.

Perception begins with inputs. A person might submit a query, or an email or Slack message might trigger the application. Text is one input format; images, voice, and video expand what the agent can observe. This implementation limits its scope to text and images.

The LLM supplies planning and reasoning, guided by a prompt. Joshi first describes planning without feedback: the model forms a plan from its initial understanding without revising it in response to tool outcomes. She presents chain-of-thought prompting in this category. A zero-shot instruction asks the model to work step by step; a few-shot prompt supplies worked examples that guide its approach to the next problem.

Planning with feedback lets new observations change the next action. The workshop uses ReAct, which interleaves reasoning and acting:

  1. Determine an action that could advance the task.
  2. Execute the selected tool.
  3. Return the result as an observation.
  4. Use that observation to decide what to do next.
  5. Continue until the model can produce a final answer.

The operational difference is the return path: tool results become inputs to another decision, rather than merely outputs of a fixed plan.

7:127:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:12 · section reference included

The model requests a tool; Python executes it

Tools can expose weather and search APIs, vector stores, or specialized machine-learning models. They are usually represented as functions. A tool-capable LLM can select a function and supply its arguments, but the application must execute the function. A generated tool call is a request for an action, not evidence that the action happened.

The model also needs a tool schema: the function’s name, a description of its purpose, and its parameters with types and descriptions. Joshi mentions JSON schemas and MCP as ways of exposing tool definitions. Her weather example accepts a city as a string. In Python, that contract can be represented directly:

python

weather_schema = {
    "name": "get_weather",
    "description": "Get today's weather for a city.",
    "parameters": {
        "type": "object",
        "properties": {
            "city": {
                "type": "string",
                "description": "The city to get weather for.",
            }
        },
        "required": ["city"],
    },
}

This describes the available operation. The weather client and the code that dispatches a model’s request to it remain separate responsibilities.

Tools slide with a weather function on the left and a schema defining its name, description, and city parameter on the right.Open full source frame
A weather tool function alongside its tool schema.
11:2111:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:21 · section reference included

Follow a weather question through the loop

Memory completes the component model. Short-term memory stores and retrieves information within one conversation. Long-term memory stores, updates, and retrieves information across conversations, supporting personalization over time. The lab implements short-term memory; cross-conversation learning is outside its scope.

Consider the workshop’s weather question: what is the weather in San Francisco today? The agent is the surrounding software system, which forwards that query to an LLM. The model has descriptions of weather and search tools, along with past interactions. It selects the weather tool and extracts San Francisco as the city argument.

Python then calls the weather API with the selected arguments and returns the API’s response to the model. At that point, the model can request another tool if it still needs information, or generate a natural-language answer from the returned temperature. This is the complete feedback loop: request, execution, observation, and a new decision. The model never has to be the component that performs the network request.

12:5012:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:50 · section reference included

Retrieve documents and interpret figures

Multimodality extends the inputs a model can process and, depending on the model, the outputs it can generate. Research papers, financial reports, organizational reports, and healthcare documents routinely mix prose with graphs, images, and tables. Understanding these documents requires preserving relationships across those forms of information.

Two model roles matter:

  • Multimodal embedding models turn different input types into vectors for retrieval through methods such as vector search, hybrid search, or graph-based retrieval.
  • Multimodal LLMs interpret those inputs and generate responses. Joshi cites DeepSeek, Claude, and ChatGPT voice mode as examples of the broader multimodal model landscape; supported input and output formats depend on the particular model.

Give a multimodal LLM tools for searching multimodal data, then let it use the results to solve a problem, and the tool loop becomes a multimodal agent.

The Python assistant has two objectives: answer questions over a large document corpus, and explain or analyze a chart or diagram supplied by the user. Joshi’s everyday example is taking a screenshot of an equation in a research paper and asking Claude to explain it. The additional challenge for this agent is searching a corpus where text, images, and tables are interleaved. Before the agent can reason about a relevant figure, the retrieval system has to find it without stripping away its context.

Objectives slide with two bullets, green emphasis on mixed modalities, and an arrow pointing to a reaction image.Open full source frame
Answer questions about documents with mixed modalities and explain charts and diagrams.
15:5716:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:57 · section reference included

What extraction pipelines lose

A text-only RAG pipeline typically chunks documents, embeds the chunks, and retrieves relevant chunks as context. Mixed documents require additional handling. Joshi names LlamaParse and Unstructured as tools for extracting text, images, and tables, then describes two ways to make the extracted elements searchable.

PipelineText handlingImage and table handling
Convert to textChunk and embed textSummarize, then embed summaries
Embed multiple modalitiesChunk and embed textEmbed visual elements directly

The first path brings everything into the text domain, using a text embedding model for both prose and visual summaries. The second uses a multimodal embedding model for the extracted elements. Both still depend on separating the document into pieces.

That separation creates two costs. Context can disappear at boundaries: a caption, chart, and explanatory paragraph may no longer travel together. Parent-document retrieval and metadata pre-filtering can help reintroduce relevant context during retrieval or generation. Preprocessing gains more stages: element detection and extraction come before chunking and embedding, and the summary-based path may add another LLM call. Each stage becomes another part of the pipeline to maintain.

19:3919:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:39 · section reference included

Embed the page with its layout intact

There is also a representation problem. CLIP uses separate networks to encode text and images. Joshi describes a modality gap in which unrelated items of the same modality can be closer in vector space than related items across modalities. An irrelevant text item may therefore be closer to a text query than a relevant image is.

The alternative is to embed a screenshot containing the text and visual elements together. Joshi motivates this with VLM-based embeddings and a more unified representation. The chosen model, Voyage Multimodal 3, uses shared-transformer vectorization. That model-specific architecture should not be read as a guarantee that every VLM eliminates modality separation. The practical benefit here is concrete: the embedding input retains the page’s text, figures, tables, and their visible arrangement. Joshi argues that this simplifies preprocessing and improves retrieval, but supplies no workshop-specific retrieval measurement.

The ingestion pipeline becomes:

  1. Render each document page as a separate screenshot.
  2. Save the screenshot locally for the lab; use blob storage such as S3 or Google Cloud Storage for a production deployment.
  3. Generate a multimodal embedding of the screenshot.
  4. Store that embedding and the image’s path or reference in MongoDB.

The vector database stores the embedding and the reference, not the raw screenshot. Rendering a PDF page means capturing the complete page, rather than extracting only the figures embedded within it. The file remains in local or blob storage, ready to be fetched after retrieval.

21:5722:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:57 · section reference included

A page preserves local context, not the whole document

Why take a screenshot of every page? The purpose is to preserve continuity between prose and visual elements so they can be retrieved together, rather than to save storage space. An audience question exposes the remaining boundary: a page is still only part of a document. Joshi favors a complete page over a small paragraph chunk, while recommending metadata pre-filtering and other familiar retrieval techniques when more context is needed. The lab uses distinct pages; overlapping captures are a possible extension.

Why not pass the entire PDF to the model? Joshi invokes the lost-in-the-middle problem: a large context window does not ensure that the model will find and use relevant information buried inside it. That motivates selective retrieval without establishing that page screenshots always outperform whole-document input. To preserve continuity across retrieved pages, store page numbers and document metadata, then expand a match to nearby pages—for example, the previous two and next two pages.

The choice of Voyage Multimodal 3 is not essential to the agent loop. Asked why that model was selected, Joshi points to the VLM-based embedding approach and avoiding the CLIP-style modality gap she described. Another suitable model could fill the retrieval role. Audio and video require their own representations and processing; screenshots fit this workshop because text and figures naturally coexist on document pages.

A second question asks whether VLM embeddings must be less accurate than embeddings from Ada 002 because of presumed model-size differences. No such comparison is established. Joshi notes that VLMs can also be large and offers to point participants toward benchmarks, without giving a measured result. Running a model locally has the same practical constraint as running an LLM locally: select one that fits the machine’s hardware.

Finally, an audience member asks about combining text, images, and a time series that is weakly aligned with the other modalities. A shared vector space may not make those records meaningfully close. Joshi suggests treating the time series as features and using a different retrieval strategy. This is a design direction for disparate data, not an implemented time-series solution or a claim that time series cannot be embedded.

26:0126:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:01 · section reference included

Search for references, then fetch the images

At runtime, the agent sends the query to a multimodal LLM. The embedding model finds relevant pages; this second model interprets their text and images. Joshi tentatively names Gemini 2.0 Flash Experimental for this role. The historical experimental endpoint is now shut down, so reproducing the architecture requires an available model with image input and tool-calling support.

The assistant exposes one tool: vector search. Its retrieval branch follows a specific sequence:

  1. The LLM requests vector search and supplies arguments.
  2. Application code executes the search.
  3. The search returns references to matching screenshots.
  4. Application code loads those screenshots from local or blob storage.
  5. The next model call includes the images, original query, and conversation history.
  6. The model produces an answer or continues the tool loop.

The extra image-loading step is essential. Returning a file reference does not give the model the image’s contents. Images already available to the conversation accompany subsequent calls, whether the model is choosing another tool or generating its answer.

Retrieval is optional. If the user supplies an image and asks for a summary, the model may already have everything needed to respond. It can answer directly without calling vector search. The same assistant therefore supports both corpus questions that require retrieval and image questions that do not.

31:3831:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

31:38 · section reference included

Persist the conversation around the tool loop

Short-term memory makes the assistant coherent across turns. Each query carries a session ID that identifies its conversation. Before asking the model what to do, the application retrieves previous turns for that session and includes them alongside the current query and any retrieved context. History can change the tool decision itself: the answer may already be available from an earlier exchange.

After generating an answer, the application appends the current user query and response to the same session. The essential database operations can be expressed with a MongoDB collection passed into two Python functions:

python

def load_history(collection, session_id: str) -> list[dict]:
    session = collection.find_one({"_id": session_id})
    return session["turns"] if session else []


def save_turn(
    collection,
    session_id: str,
    query: str,
    response: str,
) -> None:
    collection.update_one(
        {"_id": session_id},
        {
            "$push": {
                "turns": {
                    "$each": [
                        {"role": "user", "content": query},
                        {"role": "assistant", "content": response},
                    ]
                }
            }
        },
        upsert=True,
    )

Call load_history before the model loop and save_turn after an answer has been generated. User queries and model responses are the minimum persisted history. Tool calls and their outcomes can also be retained; Joshi additionally mentions recording reasoning traces where available.

34:0234:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

34:02 · section reference included

Build the loop in the notebook

The hands-on portion begins with the lab’s GitHub README: open the lab on a laptop and follow its setup instructions. Then choose how to work through the implementation:

  • lab.ipynb: fill in the code exercises, using the inline reference documentation marked with a books emoji.
  • solutions.ipynb: work through the prefilled code and explanatory comments.

The solutions are also available to consult while completing the exercises. The final slide shows the lab entry point and code-placeholder guidance.

Slide showing mdb.link/multimodal-lab above a QR code, plus notes for reference documentation, CODE_BLOCK_N placeholders, and solutions.ipynb.Open full source frame
Hands-on lab link and guidance for reference documentation, code placeholders, and the solutions notebook.

Joshi closes by encouraging participants to ask the instructor team for help and use the solutions when stuck. The next step is to connect the pieces in Python: page ingestion, tool dispatch, image loading, and session persistence. The recording hands that implementation work to the reader at the notebooks.

35:2335:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

35:23 · section reference included

Resources

From the talk

  • Python workshop materials with exercise and solution notebooks, screenshot assets, and a Codespaces setup guide.

  • Mind the GapPaper

    Research on why image and text representations can occupy separated regions within a shared embedding space.

  • Lost in the MiddlePaper

    Experiments showing how relevant information’s position affects question answering and retrieval from long contexts.

Updates since the talk

Transcript download
Full transcript

Loading transcript…