AI Engineer World's Fair 2026

On AI and Knowledge

Pablo Castro· CVP & Distinguished Engineer, Microsoft17:35

Loading slides…

Read the talk

On AI and Knowledge

Agents need more than knowledge stored in model weights: they need access to organizational context and a way to turn experience into better instructions, tools, and skills.

From a talk by Pablo Castro

Before you start: Familiarity with language models, agents, and basic retrieval concepts will help; no Azure experience is required.

What does an agent need to know?

What does it mean for an agent to know something—and how does that knowledge help it get work done? For Pablo Castro, whose work at Microsoft connects AI with knowledge representation, extraction, and search, the question has three parts: intrinsic knowledge, extrinsic knowledge, and learned knowledge. Intrinsic knowledge is what training puts into a model’s parameters: information the model can draw on without first retrieving a document or consulting another system. That stored knowledge made many early generative AI applications possible.

Black slide with Intrinsic, Extrinsic, and Learned arranged horizontally, alongside a speaker inset.Open full source frame
Three categories of knowledge: intrinsic, extrinsic, and learned.

The coding example makes the change tangible. Castro shows two pieces of code he wrote about twenty-five years apart. Despite the interval, producing them involved much the same work: sit down, recall what he knew, look up what he did not, and write the code. Writing an email or summarizing a document followed a similar pattern. The human had to turn remembered or researched information into the finished artifact.

Castro traces the subsequent acceleration through coding tools. In his timeline, IntelliSense arrived in 1996, reducing the need to remember function signatures. Twenty-two years later, machine learning helped rank its suggestions. GitHub Copilot followed three years after that, before ChatGPT. These experiences moved beyond presenting available choices toward generating useful work from knowledge already stored in models.

Cursor and GitHub Copilot X came next in the progression, followed by Opus 4.5, which Castro places late in the year before the talk, and rapid improvements across GPT, Opus, and other coding models. He points to OpenClaw as an early-year example, describing it as software with no code written by hand. That authorship claim is Castro’s; the broader mechanism is the combination of intrinsic knowledge with increasingly capable reasoning.

0:330:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:33 · section reference included

From model knowledge to organizational context

Microsoft’s platform connects that model capability to agent development: start building in GitHub, add contextualization to ground the agent, and use Microsoft Foundry for hosting, observability, and management. Castro describes a catalog containing thousands of models, allowing developers to choose a model for each task. At the time of the talk, he also highlights the previous day’s announcement that Claude in Microsoft Foundry was generally available.

Knowing how to perform a task does not mean knowing what is happening inside a company. An agent participating in an organization’s work needs information beyond its training. Retrieval-augmented generation, or RAG, began as a relatively simple way to supply that information and grew into context engineering. Two things became more demanding at once: grounding expanded from isolated datasets to the whole organization, while retrieval expanded beyond simple vector search.

An agent usually has a specific body of knowledge its developer manages. It may also need the organization’s ambient data: documents, email, chat threads, and warehouse records that surround the immediate task. Microsoft IQ is presented as a common entry point into that information, implemented as a set of capabilities rather than one feature.

CapabilityKnowledge it connects to
Work IQSharePoint documents, email, calendars, chats, and connections between people
Fabric IQData warehouses, data lakes, and Power BI reports
Foundry IQData supplied for grounding custom agents
Web IQPublic information from the web

Together, these sources extend the agent’s view beyond the information packaged specifically for it. The integrations span different release stages; the demonstration should not be read as general availability for every source, particularly Web IQ.

3:474:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:47 · section reference included

Combining retrieval methods without exposing every setting

Vector databases helped early RAG systems get off the ground. But computing cosine similarity well did not solve retrieval as a whole. Castro reports that combined retrieval methods outperformed individual methods in Azure AI Search evaluations, particularly in real-world customer scenarios. Azure AI Search is the search technology behind Foundry IQ; the result motivates combining retrieval building blocks rather than treating one similarity measure as sufficient.

The product problem is how to make those combinations available without requiring every developer to become a search specialist. Foundry IQ layers the system so the same stack supports both a managed path and detailed control:

  • Managed retrieval: Supply PDFs or pictures and let the platform handle chunking, vectorization, relevance, ranking, and agentic retrieval.
  • Search controls: Configure vector indexes, vector quantization, and lexical retrieval when the application needs more control.

Developers can move between these levels as their needs change, instead of replacing the retrieval system to reach its lower-level settings.

Above the core search system, agentic retrieval adds a decision about whether the available information satisfies the request. A quick, single-shot retrieval can be enough for an easy question. A more demanding request benefits from a system that reflects on what the dataset contains and checks whether it has met the information need before returning results.

For difficult cases in Microsoft’s internal evaluations, Castro reports that agentic retrieval improved evidence recall and answer completeness over simpler individual approaches. These are complementary concerns: retrieval must find the evidence, and the resulting answer must cover the requested information. The talk supplies no numerical scores for this comparison.

7:017:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:01 · section reference included

Building a movie knowledge base

The Foundry demonstration brings these layers together around a movie dataset. Foundry manages knowledge alongside agents and models, with a knowledge base serving as an agent’s entry point into the relevant information. Creating this one follows a short sequence:

  1. Create the knowledge base for the movie dataset.
  2. Select a model to power the agentic retrieval workflow.
  3. Set retrieval effort, trading latency against quality.
  4. Attach the sources: unstructured documents such as PDFs in blob storage, structured Parquet tables containing statistics, and web grounding.
  5. Save the knowledge base and connect it to a Foundry agent.

The saved movies configuration exposes the retrieval settings and a control for using it in an agent.

Foundry configuration page named movies, showing model, retrieval reasoning effort, output mode, and a Use in an agent button.Open full source frame
The movies knowledge base configuration in Microsoft Foundry.

The knowledge base is also a standalone asset. Castro describes each knowledge base as an MCP server, so an existing agent harness outside Foundry can connect to it without custom glue code. This separates the knowledge interface from the choice of agent harness: the same asset can serve the Foundry agent shown in the demonstration or another compatible client.

10:1410:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:14 · section reference included

Inspecting the indexes beneath the knowledge base

Some sources in the knowledge base, such as stored documents, rely on indexes and vectorized content underneath the managed interface. Castro switches to Azure, opens the same knowledge base, and finds its backing indexes. He can inspect the index structure, choose a vector quantization approach or indexing algorithm, and explore the data to see how its chunks are organized. The high-level asset does not remove access to the underlying search configuration.

Retrieval also has to respect the agent’s token budget. Castro describes evaluating the system for information-dense answers with as few tokens as possible, so the tokens consumed by retrieval carry useful evidence. This is an optimization goal, not a measured token-saving result from the movie demonstration.

11:3311:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:33 · section reference included

Turning everyday work into learned knowledge

Learned knowledge comes from the work individuals and organizations do every day. When agents perform that work, their processes can be observed, reflected on, and improved through automatic tuning. Castro invokes Satya Nadella’s description of people and agents forming a learning loop that captures what is distinctive about an organization and puts it back to work.

Agent Optimizer makes that loop concrete in Foundry. It evaluates a baseline, generates candidate configurations, evaluates and ranks those candidates, and allows a sufficiently strong result to move toward production deployment. The workflow diagram includes iteration through evaluation and ranking before deploying the winner. Deployment is the intended endpoint of the loop; the following demonstration focuses on optimization and applying configuration, not production deployment infrastructure.

Five-step Foundry Agent Optimizer diagram with an optimization loop through evaluation and ranking, followed by deployment of the winner.Open full source frame
Foundry Agent Optimizer: evaluate, generate candidates, evaluate candidates, rank, and deploy.
13:0813:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:08 · section reference included

Generate an evaluation, search candidates, apply a configuration

In VS Code with the Foundry toolkit installed, Castro starts with a simple agent whose configuration is externalized. The agent’s implementation can vary, but its instructions, tool definitions, skills, and related settings must be available for the optimizer to change. The command names here follow the demonstration; the current preview workflow also documents project prerequisites, including an azd project for applying results.

First, establish an evaluation. An existing evaluation can be reused. Otherwise, eval generate uses available agent traces, instructions, and other information to produce an evaluation focused on task adherence. Castro opens a previously generated dataset containing tasks, questions, and criteria. Those criteria supply the target against which candidate changes will be judged.

Next, run optimize. Castro reports that this example’s previously completed optimization run took approximately forty-five minutes. The optimizer hill-climbs the metric established by the evaluation. In the displayed output, it first establishes the baseline, then tests candidate combinations in what Castro calls a GEPA-style loop, looking for configurations that perform better against the rubric. The object being improved here is the agent’s configuration, not its model weights.

Once a better candidate has been found, optimize apply swaps the externalized configuration for the selected version. The terminal view shows the baseline alongside four candidates, a starred candidate_1, and the highlighted apply command. Candidate evaluation and configuration application are separate steps: finding a promising candidate does not by itself replace the active configuration.

VS Code terminal showing candidate scores and strategies, a starred candidate_1, candidate IDs, and a highlighted optimize apply command.Open full source frame
Optimization results compare the baseline with four candidates, with an apply command highlighted below.

Castro then compares the baseline with the applied version. The baseline has simple instructions; the optimized version has a larger set of instructions produced by the search process rather than written by hand. Those changes draw on the agent’s instructions, skills, and tools, together with reflection on actual traces from user activity. Experience becomes learned knowledge when it changes how the agent is configured to do the next task. The demonstrated improvement is relative to the supplied evaluation rubric.

That closes the progression from knowledge a model already has, through knowledge an agent retrieves, to knowledge accumulated by doing work. The purpose of the learning loop is to capture capabilities specific to a company or organization and make them available in subsequent work. Castro closes by directing readers to the Foundry portal at ai.azure.com to try the capabilities shown.

14:2514:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:25 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…