AI Engineer World's Fair 2026
On AI and Knowledge
Loading slides…
Read the talk
On AI and Knowledge
Agents need more than knowledge stored in model weights: they need access to organizational context and a way to turn experience into better instructions, tools, and skills.
From a talk by Pablo Castro
Before you start: Familiarity with language models, agents, and basic retrieval concepts will help; no Azure experience is required.
What does an agent need to know?
What does it mean for an agent to know something—and how does that knowledge help it get work done? For Pablo Castro, whose work at Microsoft connects AI with knowledge representation, extraction, and search, the question has three parts: intrinsic knowledge, extrinsic knowledge, and learned knowledge. Intrinsic knowledge is what training puts into a model’s parameters: information the model can draw on without first retrieving a document or consulting another system. That stored knowledge made many early generative AI applications possible.
The coding example makes the change tangible. Castro shows two pieces of code he wrote about twenty-five years apart. Despite the interval, producing them involved much the same work: sit down, recall what he knew, look up what he did not, and write the code. Writing an email or summarizing a document followed a similar pattern. The human had to turn remembered or researched information into the finished artifact.
Castro traces the subsequent acceleration through coding tools. In his timeline, IntelliSense arrived in 1996, reducing the need to remember function signatures. Twenty-two years later, machine learning helped rank its suggestions. GitHub Copilot followed three years after that, before ChatGPT. These experiences moved beyond presenting available choices toward generating useful work from knowledge already stored in models.
Cursor and GitHub Copilot X came next in the progression, followed by Opus 4.5, which Castro places late in the year before the talk, and rapid improvements across GPT, Opus, and other coding models. He points to OpenClaw as an early-year example, describing it as software with no code written by hand. That authorship claim is Castro’s; the broader mechanism is the combination of intrinsic knowledge with increasingly capable reasoning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From model knowledge to organizational context
Microsoft’s platform connects that model capability to agent development: start building in GitHub, add contextualization to ground the agent, and use Microsoft Foundry for hosting, observability, and management. Castro describes a catalog containing thousands of models, allowing developers to choose a model for each task. At the time of the talk, he also highlights the previous day’s announcement that Claude in Microsoft Foundry was generally available.
Knowing how to perform a task does not mean knowing what is happening inside a company. An agent participating in an organization’s work needs information beyond its training. Retrieval-augmented generation, or RAG, began as a relatively simple way to supply that information and grew into context engineering. Two things became more demanding at once: grounding expanded from isolated datasets to the whole organization, while retrieval expanded beyond simple vector search.
An agent usually has a specific body of knowledge its developer manages. It may also need the organization’s ambient data: documents, email, chat threads, and warehouse records that surround the immediate task. Microsoft IQ is presented as a common entry point into that information, implemented as a set of capabilities rather than one feature.
| Capability | Knowledge it connects to |
|---|---|
| Work IQ | SharePoint documents, email, calendars, chats, and connections between people |
| Fabric IQ | Data warehouses, data lakes, and Power BI reports |
| Foundry IQ | Data supplied for grounding custom agents |
| Web IQ | Public information from the web |
Together, these sources extend the agent’s view beyond the information packaged specifically for it. The integrations span different release stages; the demonstration should not be read as general availability for every source, particularly Web IQ.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combining retrieval methods without exposing every setting
Vector databases helped early RAG systems get off the ground. But computing cosine similarity well did not solve retrieval as a whole. Castro reports that combined retrieval methods outperformed individual methods in Azure AI Search evaluations, particularly in real-world customer scenarios. Azure AI Search is the search technology behind Foundry IQ; the result motivates combining retrieval building blocks rather than treating one similarity measure as sufficient.
The product problem is how to make those combinations available without requiring every developer to become a search specialist. Foundry IQ layers the system so the same stack supports both a managed path and detailed control:
- Managed retrieval: Supply PDFs or pictures and let the platform handle chunking, vectorization, relevance, ranking, and agentic retrieval.
- Search controls: Configure vector indexes, vector quantization, and lexical retrieval when the application needs more control.
Developers can move between these levels as their needs change, instead of replacing the retrieval system to reach its lower-level settings.
Above the core search system, agentic retrieval adds a decision about whether the available information satisfies the request. A quick, single-shot retrieval can be enough for an easy question. A more demanding request benefits from a system that reflects on what the dataset contains and checks whether it has met the information need before returning results.
For difficult cases in Microsoft’s internal evaluations, Castro reports that agentic retrieval improved evidence recall and answer completeness over simpler individual approaches. These are complementary concerns: retrieval must find the evidence, and the resulting answer must cover the requested information. The talk supplies no numerical scores for this comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Building a movie knowledge base
The Foundry demonstration brings these layers together around a movie dataset. Foundry manages knowledge alongside agents and models, with a knowledge base serving as an agent’s entry point into the relevant information. Creating this one follows a short sequence:
- Create the knowledge base for the movie dataset.
- Select a model to power the agentic retrieval workflow.
- Set retrieval effort, trading latency against quality.
- Attach the sources: unstructured documents such as PDFs in blob storage, structured Parquet tables containing statistics, and web grounding.
- Save the knowledge base and connect it to a Foundry agent.
The saved movies configuration exposes the retrieval settings and a control for using it in an agent.
The knowledge base is also a standalone asset. Castro describes each knowledge base as an MCP server, so an existing agent harness outside Foundry can connect to it without custom glue code. This separates the knowledge interface from the choice of agent harness: the same asset can serve the Foundry agent shown in the demonstration or another compatible client.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspecting the indexes beneath the knowledge base
Some sources in the knowledge base, such as stored documents, rely on indexes and vectorized content underneath the managed interface. Castro switches to Azure, opens the same knowledge base, and finds its backing indexes. He can inspect the index structure, choose a vector quantization approach or indexing algorithm, and explore the data to see how its chunks are organized. The high-level asset does not remove access to the underlying search configuration.
Retrieval also has to respect the agent’s token budget. Castro describes evaluating the system for information-dense answers with as few tokens as possible, so the tokens consumed by retrieval carry useful evidence. This is an optimization goal, not a measured token-saving result from the movie demonstration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turning everyday work into learned knowledge
Learned knowledge comes from the work individuals and organizations do every day. When agents perform that work, their processes can be observed, reflected on, and improved through automatic tuning. Castro invokes Satya Nadella’s description of people and agents forming a learning loop that captures what is distinctive about an organization and puts it back to work.
Agent Optimizer makes that loop concrete in Foundry. It evaluates a baseline, generates candidate configurations, evaluates and ranks those candidates, and allows a sufficiently strong result to move toward production deployment. The workflow diagram includes iteration through evaluation and ranking before deploying the winner. Deployment is the intended endpoint of the loop; the following demonstration focuses on optimization and applying configuration, not production deployment infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate an evaluation, search candidates, apply a configuration
In VS Code with the Foundry toolkit installed, Castro starts with a simple agent whose configuration is externalized. The agent’s implementation can vary, but its instructions, tool definitions, skills, and related settings must be available for the optimizer to change. The command names here follow the demonstration; the current preview workflow also documents project prerequisites, including an azd project for applying results.
First, establish an evaluation. An existing evaluation can be reused. Otherwise, eval generate uses available agent traces, instructions, and other information to produce an evaluation focused on task adherence. Castro opens a previously generated dataset containing tasks, questions, and criteria. Those criteria supply the target against which candidate changes will be judged.
Next, run optimize. Castro reports that this example’s previously completed optimization run took approximately forty-five minutes. The optimizer hill-climbs the metric established by the evaluation. In the displayed output, it first establishes the baseline, then tests candidate combinations in what Castro calls a GEPA-style loop, looking for configurations that perform better against the rubric. The object being improved here is the agent’s configuration, not its model weights.
Once a better candidate has been found, optimize apply swaps the externalized configuration for the selected version. The terminal view shows the baseline alongside four candidates, a starred candidate_1, and the highlighted apply command. Candidate evaluation and configuration application are separate steps: finding a promising candidate does not by itself replace the active configuration.
Castro then compares the baseline with the applied version. The baseline has simple instructions; the optimized version has a larger set of instructions produced by the search process rather than written by hand. Those changes draw on the agent’s instructions, skills, and tools, together with reflection on actual traces from user activity. Experience becomes learned knowledge when it changes how the agent is configured to do the next task. The demonstrated improvement is relative to the supplied evaluation rubric.
That closes the progression from knowledge a model already has, through knowledge an agent retrieves, to knowledge accumulated by doing work. The purpose of the learning loop is to capture capabilities specific to a company or organization and make them available in subsequent work. Castro closes by directing readers to the Foundry portal at ai.azure.com to try the capabilities shown.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Documentation hub for Work IQ, Fabric IQ, Foundry IQ, and Web IQ.
Introduces a prompt optimizer that reflects on execution trajectories and tests evolving candidates.
Source repository for the personal AI assistant used in Castro's coding-acceleration example.
Further reading
Explains reusable knowledge bases, enterprise and web sources, MCP access, and feature availability.
Reports evidence recall, answer completeness, token usage, and benchmark conditions for knowledge-base retrieval.
Castro explains how lexical search, vector search, and reranking contribute to retrieval quality.
Python notebooks and deployment examples for knowledge bases, IQ integrations, and MCP access, with prerequisites documented.
Updates since the talk
Current preview instructions for configuring evaluations, optimizing candidates, and applying results.
Full transcript
Loading transcript…