AI Engineer World's Fair 2025
Building Code-First AI Agents with Azure AI Agent Service — Cédric Vidal, Microsoft
Loading slides…
Read the talk
Building Code-First AI Agents with Azure AI Agent Service
Follow a Python sales agent from SQL queries to document retrieval and generated charts, including the live results that expose the difference between tool use and reliable planning.
From a talk by Cédric Vidal
Before you start: Basic Python and SQL familiarity will help you follow the code; the article explains the agent, thread, run, and tool concepts as they appear.
A sales question should not require a new dashboard
An online outdoor-equipment retailer wants its salespeople to combine sales figures with product information and generate ad hoc diagrams. Building a separate query, view, and interface for every question quickly becomes the bottleneck. The proposed application accepts the question in ordinary language, generates the necessary queries, and chooses a display suited to the answer.
An agent, in Cédric Vidal’s working definition, is semi-autonomous software that pursues a goal using tools and information from its environment. That definition spans different levels of complexity, but three capabilities matter here: reasoning over context, retrieving relevant data, and taking actions. The last capability moves an application beyond returning text toward executing operations that help accomplish a task.
The concrete interface example asks for backpacking-tent sales by region, with a brief description of each tent in the table. Answering requires sales records and product information together. A subsequent request for a sales-by-region pie chart changes the presentation. The illustrated interface is a slightly different application from the one built in the workshop, but it establishes the intended interaction: the user specifies the information and presentation they need.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Where the managed service fits
Azure AI Agent Service supplies the stateful infrastructure for this application. Frameworks such as LangChain, LangGraph, and Semantic Kernel offer other ways to assemble agent behavior; the attraction here is moving conversation storage, context, agent configuration, and supported data integrations into a managed service. Application-managed integrations can still sit alongside cloud-managed tools. Vidal also emphasizes enterprise security, while acknowledging its setup complexity. His broad description of Foundry model support should not be read as a compatibility guarantee for every model and tool combination.
The illustrated application uses Chainlit, while the workshop builds a command-line interface. Beneath either interface, a query function connects to the agent’s instructions and model. Four tool categories extend what the model can do: custom function calling, Code Interpreter, file search, and grounding with Bing Search. The interface changes; the underlying separation between application, agent, and tools remains.
Managed infrastructure still has a programming model to learn. An agent is a persistent cloud object, so starting an application does not necessarily mean creating a new agent. Vidal compares this to creating a database schema only when it does not already exist. The lifecycle is:
- Create and configure an agent, or reconnect to an existing one.
- Create a conversation thread, or reuse a thread.
- Run the agent on that thread.
- Check the run’s status.
- Display the response.
Instructions remain attached to the agent, so the application need not resend them on every request. Model configuration and data sources can be set through Foundry or the SDK; tools may execute in application code or in Azure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A tool request becomes executable work
For a request for total sales by region, the execution path starts with SQL. The model generates a query, application code executes it against the sales database, and the resulting records return to the model for an answer. SQLite is convenient for the workshop, but an application-managed function could access another relational database, a document store, or an API. Cloud-managed connectors have their own supported set. Implementing role-based access control is outside this exercise.
A request to show the results as a pie chart follows a different execution path. Code Interpreter generates Python, executes it in a sandbox with the packages available there, and saves the resulting image inside that environment. The service then makes the image available to the client application. Generating an image therefore involves actual program execution and an output artifact, not just a textual description of a chart.
The same boundary explains function calling. Early implementations could ask a model for comma-separated function names and arguments; structured JSON makes that exchange easier to interpret. Vidal prefers the term function routing: the model selects a function and supplies parameter values, while application code executes the requested operation. A generated call specification is not itself execution.
His sky-color example makes the progression concrete. An unconstrained answer might be blue. An output instruction can narrow that to color=blue. A tool schema can instead express the answer as a request to a color-setting function with a color parameter. If the available alternatives include ordering a pizza, the model uses the question and tool descriptions to select the relevant operation. This differs from ordinary one-of-n classification: the output must identify an operation and extract the values needed for its parameters.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Tool use, workflow chaining, and knowing when to stop
The hosted lab begins with an identity distinction: personal accounts use Microsoft account sign-in, while corporate accounts use Entra ID. Participants redeem a training key to access the environment. An organization’s administrator-approval requirement can block the corporate route; a personal Microsoft account, including an Outlook.com or Xbox account, is an alternative for entering the lab.
While setup proceeds, an audience question sharpens the architectural scope: could explicit workflow chaining do the same job? Yes. This application occupies an intermediate point between a simple completion and a system with a custom goal-completion loop. It combines tools, Code Interpreter, and several data sources, but it does not implement a separate process that repeatedly evaluates whether the overall goal has been achieved. A locally orchestrated LLM could implement the same application; the service manages much of its infrastructure. Vidal plans to demonstrate where the simpler arrangement falls short.
The labs are not prebuilt: Launch or Build starts provisioning, leaving time for a deeper question about termination. With a looping system such as AutoGen, the developer must define what done means. A deterministic check can enforce that condition, or an LLM can judge completion. The workflow repeats until the condition is met. Vidal’s Browser Use prototypes illustrate the difficulty: a browser agent can navigate and act yet still judge incorrectly whether its task is finished. Building a corrective feedback loop is not part of this workshop.
MCP addresses a different layer: connecting an application to tools. It is optional, and it does not turn a tool into an agent. Vidal describes the convenient local experience of selecting a filesystem or Blender server, discovering its functions, and having the client manage its process. Node, Python, or Docker can supply that executable. Package acquisition and catalog installation belong to the surrounding client tooling; MCP’s transport contract covers communication, including local subprocesses and independently running HTTP servers. Automatic downloading and universal cleanup are not protocol guarantees.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Narrow the choice of tools without isolating useful memory
Broad instructions leave more behavior available to an agent; specific instructions narrow the task and, in Vidal’s account, can improve reliability within that task. Tool selection presents a related trade-off. Giving one agent every tool asks it to choose among all of them at each decision. His proposed alternative is staged routing: first classify a question as sales-related or product-related, send it to the appropriate specialist, and then decide whether another specialist is needed.
That arrangement forms a tree of composite and leaf tools: an agent can itself expose a capability implemented by other tools or agents. Specialists may use different databases or retrieval sources. Specialization narrows the available actions; grounding supplies the evidence for an answer. Shared memory can still cross those boundaries. If several agents help the same customer, each may need access to preferences or the name of the customer’s pet. Separate responsibilities do not require separate copies of every useful fact.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect the Python project to Foundry
Once provisioning finishes, the workshop enters a Windows VM using the preselected account. The lab’s password control and T icons type supplied values automatically; Split Windows optionally separates the instructions from the desktop. Azure authentication is a second login with a different identity:
- Run
az loginin the VM. - Choose the work-or-school account option.
- Use the temporary work account supplied in the lab’s Resources pane, even if a personal account was used to enter the lab.
- Execute the provided role-assignment command, which provisioning did not automate.
- Check the returned logs and JSON before continuing.
The lab entry account and the Azure resource account serve different purposes.
The next setup command clones the workshop project, creates a Python virtual environment, installs dependencies and VS Code extensions, and prepares the editor. Edge inside the VM provides the instructions; a PDF extension supports inspecting the product documents later.
In Foundry, sign in with the temporary lab user and select the project, not the hub. Copy its project connection string, rename .env.sample to .env, and paste the complete value inside the existing double quotes. The workshop configures GPT-4o. This is the recording’s historical connection-string setup: current classic function-calling documentation uses project endpoints and now marks classic agents deprecated. Those newer examples are not a drop-in replacement for the demonstrated checkout.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep query generation separate from query execution
main.py is the entry point and configuration center. The sales-data module contains SQLite operations, while shared/instructions contains the prompts. The layout makes the mechanisms easy to inspect rather than presenting a production application template. In particular, the database function is intentionally simple: it accepts SQLite-compatible SQL generated by the model and executes it through the SQLite driver.
In Python, that execution boundary can be expressed with a small function:
python
import sqlite3
def query_sales(
connection: sqlite3.Connection,
sql_query: str,
) -> list[dict[str, object]]:
"""Execute a sales query and return rows keyed by column name."""
cursor = connection.execute(sql_query)
columns = [column[0] for column in cursor.description]
return [dict(zip(columns, row)) for row in cursor.fetchall()]
The model supplies the query text; the driver supplies the records. Vidal attributes competence with familiar database languages to the model’s training, naming PostgreSQL, MySQL, SQLite, and MongoDB as familiar examples. An unfamiliar query language demands more care than assuming the model already knows its syntax.
The configuration imports the Foundry and Azure identity packages, loads .env through dotenv, and sets up logging. Enabling the first instruction file introduces the Contoso sales-analysis role: help users answer sales questions for an outdoor-camping retailer, use the supplied sales tool, and follow formatting, localization, and clarification guidance. The tool schema describes the function name and parameters; the agent instructions add application-specific behavior. The registered tool is asynchronous, and its Python docstring and parameter documentation also reach the model. These descriptions are part of the tool’s operational interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Regional sales work; “last quarter” needs a reference date
F5 or VS Code’s Run button starts the application without a separate terminal setup step. Startup creates a cloud agent with an ID, enables automatic function calls, and creates a persistent thread. The first question asks for sales by region. The generated query sums revenue and groups it by region. Vidal corrects his reading of the result limit from three to thirty and tentatively attributes that limit to the instructions.
The demonstrated regional-sales table reports approximately 5.2 million for Africa and 5.3 million for Asia-Pacific, with currency and the complete filtering period unspecified. These are outputs of the workshop database aggregation. The next step is to ask follow-up questions against the same conversation.
A question about last quarter’s revenue exposes missing temporal context. After the request is narrowed to 2024, the generated query selects months 7, 8, and 9. An attendee suggests a model-training-cutoff explanation, but that cause is not established. Vidal then supplies the missing reference: actually, it is April. The query changes to months 1, 2, and 3.
| Conversation state | Selected months |
|---|---|
| Last-quarter request, narrowed to 2024 | July–September |
| Follow-up establishes April | January–March |
A current-date tool is proposed as an exercise, not added during the demonstration. The following question—best-selling products in Europe—changes the aggregation to product type. Both examples serve the same purpose: a nontechnical user can refine a database request through conversation, but the model still needs the context that gives the request a precise meaning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Schema grounding is necessary; instructions are not access controls
How did the model know the table and column names? Initialization loads the instruction files and inserts the database schema through a replacement step. That schema grounds the generated SQL in the actual database. Knowledge of SQL syntax alone cannot tell the model which tables this application contains. Without that context, it would have to guess.
A confidentiality question reveals a separate responsibility. An instruction can say that column X of table Y is confidential and must never be returned, but the model may fail to follow it. A prompt is not an authorization boundary. Vidal recommends IAM and column-level restrictions, alongside the safeguards used for ordinary applications. The small SQL function above illustrates an execution path; the authority of the database connection and the operations it permits must be controlled outside the model’s instructions. Conventional risks such as SQL injection do not disappear when an LLM generates the query.
A schema can also be structurally correct yet difficult to interpret. Cryptic table names need descriptions of their business meaning: for example, explain that a particular table contains orders. Those descriptions form a semantic layer between the user’s vocabulary and the database’s identifiers. They help the model choose the right data; they do not replace the access controls governing that data.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Reuse evidence, but delegate calculation
The thread changes whether another tool call is needed. Total shipping costs by region require a fresh SQL query because the earlier answers do not contain that information. Asking which regions have the highest sales can reuse the regional-sales table already in the conversation. In the demonstrated follow-up, no new SQL query appears. The agent has the evidence it needs to compare the existing values.
An audience question about numerical correctness leads Vidal to distinguish comparing existing values from calculating new ones. He recounts a team discussion about daily growth compounded over a year: a convincing GPT answer disagreed with his spreadsheet calculation. The expression for that example is:
The anecdote supplies no numerical result. Its practical point is to delegate arithmetic to executable tools rather than trust the fluency of an answer. Vidal reports poor personal results using Code Interpreter for mathematical requests, while finding it useful for charts and structured-file extraction. That is an experience report, not a lack of mathematical capability in sandboxed Python. His suggested alternative is a dedicated calculation tool accepting a mathematical expression and passing it to a calculator.
The SQL-only exercise ends with a more detailed request: tent sales in the United States in April 2022. Those records are absent from the previous answers, so another database query is necessary. Vidal then exits the application before changing its tool configuration.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add the product knowledge that SQL does not contain
The next configuration enables file search and vector-store creation in main.py. A helper uploads the product PDF so the agent can retrieve descriptions of tents and other equipment. These documents contain information missing from the sales database, including brands. Vidal describes the backing store as Azure AI Search, although the precise provisioned backend is not established in the walkthrough.
A VS Code diff shows how the instructions change. They now introduce a Contoso product-information vector store, a search tool, and additional guidance for questions such as which tent brands the retailer sells. The agent can combine structured sales records with unstructured product descriptions. Vidal contrasts this with integrating sources into an OLAP-style analytical system: the model can assemble an answer from both sources at question time.
On restart, the PDF is uploaded. Vidal revises his earlier description of the helper doing the chunking and suggests that processing is server-side. The useful boundary is service-managed document ingestion: parsing, chunking, and embedding need not be handwritten in the application. The exact backing storage depends on the service setup, so that benefit does not establish which infrastructure this lab used.
The new conversation begins with tent brands, answered from the PDF without SQL. The previous thread’s context is intentionally not reused. The response names Outdoor Living and Alpine Gear. A hiking-shoe question finds no corresponding information in the PDF, and a follow-up about these brands’ product types and categories also stays within document information. The absence of a sales question means there is no reason to invoke the sales database yet.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The cross-source query succeeds—even after a restart
Now the request asks for 2024 tent sales by product type and associated brand. Product type and revenue come from SQL; the brand mapping comes from the PDF information retrieved earlier. The resulting table combines the two sources through conversation context. Vidal predicts that asking a comparable question without the preceding product questions may fail because the application lacks a custom planning loop, while acknowledging that simple cases can still succeed.
The next query asks for Alpine Gear sales in 2024 by region. There is no brand column in the sales database. The agent uses the product information to generate a LIKE condition for family camping tents, translating the brand request into a filter the SQL database can express. This is where the cross-source dependency becomes visible: product knowledge determines which sales rows to aggregate.
Vidal restarts the agent and asks that last question again, expecting it to fall short without the earlier conversation. The live query succeeds. He suggests that the file-search instructions may already tell the agent to retrieve product information for such sales questions, but he does not establish the cause. The outcome matters: lacking a custom answer-evaluation loop does not prevent a run from making sequential tool calls. The service’s run lifecycle can support repeated tool requests.
An attendee then reports a different result: Alpine Gear was assigned to both tent types, whereas Vidal’s result distinguished Outdoor Living for backpacking tents and Alpine Gear for family camping tents. Successful execution therefore leaves another question open—whether the retrieved facts were associated correctly. The discrepancy leads directly to evaluation: an answer can look complete while containing an incorrect mapping.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate a chart from retrieved sales
Each exercise retains the previous tools and adds another. Enabling Code Interpreter adds visualization instructions that direct chart requests to the sandbox and specify PNG output. The request now asks for sales by region as a pie chart in one turn, combining the database lookup and visualization. The sequence succeeds and produces a saved revenue-by-region chart.
The underlying operation is ordinary Python plotting over returned data. For example, the chart-producing part can accept the regional totals already obtained from SQL:
python
from pathlib import Path
from collections.abc import Mapping
import matplotlib.pyplot as plt
def save_regional_sales_chart(
revenue_by_region: Mapping[str, float],
output_path: str = "regional_sales.png",
) -> Path:
fig, ax = plt.subplots()
ax.pie(
list(revenue_by_region.values()),
labels=list(revenue_by_region.keys()),
autopct="%1.1f%%",
)
ax.set_title("Revenue by region")
path = Path(output_path)
fig.savefig(path, bbox_inches="tight")
plt.close(fig)
return path
The agent’s flexibility comes from generating the program appropriate to the requested display, then returning its artifact.
Vidal’s kitesurfing example extends the same mechanism beyond charts. He exported a watch recording as XML and asked Code Interpreter to identify turns, jumps, and jump heights. Generated Python parsed the structured file and examined its data points. A later download-as-JSON example in the workshop is simpler: Vidal describes it as reformatting information already in the conversation, without a new tool invocation. The presence of an output format alone does not tell you whether fresh retrieval or computation occurred.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Multiple tool calls are not a completion criterion
Vidal connects these building blocks to deep-research systems: tools plus orchestration can pursue a larger task over many steps. He explicitly does not know those products’ implementations, so AutoGen and Semantic Kernel are examples of possible orchestration approaches, not identified dependencies. A pending request for the effect of a regional sales shock then prompts the audience to challenge the distinction: if this application already uses several tools, what is missing?
The missing layer is an explicit assessment of the answer after tools return. Suppose a question clearly points to sales data but also needs product information that its wording does not mention. Targeted instructions may teach the agent to perform that lookup. A more general evaluation loop instead asks whether the current answer is correct and complete, then chooses what to do next.
| Mechanism | Decision it makes |
|---|---|
| Tool selection | Which operation can help now? |
| Sequential tool use | What operation follows this result? |
| Answer assessment | Does the result answer the whole question? |
| Completion criterion | Is there enough evidence to stop? |
An assessor might discover that the answer is incomplete and request another database lookup or internet search. AutoGen, Semantic Kernel, or another orchestration system can support that behavior when configured for it.
A coordinator over specialized agents is one possible topology, not an automatic solution. The developer still has to specify the completion condition. Vidal identifies that definition of done as the hardest part of this kind of system.
The completed simulation applies a hypothetical 20% drop in North American sales and shows the resulting change in global sales shares. North America’s share falls while the other regions’ shares rise. That does not require their absolute sales to increase: reducing one region’s sales also reduces the global total used as the denominator. Vidal skips the suggested 50% variant. A subsequent question about regions above or below their range produces no confirmed result; he suspects a connectivity problem and relaunches the application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The web-search extension stops at a resource error
The final tool configuration adds grounding with Bing Search and switches the instruction file. A separate multilingual Code Interpreter exercise is skipped; it would address non-English text, encodings, and fonts. The Bing configuration encounters Resource not found, also reported by an attendee. Vidal suggests recent API changes around Build as a possible explanation, but the error is not diagnosed.
The intended source selection is nevertheless clear:
- Internal products: A question about beginner tents the retailer sells should consult the product PDF.
- Competitor products: A question about competitors’ beginner tents and prices needs current information from the internet.
Bing grounding would add that external evidence alongside SQL and document retrieval. The recording shows the intended questions and configuration, but no successful Bing-grounded answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate the conversation and choose what to manage
The deliberately minimal CLI keeps the workshop focused on the model, evidence, and tools instead of adding a web interface to the learning task. Once those pieces work together, evaluation has to expand beyond a single answer. A whole conversation can establish context, change the required lookup, or carry an incorrect association forward. Tool-selection accuracy is another evaluation target: did the agent choose the operation needed to answer the question? Vidal points to the Azure AI Evaluation SDK’s agent capabilities as the subject of a follow-up session.
The final questions clarify execution ownership. Tools do not have to belong to Azure. Foundry’s preconfigured integrations can be mixed with custom application code. In this workshop, the SQL query runs client-side. Bing grounding is declared by the client, but its actual search call runs server-side. Declaring a tool and executing it are separate responsibilities.
An attendee asks whether Agent Service is necessary when ordinary model endpoints can already support tool use, citing perceived pricing and model-availability differences. The answer does not settle those historical pricing or compatibility claims; model and tool usage can carry their own charges. Vidal’s architectural answer is direct: Agent Service is optional managed infrastructure. An application can use ordinary LLM endpoints and implement its own conversations, retrieval, web access, and tool execution. The service’s value is handling common infrastructure through a fuller API—especially persistence—so the application developer has less of it to build and operate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Hands-on outdoor-retail sales agent exercises covering function calling, document retrieval, and generated visualizations.
- MCP transports: March 2025 specificationDocumentation
Defines local stdio and remote Streamable HTTP communication for MCP clients and servers.
Tutorial on configuring stopping conditions for multi-agent conversations.
Updates since the talk
Explains application-executed tools and the agents, threads, and runs lifecycle, including multiple tool requests within a run.
Describes managed document ingestion and the storage differences between basic and standard agent setups.
Documents sandboxed Python execution for charts, mathematical tasks, and data analysis, with separate usage charges.
Explains web grounding setup, citations, usage charges, and data-handling boundaries.
Full transcript
Loading transcript…