AI Engineer World's Fair 2024
Creating and scaling your own custom copilots with Azure AI Studio
Loading slides…
Read the talk
Creating and scaling your own custom copilots with Azure AI Studio
A sales assistant can retrieve the right data and still draw the wrong chart. Hanchi Wang’s Azure AI Studio walkthrough follows the engineering work needed to understand and improve that behavior.
From a talk by Hanchi Wang
Before you start: Basic familiarity with Python functions, SQL queries, and language-model tool calling will help you follow the implementation.
What does it take to add AI to an application?
How do you turn a language model into an application that uses your business data, calls the right tools, and keeps working after deployment? Hanchi Wang opens with the shift from discovering generative AI in 2023 to putting it to business use in 2024. That shift exposes an engineering problem: a model cannot simply be attached to an existing application and left to operate. It needs domain-specific knowledge and tools.
The integration also needs content-safety filtering, systematic output evaluation, and continuous monitoring after deployment. The application lifecycle extends beyond generating an answer. Azure AI Studio and Prompt flow are the two complementary tools Wang uses to work through that lifecycle.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A cloud hub and local developer tools
Azure AI Studio brings models, cognitive capabilities, search, and pretrained APIs into a development hub. Azure Machine Learning supplies the deeper data-science, model-training, and LLMOps capabilities, alongside tools for trustworthy and secure AI systems. In this 2024 walkthrough, Studio had reached general availability at Microsoft Build the previous month. Wang reports more than 20,000 monthly enterprise customers at the time.
Prompt flow supplies an SDK, CLI, and VS Code extension for building, testing, tracing, and evaluating applications. Its open-source tooling is free and can run locally without an Azure account; that does not make model inference or hosted services free. Connecting it to Studio adds cloud tracking for assets, traces, service connections, and run results.
The walkthrough concentrates on three connected jobs:
- Tracing and instrumentation: capture application events and inspect what happened inside a request.
- Evaluation: measure application quality and safety so changes can be compared.
- Monitoring: observe deployed workloads and investigate production behavior.
These jobs share evidence: a trace that explains a local failure can also help interpret an evaluation result or diagnose a deployed request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two tools, a follow-up question, and an incorrect chart
The running example is a chatbot for an outdoor-equipment company’s sales data. It uses the Assistants API to coordinate two tools. A custom tool translates a natural-language question into SQL and fetches the corresponding data. A built-in code interpreter works with that data to draw charts. The initial response combines a chart with an explanation of what the assistant did.
Wang first requests monthly sales for 2024 and a line chart. He then asks for a follow-up: overlay the bar chart with a line showing the percentage difference compared with 2023. This depends on conversational context—the user does not repeat the year, grouping, and sales-analysis task from scratch. The demonstration uses the historical Assistants API; its persisted conversation history supports this interaction, although later documentation clarifies that threads can be truncated to fit the model’s context window.
The assistant produces a line chart, but Wang identifies it as incorrect. He had encountered similar behavior while preparing and switches to a prepared correct result. In that result, the percentage differences are all negative: the comparison indicates declining sales relative to the previous year. Producing a chart is not the same as answering the question correctly. The failure makes evaluation and monitoring immediate requirements for this application.
The application’s sequence diagram separates the assistant’s orchestration from the SQL retrieval tool and code interpreter. Prompt flow wraps the application as an outer instrumentation layer, enabling tracing and monitoring without making those concerns the assistant’s business logic. Wang then points viewers to a GitHub repository containing the demonstration code before moving into the Python implementation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Follow the request from Python function to generated SQL
The chatbot’s public interface is an ordinary Python function: it receives a user question and returns chat output. The addition is Prompt flow’s @trace decorator, which captures function inputs, outputs, and execution events. The configured tracing also captures LLM interactions such as Azure OpenAI calls. Prompt flow’s tracing documentation distinguishes function decoration from initialization with start_trace() for LLM instrumentation; the decorator alone is not the entire setup.
The Sales Data Insight tool follows the same pattern as a callable Python class with a traced entry point. This keeps observability attached to meaningful application boundaries. For example, an adapter around an existing question-to-SQL function can preserve that callable structure:
python
from collections.abc import Callable
from promptflow.tracing import start_trace, trace
start_trace()
class SalesDataInsight:
def __init__(self, generate_sql: Callable[[str], str]):
self.generate_sql = generate_sql
@trace
def __call__(self, question: str) -> dict[str, str]:
sql = self.generate_sql(question)
return {"question": question, "sql": sql}
The wrapped function supplies the SQL-generation behavior; the decorator makes its input and returned SQL inspectable. In Wang’s application, the tool also retrieves the data. The second tool, Code Interpreter, is built into the Assistants API and does not require an equivalent custom implementation.
Prompt flow provides a local testing UI for exercising these entry points. Wang launches it with pf flow test, pointing the command at the chat-completion function, then repeats the sales question. While the request runs, he opens the trace view and refreshes it to see new traces arrive. The UI runs on his local machine, so this inspection is part of the development loop rather than a task deferred until deployment.
Once the request completes, the trace supplies a sequence of progressively more specific questions:
- Chat boundary: what question entered the application, and what answer came back?
- Tool boundary: how did the assistant rephrase the question before passing it to Sales Data Insight, and what did the tool return?
- LLM call: what SQL did the model generate from that question?
- Code Interpreter: what Python did the interpreter generate to produce the chart?
This separation matters for the earlier failure. A wrong chart could originate in question interpretation, SQL generation, or chart-producing code; the final image alone cannot locate the problem.
The emitted instrumentation uses OpenTelemetry traces and events. Collectors can route it to destinations such as Azure Monitor, and an application that already emits OpenTelemetry can inspect its existing spans alongside Prompt flow’s instrumentation. The result is a connected request history rather than a separate AI-only debugging record.
For collaboration, Wang has configured locally generated traces to reach Azure AI Studio. Studio presents the same trace view, with a URL he can share with a colleague and storage beyond the local testing session. The instrumentation is framework agnostic: the same decorator approach can cover a single LLM call, retrieval-augmented generation, function calling, or single- and multi-agent workflows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn sales questions into repeatable evaluations
Tracing explains an individual execution. Evaluation asks whether the application behaves well across a collection of inputs, including cases where an LLM fabricates data or returns an incorrect response. It also supplies evidence for choosing among models on accuracy, cost, and performance. Wang narrows the target to the Sales Data Insight tool, where natural-language questions become SQL queries.
Because model outputs are nondeterministic, evaluation needs to run regularly against a high-quality test dataset. Prompt flow offers tools for generating synthetic test data, but Wang uses a prepared dataset. Its question column contains what a user might ask the chatbot; its ground-truth column contains the expected output from the Sales Data Insight tool. This gives each generated query something concrete to be evaluated against.
The evaluate function connects three parts: the dataset, the target application or tool, and the evaluators. It acts as the execution engine, while each evaluator defines a particular check. Wang combines a built-in safety evaluator with three custom evaluators:
| Evaluator | What it checks |
|---|---|
| Content safety | Harmful, hateful, or violent content in queries and outputs |
| Execution time | Time taken by the target |
| Error rate | Frequency of errors |
| SQL similarity | Generated SQL’s similarity to ground truth |
Safety and task quality answer different questions. A response can pass a content-safety check while still containing the wrong SQL. Likewise, SQL similarity is the demonstrated comparison measure; it should not be silently treated as proof that two queries return equivalent results.
Wang opens results from an evaluation run completed before the talk, first in the terminal and then in Studio. Studio shows aggregate metrics at the top and individual dataset rows below. He describes the displayed safety results as satisfactory. For a surprising result, the page also exposes evaluation traces, letting the developer inspect how a metric was calculated instead of accepting only its aggregate value.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a model using the application’s own dataset
At the time of the talk, Wang reports more than 1,600 models in Studio’s catalog. Catalog benchmarks help with discovery, but the more relevant experiment is to run the same sales-query dataset through different models. His earlier runs include Cohere models and the Phi-3, Mistral, Llama, and GPT families. He selects GPT-35-Turbo, GPT-4 Turbo, and Mistral Large for a closer comparison, using GPT-35-Turbo as the baseline.
For Wang’s prepared natural-language-to-SQL dataset, the displayed runs have the following relative results:
| Model | Execution time | Error rate | SQL similarity |
|---|---|---|---|
| GPT-35-Turbo | Fastest of the three | Higher than the other two | Lowest of the three |
| Mistral Large | Slower than GPT-35-Turbo | Lower than GPT-35-Turbo | Higher than GPT-35-Turbo |
| GPT-4 Turbo | Slowest of the three | Same as Mistral Large | Highest of the three |
Wang therefore prefers GPT-4 Turbo for this application if speed is not a concern. The comparison is specific to his dataset and evaluators; he gives relative rankings without numerical scores or a cost comparison. The useful decision is the tradeoff: higher SQL similarity requires accepting more execution time in these runs.
The evaluation machinery does not require every metric to come from a platform-provided catalog. Custom evaluators can be ordinary Python functions, allowing application-specific checks to run alongside built-in evaluators such as content safety. That makes the evaluation target’s actual responsibilities—not just a model’s general benchmark performance—the basis for comparison.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use production telemetry to decide what to fix
Deployment changes the scale of observation. Engineers still need the details of individual requests, but they also need system-wide telemetry to see whether the application continues to behave as expected. Wang’s team deploys the chatbot, sends Prompt flow’s OpenTelemetry spans and events to Azure Application Insights, and builds a dashboard over them.
The dashboard separates performance, usage, and failure metrics. In the displayed dashboard, GPT-4 Turbo has the highest model duration among the shown models, consistent with the earlier evaluation. Prompt tokens account for most of the displayed token usage, with comparatively few completion tokens. That distribution suggests a concrete cost investigation: examine whether the prompt templates can be shortened. It is a proposed intervention, not a demonstrated optimization.
Failure counts provide another entry point. Wang tentatively attributes some spikes to rehearsals earlier in the week. When a failure needs investigation, the developer can move from the dashboard into a specific request’s trace and stack trace. Aggregate monitoring identifies a pattern; request-level evidence helps explain its cause.
The closing product scope is broader than the sales demo: an end-to-end toolchain for efficient AI application development, with responsible AI practices, enterprise scalability, and security as stated platform priorities. In the application just examined, that lifecycle has a concrete shape: preserve the execution evidence, evaluate the SQL-generating tool against known examples, and keep inspecting real workloads after deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Microsoft's open-source Python tooling for developing, tracing, evaluating and deploying LLM applications.
Further reading
Examples of LLM instrumentation, function decorators and OpenTelemetry-compatible traces.
A historical sales-analysis example combining SQL retrieval, chart generation, local testing and cloud tracing. Setup uses older Azure APIs and model deployments.
Application Insights trace inspection and KQL dashboard examples for model duration and token usage.
Contemporary Microsoft recap confirming Studio general availability and a catalog exceeding 1,600 models.
Updates since the talk
- Azure OpenAI Assistants and conversation threadsDocumentation
Later documentation explaining Assistants, tools and persisted threads, including context-window truncation.
Full transcript
Loading transcript…