AI Engineer World's Fair 2024
Accelerate your AI journey with Azure AI model catalog
Loading slides…
Read the talk
From model selection to production with Azure AI Model Catalog
Follow model discovery, deployment, grounded chat, function calling, and flow evaluation to see how a shared inference interface supports application-specific model choices.
From a talk by Sharmila Chokalingam and Shubhi
Before you start: Familiarity with API endpoints, chat messages, and the basic idea of retrieving documents to inform an LLM response will help you follow the demonstrations.
Can AI solve the use case—and can it scale?
Can AI solve this use case? Which model fits it best? And how can the application scale into production? A growing model catalog makes experimentation easier, but it also makes these three questions harder to answer through model reputation alone. The application needs a process for establishing feasibility, improving the result, and operating it reliably.
Start by prototyping: try several models, compare benchmarks, and shortlist those that can perform the task. Then optimize for the application’s cost, latency, and regional requirements, using prompt engineering, retrieval-augmented generation (RAG), or fine-tuning as appropriate. Finally, operationalize the application with capacity planning, cost management, monitoring, scalability, privacy, and content filtering. These stages form a loop: a promising prototype still needs application-specific optimization before it becomes a production workload.
That process need not end with one model for everything. Sharmila asks whether customers use multiple models within a use case or across different use cases. Shubhi describes the catalog as a way to plug models into an application, compare them quickly, and replace one without rebuilding the surrounding code. Model choice becomes useful when the cost of testing another model is low.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose capabilities, not just model size
The catalog shown in the recording includes GPT-4o, Mistral, Llama, and Cohere models, alongside smaller models such as Phi-3 and Mistral’s open models. Selection also extends beyond text generation to multimodal tasks, image generation, and embeddings. A model’s size is only one part of its suitability; its supported operations and the data it must handle matter too.
- Tools and structured output: GPT-4o’s function calling and JSON support serve agent workflows that must connect generated responses to application operations.
- Language coverage: Shubhi highlights Mistral for European languages and Cohere’s multilingual embeddings for multilingual retrieval. Jais supplies an Arabic-language option.
- Open and proprietary choices: Hundreds of Hugging Face models broaden the selection, with Meta, Databricks, Snowflake, and NVIDIA named as partners in bringing models onto the platform.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect a model, deploy it, and compare benchmarks
The walkthrough begins in Azure AI Studio’s Model catalog. Shubhi reports more than 1,600 models in the catalog at recording time. Filters narrow that collection by deployment type, inference task, or model family, so discovery can begin with an application requirement rather than a long list of names.
Filtering for Cohere leads to Cohere Command R. Its model card describes customization, tool use, and RAG capabilities, and provides inference samples for integrations such as LangChain and LiteLLM. The card is therefore both a capability reference and a starting point for application code. Standardized APIs are intended to let those integrations work across the catalog’s supported models.
The deployment procedure then moves from model selection to a billable endpoint:
- Review pricing and select Deploy.
- Subscribe to the model’s Azure Marketplace offer, which supplies the billing arrangement for token usage.
- Continue with an existing subscription when one is already in place.
- Choose a deployment name, decide whether to enable content filtering, and submit the deployment.
Shubhi describes an expected setup time of under a minute for obtaining the endpoint URL and key.
While deployment runs, the walkthrough opens Model benchmarks. Accuracy, coherence, and groundedness provide common dimensions for narrowing the model shortlist. Returning to the deployment page, Shubhi reports that this deployment completed within a few seconds; its target URL and key are ready for use in application code. That observed deployment is the basis for the timing claim, rather than a service-level guarantee.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the same question a better source
Before moving into an IDE, the Chat Playground provides a way to test the deployed model interactively. Shubhi selects an existing Mistral Large deployment and asks how Microsoft promotes a culture of giving. Without custom data, the answer describes general initiatives, including employee matching programs and information available publicly.
Next, Add your data connects the chat to an existing index named Microsoft Give. The same question is submitted again, this time with the expectation that indexed documents will supply organization-specific context. The useful comparison holds the question and model constant while changing the information available to the answer.
While the response is pending, Sharmila asks whether Microsoft or the model providers use customer data to train models. Shubhi’s assurance is that inference prompts and completions are not shared with model providers and are not used to train the models. This concerns the handling of inference data; deliberately uploading data for a fine-tuning job is a separate operation.
The returned answer includes a concrete detail: the giving program provides a $50 new-hire credit. That detail was absent from the earlier response. The demonstration’s improvement is not simply a longer answer—it is an answer containing information from the connected index. The $50 amount is the playground’s reported output, not a separately established statement of current employee benefits.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate infrastructure choice from the inference interface
The deployment just demonstrated uses a serverless API. The catalog also offers managed compute, which changes both the billing unit and who must arrange capacity.
| Deployment option | Capacity responsibility | Billing | Recording-time selection |
|---|---|---|---|
| Managed compute | User obtains GPU capacity and manages quota | VMs per hour | Hundreds of open models |
| Serverless API | Ready-to-use hosted endpoint | Input and output tokens consumed | More than 30 flagship or premium models, according to Shubhi |
The serverless route is described through Azure OpenAI Service and Models as a Service. Managed compute makes GPU quota and capacity explicit application concerns; serverless APIs let the application begin with an endpoint and usage-based charges.
Above those deployment choices sits a shared inference contract. Azure works with model providers to standardize REST schemas and SDK structures, supporting integrations such as LangChain and provider-specific SDKs. The intended workflow is to change the endpoint, run evaluations, and compare the resulting application behavior.
A shared schema provides interface portability, not identical model capabilities. The later Azure AI Model Inference REST API reference makes that boundary explicit: models can reject unsupported parameters. Endpoint substitution therefore depends on compatible capabilities, including the tools or output formats the application requires. The recording uses the historical Azure AI Studio interface; the later reference’s API examples should not be treated as the exact version demonstrated here.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use a function for the shop’s actual bill
A stationery-shop chatbot exposes a different grounding problem. The model may know what a stapler is, but it does not automatically know this shop’s prices or current discounts. Shubhi uses Mistral Large’s function-calling capability to connect the conversation to a Get Bill Amount function. The model identifies the operation and its arguments; the application’s function supplies the business-specific result.
The tool definition describes the callable operation to the model. Separately, the system message asks the assistant to help users work out what they must pay and to offer environmentally conscious advice about using the items. Those responsibilities remain distinct: the external function supplies the bill, while the system instructions shape the accompanying explanation.
For a request about 10 staplers, the model selects the bill function and supplies the item and quantity. In Python, the application-side handoff can keep that boundary explicit: validate the requested operation, call the shop’s pricing function, and package its result as a tool message for the next model turn.
python
import json
from collections.abc import Callable
def run_bill_tool(
tool_call: dict,
get_bill_amount: Callable[..., dict],
) -> dict:
function = tool_call["function"]
if function["name"] != "get_bill_amount":
raise ValueError("Unknown tool")
arguments = json.loads(function["arguments"])
item = arguments["item"]
quantity = arguments["quantity"]
if not isinstance(item, str) or not item.strip():
raise ValueError("Item must be a nonempty string")
if type(quantity) is not int or quantity <= 0:
raise ValueError("Quantity must be a positive integer")
bill = get_bill_amount(item=item, quantity=quantity)
return {
"role": "tool",
"tool_call_id": tool_call["id"],
"content": json.dumps(bill),
}
Here, get_bill_amount belongs to the shop’s application. The model’s proposed arguments do not themselves establish a price; executing that function does.
The demo’s final answer prices 10 staplers at $45, including ongoing discounts. The notebook also shows environmental and safety reminders, reflecting the system message. This separates two sources of response content: a tool result for the amount owed and instructions for how the assistant should discuss the purchase.
The notebook returns to the broader endpoint-swapping workflow, while retaining provider-specific capabilities where needed. Its safe_prompt setting is set to true; this is a Mistral-specific option, not a universal safety switch for every model behind the common interface.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build retrieval as a flow, then vary the models
The next demonstration moves from notebook code into prompt flow, building a shopping assistant with a retrieval pipeline. A basic version takes a user question, looks up relevant information in an index, and sends that context to an LLM for generation. Shubhi adds a query-transformation step before retrieval:
- Rephrase the user’s succinct question into a more explicit search request.
- Lookup relevant context in the index.
- Generate an answer using the retrieved information.
Shubhi says that expanding short questions has produced better RAG results in their experience; no measured comparison is supplied.
With the compute session running, the example starts with: “Do you have any new hiking shoes?” The rephrase step expands the request to ask about hiking-shoe availability, materials, and features. The lookup output then exposes the context retrieved from the supplied index before that information is passed to the generation step. Inspecting these intermediate outputs makes it possible to distinguish a retrieval problem from an answer-generation problem.
The narration next points to a women’s jacket recommendation as the generated output. That recommendation does not match the earlier hiking-shoes request, so this sequence demonstrates the flow’s stages without establishing a successful answer to that particular question. The intermediate query, retrieved context, and final response need to be evaluated together.
To compare models without rebuilding the flow, the generate node supports variants. Its initial connection uses Cohere Command R; another variant can point the same node at a different model connection. Embeddings are another choice: the demonstration uses an Ada embedding model and proposes pairing Command R with Cohere embeddings. Shubhi calls the latter “Command Embed”; the public product name is Cohere Embed, and the exact proposed deployment identifier is not established here.
Previously run evaluations compare Cohere Command R, Llama 3, and Mistral Large. The purpose is to compare outputs from the same application flow across evaluation parameters. No numeric scores or winning model are reported, so the useful result is the evaluation setup: preserve the workflow, vary a model connection, and inspect how the application’s outputs change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Control data handling, model integrity, and access
The enterprise discussion organizes the platform’s controls into three areas: data privacy, security and compliance, and responsible AI and content safety. Returning to the earlier inference-data assurance, Shubhi explicitly says it applies to both managed compute and serverless APIs: prompts and completions are not shared with model providers or used to train the models.
Model integrity adds another layer. HiddenLayer scanning is described as checking for embedded malware and backdoors, common vulnerabilities and exposures, and tampering or corruption across model layers. Shubhi ties the required checks to models labeled curated by Azure AI. That scope matters: a curated-model scanning process is not a guarantee that every model in the catalog is free of vulnerabilities.
- Network isolation: Private networking allows ingress and egress to be routed through virtual networks, with fully qualified domain name (FQDN) rules approving outbound access to non-Azure resources.
- Model access: Azure Policy integration supports allow-list and deny-list patterns for model collections.
- Environment separation: Different policies can govern development, test, and production environments.
These controls address different boundaries: where traffic can travel, which models an application may use, and which rules apply at each deployment stage.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From internal assistance to customer-care automation
The customer examples return to model choice across applications. EY began with models available through Azure OpenAI Service and was exploring Llama for more specific documentation and summarization tasks. Its EY.ai platform supports professional work where productivity and accuracy matter. Sharmila reports that EYQ chat had been adopted by 275,000 internal employees. The recording does not define whether adoption means access, registration, or active use.
EY’s operational lesson is that deploying AI is not a one-time selection exercise. Performance needs continuous evaluation, and production use must remain aligned with responsible-AI practices. Sharmila cites the catalog and Studio’s evaluation capabilities as reasons EY uses the platform: model selection continues after the first application ships.
CMA CGM, operating across sea, land, air, and logistics, provides a different example. Building on traditional robotic process automation, it used a Mistral model from the catalog for a chatbot scenario supporting customer-care agents. Sharmila reports reduced response latency and increased customer satisfaction for that use case, without numeric measurements. Planned extensions include invoicing, customer-document analysis, interpretation of free-form client text, and email writing.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Forecasting broadens the catalog beyond chat
Bridgestone’s example moves beyond conversational applications to monthly demand forecasting across a large product portfolio. It uses Nixtla’s TimeGEN time-series model, which Sharmila describes as having launched on Azure at Build the month before the recording. The goals are to simplify forecasting pipelines, improve accuracy, and reduce operational complexity.
Sharmila reports that Bridgestone reduced forecasting errors by nearly 30% on average using TimeGEN. The recording does not specify the error metric, comparison baseline, or evaluation procedure, so the figure describes this reported customer result rather than an expected improvement for other forecasting workloads.
The closing claim connects these customer applications to the opening lifecycle. Sharmila attributes a significantly shorter path from initial model use to business impact to the catalog and Studio’s integrated tools, without giving an elapsed-time measurement. The connection is practical: discovery and prototyping establish feasibility, optimization adapts the model to the task, and operational controls support production use. Keeping those activities connected lets an application continue evolving as its data, requirements, and model choices change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Nixtla's overview of deploying its TimeGEN-1 forecasting model through Azure.
Further reading
- Cohere EmbedArticle
March 2024 announcement introducing compressed embeddings and asynchronous embedding computation.
Historical partnership announcement with customer accounts, including CMA CGM's internal assistant.
Updates since the talk
Reference for shared inference requests, SDKs, model-specific parameters, and capability errors.
Full transcript
Loading transcript…