AI Engineer World's Fair 2025
Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive — Emma Ning, Microsoft
Loading slides…
Read the talk
Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive
From unreliable conference Wi-Fi to private document summaries and receipt extraction, Foundry Local brings model management, hardware acceleration, and tool use onto the device.
From a talk by Emma Ning and Tsavo Knott
Before you start: Familiarity with command-line tools and JavaScript chat-completion requests will help you follow the application integration.
When the network cannot be a dependency
What happens to a cloud AI demo when the conference Wi-Fi fails? The model may be powerful, but the application still depends on reaching it. Microsoft product manager Emma Ning opens with that familiar problem: presenters hoping their connection survives the next request. Her Foundry Local demonstration runs inference on the device instead.
Network reliability is only the first reason to move inference closer to the application. The other constraints concern what data can leave, what repeated requests cost, and how quickly the application must respond:
- Privacy and security: Legal documents and patient information may need to be processed entirely on-device.
- Cost efficiency: Ning asks developers to imagine a game deployed to millions of devices, generating hundreds of millions of inference calls each day. This is a workload scenario motivating local execution, rather than a measured cost comparison.
- Real-time latency: Interactive applications can have response requirements that leave little room for a cloud round trip.
Local inference makes the device an execution environment, rather than just a client of a remote model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From capable hardware to a usable local stack
Local AI becomes practical when hardware, models, and runtime software improve together. Modern client GPUs and NPUs provide more compute; smaller models such as Phi-4 Mini and small DeepSeek variants reduce the resources required for useful generation; runtime optimizations make better use of the available hardware. None of those pieces alone supplies an application developer with the whole experience.
Microsoft assembles Foundry Local from existing assets. Ning describes Azure AI Foundry, introduced at the previous year's Microsoft Ignite, as serving more than 70,000 organizations and offering more than 1,900 models at the time of the talk. The Build 2025 announcement places that model count in the cloud platform's partner-hosted and Microsoft-hosted catalog; it is not a count of models available for local execution.
ONNX Runtime supplies the cross-platform inference engine. Ning reports more than 10 million ONNX Runtime downloads per month and describes customer-reported acceleration over PyTorch, without presenting a quantified performance comparison. Windows contributes distribution and reach across millions of client devices and users. Foundry Local combines these assets into a managed path from model selection to on-device execution.
The stack separates execution from model management and application integration:
| Layer | Responsibility |
|---|---|
| ONNX Runtime | Accelerate inference across supported hardware |
| Foundry Local management service | Host and manage models on the client device |
| Model acquisition | Download open-source models from Azure AI Foundry on demand |
| CLI | Explore and run models locally |
| SDKs | Integrate local inference into applications |
This division lets a developer explore models through the CLI before embedding the selected model in an application.
At the time of this June 2025 talk, Foundry Local had recently been announced at Microsoft Build and was available on Windows and macOS. Ning emphasizes its Windows integration; the launch announcement distinguishes the installable preview from direct Windows 11 and Windows App SDK availability, which was still forthcoming. Hardware work with NVIDIA, Intel, AMD, and Qualcomm connects the runtime to their accelerators. Ning says more than 100 customers joined the private pre-preview and supplied feedback before the public announcement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Developer memory and sensitive speech
Pieces provides one example of why packaging and lifecycle management matter alongside inference speed. Tsavo Knott, its CEO and co-founder, describes a product that gives developers artificial long-term memory across the operating system. Offline-first AI is central to that goal, and the team began exploring small language models across major platforms in late 2022. It moved from managing its own llama.cpp setup to Ollama, but still encountered difficulties with versioning, performance, and the end-user experience.
Knott reports a rapid move from access to Foundry Local documentation to a production-ready build, with improvements in memory management, time to first token, and tokens per second. The testimonial supplies no measurements, but identifies the operational concerns that a local inference platform needs to address: loading and managing models, starting responses promptly, and sustaining generation within a desktop application's resource budget.
A second customer example starts from a stricter data boundary: clients want generative AI but cannot send sensitive information to external services. The demonstration combines Foundry Local with a speech-to-text service that also runs locally. Its sample speech includes the name Maria Rodriguez and a redacted savings account number—the sort of input that makes the location of processing consequential. The customer highlights straightforward installation and model use, then describes hybrid applications in which only part of the workload runs locally.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover hardware variants and prepare for offline use
The CLI demonstration starts with Foundry Local already installed. Windows installation uses WinGet; macOS installation uses Homebrew. The first exploration command is:
bash
foundry model list
The catalog exposes models together with variants optimized for different hardware. Ning's device shows CPU, CUDA, and integrated-GPU options. A Qualcomm NPU variant does not appear because that device lacks the corresponding NPU. Model discovery therefore includes hardware compatibility, not just a list of model families.
Offline inference still requires preparing the device. When a model is not cached, Foundry Local downloads it from the cloud before running it, which requires internet access. Ning has downloaded the models in advance and inspects them with:
bash
foundry cache list
Her cache contains four models. Keeping several models locally allows her to compare response quality and performance without depending on the conference network for model acquisition.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Compare speed and answer detail with the same prompt
Ning first uses foundry model run to load a cached Qwen 2.5 1.5B model. Once its interactive session is ready, she asks, “What's ONNX Runtime?” The initial answer arrives quickly enough that she exits and repeats the request in verbose mode to inspect the generation rate.
For the Qwen 2.5 1.5B response to “What's ONNX Runtime?”, Ning reports approximately 90 tokens per second in verbose mode on her device. The hardware specification, quantization, execution provider, runtime build, measurement boundaries, and repeated trials are not supplied, so this number describes the displayed run rather than a portable performance expectation.
She then loads Phi-4 Mini in verbose mode and asks the same question. In this demonstration, the larger Phi-4 Mini model generates more slowly but supplies more detailed information. Ning prefers that answer for the application she wants to build.
| Model | Observed behavior in the demo | Selection implication |
|---|---|---|
| Qwen 2.5 1.5B | Faster, briefer response | Useful candidate when speed and brevity matter |
| Phi-4 Mini | Slower, more detailed response | Ning's choice for the summarizer |
The comparison makes model selection tangible: run the intended style of request, inspect both the output and the execution behavior, and choose according to the application's needs. Here, detail in one answer drives the choice; the demonstration does not establish general model superiority.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Summarize internal documents without uploading them
The application problem is onboarding into an unfamiliar project. A team has long, detailed documents to read and wants high-level summaries to get oriented. Those documents are internal and cannot be uploaded to the cloud. Some teammates use Windows and others use Macs, so the solution needs both local inference and a cross-platform application.
The prepared summarization app accepts either a URL or a local file. Its settings tab lets the user select a model by ID. Ning chooses Phi-4 Mini, supplies the project document, and submits it for summarization. The response describes Foundry Local as a way to build cross-platform AI applications that run directly on-device. The workflow carries the earlier model experiment into a concrete task: choose a model, provide the document, and receive a summary inside the application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connect the JavaScript app and run it on a Mac
Foundry Local offers Python and JavaScript SDKs; the summarizer uses JavaScript. The demonstrated integration creates a Foundry Local manager, initializes it with the selected Phi-4 Mini model, and uses the manager's local endpoint to construct a chat client. The application then awaits a chat completion and displays the result. This is the talk's manager-and-endpoint integration; today's SDK examples emphasize in-process inference with optional serving, so the two integration styles should not be mixed.
Once the client points at the local service and the selected model is initialized, the summarization request can remain a small JavaScript function:
javascript
async function summarizeDocument(client, modelId, documentText) {
const completion = await client.chat.completions.create({
model: modelId,
messages: [
{
role: "system",
content: "Summarize the document's main ideas and key details."
},
{
role: "user",
content: documentText
}
]
});
return completion.choices[0].message.content;
}
The manager handles model initialization, while this function handles the application's request and response. The model ID must refer to the selected local model, and the client must use the Foundry Local endpoint.
Ning then packages the Windows project and shares it with a teammate who records the Mac experience. The teammate starts the same code with:
bash
npm run start
The application and UI are unchanged. On the Mac, the teammate selects Qwen and submits the same document. Its summary is shorter than the earlier Phi-4 Mini output, echoing the difference seen in the CLI comparison. The handoff demonstrates reuse of the application across operating systems while leaving model choice available to the user.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give a local model tools through MCP
The final demonstration moves from generating a summary to using tools. Foundry Local's agent feature is in private preview in the recording. Its composition is straightforward: one local model plus one or more MCP servers. The model supplies the reasoning and tool selection; the configured servers supply the available operations.
Ning returns to the terminal and runs:
bash
foundry agent list
The output shows three sample agents: a file organizer, an OCR agent, and a web interactor, with their server counts and model names. A developer can compose an agent by choosing a model from the catalog and the MCP servers needed for the task.
For this walkthrough, Ning selects the existing OCR agent and inspects it with foundry agent info. It extracts text from images on the local device and combines a Phi-4 Mini variant with two servers: a filesystem MCP server and an OCR Mini MCP server. She then launches it with foundry agent run. These are the private-preview commands shown in the recording.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Authorize access, find the receipt, and extract its total
Launching the agent does not immediately grant it unrestricted access. Startup first checks its dependencies and, if any are missing, offers to install them with permission. Ning has already installed the dependencies. She then works through the access prompts:
- Approve the filesystem MCP server.
- Supply the allowed directory: the folder containing the demonstration photos.
- Approve the OCR MCP server.
After setup, the agent lists the tools provided by those servers, including filesystem and OCR operations. Dependency installation, server approval, and directory selection are separate parts of preparing the agent to act.
The user request is to find a receipt, process it, and obtain the total amount. The agent first selects file search to locate the receipt image. It then uses OCR to extract the text and produces the total. The completed terminal view shows the located file, extracted receipt text, and a final result of Total: $8.39. This is a concrete division of labor: filesystem tools locate the input, OCR turns image content into text, and the model uses those results to answer the request.
The receipt task also gives the closing capability warning a practical setting. Ning cautions that local models are generally less capable than cloud models and should not be expected to perform the same sophisticated model or agent work. A bounded task with relevant tools can still be useful, as the receipt workflow shows. She closes by pointing attendees to further information and a signup form for the agent private preview. The engineering opportunity is to choose work that fits the local model's capabilities while benefiting from on-device data handling and execution.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Documentation for the inference runtime underlying Foundry Local.
Further reading
- Foundry Local at Build 2025Article
The launch-era account of Foundry Local preview and its relationship to Windows ML and Windows AI APIs.
Microsoft tooling for converting, quantizing and optimizing models for different hardware targets.
Microsoft's model card for the instruction-tuned Phi-4 Mini model.
The Qwen team's model card and usage guidance for its 1.5-billion-parameter instruction model.
Updates since the talk
Current SDK quickstarts, model management, hardware acceleration and application samples.
Current guidance for local agents, model-dependent tool calling and local MCP integration.
Full transcript
Loading transcript…