AI Engineer World's Fair 2025

Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive — Emma Ning, Microsoft

Loading slides…

Read the talk

Foundry Local: Cutting-Edge AI Experiences on Device with ONNX Runtime and Olive

From unreliable conference Wi-Fi to private document summaries and receipt extraction, Foundry Local brings model management, hardware acceleration, and tool use onto the device.

From a talk by Emma Ning and Tsavo Knott

Before you start: Familiarity with command-line tools and JavaScript chat-completion requests will help you follow the application integration.

When the network cannot be a dependency

What happens to a cloud AI demo when the conference Wi-Fi fails? The model may be powerful, but the application still depends on reaching it. Microsoft product manager Emma Ning opens with that familiar problem: presenters hoping their connection survives the next request. Her Foundry Local demonstration runs inference on the device instead.

Network reliability is only the first reason to move inference closer to the application. The other constraints concern what data can leave, what repeated requests cost, and how quickly the application must respond:

  • Privacy and security: Legal documents and patient information may need to be processed entirely on-device.
  • Cost efficiency: Ning asks developers to imagine a game deployed to millions of devices, generating hundreds of millions of inference calls each day. This is a workload scenario motivating local execution, rather than a measured cost comparison.
  • Real-time latency: Interactive applications can have response requirements that leave little room for a cloud round trip.

Local inference makes the device an execution environment, rather than just a client of a remote model.

The “Why local AI” slide shows four illustrated cards linking these motivations to warehouses, healthcare, video games, and creativity tools.Open full source frame
Four reasons for local AI: low bandwidth, privacy and security, cost efficiency, and real-time latency.
0:160:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

From capable hardware to a usable local stack

Local AI becomes practical when hardware, models, and runtime software improve together. Modern client GPUs and NPUs provide more compute; smaller models such as Phi-4 Mini and small DeepSeek variants reduce the resources required for useful generation; runtime optimizations make better use of the available hardware. None of those pieces alone supplies an application developer with the whole experience.

Microsoft assembles Foundry Local from existing assets. Ning describes Azure AI Foundry, introduced at the previous year's Microsoft Ignite, as serving more than 70,000 organizations and offering more than 1,900 models at the time of the talk. The Build 2025 announcement places that model count in the cloud platform's partner-hosted and Microsoft-hosted catalog; it is not a count of models available for local execution.

ONNX Runtime supplies the cross-platform inference engine. Ning reports more than 10 million ONNX Runtime downloads per month and describes customer-reported acceleration over PyTorch, without presenting a quantified performance comparison. Windows contributes distribution and reach across millions of client devices and users. Foundry Local combines these assets into a managed path from model selection to on-device execution.

The stack separates execution from model management and application integration:

LayerResponsibility
ONNX RuntimeAccelerate inference across supported hardware
Foundry Local management serviceHost and manage models on the client device
Model acquisitionDownload open-source models from Azure AI Foundry on demand
CLIExplore and run models locally
SDKsIntegrate local inference into applications

This division lets a developer explore models through the CLI before embedding the selected model in an application.

At the time of this June 2025 talk, Foundry Local had recently been announced at Microsoft Build and was available on Windows and macOS. Ning emphasizes its Windows integration; the launch announcement distinguishes the installable preview from direct Windows 11 and Windows App SDK availability, which was still forthcoming. Hardware work with NVIDIA, Intel, AMD, and Qualcomm connects the runtime to their accelerators. Ning says more than 100 customers joined the private pre-preview and supplied feedback before the public announcement.

2:272:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:27 · section reference included

Developer memory and sensitive speech

Pieces provides one example of why packaging and lifecycle management matter alongside inference speed. Tsavo Knott, its CEO and co-founder, describes a product that gives developers artificial long-term memory across the operating system. Offline-first AI is central to that goal, and the team began exploring small language models across major platforms in late 2022. It moved from managing its own llama.cpp setup to Ollama, but still encountered difficulties with versioning, performance, and the end-user experience.

Knott reports a rapid move from access to Foundry Local documentation to a production-ready build, with improvements in memory management, time to first token, and tokens per second. The testimonial supplies no measurements, but identifies the operational concerns that a local inference platform needs to address: loading and managing models, starting responses promptly, and sustaining generation within a desktop application's resource budget.

A second customer example starts from a stricter data boundary: clients want generative AI but cannot send sensitive information to external services. The demonstration combines Foundry Local with a speech-to-text service that also runs locally. Its sample speech includes the name Maria Rodriguez and a redacted savings account number—the sort of input that makes the location of processing consequential. The customer highlights straightforward installation and model use, then describes hybrid applications in which only part of the workload runs locally.

6:467:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:46 · section reference included

Discover hardware variants and prepare for offline use

The CLI demonstration starts with Foundry Local already installed. Windows installation uses WinGet; macOS installation uses Homebrew. The first exploration command is:

bash

foundry model list

The catalog exposes models together with variants optimized for different hardware. Ning's device shows CPU, CUDA, and integrated-GPU options. A Qualcomm NPU variant does not appear because that device lacks the corresponding NPU. Model discovery therefore includes hardware compatibility, not just a list of model families.

Offline inference still requires preparing the device. When a model is not cached, Foundry Local downloads it from the cloud before running it, which requires internet access. Ning has downloaded the models in advance and inspects them with:

bash

foundry cache list

Her cache contains four models. Keeping several models locally allows her to compare response quality and performance without depending on the conference network for model acquisition.

8:419:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:41 · section reference included

Compare speed and answer detail with the same prompt

Ning first uses foundry model run to load a cached Qwen 2.5 1.5B model. Once its interactive session is ready, she asks, “What's ONNX Runtime?” The initial answer arrives quickly enough that she exits and repeats the request in verbose mode to inspect the generation rate.

For the Qwen 2.5 1.5B response to “What's ONNX Runtime?”, Ning reports approximately 90 tokens per second in verbose mode on her device. The hardware specification, quantization, execution provider, runtime build, measurement boundaries, and repeated trials are not supplied, so this number describes the displayed run rather than a portable performance expectation.

She then loads Phi-4 Mini in verbose mode and asks the same question. In this demonstration, the larger Phi-4 Mini model generates more slowly but supplies more detailed information. Ning prefers that answer for the application she wants to build.

ModelObserved behavior in the demoSelection implication
Qwen 2.5 1.5BFaster, briefer responseUseful candidate when speed and brevity matter
Phi-4 MiniSlower, more detailed responseNing's choice for the summarizer

The comparison makes model selection tangible: run the intended style of request, inspect both the output and the execution behavior, and choose according to the application's needs. Here, detail in one answer drives the choice; the demonstration does not establish general model superiority.

10:4110:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:41 · section reference included

Summarize internal documents without uploading them

The application problem is onboarding into an unfamiliar project. A team has long, detailed documents to read and wants high-level summaries to get oriented. Those documents are internal and cannot be uploaded to the cloud. Some teammates use Windows and others use Macs, so the solution needs both local inference and a cross-platform application.

The prepared summarization app accepts either a URL or a local file. Its settings tab lets the user select a model by ID. Ning chooses Phi-4 Mini, supplies the project document, and submits it for summarization. The response describes Foundry Local as a way to build cross-platform AI applications that run directly on-device. The workflow carries the earlier model experiment into a concrete task: choose a model, provide the document, and receive a summary inside the application.

13:0613:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:06 · section reference included

Connect the JavaScript app and run it on a Mac

Foundry Local offers Python and JavaScript SDKs; the summarizer uses JavaScript. The demonstrated integration creates a Foundry Local manager, initializes it with the selected Phi-4 Mini model, and uses the manager's local endpoint to construct a chat client. The application then awaits a chat completion and displays the result. This is the talk's manager-and-endpoint integration; today's SDK examples emphasize in-process inference with optional serving, so the two integration styles should not be mixed.

VS Code displays model initialization, client creation, and the beginning of a summarization prompt in the Brevity project.Open full source frame
JavaScript code initializes the model and creates a client using the Foundry Local endpoint.

Once the client points at the local service and the selected model is initialized, the summarization request can remain a small JavaScript function:

javascript

async function summarizeDocument(client, modelId, documentText) {
  const completion = await client.chat.completions.create({
    model: modelId,
    messages: [
      {
        role: "system",
        content: "Summarize the document's main ideas and key details."
      },
      {
        role: "user",
        content: documentText
      }
    ]
  });

  return completion.choices[0].message.content;
}

The manager handles model initialization, while this function handles the application's request and response. The model ID must refer to the selected local model, and the client must use the Foundry Local endpoint.

Ning then packages the Windows project and shares it with a teammate who records the Mac experience. The teammate starts the same code with:

bash

npm run start

The application and UI are unchanged. On the Mac, the teammate selects Qwen and submits the same document. Its summary is shorter than the earlier Phi-4 Mini output, echoing the difference seen in the CLI comparison. The handoff demonstrates reuse of the application across operating systems while leaving model choice available to the user.

15:3515:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:35 · section reference included

Give a local model tools through MCP

The final demonstration moves from generating a summary to using tools. Foundry Local's agent feature is in private preview in the recording. Its composition is straightforward: one local model plus one or more MCP servers. The model supplies the reasoning and tool selection; the configured servers supply the available operations.

Ning returns to the terminal and runs:

bash

foundry agent list

The output shows three sample agents: a file organizer, an OCR agent, and a web interactor, with their server counts and model names. A developer can compose an agent by choosing a model from the catalog and the MCP servers needed for the task.

Terminal tables list available MCP servers and the sample file organizer, OCR agent, and web interactor, alongside server counts and model names.Open full source frame
Foundry Local lists MCP servers and three sample agents with their server counts and models.

For this walkthrough, Ning selects the existing OCR agent and inspects it with foundry agent info. It extracts text from images on the local device and combines a Phi-4 Mini variant with two servers: a filesystem MCP server and an OCR Mini MCP server. She then launches it with foundry agent run. These are the private-preview commands shown in the recording.

17:5518:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:55 · section reference included

Authorize access, find the receipt, and extract its total

Launching the agent does not immediately grant it unrestricted access. Startup first checks its dependencies and, if any are missing, offers to install them with permission. Ning has already installed the dependencies. She then works through the access prompts:

  1. Approve the filesystem MCP server.
  2. Supply the allowed directory: the folder containing the demonstration photos.
  3. Approve the OCR MCP server.

After setup, the agent lists the tools provided by those servers, including filesystem and OCR operations. Dependency installation, server approval, and directory selection are separate parts of preparing the agent to act.

The user request is to find a receipt, process it, and obtain the total amount. The agent first selects file search to locate the receipt image. It then uses OCR to extract the text and produces the total. The completed terminal view shows the located file, extracted receipt text, and a final result of Total: $8.39. This is a concrete division of labor: filesystem tools locate the input, OCR turns image content into text, and the model uses those results to answer the request.

Terminal output shows a receipt image search result, OCR text, and a final result reading “Total: $8.39.”Open full source frame
The receipt workflow shows the located file, extracted receipt text, and final total.

The receipt task also gives the closing capability warning a practical setting. Ning cautions that local models are generally less capable than cloud models and should not be expected to perform the same sophisticated model or agent work. A bounded task with relevant tools can still be useful, as the receipt workflow shows. She closes by pointing attendees to further information and a signup form for the agent private preview. The engineering opportunity is to choose work that fits the local model's capabilities while benefiting from on-device data handling and execution.

20:0320:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:03 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…