AI Engineer Europe 2026

Why MLX

23:10

Loading slides…

Read the talk

Why MLX: Building Agents That See, Hear, and Speak Locally

Local inference turns accessibility, voice control, and visual perception into applications that can work without a reliable connection, with MLX providing the building blocks on Apple silicon.

Before you start: Basic familiarity with model inference and Python command-line tools will help with the demonstrations; no prior MLX experience is required.

What happens when an agent cannot depend on the internet?

Have you tried running AI on your phone or MacBook—and how good was the experience? The practical goal here is to deploy AI agents, including voice agents, entirely on those devices. Moving some workloads off subscriptions shifts their inference cost toward the hardware and electricity already available to the user. The route runs from the reasons for local execution through MLX, live demonstrations, and applications built by its community.

The need became personal in 2020. The speaker’s father lost his sight that year, while Apple introduced its first M1 hardware. A voracious reader could no longer access information as before, and his son promised to help him read again. His father lives in Africa, where unreliable connectivity and poor internet subscription options made a cloud-dependent solution a difficult fit. On-device inference was an accessibility requirement: assistance needed to remain available when a connection was not.

Slide titled Apple Silicon beside an M1 chip image and a labeled component diagram.Open full source frame
Apple Silicon: the M1 chip and its component layout.
0:170:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:17 · section reference included

From an array framework to accessible perception

In 2023, a GitHub search led the speaker to MLX, an array framework for Apple silicon. Think of the role PyTorch or TensorFlow plays in a machine-learning application: arrays, operations, and the machinery for running models. The initial examples suggested a practical foundation for local intelligence, and he began contributing. The speaker reports more than 1.5 million downloads and more than 4,000 ported models three years later. He also describes collaboration with frontier labs and day-zero support for Gemma 4, bringing supported open models to MacBooks, iPhones, and iPads.

Neywa Labs slide titled MLX VLM + Audio with three metrics: 6 Frontier Labs partners, 1.5m+ downloads, and 4,000+ Models ported.Open full source frame
MLX VLM + Audio: 6 frontier lab partners, 1.5m+ downloads, and 4,000+ models ported.

Vision was the first application: help someone understand what is in front of them and navigate their surroundings. An earlier attempt, built with a team at a 2021 hackathon, used goggles that described nearby objects. MLX-VLM became a second iteration of that idea, making the same kind of assistance possible through a phone camera. Point the phone at something and ask what it sees.

Omni models add audio input to that interaction. For a user who cannot readily type, speech can control the camera and request information about the scene. The speaker identifies MLX-VLM as an engine used by LM Studio and for Liquid AI models, then challenges the assumption that local inference necessarily means small models. He reports examples of models with hundreds of billions of parameters running on an early M1 MacBook, and Gemma 4 26B running on an iPhone using storage. These capacity claims come without a specified throughput or configuration; fitting a model and achieving an acceptable interactive experience remain separate questions.

2:272:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:27 · section reference included

Assembling a voice agent for the available hardware

Audio followed vision, both for accessibility and for a more everyday desire: controlling a computer without sitting in front of it. Speak a command and have the computer act—the JARVIS-style interaction. Work began with text-to-speech and Marvis. The speaker reports audio generation in less than 100 milliseconds, without specifying the model version, hardware, or latency measurement boundary. Speech-to-text supplies the other direction, transcribing the user’s speech in real time.

Wispr Flow and superwhisper illustrate the kind of dictation product these components can support. The speaker suggests pointing Claude Code or Codex at MLX Audio to build a comparable application in about ten minutes; that is an estimate, not a demonstrated build. Speech-to-speech then closes the conversational loop, allowing the computer to respond audibly. Python came first for development speed, followed by Swift support for native applications combining audio and vision.

There are two ways to assemble the conversation:

ApproachHow it worksMain choice
Native speech-to-speechA model handles the speech interactionSelect the integrated model
Modular pipelineSpeech recognition → language model → speech synthesisSelect each component separately

In the modular version, automatic speech recognition, the language model, and text-to-speech are independent choices. That lets the application adapt to its hardware budget, from a first-generation M1 to a newer machine, without requiring the same model combination everywhere. The remaining question is whether the resulting speech sounds good enough for the experience being built.

5:235:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:23 · section reference included

Detecting objects and masking a background

The first live demonstration starts a camera-analysis application with a Python command. It runs Roboflow’s RF-DETR locally, detecting objects as the scene changes. Swift support for this particular demonstration is described as forthcoming. During an attempt to show operation without an internet connection, processing continues. When the speaker lifts a glass, the detector identifies it as wine, prompting a joke about being sober. The visible demonstration runs on a Mac; phone execution is also described as possible.

The next variation applies perception to a familiar meeting feature: background blur. A real-time person mask separates the speaker from the surrounding scene so the background can be blurred while the foreground remains visible. The speaker notes that the blur may be difficult to see in the presentation, but the mechanism is the useful part: a local segmentation result becomes an input to a product’s rendering behavior.

8:008:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:00 · section reference included

Opening a multimodal chat interface

The next demonstration moves from live detection to a larger multimodal model. Gemma 4 is described as having arrived the previous week; its published release date is April 2, 2026. MLX-VLM’s mlx_vlm.chat_ui entry point accepts a model selection and opens a Gradio interface. With MLX-VLM’s chat dependencies installed, a small Python launcher can pass the chosen model through to that module:

python

import subprocess
import sys

model = input("MLX-VLM model ID or local path: ").strip()
if not model:
    raise SystemExit("A model ID or local path is required.")

subprocess.run(
    [sys.executable, "-m", "mlx_vlm.chat_ui", "--model", model],
    check=True,
)

The interface provides a quick way to test a model before building a custom application around it.

The launch initially stalls. On retry, it attempts to download model files before the speaker switches the internet off again. That sequence exposes an operational distinction: local inference still requires the model files to be available locally. Once the chat window opens, he selects an image and enters “Describe this image in detail.” The response identifies a profile and describes details including the biography.

The speaker describes the demonstration machine as having 96 GB of memory and says it can run all the models shown so far simultaneously in real time. On Apple silicon, this is unified memory shared by CPU and GPU, rather than a separate dedicated VRAM pool; the demonstrated inference uses the GPU. The planned live audio demonstration does not follow: an unresolved problem in the Swift branch that morning prevents it, so community recordings provide the audio examples instead.

9:279:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:27 · section reference included

Inspecting footage and continuing a generated story

Grounded visual reasoning turns model output into a way to inspect video. Gemma 4, RF-DETR, or another perception model can look for fires or particular requested objects without sending the footage to a remote service. The displayed example asks to segment every jet and shows colored detection overlays on aircraft and their contrails. The proposed applications include home security running on a MacBook and analysis of dashcam footage.

Slide titled Grounded Visual Reasoning using mlx-vlm showing jets and contrails with colored detection overlays and the prompt Segment every jet.Open full source frame
Grounded visual reasoning demo highlighting jets in flight.

The speaker already uses a system like this to review dashcam footage after incidents. He then switches from analyzing video to generating it with MLX Video, showing a community-created cartoon that starts from a text prompt. The sequence is not generated as one long clip: successive generations continue from previous video, allowing a story to develop across multiple steps.

The speaker says this particular chained-generation system can run on a MacBook with 16 GB of unified memory. The model, settings, and generation time are not given. He promises to reshare more of the creator’s videos through his social profile; the demonstrated contribution is the chaining strategy that turns individual generations into a continuing sequence.

12:0712:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:07 · section reference included

Giving applications—and a robot—a voice

The next recording, located in Drive under Neywa Labs, comes from Adrian, a participant in the audience who built an application called Locally. MLX Audio and Marvis TTS give it spoken output. The sample reads prose aloud, beginning with a small voice and gesture and continuing into a passage about the Chatsubo bar. It connects the speech components to a recognizable native reading experience. The speaker emphasizes that product design matters alongside the model integration.

Robotics extends the same combination of perception and speech into a physical device. The speaker acquired a Reachy Mini the previous year and describes using MLX Audio and MLX Vision to supply capabilities based on its camera and audio input. A recording demonstrates what he describes as real-time cloning of the Iron Man JARVIS voice. After a greeting, the robot responds: “Hey there. Great to see you. How’s it going?”

That is the prepared talk’s destination: agents that can hear, see, and speak in a familiar voice, running on an iPhone, iPad, Mac, or robot. The individual libraries become useful when they are composed into an interaction that serves the person using it. The questions that follow turn to the execution backend, observability, model limits, and the memory costs of longer contexts.

14:2714:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:27 · section reference included

Where inference runs, and how to observe it

An audience member has noticed that Neural Engine utilization stays at zero while MLX is running. The explanation is that the demonstrated inference path uses the GPU, not the Neural Engine. MLX also supports CPU execution; it should not be understood as a GPU-only framework. For Neural Engine access, the speaker points to Core ML, while describing its developer experience and private-API issues as obstacles.

He describes internal work toward hybrid GPU and Neural Engine inference, contingent on Apple resolving API issues, and hopes for progress at WWDC. He also speculates about Neural Engine components moving into the GPU, citing the M5 generation, while acknowledging that Apple’s direction is unknown. These are prospective paths, not capabilities demonstrated in the session.

For observing the current path, he recommends mactop, whose author is Carsen Klock. It displays CPU and GPU activity, and the demonstration uses a monitoring view alongside the application. To reproduce the observation:

  1. Open mactop and watch the utilization display.
  2. Start the local inference application.
  3. Submit a prompt and observe the GPU activity as the model responds.

In the demonstration, GPU usage rises after inference starts and a greeting is entered. This connects the application’s response to visible local compute activity.

17:0717:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:07 · section reference included

Choosing an omni model without overpromising

Asked which omni models to try, the speaker names two families:

  • Gemma 4 E2B and E4B: The edge variants accept image, audio, and text inputs in combinations. These are the published names of the variants referred to aloud as E2 and E4.
  • Qwen3-Omni: A larger local option, described by the speaker as approximately 30 billion parameters, without identifying the precise variant or parameter accounting.

The distinction is useful for choosing a starting point: the input modalities must fit the application, and the model must fit the device.

The main limitation he identifies is capability relative to the task. Do not expect these local models to match the contemporary Claude/Opus models he invokes as a comparison. His suggestion that open models might reach that level within six months is a forecast. The application’s promises need to match the model’s current behavior.

He also describes inference over hundreds of images in parallel and many documents, then credits TurboQuant with making much longer contexts feasible. The speaker claims local operation with context lengths up to one million, depending on model size and hardware. Cache compression reduces the memory required to retain context; it does not automatically extend a model’s supported context window. For example, Google specifies 128K for Gemma 4’s edge variants and up to 256K for its larger variants, so the million-context claim needs a separate model and workload configuration.

19:3219:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:32 · section reference included

Reducing the cache cost of long contexts

A final question asks whether Google’s general TurboQuant approach is being implemented. The speaker says he produced one of the earliest public implementations, about thirty minutes after encountering what he calls the paper’s release. Searching his profile brings up a March 25 post that he says attracted roughly 700,000 views after being posted late at night. The publication chronology needs one distinction: the paper was submitted to arXiv on April 28, 2025, while Google’s research announcement followed on March 24, 2026. The implementation account belongs to the speaker’s recollection of that publicity cycle.

The result he highlights is a reduction in KV-cache memory, the stored attention keys and values retained during inference. He describes a cache approaching 1 GB becoming roughly four times smaller. The inspected table gives a more precise view of the displayed comparison:

ConfigurationExact matchAverage cache, GBDisplayed reduction
Full6/60.703—
TurboQuant 2.56/60.1434.9× smaller
TurboQuant 3.56/60.1853.8× smaller

The speaker interprets the exact matches as agreement with full-model responses and describes the quality as similar. The model, hardware, and six evaluation cases are not specified here, so those matches establish agreement for the displayed test, not general quality equivalence. A smaller KV cache also does not mean total application memory falls by the same factor: model weights and other allocations remain.

Table with full, TurboQuant 2.5, and TurboQuant 3.5 rows. All show 6/6 exact match; average cache GB values are 0.703, 0.143, and 0.185, with reductions labeled 4.9x smaller and 3.8x smaller.Open full source frame
TurboQuant results compare exact matches and average cache size against the full configuration.

The speaker reports almost doubled throughput at around 300,000 context in his published results. The workload and hardware behind that comparison are not specified in the discussion. It closes the session on a concrete engineering constraint: as context grows, cache memory and the cost of accessing it matter alongside model size. Reducing that burden is one way to extend what an on-device application can do.

21:0621:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

21:06 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…