AI Engineer Europe 2026
Frontier AI at Home (literally)
Loading slides…
Read the talk
Frontier AI at Home (literally)
Running capable models at home means solving more than memory capacity: kernels, caching, device placement and communication determine whether local inference is actually useful.
From a talk by Alex Cheema
Before you start: Basic familiarity with LLM inference and GPU memory is helpful; prefill, decode, KV caching and tensor parallelism are explained as they arise.
What happens when your AI is no longer just a chat window?
If an AI system becomes an extension of your working memory—holding your data, operating your tools and helping you make decisions—what happens when its provider cuts off access? That is the opening problem behind EXO, where Alex Cheema works across models, software and hardware to reduce the cost of running frontier AI locally. Most people still access those capabilities through models in somebody else’s data center.
The name comes from exocortex: an external extension of the brain. Cheema invokes Andrej Karpathy’s line, “Not your weights, not your brain.” As systems such as OpenClaw move beyond answering questions into acting on a user’s behalf, ownership starts to include continuity of access, control over personal data and the ability to choose what the system can do.
Cheema describes a cybersecurity friend who lost access to Claude, Gemini and ChatGPT while doing penetration testing intended to secure a system. The anecdote makes the dependency concrete: useful work can become contingent on a handful of providers’ decisions. An open alternative needs both available weights and an affordable way to run them; otherwise the economic concentration remains even when the model can be downloaded.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The hardware lottery reaches into inference
The engineering question begins after training: given the weights, how do you execute the model? A system that requires hardware on the order of a million dollars remains inaccessible to most people. Meanwhile, much of the hardware and research ecosystem has developed around training, NVIDIA data-center GPUs and the economics of floating-point operations.
Sara Hooker’s The Hardware Lottery explains why available hardware and software influence which research ideas succeed. An idea can remain unexplored because it does not fit the dominant stack. Inference on different hardware opens another design space: the best arrangement for training need not be the best arrangement for serving one person’s model.
A recent Qwen 3.5 investigation supplies a concrete example. Cheema reports that local inference on Apple Silicon was roughly 50% slower than his team’s theoretical expectation. They found unnecessary, separately launched kernels adding overhead. At the illustrative target of 150 tokens per second, each token has less than ten milliseconds of total execution time; a few milliseconds of launch overhead can consume a substantial part of that budget.
Cheema reports a 30% inference-performance improvement after fusing unnecessary separate kernels in that Qwen 3.5 Apple Silicon investigation. The workshop does not specify the model size, quantization or timing protocol for this result. The mechanism is nevertheless clear: combining work eliminates launches that would otherwise sit on the critical path of every token.
The same reasoning extends upward through orchestration, device connections and communication, then into the agent harness. Claude Code and OpenCode can produce different results with the same model because they organize the work differently. On resource-constrained local hardware, a harness that understands the machine’s limits can matter alongside kernel optimization. The useful unit of optimization is the whole running system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prefill builds context; decode repeatedly reads it
Personal inference usually lacks the large batches available to a cloud provider. That changes which hardware resource limits performance. Cheema contrasts compute-heavy training with low-batch inference, then separates inference into two phases.
| Phase | Work performed | Typical constraint in this discussion |
|---|---|---|
| Prefill | Process the prompt and build the KV cache | Compute |
| Decode | Generate tokens autoregressively, one at a time | Memory bandwidth at low batch size |
Loading a PDF into the prompt requires prefill before the model can answer. The resulting KV cache holds attention keys and values that decode reuses while extending the response.
A good harness can avoid repeatedly prefilling an unchanged prefix. Cheema points to Claude Code’s /context command, which exposes the prompt’s composition: system instructions and tool definitions occupy a substantial region that usually stays stable until an update. Preserving that prefix creates opportunities for cache hits. His emphasis on decode assumes this kind of reuse; a newly introduced long document still requires substantial prefill.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Capacity, bandwidth and energy impose different limits
For memory-bound decode, three questions determine whether a local setup is practical:
- Capacity: Can the model fit in memory? Repeatedly loading weights from disk makes execution slow.
- Bandwidth: How quickly can the system move model weights and KV-cache data into the GPU?
- Energy per byte: How much energy does moving that data consume?
Capacity is the admission requirement. Bandwidth governs speed once the model fits. Energy connects that speed to battery life, cooling and the cost of sustained use.
Cheema’s phone example pairs a roughly 10–15 Wh battery with roughly 10–15 W of inference power to illustrate battery life on the order of an hour. In his team’s phone experiments about eighteen months earlier, the device became too hot for him to hold. Future phones may improve, but a demonstration that produces tokens does not by itself establish a usable mobile assistant.
A task-based efficiency metric brings model quality into the picture. Cheema introduces work associated with Stanford’s Hazy Research, known also for ThunderKittens, and describes dividing task performance by the energy used. He calls this intelligence per joule, distinguishing energy spent completing a task from instantaneous power. The related Intelligence per Watt study uses a power-based metric; the two denominators should not be treated as interchangeable.
Consumer memory capacity is also changing what can fit locally. The demonstration uses 512 GB machines; Cheema says that configuration had recently become unavailable to buy, while 256 GB remained an off-the-shelf option. He also cites the M5 Max MacBook’s increased capacity and bandwidth. Apple’s specification supplies the unit omitted in the spoken figure: up to 128 GB of unified memory and 614 GB/s of memory bandwidth.
The slide’s complete reveal attributes improvements in accuracy per joule to both hardware and models, while Cheema verbally rounds the contributions to roughly fivefold and threefold. Those figures differ from the likely related study’s historical companion analysis, which reports 5.3× total improvement in intelligence per watt over 2023–2025, decomposed into 3.1× from models and 1.7× from accelerators, on single-turn chat and reasoning workloads. The slide remains useful for the structural point: model and hardware advances can compound, but its accuracy-per-joule chart is not the same result as the paper’s power-based metric.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When does an inference box become a household appliance?
An audience member frames the purchase as an appliance: if someone will spend money on a fridge, might they also spend $5,000 on an inference box? The question includes supply-chain constraints, Apple’s RAM purchasing and whether the company sees a new market. Cheema answers primarily through the trajectory of useful inference rather than attempting to explain Apple’s pricing strategy.
The difficulty is that the frontier keeps moving. Cheema introduces GLM-5.1 as a newly released model and describes it loosely as a trillion-parameter, FP16 system requiring about 1.5 TB. The official checkpoint metadata is more precise: 754B stored parameters in BF16. At two bytes per stored parameter, that is approximately 1.5 TB of raw weights, before runtime allocations. His practical point survives that correction: an unquantized checkpoint of this size exceeds a single consumer machine.
Cheema estimates roughly $40,000 of 512 GB Mac Studio hardware and about 20 tokens per second for the large-model setup he describes, against a presumed cloud-user expectation nearer 50 tokens per second. His forecast is that compounding improvements across harnesses, models and kernels leave a 100× price/performance opportunity, potentially bringing a fast, near-frontier appliance to about $5,000 within eighteen months. These are purchase-planning estimates and a research thesis, rather than a demonstrated configuration or guaranteed delivery schedule.
Reaching that point requires co-design: choosing the model architecture, hardware and execution strategy together. Agents and large mixture-of-experts models provide somewhat more stable targets for specialization. Cheema points to NVIDIA and Groq as an example, although the announced arrangement was non-exclusive technology licensing and personnel joining NVIDIA, with Groq remaining independent—not an acquisition of Groq.
For now, Cheema would recommend the purchase mainly to people who want to experiment; he broadens the consumer-product horizon to about two years. He also raises OpenClaw access restrictions involving Opus, without establishing an outright API prohibition, and describes acquaintances spending around $1,000 a day on tokens. Owned hardware removes that per-token bill, but electricity remains a running cost. When pressed on when to buy, he explicitly calls his timeline an estimate.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Useful intelligence can saturate before the frontier stops moving
The next question sets several trends against one another: larger frontier models, smarter small models, changing hardware prices and energy costs. Cheema mentions explicitly unverified size rumors for Mythos and Gemini, followed by speculation about still larger training runs. At the other end, he offers Gemma 4 as an example of a small model that appears stronger than much larger predecessors from a few years earlier. Both directions can advance at once.
The proposed resolution is an S-curve for each use case. More intelligence initially creates large gains, then additional capability produces diminishing utility. Wispr Flow-style transcription makes the tradeoff tangible: the user needs accurate words quickly. An enormous model reasoning for minutes would violate the latency requirement, and once transcription is good enough, extra reasoning may add little value. Email summaries, to-do lists and ordinary summarization can reach similar thresholds.
Disease research and other difficult scientific work could continue absorbing enormous amounts of compute while everyday consumer tasks become locally tractable. Cheema’s suggestion that almost all consumer work might eventually run locally is conditional on those utility curves flattening. He also acknowledges how hard prediction has become: Claude Code with Opus 4.5 around Christmas surprised him by handling work he previously considered out of reach. Harness improvements contributed to that jump. The frontier can move abruptly even if particular tasks eventually stop benefiting from a larger model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Stable tasks change the economics of specialized chips
An audience member extends the argument: if a Mac mini-style device eventually satisfies ordinary tasks, could a sufficiently capable model be baked into a chip? Today’s rapid obsolescence makes that difficult. By the time a model-specific chip is ready, users may already want a different model.
| Hardware position | Examples in the discussion | Tradeoff |
|---|---|---|
| General-purpose GPUs | NVIDIA RTX, H100 | Broad flexibility |
| More specialized accelerators | Cerebras, Groq | Narrower execution design |
| Model-specific hardware | Taalas | Strong commitment to a particular model |
Cheema describes this as a spectrum. The economic test is whether deployment savings repay the chip’s development cost before the relevant model becomes obsolete. A frontier moving every few months makes that amortization difficult; a task whose requirements have stabilized could make it attractive.
EXO is evaluating hardware rather than manufacturing its own chips. A future system could be more opinionated about workloads: instead of one generic model serving every API use case, Cheema imagines twenty task-specialized models covering similar ground, with a few chips designed to execute them efficiently. That possibility depends on knowing which tasks matter and having models that remain sufficient long enough to justify the hardware.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A large memory pool and a fast GPU solve different problems
An audience member challenges the appeal of unified memory: fitting a large model on a Mac does not make it fast. Cheema agrees. A Mac exposes a large shared memory pool, while an RTX 5090 offers less capacity but more bandwidth and compute. His approximate comparison separates those resources explicitly.
| Device in Cheema’s comparison | Memory capacity | Memory bandwidth |
|---|---|---|
| RTX 5090 | 32 GB GDDR7 VRAM | Approximately 1.5–2 TB/s |
| Demonstrated Mac Studio | 512 GB unified memory | Approximately 800 GB/s |
He also estimates much less compute on the Mac. The answer is to consider placing different parts of a model on different devices, rather than treating a model as an indivisible object that must run entirely on one hardware type.
The questioner describes a roughly 120 GB Mac becoming hot and difficult to use while running a 120B-parameter GPT model, with battery drain even when plugged in, whereas smaller models perform well on an RTX. Those are the owner’s observations, not a controlled comparison. They sharpen the distinction between fitting weights and sustaining a useful workload.
Cheema then brings out an ASUS version of the NVIDIA Spark platform, priced in his discussion at about $4,000. Its compute makes it attractive for prefill; the Mac’s higher memory bandwidth makes it attractive for decode. When an audience member reverses those assignments, he corrects them: Spark prefill, Mac decode. An RTX changes the balance again, because it offers both more compute and more bandwidth but a much smaller memory pool, inviting finer-grained placement.
A directly connected RTX delivering a threefold large-model speedup is presented as work underway, not a result from this demonstration. Cheema compares the direction with heterogeneous data-center systems, citing NVIDIA/Groq and Cerebras/Trainium arrangements. Locally, the software and physical packaging are still awkward: stacks of Macs and Thunderbolt cables expose integration work that an appliance would need to hide.
Model architecture also changes the recommendation. In response to concerns about running capable local coding agents, Cheema suggests trying a quantized dense Gemma 4 on RTX. A dense model that fits benefits from a small, fast memory pool because its weights are repeatedly used. A sparse mixture-of-experts model, or MoE, can benefit from the Mac’s ability to hold many weights while activating only a subset. Future local SRAM could serve small models extremely quickly, but its cost and low density limit capacity. The design opportunity is to combine the useful properties of these devices.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discover the cluster, then access it from anywhere
How does someone actually coordinate hardware scattered around a house? EXO’s intended workflow is a background application on every device. The applications discover peers, maintain a mesh and choose a model distribution based on the available hardware and connections. This removes the need for the user to manually assemble a separate serving arrangement for every combination of machines.
The prepared demonstration uses four Mac Studios. Cheema cautions that collecting arbitrary devices is not enough: a cluster of Raspberry Pis may produce an impressive headline while depending on aggressive quantization and painfully slow prefill. Spare GPUs are a more promising starting point for a capable system.
A phone can remain the interface while inference runs on owned hardware at home. Each EXO node exposes an HTTP API endpoint, and Cheema uses Tailscale to access those endpoints securely from elsewhere. Tailscale is separate, not bundled into EXO; an integrated remote-access option is only exploratory in this discussion. The proposed appliance holds the user’s data, runs an agent such as OpenClaw and accepts instructions from the phone without requiring the phone to perform inference itself.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One user can generate a batch—and personalized weights can break one
Cloud providers combine requests from many users to increase hardware utilization. Instead of executing each inference sequentially, they schedule work together and amortize costs. A single local user normally cannot supply that volume, which creates an apparent structural advantage for the cloud. Cheema’s hypothetical hundredfold cloud advantage is a way of posing the objection, not a measured comparison.
Multi-agent workloads offer one way to generate concurrency locally. Cheema uses Grok 4.20 as an example: a single user request can engage several collaborating agents. The displayed diagram connects Grok with Harper, Lucas, Benjamin and the user input. A workload of this kind could turn one user into several simultaneous model requests; his illustrative batch size of eight is potentially a better match for Mac utilization than batch size one.
Search supplies another source of parallel work. In best-of-N, the system generates multiple candidates—ten in Cheema’s example—then uses a verifier to score them and select a response. More sophisticated searches can branch and evaluate intermediate steps. He points to Hugging Face’s test-time compute work, describing a roughly 1B model improving toward larger-model performance as search increases. The exact benchmark and verifier budget are not established in the workshop discussion. Multi-agent inference is itself one form of this broader test-time scaling.
Continual learning changes the economics in a different direction. Instead of only performing forward passes through fixed weights, test-time training updates a model during use. Each person could acquire a different version of the weights through their own interactions and data. Cheema suggests this could address forgetting and bounded sessions, although updating weights does not by itself establish that all context limitations disappear.
If every user’s full model differs, ordinary shared-weight batching across users becomes unavailable. If only a small component changes, more sharing may remain possible. Cheema’s forecast of a possible tenfold relative economic improvement for local inference depends on the stronger, full-model-personalization scenario; it is not a demonstrated savings figure. The key question is how much model state remains common across requests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Idle capacity is a workload question before it is a marketplace
An audience question begins with uptime but resolves into utilization: how much does Cheema actually use his cluster, and could he rent out the spare capacity? He still uses Opus and other cloud services, and says his local utilization is low, although the set of sufficiently capable local use cases is growing. A future fleet of a million EXO clusters could contribute idle inference to scientific work, but that is not the team’s immediate focus.
Better local agents might consume that capacity themselves. Cheema imagines an always-on assistant scanning for relevant information, thinking about EXO’s future and proactively bringing useful findings to his attention. Without a per-token charge, he would want it working continuously. In that scenario, little capacity may remain idle. If spare capacity does exist, independent clusters already capable of running complete models can receive separate tasks in a data-parallel arrangement; they need not shard a single model across the internet.
Another audience member compares this with home cryptocurrency mining, where industrial operations eventually displaced small GPU owners. The distinction is the reason for buying the machine: local AI can provide privacy and personal utility before it earns anything from outsiders. Cheema is skeptical of a business whose entire premise is paying people to rent out hardware. He prefers moving the product boundary upward, selling tokens or completed tasks.
His example is an inexpensive API that accepts a task for completion within twenty-four hours. It tolerates retries and does not promise interactive latency, so a scheduler can exploit intermittent spare capacity. That only becomes interesting after capable hardware is widespread: he contrasts a small collection of Macs with a hypothetical gigawatt-scale fleet, whose aggregate compute would otherwise require substantial capital and permitting. It remains a deferred possibility, with household usefulness coming first.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why tensor parallelism needs microsecond communication
The live demonstration begins through Tailscale access to the prepared cluster. Each of four Macs runs the EXO application, and each Mac is connected to every other Mac through Thunderbolt 5. Cheema describes Thunderbolt as a transport around PCIe. The important change is EXO’s integration of RDMA, which sharply reduces the latency of communication between machines.
Low latency matters because tensor parallelism splits work within model layers across devices. The machines must synchronize repeatedly for every generated token. In Cheema’s example, a model with sixty layers requires two synchronizations per layer, giving 120 synchronization points per token. At the earlier inter-Mac latency of approximately 0.3 milliseconds, the illustrative synchronization budget alone is 36 milliseconds per token.
The arithmetic is small enough to expose directly:
python
layers = 60
synchronizations_per_layer = 2
latency_ms = 0.3
synchronizations_per_token = layers * synchronizations_per_layer
communication_ms = synchronizations_per_token * latency_ms
print(f"{synchronizations_per_token} synchronizations per token")
print(f"{communication_ms:.1f} ms per token in synchronization latency")
This is the latency component of the example, before accounting for computation and other transfer costs. Cheema rounds the burden to roughly forty milliseconds: enough to erase the benefit of dividing computation across machines. A cluster could fit a larger model while generating more slowly.
Cheema reports that RDMA reduced communication latency by about 100× into single-digit microseconds, bringing the example’s total communication overhead below one millisecond per token. That is the change that makes the cluster useful for speed as well as capacity. The claim concerns the communication path; it does not mean the complete model runs one hundred times faster.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Running GLM-5.1, then adding a smaller instance
The first model is GLM-5.1, already loaded across the four machines. EXO calls one running model configuration an instance. For turnaround time, the team downloaded a four-bit version that Cheema describes as almost 400 GB; he reads roughly 112 GB of memory use on each machine. These are the workshop’s approximate download and allocation figures, distinct from the raw full-precision weight size discussed earlier.
Cheema asks the model about AI Engineer while explaining the conversion work. Leo converted the weights to MLX the previous day. Because GLM-5 already had architecture support, this checkpoint required weight conversion rather than a new implementation. Gemma 4 had been harder: its different KV-cache architecture required additional work. Supporting a file format and supporting a model’s execution semantics are separate jobs.
The answer identifies AI Engineer as a community and names swyx. Cheema points to all four machines being active under tensor parallelism with RDMA. He attributes the choppy visible response to the Wi-Fi connection between his laptop and the cluster dashboard, reached through the Tailscale host James. That interface path is separate from the Thunderbolt communication carrying model execution between the cluster nodes.
He then launches a smaller Qwen 3.5 instance on Mike and S13, illustrating that different instances can coexist on the cluster. Memory use rises on those two machines as the weights load. The smaller Qwen 3.5 demonstration displays approximately 77 tokens per second across Mike and S13. Both nodes become active during a longer response, while Wi-Fi still makes the displayed output uneven. This is a separate run from the earlier kernel-fusion example.
The user-facing action is simply to launch a configuration and chat. Underneath, EXO must select machines, load weights and establish the appropriate communication path. Cheema next turns toward a Spark–Mac arrangement, where placement can become more granular than the tensor-parallel distribution just demonstrated.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A new laptop exposes the cluster’s state model
An audience member offers a MacBook to join the cluster. Discovery over the shared Wi-Fi fails, with the network or firewall suspected, so the team switches to a direct Thunderbolt cable. App-version and model-availability questions surface immediately. Cheema also warns against the Mac Studio port beside Ethernet, an unexplained restriction corroborated by EXO’s setup documentation; his suggestion that Apple is fixing it does not establish a cause or a delivery date.
The laptop is identified as an M4 Max with Thunderbolt 5, but RDMA requires separate enablement. Pipeline operation is suggested as an alternative, and after waiting on the wired connection, automatic discovery succeeds. Joining the network is only the beginning: the new node must learn what has already happened.
EXO uses event sourcing. Each machine maintains an append-only log, and the joining laptop replays cluster history before catching up to the current state. Cheema connects that choice to dynamic membership: laptops can sleep, devices can power off and machines can leave during a request. Retaining events provides a basis for reconstructing state and reasoning about requests instead of treating an in-memory view on one machine as the entire cluster. The workshop does not specify a complete failure-recovery protocol.
The setup sequence reveals several distinct requirements:
- Transport capability: Cheema lists M3 Ultra, M4 Pro/Max and M5 Pro/Max machines as Thunderbolt 5 options for RDMA.
- RDMA enablement: The demonstrated setup requires booting into macOS Recovery to enable the feature. This laptop has not done so, so the team attempts slower TCP/IP operation.
- Topology discovery: Once replay finishes, the dashboard shows the MacBook attached to only one existing node. EXO uses that physical topology when considering placement.
- OS compatibility: The laptop runs macOS 26.2 while the cluster runs 26.3, producing an incompatibility warning. Current setup documentation calls for matching versions.
Every node exposes the same API and dashboard, so Cheema can manage the newly joined laptop from the machine already connected to the projector.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discovering a node does not discover all its model files
The MacBook owner has downloaded models through MLX and LM Studio, but EXO shows none available. In the demonstrated build, those downloads live outside the model directory EXO expects, discussed as exo/models. Cheema selects nodes to filter possible configurations, then launches a small model and discovers that it is downloading rather than immediately executing.
Provisioning is a substantial part of local inference. A large checkpoint can require a terabyte-scale download. Cheema says the current arrangement keeps essentially the complete model weights on each machine’s storage, even though runtime execution is split. If a node disappears, the remaining machines may need to assume different responsibilities; storing only the current shard would complicate that reassignment. Storage replication and distributed memory allocation therefore solve different problems.
The test model is Qwen 0.6B, described as about 0.3 GB. Existing downloads from other applications require manual migration in the workshop build, with automatic recognition proposed for later; current EXO documentation now includes additional read-only model-directory options. The small model reaches its ready state and produces output on the MacBook despite the earlier OS warning. Cheema notices that the utilization display may be broken, so the successful response should not be read as validation of that telemetry.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Moving prefill also means moving the KV cache
The Spark–MacBook demonstration assigns prefill to Spark and decode to the MacBook. Cheema cites 546 GB/s of memory bandwidth for the MacBook versus 273 GB/s for Spark, while describing Spark as having four times the compute. Those approximate specifications explain the placement: use Spark for substantial prompt processing, then use the Mac’s bandwidth for repeated decode steps. His additional twelvefold ratio claim does not follow from those two stated ratios; on a comparable basis they imply an eightfold contrast in compute-to-bandwidth ratio.
The split introduces a new dependency. Prefill constructs the KV cache on Spark, but decode needs that cache on the MacBook. The intended transport is 10 Gb Ethernet, with an adapter supplying the MacBook’s Ethernet connection. A simpler USB-C arrangement is experimental and requires scripts in this version.
The demonstration initially sends cache data over Wi-Fi instead, making the network the bottleneck. Cheema explains the required schedule: stream cache data while computation proceeds, so communication overlaps useful work. If the link cannot keep up, the stages become effectively serial—prefill, wait for the remaining transfer, then decode. Faster prefill hardware cannot help enough if it merely creates a longer queue of cache data waiting to cross the network.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Measure quality alongside speed
The proposed closing project addresses a practical information problem: much local-inference advice comes from Reddit and Twitter, where useful experimentation mixes with poorly understood results. AI tools make experiments easier to run, but they can also encourage an experimenter to mistake runnable output for a breakthrough.
Cheema is particularly skeptical of extreme quantization claims. A one-bit version of a large model may be less useful than a smaller model kept at higher precision. A headline about running Kimi on a MacBook can also hide changes to expert activation—for example, activating two experts instead of eight. Those modifications alter model quality as well as resource use.
EXO’s proposed response is an open benchmark site, announced for roughly the following month, with thousands of results being collected continuously across hardware, models, quantizations and pruning choices. The aim is to track progress in software and hardware while measuring tokens per second, prefill time, task quality and energy efficiency together. The accompanying slide shows hardware photographs and the benchmark-site announcement, not completed benchmark results.
A budget selector would expose a Pareto frontier: configurations for which improving one objective requires sacrificing another. At an illustrative $10,000 budget, Cheema contrasts accepting slower GLM-5.1 output for quality with choosing a smaller Gemma model for speed. His illustrative 20-versus-100-tokens-per-second comparison is not tied to specified measured configurations. The useful purchasing question is which quality, latency and energy tradeoff serves the workload, rather than which setup can claim to run the largest model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Autoresearch still needs a hypothesis and a quality check
A final audience question references Simon Willison discussing Karpathy’s autoresearch loop, reportedly used to optimize disk-backed inference on a 48 GB M3 Mac. Cheema responds cautiously: he has seen results whose apparent success depends on cherry-picking or severe model degradation. His example is a one-bit Kimi running from disk at roughly five tokens per second. An optimizer can improve a speed measurement while silently damaging the model’s usefulness.
He uses and values autoresearch, but places it inside an experimental process:
- Form a reasoned hypothesis about the bottleneck and the proposed change.
- Run an experiment that tests that hypothesis, including the model-quality consequences.
- Inspect the result and iterate on what the experiment actually establishes.
An unrestricted instruction to make inference faster can become what Cheema calls a “slot machine”: repeatedly trying changes and trusting a favorable-looking number. Brute-force search may occasionally find something useful, but it does not remove the need to understand what changed.
His skepticism about disk is narrower than a rejection of memory hierarchies. A MacBook’s single SSD is too slow for the role he wants memory to play. He allows that changing RAM prices might make a large SSD array economically interesting—perhaps a box with sixty-four SSDs, with RAID 0 suggested by an audience member—but that is a different machine from a laptop.
MoE expert prediction can improve a hierarchy by preloading the expert likely to be needed next. In Cheema’s illustrative example, a predictor that is right 90% of the time gets useful data ready early; the remaining misses require additional loading. He would first apply such techniques to ordinary memory, where a successful prediction avoids a smaller underlying latency penalty and the system retains much higher bandwidth. The question is whether the complete hierarchy is economical and useful, not merely whether weights can be streamed from disk.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The resumed Spark–Mac comparison
After the apparent ending, a collaborator reports that the heterogeneous setup has worked in the background. The team returns to the comparison, showing the combined arrangement first: Spark processes the prompt and the MacBook decodes. The input consists of paper text, described at different points as a 38 KB paper, two papers and roughly 100 KB. Those changing descriptions matter because prompt size determines how much prefill work can be offloaded.
A greeting such as “hello” offers little work to amortize the split. A large, uncached prompt can spend substantial time in prefill; full-attention computation includes a quadratic component in sequence length, though that is not a universal scaling law for all prefill work or architectures. During the attempt, Cheema points to Spark utilization while cache data streams toward the MacBook. Another Wi-Fi error interrupts the run.
Cheema and his collaborator report approximately 2× faster end-to-end prefill-plus-decode execution for the referenced large-prompt Spark–MacBook workload compared with the MacBook alone. The exact model, prompt size, cache state and timing protocol are not established. He then suggests an RTX 5090 as a cheaper alternative with more compute and bandwidth, opening further opportunities to split model components.
The visible single-node output includes a summary of “Attention Is All You Need” and a table of Transformer components. The collaborator then reports about seven seconds for one paper on the MacBook alone. This is the baseline for the final retry, rather than visual evidence of a two-device speedup.
After another attempt with Spark enabled, Cheema reads 4.8 seconds for the same paper across both devices. The final narrated comparison is about seven seconds on the MacBook alone versus 4.8 seconds across Spark and MacBook, or approximately 1.46× faster. Those timings follow retries and are distinct from the earlier reported 2× result. Cheema closes by returning to the workload dependency: the benefit grows as prompt processing becomes a larger part of the job. A useful home inference system must fit the model, place the work on suitable hardware and deliver the data to that hardware quickly enough.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
EXO's local inference application connects machines, discovers their topology and distributes model execution.
Sara Hooker's essay explains how hardware and software availability influence which research ideas succeed.
Official model information, checkpoint access and local serving guidance for GLM-5.1.
Further reading
A study of task accuracy and power efficiency across local models and accelerators, including progress from 2023 to 2025.
Hugging Face recipes for inference-time search, including Best-of-N, beam search and verifier-guided tree search.
Updates since the talk
Current installation instructions, model-directory settings and Thunderbolt RDMA requirements for EXO clusters.
Full transcript
Loading transcript…