AI Engineer Summit 2023

Harnessing the Power of LLMs Locally

Mithun Hunsur· Senior Engineer, Ambient17:09

Loading slides…

Read the talk

Harnessing the Power of LLMs Locally

Local inference puts model selection, session state, sampling, and deployment inside your application. Mithun Hunsur’s llm.rs examples show what that control enables—and what it costs.

From a talk by Mithun Hunsur

Before you start: Basic familiarity with language-model prompts and Rust functions, closures, and threads will help with the implementation examples.

What changes when the model runs on your computer?

What changes when an application runs its own language model instead of sending prompts to a hosted service? With ChatGPT, Claude, or Bard, the provider operates the model. With local inference, the model runs on your computer, giving your application control over the hardware, inference process, and data it sends into that process. Mithun Hunsur, introduces this approach through llm.rs, the Rust inference library he maintains alongside his work building a game engine at Ambient. The recording describes the historical library; its repository is now archived and unmaintained.

0:150:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:15 · section reference included

Model capacity meets available hardware

The first constraint is model size. Hunsur uses parameter count as a rough proxy for capacity: the hosted models in his comparison—GPT-3, GPT-4, Gopher, and PaLM 2—occupy the large end of the chart, while Llama and Falcon represent the growing local-model ecosystem. Size is not a task-quality guarantee, however. A smaller model focused on one problem may solve it better than a larger general model. The chart’s GPT-4 size is explicitly speculative; Hunsur says its actual size is unknown outside OpenAI.

Slide titled “Differences — Size” with a bar chart labeled “Parameter count (billions)” and a Local/Remote legend.Open full source frame
Local and remote models compared by parameter count.

Hardware introduces a separate limit. Hosted inference uses specialized infrastructure, while local inference uses whatever equipment you can obtain, including rented machines. More capable infrastructure can improve generation speed, support more simultaneous inference, or both. Hunsur’s illustrative hardware-cost axis runs from a few hundred dollars to a few hundred million dollars. This is a comparison of infrastructure scale, not a measured price-performance benchmark.

1:111:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:11 · section reference included

Latency, cost, customization, and privacy

Latency is not just the speed at which a model generates tokens. In the hosted request pattern Hunsur describes, the application submits a completed prompt and waits for the network exchange. A local application can feed input into the model incrementally as it arrives. For conversation, that means processing what someone is saying while they are still saying it. Removing the network trip and overlapping prompt processing with input can reduce waiting, although inference itself still takes time.

The economic comparison is similarly workload-dependent:

QuestionHosted inferenceLocal inference
What determines the bill?Provider’s per-token priceCost of operating the machine
Who supplies the hardware?ProviderYou, or a hardware rental service
When can local be cheaper?Compare the workload’s API billWhen local operating cost is lower

An existing computer reduces additional acquisition cost; it does not make electricity, machine time, or operation literally free. The relevant comparison is the cost of serving your workload, not simply whether you already own a laptop.

Model choice adds another reason to run locally. Hosted applications select from a provider’s catalog, and fine-tuning may introduce both training fees and higher usage charges. Local applications can choose custom models for retrieval, storytelling, conversation, or tool use, then fine-tune further when necessary. Hunsur recommends Axolotl for that training work. Finally, local inference can keep private prompts on the machine instead of submitting them to a cloud model—a practical benefit even when the question is merely embarrassing.

2:242:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:24 · section reference included

Fitting billions of parameters into memory

Owning the inference process only helps if the model fits the available hardware. Billions of parameters mean billions of stored numerical values, and consumer machines have limited memory. Quantization addresses this by representing parameters at lower precision: a lossy compression that reduces the model’s memory requirement while preserving much of its useful capability.

The benefit extends beyond storing a smaller file. A smaller representation can let the computer process more of the model at a time, potentially improving inference speed. The trade-off is information loss, so reduced size and retained quality have to be considered together. Hunsur presents the mechanism rather than a particular compression ratio or speed benchmark.

3:584:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:58 · section reference included

From a Rust rewrite to a reusable library

The project began with Hunsur encountering llama.cpp in March. The repository snapshots he presents contrast 98 stars early on with roughly 42,000 at presentation time. His response was to rewrite it in Rust, partly for the familiar programming joke and partly because he wanted to reuse inference in other applications. Another developer finished a parallel implementation first. They merged their projects, and Hunsur became maintainer of the resulting llm library.

Why build another implementation? Hunsur describes the March version of llama.cpp as an application rather than a reusable library. His design starts with embedding inference in other software and develops six requirements:

  • Library first: expose inference as functionality an application can call.
  • Customizable behavior: avoid coupling the library to one application or assuming how it will be used.
  • Multiple architectures: support more than one model family.
  • Idiomatic Rust: make the interface feel native to Rust, rather than like foreign-language bindings.
  • Multiple backends: allow CPU, GPU, and future compute implementations.
  • Portable deployment: keep application behavior consistent across Windows, Linux, macOS, and other platforms.

Together, these requirements make inference a component whose behavior the application can shape.

Library slide with llm.rs surrounded by customizable, library, models, rust-native, platforms, and backends.Open full source frame
The llm.rs library’s six design themes.

The supported architectures, including Llama and Falcon, share an interface. Switching architectures therefore does not require rewriting the application’s integration around a completely different API. Hunsur credits Lucas, Dan, and the wider contributor community for making that breadth possible.

4:524:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:52 · section reference included

Load a model, then keep state in sessions

The demonstrated API separates the loaded model from an inference session, which tracks one ongoing use of that model. This is the historical rustformers API, not the provider interface now documented under the latest llm crate. Its application flow is straightforward:

  1. Load the model.
  2. Create a session for an ongoing interaction.
  3. Pass a prompt into that session and run inference.
  4. Receive generated tokens through a callback.

You can create multiple sessions, but each carries a memory cost. Keeping a session alive lets a conversation continue without feeding its previous context back into the model on every turn. Session lifetime therefore affects both memory consumption and conversational behavior.

The inference function is a convenience helper: it repeatedly calls the model until its boundary conditions say to stop. An application that needs different control flow can replace that loop. Likewise, the sample’s repeated Default values are customization points for model loading, inference, and sampling. The defaults provide a short path to inference without making the default behavior mandatory.

Hunsur then demonstrates Llama 7B running on his MacBook CPU before showing GPU acceleration. The GPU version makes the interaction visibly more usable in his presentation; he also reports faster performance on NVIDIA GPUs. These are qualitative comparisons, with no tokens-per-second result or complete benchmark configuration. AMD and Intel support is described as pending at the time of the demonstration.

7:187:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:18 · section reference included

Three ways to package local inference

The community projects show how the same inference engine fits different kinds of software:

  • local.ai: louisgv’s desktop application makes local inference available through a downloadable app, without requiring the user to build an integration.
  • llm-chain: a Rust workflow library that Hunsur describes as LangChain for Rust, with llm.rs as an inference option.
  • Floneum: a flowchart-based application in which users connect nodes into workflows, using the library as an inference engine.

The reusable part is model execution; the surrounding product can be a simple app, a programmatic workflow, or a visual editor.

8:549:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:54 · section reference included

Separating generation from response delivery

Hunsur’s first proof of concept, llmcord, puts the library behind a Discord bot. A user sends a prompt and receives generated text. Hunsur attributes the visible hitches in the demonstration to Discord’s limits rather than inference. Underneath the chat interface, a generation thread and a channel connect the request to a response task responsible for sending output to the user.

Flow diagram connecting User request to a Response channel above a Generation thread, then to a Response task and two response messages.Open full source frame
The llmcord request and response pipeline.

The separation can be expressed directly in Rust. This adapter accepts a generation function that emits text chunks and a response function that delivers them. The channel carries output between the generation thread and the response side:

rust

use std::sync::mpsc::{self, Sender};
use std::thread;

pub fn stream_response<G, R>(generate: G, mut respond: R)
where
    G: FnOnce(Sender<String>) + Send + 'static,
    R: FnMut(String),
{
    let (output, incoming) = mpsc::channel::<String>();
    let generation = thread::spawn(move || generate(output));

    for chunk in incoming {
        respond(chunk);
    }

    generation.join().expect("generation thread panicked");
}

An inference callback supplies chunks to output; respond handles delivery. Once generation finishes and its sender is dropped, the receiver loop ends. Keeping those responsibilities separate lets the response side accommodate a transport’s delivery behavior without putting that behavior inside model generation.

The bot creates and discards a session for each query. Retaining a session instead would let later requests continue the conversation. The transport is another replaceable part: remove Discord, put HTTP at the boundary, and the core flow remains request, generation, response. The session policy and the transport policy are independent application decisions.

9:369:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:36 · section reference included

Autocomplete beyond the code editor

Alpa, Hunsur’s second proof of concept, starts with a limitation he encounters using GitHub Copilot: completion is tied to his editor and requires an internet connection. Alpa takes text to the left of the cursor and uses a local model to continue it elsewhere in the operating system. Model selection remains open, including a model fine-tuned on the user’s own writing.

The application loop is small:

  1. Listen for input.
  2. Copy the relevant text into a prompt.
  3. Generate a continuation with the model.
  4. Type the response into the application.

The inference engine supplies text; the operating-system integration decides where that text comes from and where the completion goes.

10:4511:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:45 · section reference included

Turning Wikipedia dates into structured data

The third application moves from proofs of concept to a data-extraction workload. Hunsur wants to build a world-history timeline from dates scattered across Wikipedia. The scale is millions of pages, but the harder problem is interpretation: date expressions are unstructured, and finding a date-like substring does not recover the context needed to understand it. Regular expressions can locate patterns without resolving what those patterns mean.

He first tries GPT-3 and GPT-4. Even after prompt engineering, the results remain imperfect, and processing millions of dates is too slow and expensive for his workload. That leads to a specialized model and a data preparation loop:

  1. Generate a representative dataset with GPT-3.
  2. Inspect examples in a custom tool.
  3. Correct erroneous data points and assemble the corrected dataset.
  4. Fine-tune a model with Axolotl.

The correction step matters: generated examples are material to review and repair before training.

The resulting model accepts a date expression and returns a structured representation that Rust can parse. Hunsur describes it as small, fast, and consistent, without giving measured accuracy or throughput. Once the inference and output parsing are wrapped behind an fn parse-style interface, the rest of the ingestion pipeline can treat the model as a component: pass dates in, receive structured dates back. Specialization turns a broad language-model capability into a narrow application interface.

11:2011:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:20 · section reference included

Shipping inference and controlling its behavior

Embedding inference in Rust also changes distribution. Hunsur contrasts managing Python environments through conda, pip, and Pipenv with using Rust’s build system and platform support to ship self-contained application binaries. Users need not install Torch to run these applications, making local models easier to incorporate into desktop software.

The surrounding ecosystem matters as much as the model interface. Rust libraries let Hunsur build Discord integration, operating-system autocomplete, a data-ingestion pipeline, and tools for exploring and correcting datasets in the same language. Inference becomes another library in that application, rather than a separate service around which the entire program must be organized.

Control also extends to token sampling. Hunsur contrasts direct access to local generation with hosted interfaces that expose limited probability information and require repeated network exchanges to influence the process. Owning the inference loop lets application code decide how to sample at the point where the next token is selected.

That access creates room to experiment with new research before a hosted provider offers equivalent functionality. A developer can implement a technique, try it against the application’s problem, and decide whether it is an improvement. The opportunity is earlier experimentation—not an assurance that every new paper will help.

12:5012:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:50 · section reference included

Hardware, compatibility, and licensing still constrain the application

Local inference does not remove hardware requirements. Hunsur explicitly qualifies the earlier suggestion that models can run on almost anything: an old computer, smartphone, or Raspberry Pi has limits that smaller models and better inference implementations can ease but not erase. The application still has to choose among speed, cost, and quality. A larger model may improve results while making the interaction slower; whether that is acceptable depends on the task.

Rapid innovation also creates compatibility churn. A new model can break a previously working workflow. Hunsur points to his work on GGUF standardization as one way to reduce that friction, not eliminate it. Format standardization should not be confused with support in this particular library: the archived repository’s main branch lacks GGUF support, while its experimental GGUF branch has limited Llama support and lacks quantization.

Licensing is a separate constraint. Downloadable weights and permission for personal use do not establish unrestricted rights; model licenses can contain consequential clauses and exceptions. Hunsur points to Mistral 7B and StableLM as encouraging developments in smaller models, compared with the larger Llama and Falcon models in his discussion. Their licenses still need to be distinguished: Mistral 7B was released under Apache 2.0, while StableLM terms vary by checkpoint, including attribution/share-alike and noncommercial terms. The model family’s name alone is not enough to determine whether a release fits an application.

14:2914:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:29 · section reference included

More accessible models, application-specific choices

Smaller, more capable models make local inference easier to adopt, and a reusable library makes that capability easier to embed. Neither development produces one obvious answer for every problem. The practical choice remains tied to the application: what it needs the model to do, how quickly it must respond, and which constraints its deployment can accept.

Conclusion slide describing local LLMs as great, powerful, and controllable, saying llm.rs makes them easier to use, and ending with “but not perfect.”Open full source frame
Local LLMs are becoming easier to run, but remain imperfect.

Hunsur closes by directing viewers to llm.rs and inviting contributions from people interested in Rust or language models. Sponsorship would help test additional hardware—a concrete requirement for a project aiming to support diverse machines. He offers email, Mastodon, and conversation at the conference as ways to get involved.

16:2016:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:20 · section reference included

Resources

From the talk

  • Archived source for the library in the talk, including CLI examples, session persistence and model-format limitations.

  • Desktop application built around llm, with model downloads, notes and a streaming inference server.

  • llmcordRepository9:45

    Archived Rust Discord bot demonstrating integration with the llm inference library.

  • Fine-tuning framework with dataset preparation, configuration examples and training instructions. The repository documents its current capabilities.

  • The September 2023 announcement describes the 7.3B model, Apache 2.0 licensing and the team's benchmark methodology.

  • Model-family documentation distinguishing base-model, tuned-model and source-code licenses.

Transcript download
Full transcript

Loading transcript…