AI Engineer World's Fair 2026
Frontier results, on device - RL Nabors, Arize
Loading slides…
Read the talk
Frontier results, on device
A local model can replace a frontier call when the task is narrow, the acceptance criteria are explicit, and evaluation covers the whole application workflow.
From a talk by Rachel Lee Nabors (RL Nabors)
Before you start: Familiarity with JSON, language-model prompts, and basic application testing will help; no model-training experience is required.
Does this task need a frontier model?
What does every call to a frontier model cost your application—and does the task need that model at all? Rachel Lee Nabors approaches this question from web development: Firefox DevTools, W3C standards, Microsoft Edge, React, and three years consulting on web, AI, and UI before joining Arize. A seemingly small prompt edit or model migration can break an agent workflow, making evaluation as important as the initial model choice. Phoenix, Arize’s evaluation and observability tooling, provides the experiment environment used here.
Reaching for GPT-5 or Claude brings costs beyond the token bill. Remote inference sends data to another organization’s servers, introducing exposure, interception, and retention risks; Nabors points to business-data leaks as a consequence. Latency also changes the experience. Nabors cites research on response delays in LLM-powered virtual agents when describing four seconds as a believability threshold. That study concerns spoken interactions in VR, so it motivates a responsiveness budget rather than establishing a universal deadline for every AI interface.
The bill can grow even as individual tokens become cheaper: agentic and reasoning workflows make more calls and consume more tokens. Remote inference also makes connectivity a product dependency. An outage, unavailable Wi-Fi, or a deliberately disconnected secure environment can disable the feature entirely. These business, user, and environmental costs lead to the first design decision: identify the capability the task actually requires before choosing the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Match the capability, then the device
A camera input does not automatically require a general multimodal model, and a microphone does not automatically require a language model. Nabors offers a practical starting map:
| Input or task | Candidate families |
|---|---|
| Camera and vision | MobileNet, YOLO, MediaPipe |
| Recorded audio | Whisper, wav2vec2 |
| Chat, translation, language analysis | Gemma, Qwen |
Choose a specialized model when the job is specialized. Choose a smaller language model when the application needs language generation or interpretation without the full breadth of a foundation model.
The boundary around small language models, or SLMs, is fuzzy. Nabors describes them as ranging from millions to billions of parameters, overlapping with the low end of larger models. The parameter-count dots in the presentation illustrate why this matters: the machine needed to hold and run a model changes dramatically with its size. A thread summary or hostile-behavior detector does not necessarily need broad historical and philosophical knowledge, or the ability to analyze images and audio together.
Quantization makes those smaller models easier to distribute. At FP16, one billion parameters requires about 2 GB for weights; 8-bit and 4-bit representations can roughly halve or quarter that storage. Runtime memory must also accommodate more than the weights, so the presentation’s device-fit grid is a sizing guide, described by Nabors as upper-bound estimates, rather than a guarantee that a particular model will run comfortably.
Nabors bought a Pixel 10 Pro specifically to explore its on-device model. The broader case for this deployment pattern appears in the NVIDIA-associated position paper Small Language Models are the Future of Agentic AI: many agent invocations are narrow enough for smaller, specialized models. Nabors’s illustrative energy chart places an SLM at about 25% of the LLM’s task energy and a task-specific model at about half the SLM’s amount. The paper supports the efficiency argument, but does not establish those fixed ratios for an identified workload and hardware setup.
Running locally removes the network round trip and the provider’s per-call inference fee. It also enables offline operation after the model is available and keeps model inputs on the device. Those advantages make local inference attractive, but actual speed still depends on the model and the hardware executing it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Can a local model reason about ichthyosaurs?
A side-by-side conversation makes the capability question concrete. Nabors runs Gemma in Goose, an open-source agent harness, beside Claude. The local response takes longer in this example—a reminder that removing network latency does not eliminate generation time.
The question is whether ichthyosaurs, marine reptiles with dolphin-like skeletons, could have used echolocation. A useful response must move beyond visual resemblance and reason about the biological structures involved. The local model, which Nabors tentatively identifies as Gemma 3, points to the lack of fossil evidence for a melon and the specialized sound-conducting bones found in dolphins. Nabors finds this a good response; Claude sometimes retreats to uncertainty about what can be known. This informal probe suggests that a local model can supply useful reasoning, but choosing a production model requires a repeatable task evaluation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prototype the feature and capture acceptable outputs
Nabors’s model-selection framework, developed with Google and described in Choose a small, efficient model, begins with prototype big, deploy small. Start with a highly capable model to find out whether the feature is feasible. For handwriting or writing-style recognition, that might mean Gemini or a strong task-specific model. Success establishes a target that smaller candidates can then attempt to meet; it does not establish in advance which candidate will succeed.
The worked application is Mima, Nabors’s client for multiple social networks. Its summary feature addresses a familiar problem: waking up to 50 comments on a post and wanting to know what people are discussing—and whether they are angry. Nabors first prototypes the feature with Claude and judges its summaries good enough to make the feature useful. Those accepted outputs become the starting point for evaluation.
A golden dataset pairs representative inputs with the outputs the application should produce. Ideally, people curate and label those examples. Here, Nabors accepts Claude-generated summaries after reviewing their adequacy. The public thread records include author handles, message content, creation times, and a flag identifying messages written by Nabors. The dataset is exported as JSONL, with two output tasks for each thread: a short summary and a summary containing references deeper into the conversation. The dataset contains 14 threads and 28 examples across those two output tasks.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn product requirements into evaluation checks
Before comparing models, define what a usable summary must do. Some requirements are deterministic; others require judgment.
| Requirement | Evaluation |
|---|---|
| Parseable JSON | Attempt JSON.parse |
| Valid references | Check that referenced messages exist |
| Factual consistency | Compare summary meaning with the thread |
| Length compliance | Check the word count |
| Responsiveness | Measure P50 and P95 latency |
Parsing checks JSON syntax, not whether the result has the right fields or points to real messages. Factual consistency is a different question again: a conversation about cats should not become a summary about puppies. Nabors uses an LLM judge for that semantic assessment; human review is another option. P50 is median latency, while P95 describes the tail of the latency distribution, not its absolute worst case.
With those checks in place, test candidates from small to large. In Phoenix, a capability evaluation compares what each candidate can do against the reference outputs and acceptance criteria. Nabors introduces this with an illustrative comparison between Claude Opus and Gemma; the actual experiment then establishes a Sonnet baseline. Phoenix is presented as free and open source in the talk; its current distribution specifies the Elastic License 2.0.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Find the small and good enough model
The actual baseline uses Claude Sonnet. Nabors reports an average latency of 2.9 seconds and a cost of about $0.22 for 14 tasks. Extrapolating from personal Mima usage gives about $1 per day in inference spending, a meaningful subsidy if the application grows. The local candidates show zero provider inference charges because computation moves to the user’s device. That transfers the energy cost to the user’s battery and charging, rather than making computation free; battery drain was left for future evaluation.
The local shortlist spans several sizes. These are the parameter counts and disk sizes Nabors reports for the candidates:
| Candidate | Parameters | Disk size |
|---|---|---|
| Qwen 2.5 Instruct | 1.5B | About 1 GB |
| Qwen 3 | 1.7B | Not stated |
| Llama 3.2 | 3B | About 2 GB |
| Gemma 4 E2B | About 5B | 3.1 GB |
The later official Gemma 4 model card distinguishes effective parameters from total parameters including embeddings; it does not identify the exact artifact tested here. Gemma was the enthusiastic recommendation from several engineers Nabors consulted, but its performance in this application made that recommendation a poor substitute for measurement.
SAGE means “small and good enough”: select the smallest model that produces acceptable responses for the application. Qwen 2.5 initially looks attractive on speed. In the model comparison, Nabors reports roughly one-second P50 latency for Qwen 2.5, roughly three-second latency for Sonnet, roughly 90% accuracy for Llama 3.2, and roughly eight-second latency for Gemma 4 E2B. On the plot, these appear as the gray Qwen circle, blue Sonnet diamond, large green Llama circle, and orange Gemma square. Qwen’s weaker accuracy changes the decision; Llama offers the more useful balance.
Nabors tested each model three times and averaged the results. A different prompt might have helped Gemma, but under the tested prompts its slower behavior was consistent. These are application-specific measurements: hardware, runtime, quantization, exact cloud-model versions, and accuracy aggregation are not specified. The next check is therefore to open the Phoenix experiment and inspect raw outputs beside expected outputs. Nabors finds many Llama summaries effectively indistinguishable from Claude’s and selects Llama 3.2. Meta’s interest in social-network content offers a possible explanation for the fit, but that is Nabors’s interpretation rather than evidence about the cause of its performance.
The selection procedure is repeatable:
- Prove feasibility with a sufficiently capable model.
- Set the bar with representative inputs, acceptable outputs, and success criteria.
- Test from small to large, comparing each candidate against that bar.
- Select SAGE, the smallest candidate that meets the application’s requirements.
This gives model size a concrete role in the decision without allowing it to replace output quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Improve the prompt before replacing the weights
Selecting Llama leaves a quality gap to close. A distilled model is one possible route, but changing model weights carries a distribution cost: adding capabilities may require retraining and asking mobile users to download another large model. Data plans and download friction constrain that approach. An established team may also be committed to Gemma 3 until a later model update passes its evaluations. Prompt optimization is useful when the deployed model is difficult to replace.
Nabors focuses on JSON and reference structural validity, factual consistency, and latency. The displayed acceptance targets are at least 99% structural validity, at least 95% factual consistency, P50 latency at most 1.5 seconds, and P95 latency at most 3.5 seconds. The latency budgets leave room below the response-delay threshold that motivated the feature’s experience requirements. These targets turn “close the gap” into a set of checks for the next experiment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test one prompt hypothesis at a time
The prompt experiment keeps the original baseline and adds four variants, each testing a specific hypothesis:
- Numbered input: Replace the JSON thread with numbered messages. The hypothesis is that a smaller model may track natural-language indexing more reliably than array offsets inside JSON.
- Few-shot examples: Add thread-and-output pairs. The hypothesis is that examples teach the desired format more effectively than rules alone.
- Strict rules: Add literal constraints, including no preamble and counting words before responding. The hypothesis is that explicit commands improve compliance.
- Chain of thought: Ask the model to identify key moments before writing the summary. The hypothesis is that this intermediate work improves grounding.
Nabors runs the variants locally with Llama 3.2, holding the model fixed so the prompt changes can be compared. The resulting measurements can also be taken into a conversation with Claude to explore the tradeoffs.
The baseline struggles to keep summaries short enough for their allotted interface space. Nabors reports baseline reference accuracy of 91.2%, factual correctness of 87.1%, and latency of about one second. Reformatting the input makes little difference, and explicit rules worsen the results. Chain of thought improves length slightly but adds about 600 ms of latency. Few-shot prompting improves length, references, and agreement with Claude while adding about 200 ms. Examples provide the best quality improvement for the added delay.
In the subsequent comparison, Llama 3.2 3B with the few-shot prompt reaches 91.7% structural validity, 92.9% factual consistency, and P50 latency below 1.5 seconds. Nabors considers this close enough to investigate the remaining failures individually. No statistical margin calculation is supplied, so the useful next step is to inspect which outputs received penalties and why.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Inspect the judge and enforce deterministic constraints
A factual-consistency score can hide disagreements about interpretation. Here, Claude Opus judges Llama’s output against Claude Sonnet’s reference summary. Nabors finds penalties for distinctions such as describing Jenna as “angsty” rather than “cross,” and interprets some of them as excessive strictness or favoritism toward the related model. The inspection establishes that the judge’s decisions deserve review; it does not establish that every penalty is wrong or that model-family bias caused it.
Reference validity and output length are more directly controllable in the application harness. Nabors checks references against the size of the thread and truncates overlong summaries. A count check can catch obvious problems, but each reference should also point to an existing message. The following JavaScript illustrates that boundary using a three-message thread about cats and a proposed summary containing an invalid reference. It validates the object shape, removes nonexistent message references, and applies a word limit:
javascript
const thread = [
{ id: 1, content: "My cat keeps waking me up." },
{ id: 2, content: "Mine wants breakfast early." },
{ id: 3, content: "An automatic feeder helped us." }
];
const raw = JSON.stringify({
summary: "People discuss cats waking them early and suggest an automatic feeder.",
refs: [1, 3, 9]
});
function normalizeSummary(raw, thread, maxWords) {
if (!Number.isInteger(maxWords) || maxWords < 1) {
throw new Error("maxWords must be a positive integer");
}
const result = JSON.parse(raw);
if (
result === null ||
typeof result !== "object" ||
typeof result.summary !== "string" ||
!Array.isArray(result.refs) ||
!result.refs.every(Number.isInteger)
) {
throw new Error("Invalid summary shape");
}
const validIds = new Set(thread.map(message => message.id));
const words = result.summary.trim().split(/\s+/).filter(Boolean);
return {
summary: words.slice(0, maxWords).join(" "),
refs: [...new Set(result.refs.filter(id => validIds.has(id)))]
};
}
const normalized = normalizeSummary(raw, thread, 8);
// {
// summary: "People discuss cats waking them early and suggest",
// refs: [1, 3]
// }
The example also exposes the limit of this repair: the word count passes, but truncation leaves an unfinished sentence. Likewise, an existing reference can still fail to support a claim. Deterministic enforcement should therefore remain separate from the factual and readability evaluations.
After adding post-processing, Nabors reports 100% JSON validity and 100% structural validity on this evaluation. Some factual disagreement remains, which Nabors attributes to the judge’s strictness. The completed workflow has reported P50 latency of about one second and P95 latency below 3.5 seconds. Nabors describes the result as meeting or beating Sonnet for this summary feature and saving about $1 per day in personal inference costs. The improvement belongs to the complete workflow—model, prompt, validation, and post-processing—not just the model weights.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep the gains and move one feature
Once the feature works, preserve those gains with regression evaluations. Run the same checks when changing a prompt or upgrading a model, much as CI/CD runs application tests. The failure modes are concrete: short summaries expand into paragraphs, references break, or unsupported claims appear. This returns to the opening problem of a seemingly harmless change disrupting an agent experience—an incident Nabors attributes to a founder friend.
The next opportunity is already in the application’s call history: which Claude calls could become Llama calls, and what would that save? Look for models already managed by the user’s platform. Chrome’s Prompt API exposes Gemini Nano, allowing Chrome to manage model distribution instead of requiring the application to ship its own weights. Current documentation still requires an initial model download, availability checks, and eligible hardware; these foundation-model APIs exclude mobile Chrome. Once downloaded and available, inference can run offline.
Local inference keeps the model’s processing on the device, reducing the need to transmit user information for that step. It does not determine how the rest of the application handles personal data. The practical migration can remain narrow: treat the existing frontier implementation as a prototype, choose one feature, establish its acceptance criteria, and test smaller candidates. Use prompt engineering to improve the selected model where needed, then retain the evaluations as the feature evolves. Nabors’s closing challenge is to make that one-feature conversion and measure the result.
For follow-up, Nabors points readers to the Nearest Neighbors site, Phoenix for trying evaluations, and the Mima beta for the application behind the case study.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Nabors’s guide to defining success criteria and selecting the smallest model that meets them.
Tools for tracing model calls, assembling evaluation datasets, and comparing experiments locally or on hosted infrastructure.
A position paper arguing for smaller models on specialized agent tasks and mixed-model systems where broader capabilities are needed.
- Mitigating Response Delays in Free-Form Conversations with LLM-powered Intelligent Virtual AgentsPaper1:38
A VR user study examining how response delays and conversational fillers affect perceived responsiveness and interaction quality.
Use Gemini Nano through Chrome, check device eligibility, and handle model downloads and session creation.
An extensible AI agent harness for working with language models and tools.
Updates since the talk
- Gemma 4 model cardDocumentation
Official model specifications, including the distinction between effective parameters and total parameters with embeddings for E2B.
Full transcript
Loading transcript…