AI Engineer World's Fair 2026
OpenClaw in Your Hand: Building a Physical AI Terminal for Local LLM Agents
Loading slides…
Read the talk
OpenClaw in Your Hand: Building a Physical AI Terminal for Local LLM Agents
A keyboard, OLED and e-paper display make a quiet interface to OpenClaw, while a separate DGX Spark handles inference, agent actions and generated adventures.
From a talk by Lech Kalinowski
Before you start: Basic familiarity with microcontrollers, LLM inference and agents that can act on a computer is helpful.
A terminal that answers back
During a basement cleanup, a dusty device appears in a corner: an old-looking keyboard terminal with unusual displays. Its shell is still running. Type “Who am I?” and it answers in natural language. Lech Kalinowski opens with this imagined discovery, then reveals that it is a possible marketing story. The real ambition is straightforward: build a physical, AI-native device that feels as though it came from the future.
Kalinowski, who holds a PhD in physics, spent several months building the device, its backend and the surrounding system. The starting requirement was narrower than the eventual product: a remote controller for an OpenClaw instance on his DGX Spark. That controller became Vault.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate typing from reading
For this interface, the useful starting point is text: a place to type requests and read model responses. Electronic paper suits sustained reading, but its refresh behavior makes it less suitable for updating the screen with every keystroke. Kalinowski pairs it with a small monochrome OLED, dividing the work between two surfaces instead of asking one display to serve both purposes. The intended benefit is a responsive terminal with economical display power use.
The remote controller also becomes an experiment in an AI-native operating interface built around a microcontroller. Its four modes are shell, assist, control and RPG, as enumerated in the recording’s description. The internal shell handles the terminal’s own settings, basic configuration and Wi-Fi connection; the other modes extend the interface toward assistance, remote actions and play.
In the prototype, the OLED is the live typing surface. Entering text updates that display; pressing Enter triggers a render on the bistable e-paper display. This gives the interaction a clear boundary between composing an input and committing a page for reading.
| Surface | Job | Update behavior |
|---|---|---|
| OLED | Live text entry | Updates while typing |
| E-paper | Committed page | Renders after Enter |
The same interface carries requests and responses for LLM interaction and operations on the DGX Spark through OpenClaw.
Making that division practical requires a deliberately small rendering contract. Pages live as one-bit images in fixed, pre-allocated buffers. The microcontroller displays those images without a Markdown engine or calls to malloc. The page representation is therefore bounded in memory, and the terminal does not need to interpret a rich document format to put a response on screen.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Power delivery is part of the interface
The interior is more complicated than a keyboard attached to two screens. Much of that complexity comes from power management: Kalinowski reports destroying two displays during prototyping and needing much more stable current and voltage. A reliable interface depends on keeping its fragile display electronics alive.
The hardware centers on a dual-core ESP32 microcontroller, with the OLED, keyboard, rotary encoder and power-supply components around it. The opened enclosure shows how much wiring and supporting circuitry sits behind the restrained exterior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep inference off the handheld
Local inference happens on the backend, not inside the terminal. The compute and power requirements of the LLM would overwhelm the handheld design. Vault firmware runs on the device, while the backend handles both OpenClaw’s agent work and model inference. The terminal is a physical entry point into that system.
For the demonstration, Kalinowski describes an open-source GPT model in the 120-billion-parameter class; the recording’s description identifies it as gpt-oss:120b. He serves it using TensorRT, exposing an OpenAI-style interface through an LLM proxy. That adapter matters because other open-source models and serving setups had not matched the API behavior he needed. Current deployment guides describe available serving paths, but do not establish the exact proxy or server configuration used here.
A concrete example is asking the agent to write a Java example and save it on the local machine. Kalinowski reports that OpenClaw, with the model’s help, completed the request. The important transition is from generating an answer to creating a file on the backend host. A small Java artifact for that kind of request could be Hello.java:
java
public class Hello {
public static void main(String[] args) {
String name = args.length > 0 ? args[0] : "Vault";
System.out.println("Hello, " + name + "!");
}
}
The keyboard supplies the request and the displays present the interaction; the agent performs the filesystem action on the machine it controls.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What failed on the workbench
The prototype’s field notes are mostly electrical rather than model-related:
- Software I2C: The initial wiring did not work correctly. Kalinowski reports eventually controlling the bus without adding physical pull-ups. Treat that as a property of this build, not a general wiring recipe: Espressif’s I2C guidance generally recommends external pull-ups because the internal ones are weak.
- GPIO 13: A silent failure led him to move the connection to another port.
- Power regulation: A regulator damaged the OLED and other fragile display parts. Reworking the power-supply unit was necessary, and replacement parts took a couple of weeks to arrive.
- Encoder noise: A cheap rotary encoder produced noisy rotational input. He added pull-ups and capacitors to address it.
These failures explain why the supporting circuitry occupies so much of the hardware design: a clean text interface still depends on clean electrical signals.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The remote controller becomes an RPG console
The unexpected fit is a text-based role-playing game. Kalinowski had never played a tabletop RPG with friends gathered around a book and a game master. Building Vault gave him another route into that experience: a dedicated console where an LLM supplies the unfolding fiction.
NPCs, memory and the mood of the world give the game continuity beyond a single generated response. He creates four worlds, drawing on cyberpunk, Witcher-like fantasy with dragons, and a favorite deep-space setting. The text-focused hardware suits an experience in which reading, choosing and imagining are the main activities.
Generation extends to characters, worlds, maps and skills. Pictures then pass through the same rendering constraint introduced earlier: they are converted into one-bit matrices or image maps for the display. The generated fantasy imagery and its physical e-paper output show the two sides of the pipeline—rich material produced upstream, reduced to a representation the terminal can hold and render.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Preserve a usable path—and a quiet environment
The paired components also provide alternative interaction paths. Kalinowski describes the device as resilient because an individual failure need not make it completely unusable.
| Failure | Remaining path |
|---|---|
| OLED stops working | Read through e-paper |
| Keyboard stops working | Use the encoder |
| Wi-Fi connection fails | Use the local shell |
These paths preserve parts of the experience; the local shell, for example, remains useful without implying that disconnected backend inference continues.
That restrained interaction model is also the product idea. In his intellectual-property research, Kalinowski found much of the attention going toward audio interfaces and video capture. Vault instead makes room for quiet reading, writing, agent work and RPG play without bright moving graphics, advertisements or web browsing competing for attention. He reports pursuing a provisional patent, with the possibility of a commercial device in the future.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the build adds up to
Kalinowski reports 130 commits over roughly three months. His project inventory includes two displays, four modes, four gaming worlds and sixteen classes associated with the RPG system. He describes the firmware as small and fast, without giving a measured size or latency. The handheld uses a single battery cell; the spoken recap calls it lithium-polymer, while the earlier hardware slide labels it an 18650.
The resulting design principles follow directly from those choices: match each display to its job, keep heavyweight LLM inference off the tiny microcontroller, and preserve narrative context in the game rather than tracking only numbers. The display architecture controls the physical experience; the backend carries the computation and the remembered story.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From boot to a remote disk-space query
The recorded demonstration starts with the ordinary operations that make a dedicated device approachable:
- Boot the terminal and connect to the local Wi-Fi network.
- Move through the welcome screen to the main interface.
- Type
helpon the keyboard to display the available commands.
The command reference fills the e-paper screen in multiple columns, showing how the larger reading surface complements the small live display.
There are separate tutorials for the device and for gaming. The first-use tutorial is optional, so a returning or confident user can proceed directly to the interface. Kalinowski then uses OpenClaw to ask about disk space on the DGX Spark. A response appears in the recording, completing the path from handheld input to inspection of the remote machine and back to the display.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing an adventure, finding an LLM Game Boy
The demonstration then switches to gaming. Four worlds are offered—named in the narration as Alderan, Neon Nabas, The Hollows and Voidreach—followed by character selection. The LLM generates the adventure’s material, including pictures and character personalities. The deep-space scene introduces a breach and a distinctive mood: the terminal is now a window into a generated setting rather than a remote system console.
After the video, Kalinowski points out the amusing contrast: this quiet text game has no three-dimensional graphics, yet the demonstration runs against substantial NVIDIA hardware in a DGX Spark. That is the backend he chose for the demonstration, not a minimum hardware requirement for text RPGs. The computation has moved out of sight, leaving a small device organized around reading and writing.
What began as a handheld controller for OpenClaw and local models has become something closer to a Game Boy for playing with LLMs. Kalinowski closes by crediting Callstack, whose technological incubator supported the work. The change in purpose is the revealing part of the build: once model interaction has a dedicated physical form, operating an agent and inhabiting a generated world can share the same keyboard and paper-like screen.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Hardware specifications for the compact GB10 Grace Blackwell system used as the terminal's inference backend.
Further reading
Espressif's versioned guide explains I2C configuration, pull-up requirements, and error handling for the ESP32-S3.
Updates since the talk
NVIDIA's current deployment playbook includes gpt-oss-120b support and OpenAI-compatible serving on DGX Spark.
Current configuration guidance for connecting OpenClaw to local inference servers and OpenAI-compatible proxies.
Full transcript
Loading transcript…