AI Engineer World's Fair 2026

OpenClaw in Your Hand: Building a Physical AI Terminal for Local LLM Agents

Loading slides…

Read the talk

OpenClaw in Your Hand: Building a Physical AI Terminal for Local LLM Agents

A keyboard, OLED and e-paper display make a quiet interface to OpenClaw, while a separate DGX Spark handles inference, agent actions and generated adventures.

From a talk by Lech Kalinowski

Before you start: Basic familiarity with microcontrollers, LLM inference and agents that can act on a computer is helpful.

A terminal that answers back

During a basement cleanup, a dusty device appears in a corner: an old-looking keyboard terminal with unusual displays. Its shell is still running. Type “Who am I?” and it answers in natural language. Lech Kalinowski opens with this imagined discovery, then reveals that it is a possible marketing story. The real ambition is straightforward: build a physical, AI-native device that feels as though it came from the future.

VAULT title beside a photograph of a gray terminal with a keyboard, display and knob; the presenter holds a device in the inset.Open full source frame
Vault’s physical terminal, pictured on the slide and held up by the presenter.

Kalinowski, who holds a PhD in physics, spent several months building the device, its backend and the surrounding system. The starting requirement was narrower than the eventual product: a remote controller for an OpenClaw instance on his DGX Spark. That controller became Vault.

0:020:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:02 · section reference included

Separate typing from reading

For this interface, the useful starting point is text: a place to type requests and read model responses. Electronic paper suits sustained reading, but its refresh behavior makes it less suitable for updating the screen with every keystroke. Kalinowski pairs it with a small monochrome OLED, dividing the work between two surfaces instead of asking one display to serve both purposes. The intended benefit is a responsive terminal with economical display power use.

The remote controller also becomes an experiment in an AI-native operating interface built around a microcontroller. Its four modes are shell, assist, control and RPG, as enumerated in the recording’s description. The internal shell handles the terminal’s own settings, basic configuration and Wi-Fi connection; the other modes extend the interface toward assistance, remote actions and play.

In the prototype, the OLED is the live typing surface. Entering text updates that display; pressing Enter triggers a render on the bistable e-paper display. This gives the interaction a clear boundary between composing an input and committing a page for reading.

SurfaceJobUpdate behavior
OLEDLive text entryUpdates while typing
E-paperCommitted pageRenders after Enter

The same interface carries requests and responses for LLM interaction and operations on the DGX Spark through OpenClaw.

Making that division practical requires a deliberately small rendering contract. Pages live as one-bit images in fixed, pre-allocated buffers. The microcontroller displays those images without a Markdown engine or calls to malloc. The page representation is therefore bounded in memory, and the terminal does not need to interpret a rich document format to put a response on screen.

2:312:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:31 · section reference included

Power delivery is part of the interface

The interior is more complicated than a keyboard attached to two screens. Much of that complexity comes from power management: Kalinowski reports destroying two displays during prototyping and needing much more stable current and voltage. A reliable interface depends on keeping its fragile display electronics alive.

The hardware centers on a dual-core ESP32 microcontroller, with the OLED, keyboard, rotary encoder and power-supply components around it. The opened enclosure shows how much wiring and supporting circuitry sits behind the restrained exterior.

Slide titled “Runs entirely off a single 18650 cell,” with a component table and a photograph of the opened enclosure showing circuit boards and wiring.Open full source frame
Vault’s component list and internal wiring, powered by a single 18650 cell.
7:177:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:17 · section reference included

Keep inference off the handheld

Local inference happens on the backend, not inside the terminal. The compute and power requirements of the LLM would overwhelm the handheld design. Vault firmware runs on the device, while the backend handles both OpenClaw’s agent work and model inference. The terminal is a physical entry point into that system.

For the demonstration, Kalinowski describes an open-source GPT model in the 120-billion-parameter class; the recording’s description identifies it as gpt-oss:120b. He serves it using TensorRT, exposing an OpenAI-style interface through an LLM proxy. That adapter matters because other open-source models and serving setups had not matched the API behavior he needed. Current deployment guides describe available serving paths, but do not establish the exact proxy or server configuration used here.

A concrete example is asking the agent to write a Java example and save it on the local machine. Kalinowski reports that OpenClaw, with the model’s help, completed the request. The important transition is from generating an answer to creating a file on the backend host. A small Java artifact for that kind of request could be Hello.java:

java

public class Hello {
    public static void main(String[] args) {
        String name = args.length > 0 ? args[0] : "Vault";
        System.out.println("Hello, " + name + "!");
    }
}

The keyboard supplies the request and the displays present the interaction; the agent performs the filesystem action on the machine it controls.

8:168:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:16 · section reference included

What failed on the workbench

The prototype’s field notes are mostly electrical rather than model-related:

  • Software I2C: The initial wiring did not work correctly. Kalinowski reports eventually controlling the bus without adding physical pull-ups. Treat that as a property of this build, not a general wiring recipe: Espressif’s I2C guidance generally recommends external pull-ups because the internal ones are weak.
  • GPIO 13: A silent failure led him to move the connection to another port.
  • Power regulation: A regulator damaged the OLED and other fragile display parts. Reworking the power-supply unit was necessary, and replacement parts took a couple of weeks to arrive.
  • Encoder noise: A cheap rotary encoder produced noisy rotational input. He added pull-ups and capacitors to address it.

These failures explain why the supporting circuitry occupies so much of the hardware design: a clean text interface still depends on clean electrical signals.

10:2810:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:28 · section reference included

The remote controller becomes an RPG console

The unexpected fit is a text-based role-playing game. Kalinowski had never played a tabletop RPG with friends gathered around a book and a game master. Building Vault gave him another route into that experience: a dedicated console where an LLM supplies the unfolding fiction.

NPCs, memory and the mood of the world give the game continuity beyond a single generated response. He creates four worlds, drawing on cyberpunk, Witcher-like fantasy with dragons, and a favorite deep-space setting. The text-focused hardware suits an experience in which reading, choosing and imagining are the main activities.

Generation extends to characters, worlds, maps and skills. Pictures then pass through the same rendering constraint introduced earlier: they are converted into one-bit matrices or image maps for the display. The generated fantasy imagery and its physical e-paper output show the two sides of the pipeline—rich material produced upstream, reduced to a representation the terminal can hold and render.

Slide titled “Narrated on the fly · rendered to paper,” showing five fantasy images followed by a photograph of an e-paper display.Open full source frame
Generated role-playing imagery alongside its physical e-paper output.
12:0812:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:08 · section reference included

Preserve a usable path—and a quiet environment

The paired components also provide alternative interaction paths. Kalinowski describes the device as resilient because an individual failure need not make it completely unusable.

FailureRemaining path
OLED stops workingRead through e-paper
Keyboard stops workingUse the encoder
Wi-Fi connection failsUse the local shell

These paths preserve parts of the experience; the local shell, for example, remains useful without implying that disconnected backend inference continues.

That restrained interaction model is also the product idea. In his intellectual-property research, Kalinowski found much of the attention going toward audio interfaces and video capture. Vault instead makes room for quiet reading, writing, agent work and RPG play without bright moving graphics, advertisements or web browsing competing for attention. He reports pursuing a provisional patent, with the possibility of a commercial device in the future.

14:4215:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:42 · section reference included

What the build adds up to

Kalinowski reports 130 commits over roughly three months. His project inventory includes two displays, four modes, four gaming worlds and sixteen classes associated with the RPG system. He describes the firmware as small and fast, without giving a measured size or latency. The handheld uses a single battery cell; the spoken recap calls it lithium-polymer, while the earlier hardware slide labels it an 18650.

The resulting design principles follow directly from those choices: match each display to its job, keep heavyweight LLM inference off the tiny microcontroller, and preserve narrative context in the game rather than tracking only numbers. The display architecture controls the physical experience; the backend carries the computation and the remembered story.

16:2516:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:25 · section reference included

From boot to a remote disk-space query

The recorded demonstration starts with the ordinary operations that make a dedicated device approachable:

  1. Boot the terminal and connect to the local Wi-Fi network.
  2. Move through the welcome screen to the main interface.
  3. Type help on the keyboard to display the available commands.

The command reference fills the e-paper screen in multiple columns, showing how the larger reading surface complements the small live display.

A close-up of Vault’s e-paper screen filled with a multi-column command listing beneath the slide title “Vault, running.”Open full source frame
The recorded demo shows a multi-column command reference on Vault’s display.

There are separate tutorials for the device and for gaming. The first-use tutorial is optional, so a returning or confident user can proceed directly to the interface. Kalinowski then uses OpenClaw to ask about disk space on the DGX Spark. A response appears in the recording, completing the path from handheld input to inspection of the remote machine and back to the display.

18:0318:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:03 · section reference included

Choosing an adventure, finding an LLM Game Boy

The demonstration then switches to gaming. Four worlds are offered—named in the narration as Alderan, Neon Nabas, The Hollows and Voidreach—followed by character selection. The LLM generates the adventure’s material, including pictures and character personalities. The deep-space scene introduces a breach and a distinctive mood: the terminal is now a window into a generated setting rather than a remote system console.

After the video, Kalinowski points out the amusing contrast: this quiet text game has no three-dimensional graphics, yet the demonstration runs against substantial NVIDIA hardware in a DGX Spark. That is the backend he chose for the demonstration, not a minimum hardware requirement for text RPGs. The computation has moved out of sight, leaving a small device organized around reading and writing.

What began as a handheld controller for OpenClaw and local models has become something closer to a Game Boy for playing with LLMs. Kalinowski closes by crediting Callstack, whose technological incubator supported the work. The change in purpose is the revealing part of the build: once model interaction has a dedicated physical form, operating an agent and inhabiting a generated world can share the same keyboard and paper-like screen.

20:4020:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:40 · section reference included

Resources

From the talk

  • Hardware specifications for the compact GB10 Grace Blackwell system used as the terminal's inference backend.

Updates since the talk

Transcript download
Full transcript

Loading transcript…