AI Engineer World's Fair 2026
I Run a Fleet of AI Agents Across Three Machines. Here's What Broke.
Loading slides…
Read the talk
I Run a Fleet of AI Agents Across Three Machines. Here's What Broke.
Kyle Jaejun Lee’s agent fleet turns attention, state, approvals and machine placement into separate engineering problems—and exposes what remains unsolved between them.
From a talk by Kyle Jaejun Lee
Before you start: Familiarity with terminal sessions, Git and AI coding agents will help; no Kubernetes experience is required.
The first bottleneck was attention
A MacBook sleeps. Long-running coding jobs need somewhere that does not. Kyle Jaejun Lee’s everyday fleet spans that laptop and two headless, always-on Linux boxes, connected through one control plane. The MacBook handles heavy coding and personal projects; Linux A takes long-running coding tasks, while Linux B runs short-lived personal projects. He opens with the fleet running live: daily infrastructure across two operating systems.
Before distributing the work, though, Lee ran into a limit on one machine: his own attention. A few tmux panes became four, five, then six simultaneous agent contexts. Every additional agent gave him three jobs: scheduler, deciding who should do what; memory, remembering what each agent was doing; and reviewer, checking the results. Six active contexts were more than he could keep in his head.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate context, then persist it
How can a handful of executives oversee thousands of people? They do not retain everyone’s working context. Lee applied that separation to his agents, building a hierarchy with four actual entity types: CEO, VP, manager and worker. Each entity is an agent with its own scoped context and approval boundary. Context flows downward in the slices needed for each task; results flow upward, and Lee reviews what reaches the top.
Lee describes his supervisory context load falling from six contexts to one. That is his account of what he has to hold in mind, rather than a measure of fleet throughput. The hierarchy reduces the context reaching the human, but each agent still has a finite context window. The next question is where its state should live when that window fills.
His answer is a workspace on disk for every entity. The layout separates common obligations, machine-specific information and the work an individual agent needs to continue:
| Location or content | Purpose |
|---|---|
shared | Context every entity must honor |
machines | Machine-bound state |
| Entity workspace | The entity’s mission and current status |
| Handoff folder | Work product passed to the next agent or session |
The durable state lives in files. A model’s current conversation is no longer the only place that knows what the agent is doing.
That changes how Lee handles a full context window. He finds compaction slow and dislikes leaving retention decisions to a summary, so inside Claude he clears the context completely. The fresh context then reads the handoff and history files the agent wrote for itself and resumes from those records. Current Claude Code session guidance does allow focus instructions to steer compaction, although the summary remains lossy; that does not establish what Lee’s installed version supported. His preference is to make the continuity material explicit before resetting.
A context wipe—or even a machine crash—can therefore leave the written work available for recovery. This preserves recorded state and work product; it does not keep a running process alive through a crash. With continuity moved onto disk, the next problem is whether agents remain aligned with the intended plan.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make approval a blocking operation
Plans drift as they move through the hierarchy. Initially, Lee detected that drift by entering individual panes and inspecting the work. He replaced those visits with a review gateway: any layer that wants to act submits its plan and blocks. The plan is pending until he approves it; approval fires a hook that starts or resumes the work automatically. One web inbox becomes the control point.
The demonstrated cycle has three steps:
- An agent submits a plan to the gateway.
- Lee reviews and approves it.
- The approval hook resumes execution.
An infrastructure team inside the fleet built the gateway itself. Agents were now building the tools used to coordinate agents, and the arrangement worked well for Lee on one machine. Scaling the workload exposed a different set of limits.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Five failures under load
1. Orchestrators did the work themselves. An agent assigned to coordinate a task would start implementing it instead of dispatching it to a worker. Lee responded with a CLI harness and skills that invoke those CLIs. In his implementation, dispatch became the only available path for the orchestrator. The intended division of responsibility was enforced through the available operations.
2. Worker panes became unreadable. As managers received more tasks, they created more worker panes inside a single tmux window. Eventually, each pane was too small to inspect. Lee reports that even tmux capture-pane, his programmatic way of reading a pane, stopped returning meaningful content in this arrangement. The fleet had exhausted its display space as well as his ability to observe it; he does not describe a specific pane-layout fix.
3. Accumulated sessions exhausted memory. Lee’s Activity Monitor view shows overlapping process lists with repeated MCP entries. He describes Claude Code and MCP processes stacking up until swap was almost full and almost no free memory remained. The failure here is session accumulation consuming the host’s resources.
4. Git credentials crossed workspace boundaries. The desired mapping was simple: credential A belonged to workspace A, and credential B to workspace B. Instead, credentials collided and became associated with the wrong workspaces. Lee’s fix was a clean, fully separated environment for each workspace.
5. The laptop could disappear. Lee reports losing in-progress jobs when his MacBook lost power or dropped off the network. Network disconnection is a deployment-specific part of that account: tmux itself is designed to preserve sessions across an SSH disconnect, so losing access is not inherently the same as losing running processes. In a more decisive failure, Lee loaded the laptop with so many workers that he later opened its lid to find it had restarted. The in-flight execution was gone.
His first response was a fleet recovery command:
bash
overlord boot
The command brings the fleet back up using the state already stored in files. Recovery was possible because the work was not held solely in model contexts. But restartability could not make a laptop continuously available. Lee still needed somewhere else to run the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move execution without losing context
Lee added the two always-on Linux machines, offloading long-running coding tasks to Linux A and short-lived personal projects to Linux B. That added resources and took long-running work off the laptop while retaining one control plane. It also created a new requirement: an agent on one machine needed the context produced on another.
File-backed state made the transfer possible with familiar tools:
- Commit the context files on the source machine.
- Push the commit through Git.
- Connect over SSH and use
tmux send-keysto tell the destination to pull. - Have the destination agent read the transferred files and continue the work.
Git carries the durable context; SSH and tmux provide the signal that prompts the other machine to retrieve it.
The first version allowed two machines to change the same directory. Conflicts followed, and supposedly shared context diverged. Lee then separated machine-specific state into per-machine directories and required pull requests for shared-state changes. The separation gives local state an explicit home, while the pull-request rule puts a review boundary around changes that affect everyone.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring approvals and routing back together
Distributing execution also distributed the gateway: Lee now had an inbox on every machine. That recreated the attention problem the gateway was supposed to solve. He collapsed the inboxes into one main gateway, with every machine forwarding review requests over SSH. The gateway lives on an always-on Linux box because the sole point of control cannot depend on the MacBook staying awake.
Machine selection remained another burden. While developing across the fleet, Lee went looking for a feature and could not remember which machine he had built it on. He made Discord the single routing interface, with one bot for the MacBook, one for Linux A and one for Linux B. His phone became a remote control for the fleet. That unified the place from which he issued requests, though it did not yet eliminate the need to know where work belonged.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let agents declare requirements
The distributed fleet still leaves Lee with four open engineering problems:
- Consistency: keeping state coherent across machines.
- Tool access: abstracting tools that remain local to the Mac, including MCP servers and the browser.
- Credential handoff: securely moving access between instances.
- Resource management: deciding what should run where.
He still makes placement decisions himself. The contract he wants is for an agent to declare what it needs and let the system choose where it runs.
That suggests a boundary between agent orchestration and execution infrastructure. The logical hierarchy, orchestrator and review gate sit above compute, secrets and tools; a scheduler places the workloads underneath them. Lee identifies Kubernetes as the foundation for that direction. Its scheduler places Pods using resource requirements and constraints, which fits the requirement-based placement he wants.
This is a proposed architecture, not a completed migration. Lee intends to put existing infrastructure underneath his orchestration manager and keep the custom work focused on task orchestration, review flow and context management. Kubernetes supplies useful primitives, but tool portability and secure credential handling still need engineering. In particular, Kubernetes Secrets are stored unencrypted in etcd by default; encryption at rest and restricted access require configuration.
Lee’s closing assessment is that single-machine operation works for him, while the cross-machine system remains rough and under construction. The next step is to reuse established infrastructure below the agent layer while continuing to build the coordination above it. He ends by inviting other people running agent fleets to compare notes on those unresolved operational problems.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The project manual for terminal sessions, windows, panes, detaching and reattaching.
How Kubernetes assigns Pods to nodes using resource requirements, constraints, filtering and scoring.
Further reading
A bilingual account of hierarchical Claude Code orchestration using tmux, file handoffs and SSH.
Anthropic explains compaction, fresh sessions, written handoffs and separate subagent contexts.
Credential delivery to workloads and the encryption and access controls needed to protect Secrets.
Full transcript
Loading transcript…