AI Engineer World's Fair 2026

Local Agentic Theory For Mobile Games — Shafik Quoraishee & Joanne Song, The New York Times

Loading slides…

Read the talk

Local Agentic Theory For Mobile Games

On-device agents could make mobile games more responsive and accessible, but their decisions must share memory, frame time, and battery with the game itself.

From a talk by Shafik Quoraishee and Joanne Song

Before you start: Familiarity with basic game loops and model inference is helpful; reinforcement learning and accessibility concepts are introduced as needed.

What can a local agent do for a human-made game?

Can an agent help determine whether a puzzle is solvable, understand how it plays, or support a player without generating the puzzle itself? That is the experimental territory Shafik Quoraishee and Joanne Song explore. Quoraishee establishes the boundary first: the New York Times puzzles discussed here are made by people, and he describes the published games as having no AI features, explicitly excluding WordleBot from that classification. The work concerns possible supporting capabilities, not an announcement of deployed game features.

The starting point is solvability and playability. Quoraishee’s earlier Connections solver presentation explored reinforcement learning in greater depth. Here, the question broadens from how to solve a particular game to what an intelligent system could do while running on the player’s own device.

0:320:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:32 · section reference included

From Pac-Man’s ghosts to intelligence on the phone

Game AI long predates language models. Pac-Man’s ghosts provide a familiar example: a finite-state machine selects behavior through explicit states and conditions. Events in the game change what the ghosts do. This is conditional, symbolic intelligence, with the rules authored in advance. The move toward local agents builds on that history while asking whether more capable decision-making can also live on a personal device.

Slide titled “AI has always been part of games,” showing a Pac-Man maze with pellets and ghosts beside a connected state diagram.Open full source frame
Pac-Man gameplay alongside a ghost state-machine diagram.

A cloud-backed mobile application sends an inference request upstream and waits for the result. That adds service cost and a network round trip to the computation. Moving inference onto the phone removes that round trip; the device still has to perform the work, so local execution is an opportunity to reduce delay rather than a guarantee of instant decisions.

Local execution also changes where gameplay information needs to travel. If the system is configured appropriately, computation and the data it uses can remain inside the device’s security boundary. Information needed only for the immediate gameplay experience need not be sent upstream as telemetry.

The practical example is a subway tunnel. A game may still require connectivity for some features, but its local intelligence can continue operating when HTTP requests cannot get through. Keeping relevant features on the device also creates room for personalization: the game can tailor its behavior to the person using it without making every adaptation depend on a remote service.

1:502:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:50 · section reference included

Training a player versus reasoning during play

AlphaGo and AlphaZero anchor the reinforcement-learning history. Through repeated training iterations, a model becomes tuned to playing a game. EfficientZero and EfficientZero V2 extend that direction through sample efficiency: learning with less game-interaction data. They are examples of progress in limited-data reinforcement learning, not evidence that the same models meet a phone’s inference budget. This training approach continues alongside the agent systems discussed here.

The useful distinction is where adaptation happens:

ApproachWhat changesHow the game informs behavior
Reinforcement-learning trainingModel weightsRepeated training cycles
The described language-model agentContext and subsequent decisionsObservations and local tool calls

In the first case, training changes the model itself. In the second, a language model reasons over the current game state through in-context learning, chooses actions, and uses device functionality through tools. The contrast concerns the execution loop; it does not imply that a language model could never have undergone reinforcement learning during training.

4:575:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:57 · section reference included

Inside the Space Invaders agent

The Space Invaders demonstration makes the loop concrete. The local framework controls a player that must shoot approaching ships while avoiding their bullets. Although reinforcement-learning players can perform the same task, Quoraishee identifies this prototype as an agentic system.

Its behavior follows four stages:

  1. Perceive: inspect what is happening in the scene.
  2. Predict: anticipate the attacking ships’ next moves and incoming threats.
  3. Decide: determine whether a nearby ship or bullet calls for an attack or evasive action.
  4. Act: execute the chosen action, then return to perception to inspect the resulting state.

The return path matters. An action does not complete the task; it changes the situation the agent must evaluate next.

Slide titled “Inside the agent that plays the shooter,” with four stacked stages connected to an arcade-shooter game box by a return loop.Open full source frame
The shooter agent’s perceive, predict, decide, and act loop.
7:037:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:03 · section reference included

The agent shares the game’s resource budget

Running that loop on a device means fitting more than model weights into memory. The agent also needs a history of prior states and a working set of plans or intermediate artifacts. Those may require compression, and the game still needs enough headroom to render and respond to the player. Agent capacity is only useful if the game remains playable.

At 60 Hz, a frame interval is approximately 16.7 milliseconds, rounded to 16 milliseconds in the talk. That interval is a deadline shared with rendering and other game work, not an entire budget reserved for inference or a measured planning latency. Agent execution that overruns the relevant budget can produce visible jank.

Energy imposes another limit. Repeated inference and planning consume battery, and the speakers describe mobile hardware as still evolving toward these workloads. Neural processing units provide specialized inference hardware, but the agent loop must still minimize unnecessary processing. A system that makes good decisions while rapidly draining the phone is not a satisfactory mobile design.

The proposed constraint graph balances these demands rather than optimizing one in isolation:

ConstraintResource being protectedDesign pressure
SpaceWeights, history, working stateCompress and preserve render headroom
TimeResponsive framesPenalize disruptive overruns
EnergyBattery lifeLimit agent processing

The penalties need not be equally strict. Quoraishee describes allowing a softer penalty when a plan needs a little extra space while applying stronger pressure against time overruns that disrupt the user experience. The graph expresses a tradeoff policy, not simply three independent maxima.

8:098:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:09 · section reference included

From gaze signals to crossword constraints

Gaze estimation adds another input to the decision process. A convolutional neural network can estimate gaze and supply information that an agent translates into behavior. Richer sensing does not, by itself, supply better reasoning: Quoraishee separately raises base-model intelligence as a limit on complex game decisions. He mentions ARC-AGI as a capability reference without reporting a model score. Its reasoning tasks are distinct from measuring mobile responsiveness or accessibility.

The Mini Crossword prototype shifts from reacting to moving threats to satisfying a set of linked constraints. Its constraint-satisfaction graph guides word placement so that the puzzle can be filled consistently. A candidate can be wrong; when placements conflict or the constraints become difficult to satisfy, the agent backtracks and tries to recover a valid layout. Quoraishee describes the graph as optimized for efficient revision, with natural-language information from the graph helping the agent choose how to backtrack.

These experiments sit within a wider landscape. SIMA is the named example of a generalist game agent; its research direction should not be confused with an established mobile deployment of this framework. Phone-based visual estimation supplies another capability, while AI-generated worlds create additional environments for agents to operate in. Together, these directions expand both what an agent can perceive and the kinds of worlds it may need to understand.

Three cards labeled “One generalist agent, many games,” “Agents that see, on the phone,” and “The engine becomes a neural net,” each with a diagram.Open full source frame
Three research directions on the “Where we are now” slide.
10:2910:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:29 · section reference included

Accessibility needs more than an easy-mode toggle

Joanne Song turns the same local-agent architecture toward the player. A fixed state model can offer a toggle or static menu, but it cannot necessarily respond when someone’s needs change during play. The design foundation is WCAG 2.2, organized around four requirements: an experience should be perceivable, operable, understandable, and robust. These provide a framework for evaluating interaction rather than treating accessibility as an isolated game setting.

A mobile crossword makes the failures tangible. Tiny grid cells can cause mis-taps. An active-cell highlight can be too difficult for a low-vision player to see. Manual logins can create cognitive barriers, and unforgiving timers can exclude people who need more processing time. Lowering puzzle difficulty does not repair any of those interface problems.

Song uses the proposed Bronze, Silver, and Gold levels in the WCAG 3.0 Working Draft to motivate a broader range of assistance. The draft is not a finalized continuous scoring standard, and its proposed levels still include core requirements. The gameplay idea is to move beyond a single binary setting toward adjustable dimensions of support.

Two such dimensions are:

  • Input tolerance: how forgiving the interaction is of physical constraints or imprecise input.
  • Step granularity: how operations are divided within a sequence, allowing complex tasks to be broken into manageable steps or presented with more nuance.

The agent’s task is to adjust these dimensions together against the player’s live context. A useful intervention depends on their combination, not just turning every assistance setting upward.

12:3012:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:30 · section reference included

Sense friction, then repair the interaction

On-device sensing could help identify where an interaction is becoming difficult. Gaze may indicate search friction, shaky taps may indicate trouble operating controls, and handwriting recognition can offer another input method. These are cues and alternatives, not definitive diagnoses of the player. The vision pipeline shown here connects gaze dwell, handwriting ink, and touch jitter through LiteRT to an agent.

Slide titled “Computer-vision agents sense the player,” showing gaze dwell, handwriting ink, and touch jitter connected through LiteRT to an agent.Open full source frame
Gaze dwell, handwriting ink, and touch jitter feed an on-device vision pipeline.

The next example turns observation into action: a simulated broken dialog traps a keyboard or switch user in a focus loop with no usable way back. The agent monitors the focus path, flags the problem, and supplies an exit locally. The defect is the missing exit, not focus cycling alone: the accessible modal-dialog pattern intentionally contains focus while also providing dismissal and restoring focus afterward.

A small TypeScript example makes the distinction explicit. Repeated navigation is a reason to inspect the dialog; a repair is proposed only when that evidence coincides with no usable dismissal route:

typescript

type DialogObservation = {
  focusPath: string[];
  hasUsableDismissal: boolean;
  returnFocusId: string;
};

type ExitProposal = {
  kind: "add-exit";
  controlId: string;
  returnFocusId: string;
};

function proposeExit(
  dialog: DialogObservation
): ExitProposal | null {
  const path = dialog.focusPath;
  const repeatedCycle =
    path.length >= 4 &&
    path[path.length - 4] === path[path.length - 2] &&
    path[path.length - 3] === path[path.length - 1];

  if (!repeatedCycle || dialog.hasUsableDismissal) return null;

  return {
    kind: "add-exit",
    controlId: "dialog-close",
    returnFocusId: dialog.returnFocusId,
  };
}

const proposal = proposeExit({
  focusPath: ["hint", "settings", "hint", "settings"],
  hasUsableDismissal: false,
  returnFocusId: "crossword-cell",
});

Here, proposal describes an exit to add and where focus should return; it does not mutate the interface. The repeated-pair check illustrates the observation-to-proposal boundary, rather than a complete detector for every keyboard trap.

The next intervention concerns target size. For someone with limited dexterity, a tiny control can turn an intended selection into a miss. Song describes an agent acting as a live layout auditor: it measures the interface, identifies inadequate controls, and dynamically resizes them. The layout changes to accommodate the person during interaction.

This produces a continuously adjusted relationship between assistance and challenge. Song frames them as ends of a dial that responds to the moment, with the broader goal of a device that senses frustration and responds usefully. In these experiments, responsive accessibility means changing the interaction itself, not merely observing that the player is struggling.

14:5815:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:58 · section reference included

What useful local agents still need

The remaining work starts with speed. Planning must respect the frame deadline already discussed so that assistance does not introduce stutter. It also needs prediction: before applying a layout change, the system should be able to anticipate what that change will do. Reacting to difficulty is only part of the problem; avoiding a disruptive intervention is another.

Long-term memory would let an agent learn one person’s habits and needs over time instead of treating every interaction as new. A shared game-state language would address a different limitation: portability. If games exposed a common representation, an agent could operate across releases without being rebuilt separately for each one.

Better chips must be paired with honest benchmarks that establish whether the agent actually improves the experience. The destination is not necessarily one centralized intelligence serving every player in the same way. It could be many small local agents, each running on a personal device and shaped by the individual it serves.

16:4216:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:42 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…