AI Engineer World's Fair 2026

Perception Agents — Antje Barth, Amazon AGI Lab

Antje Barth· Member of Technical Staff, Amazon AGI Lab21:45

Loading slides…

Read the talk

Perception Agents

An agent that can click through an application still needs to understand what changed. Antje Barth explores shared context, visual annotation and verification as foundations for reliable work.

From a talk by Antje Barth

The work between applications

A new colleague starts on Monday. Someone needs to create their accounts, add them to a Slack channel, book introductions and order a laptop. An agent may be able to perform each operation, yet still struggle to complete the onboarding. The job crosses perhaps five systems, and nobody necessarily owns the whole process. Competence at individual steps does not guarantee completion of the work between them.

Slide with that statement beside an illustration of overlapping application windows, with the speaker inset at left.Open full source frame
The agent can use every tool. It still can’t do the work.

That gap is the starting point for Antje Barth’s discussion of perception agents. Operating every tool in a workflow is useful, but the user delegates an outcome: a colleague ready to work on Monday. The seams between applications are where that outcome can fall apart.

1:201:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:20 · section reference included

From tool capability to trustworthy delegation

Computer use builds on concrete operations: clicking, scrolling, typing, calling an API and filling out a form. Those operations can be chained into workflows. This progress takes agents from finding controls on unfamiliar screens toward operating browsers and desktop applications—but it leaves a harder question: when can you hand over a job and walk away?

Adding tools, improving models and connecting agents expands capability. Delegation depends on reliability, because trust comes from expecting the whole job to succeed. Barth asks the audience to imagine end-to-end success rates of 60–80%; these are hypothetical guesses, not measured results. Her deliberately severe follow-up is an agent that deletes a database one time in four. A reassuring-looking aggregate success rate means little when a failure has that consequence. She calls for reliability “in the nines,” without specifying a measured target.

2:262:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:26 · section reference included

Why coding moved faster

Coding offers a useful progression: autocomplete became function generation, then agents began writing code and opening pull requests. Barth describes a corresponding change in review habits. Early outputs invited line-by-line inspection; the volume of generated code now makes that difficult, while confidence in coding agents has grown.

Her explanation is verifiability. Code can be run, tested and checked, giving the agent and its user feedback about whether a change worked. Those feedback mechanisms help turn a capability into something people can trust.

Slide stating that code can be verified, beside a code-window illustration with a check mark.Open full source frame
Code crossed it first because you can verify it.

Broader knowledge work often lacks such a direct check. Did the report land with its audience? Is the design on brand? Did the agent understand what the user meant? A unit test cannot settle those questions on its own. Across applications, success also depends on intent and judgment, making easy verification an open problem precisely where much everyday work happens.

4:364:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:36 · section reference included

Shared context and a continuous feedback loop

People already navigate this ambiguity together. You bring a colleague into a Zoom meeting, discuss the problem and point at the systems involved. In Barth’s example, a brief conversation resolves something that had no directly verifiable answer. Looking at the same screen lets both people refer to the same objects and notice the same state. The missing ingredient for an agent may therefore be shared context, rather than necessarily a bigger model: less of the situation has to be translated into a prompt.

That context must persist after an action. An agent that sees a screen, clicks and moves on can miss whether the operation succeeded or whether recovery is needed. Robotics supplies the relevant pattern: perceive → plan → act, then perceive again. On a computer, perception means attending to the rendered layout, the current state and what just changed. The screen is the grounding for the task, even when implementation tools also inspect the page’s underlying structure.

The interaction rhythm matters too. A chatbot encourages a sequence of prompt, wait, inspect and correct. If the result misses the user’s intent, another prompt sends the agent away again. Barth’s proposed alternative is an agent that can react and suggest changes while the user is still working. Simultaneous assistance without waiting is the desired interaction, not a latency result established by these demonstrations.

7:297:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:29 · section reference included

What perception adds to computer use

Barth calls this class of system a perception agent. Its first benefit is closing the action loop: after clicking, typing or scrolling, it reads the rendered result and checks whether its output actually appeared. Sending an action is no longer treated as sufficient evidence that the action worked.

The second benefit is access through an application’s visible interface without requiring that application to expose an API. Barth motivates this by saying that most everyday software does not expose APIs, although she supplies no measured population for that claim. The practical distinction is between needing a target application’s API and needing infrastructure to operate the agent: the released verifier’s current setup documentation requires Nova Act authentication and an MCP server.

The third benefit runs in the other direction, from the person to the agent. Instead of describing a website heading through a long paragraph, the user can point at it and request a change. A marked heading or section supplies a precise target alongside the instruction, reducing the information lost when a visual intention is converted into text.

11:3611:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:36 · section reference included

Mark the target, then hand off the context

The first two components of Amazon AGI Lab’s perception agent harness, released in May 2026, address opposite ends of the loop: annotation communicates what the user wants, and verification checks the agent’s work.

The annotation demonstration uses the Nova Act Annotator, a Chrome extension. The workflow is concrete:

  1. Draw around a heading or mark a section on the page.
  2. Attach a request, such as changing the heading to red.
  3. Alternatively, hover over an element, click to select it and request a change such as doubling its font size.
  4. Hand the resulting summary to the implementation agent.

The summary captures the feedback, its location and style information. Rendered-screen grounding does not mean a pixels-only implementation: the launch description documents DOM selectors, bounding boxes and computed styles as part of the captured context.

The completed handoff appears beside the Some Podcast webpage in the demonstration. The useful output is not merely a sentence saying to change a heading; it ties the requested change to what the user selected. Barth presents that shared capture as a way to remove the clarification back-and-forth otherwise needed to establish which element the user meant.

Annotation slide showing the Some Podcast webpage and a populated sidebar headed “Hand this off to your agent.”Open full source frame
The annotation demo displays its agent handoff summary.
13:2313:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:23 · section reference included

Check both the design and the experience

The second demonstration uses verification from Nova Act Agent Skills. Its starting point is a design.md file describing the application’s design rules. Colors, components and layout expectations become criteria against which the agent checks its work; if those rules have not already been written down, the workflow can turn the design description into rules.

A small design.md example makes the role of that input concrete:

markdown

# Design rules

## Colors
- Use a white page background.
- Use black text for the main heading.

## Layout
- Keep the main heading above the task list.

## Components
- Give each task a visible delete control.

These illustrative rules give verification explicit expectations to inspect. The visual check asks whether the result follows the intended brand and layout. In the released tool, that includes deterministic checks of live DOM computed styles, rather than relying solely on a model’s judgment of a screenshot.

The behavioral check then walks through the application like a user. Depending on the available tasks, it might add a task and delete one.

CheckWhat it examines
DesignColors, components and layout against the rules
User flowWhether actions work through the interface

These checks answer different questions: a page can look right while its controls fail, or function correctly while violating the design. The workflow produces a reviewable report identifying passing checks and failures, taking over the manual walkthrough that might otherwise happen at the end of a long day.

15:0715:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:07 · section reference included

When the shared context is a conversation

Perception need not begin with a screen. For another experiment, Barth and her colleague Giovanni held a design meeting in a booth at the conference. Both wore Bee devices, supplied through Bee’s sponsorship, to transcribe the conversation. The experiment extends the same workflow to intentions expressed aloud during a meeting.

The demonstration imports the meeting transcript and displays a detailed summary with the captured insights. At this point, those insights are proposed changes. Clicking Apply sends them to the agent. The website then receives the requested yellow background, red heading and changed emoji, and verification starts immediately.

The resulting report accepts the color scheme because it fits the demo’s approved rules. That detail matters: verification checks the specification it has been given. A rule set that disallows yellow backgrounds would instead flag the change. The human then has a judgment call—revise the specification because yellow is now desired, or ask the agent to fix the violation. The loop brings the disagreement into view without deciding the design preference on the user’s behalf.

16:4817:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:48 · section reference included

Building the rest of the loop in the open

Annotation and verification are the first pieces of the harness, not a completed system for general knowledge work. Barth invites developers to try the open-source repositories, build on them, break things and report what is missing. Further development depends on discovering where these patterns help real collaboration and where they still fail. The goal is AI that makes people more effective together.

For the human-agent interaction questions behind this work, Barth recommends the podcast hosted by her colleague Danielle Perszyk, a cognitive scientist leading the lab’s HCI team. Perszyk’s Making a Mind explores these questions with industry experts. Barth also points conference attendees to expo talks, the lab’s booth and Gaurav Mishra’s From RL to IRL, scheduled for 1:30 in the computer use track. The closing invitation extends beyond the venue: continue the conversation through the lab’s GitHub projects and website as the remaining pieces take shape.

“Come, find us” slide listing expo access, github.com/amazon-agi-labs, and labs.amazon.science beside a booth photograph.Open full source frame
Find Amazon AGI Lab at the expo, on GitHub, or at labs.amazon.science.
19:3019:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:30 · section reference included

Resources

From the talk

  • Chrome extension for marking page elements, drawing feedback and exporting structured instructions for coding agents.

  • Browser automation and UI verification skills, including design checks, user-flow execution and reports.

  • Danielle Perszyk hosts conversations about perception, reinforcement learning and useful general intelligence.

  • BeeDocumentationTry it ↗17:35

    Wearable personal AI that turns conversations into transcripts, summaries and insights.

Transcript download
Full transcript

Loading transcript…