AI Engineer World's Fair 2026
Perception Agents — Antje Barth, Amazon AGI Lab
Loading slides…
Read the talk
Perception Agents
An agent that can click through an application still needs to understand what changed. Antje Barth explores shared context, visual annotation and verification as foundations for reliable work.
From a talk by Antje Barth
The work between applications
A new colleague starts on Monday. Someone needs to create their accounts, add them to a Slack channel, book introductions and order a laptop. An agent may be able to perform each operation, yet still struggle to complete the onboarding. The job crosses perhaps five systems, and nobody necessarily owns the whole process. Competence at individual steps does not guarantee completion of the work between them.
That gap is the starting point for Antje Barth’s discussion of perception agents. Operating every tool in a workflow is useful, but the user delegates an outcome: a colleague ready to work on Monday. The seams between applications are where that outcome can fall apart.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From tool capability to trustworthy delegation
Computer use builds on concrete operations: clicking, scrolling, typing, calling an API and filling out a form. Those operations can be chained into workflows. This progress takes agents from finding controls on unfamiliar screens toward operating browsers and desktop applications—but it leaves a harder question: when can you hand over a job and walk away?
Adding tools, improving models and connecting agents expands capability. Delegation depends on reliability, because trust comes from expecting the whole job to succeed. Barth asks the audience to imagine end-to-end success rates of 60–80%; these are hypothetical guesses, not measured results. Her deliberately severe follow-up is an agent that deletes a database one time in four. A reassuring-looking aggregate success rate means little when a failure has that consequence. She calls for reliability “in the nines,” without specifying a measured target.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Why coding moved faster
Coding offers a useful progression: autocomplete became function generation, then agents began writing code and opening pull requests. Barth describes a corresponding change in review habits. Early outputs invited line-by-line inspection; the volume of generated code now makes that difficult, while confidence in coding agents has grown.
Her explanation is verifiability. Code can be run, tested and checked, giving the agent and its user feedback about whether a change worked. Those feedback mechanisms help turn a capability into something people can trust.
Broader knowledge work often lacks such a direct check. Did the report land with its audience? Is the design on brand? Did the agent understand what the user meant? A unit test cannot settle those questions on its own. Across applications, success also depends on intent and judgment, making easy verification an open problem precisely where much everyday work happens.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What perception adds to computer use
Barth calls this class of system a perception agent. Its first benefit is closing the action loop: after clicking, typing or scrolling, it reads the rendered result and checks whether its output actually appeared. Sending an action is no longer treated as sufficient evidence that the action worked.
The second benefit is access through an application’s visible interface without requiring that application to expose an API. Barth motivates this by saying that most everyday software does not expose APIs, although she supplies no measured population for that claim. The practical distinction is between needing a target application’s API and needing infrastructure to operate the agent: the released verifier’s current setup documentation requires Nova Act authentication and an MCP server.
The third benefit runs in the other direction, from the person to the agent. Instead of describing a website heading through a long paragraph, the user can point at it and request a change. A marked heading or section supplies a precise target alongside the instruction, reducing the information lost when a visual intention is converted into text.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Mark the target, then hand off the context
The first two components of Amazon AGI Lab’s perception agent harness, released in May 2026, address opposite ends of the loop: annotation communicates what the user wants, and verification checks the agent’s work.
The annotation demonstration uses the Nova Act Annotator, a Chrome extension. The workflow is concrete:
- Draw around a heading or mark a section on the page.
- Attach a request, such as changing the heading to red.
- Alternatively, hover over an element, click to select it and request a change such as doubling its font size.
- Hand the resulting summary to the implementation agent.
The summary captures the feedback, its location and style information. Rendered-screen grounding does not mean a pixels-only implementation: the launch description documents DOM selectors, bounding boxes and computed styles as part of the captured context.
The completed handoff appears beside the Some Podcast webpage in the demonstration. The useful output is not merely a sentence saying to change a heading; it ties the requested change to what the user selected. Barth presents that shared capture as a way to remove the clarification back-and-forth otherwise needed to establish which element the user meant.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Check both the design and the experience
The second demonstration uses verification from Nova Act Agent Skills. Its starting point is a design.md file describing the application’s design rules. Colors, components and layout expectations become criteria against which the agent checks its work; if those rules have not already been written down, the workflow can turn the design description into rules.
A small design.md example makes the role of that input concrete:
markdown
# Design rules
## Colors
- Use a white page background.
- Use black text for the main heading.
## Layout
- Keep the main heading above the task list.
## Components
- Give each task a visible delete control.
These illustrative rules give verification explicit expectations to inspect. The visual check asks whether the result follows the intended brand and layout. In the released tool, that includes deterministic checks of live DOM computed styles, rather than relying solely on a model’s judgment of a screenshot.
The behavioral check then walks through the application like a user. Depending on the available tasks, it might add a task and delete one.
| Check | What it examines |
|---|---|
| Design | Colors, components and layout against the rules |
| User flow | Whether actions work through the interface |
These checks answer different questions: a page can look right while its controls fail, or function correctly while violating the design. The workflow produces a reviewable report identifying passing checks and failures, taking over the manual walkthrough that might otherwise happen at the end of a long day.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When the shared context is a conversation
Perception need not begin with a screen. For another experiment, Barth and her colleague Giovanni held a design meeting in a booth at the conference. Both wore Bee devices, supplied through Bee’s sponsorship, to transcribe the conversation. The experiment extends the same workflow to intentions expressed aloud during a meeting.
The demonstration imports the meeting transcript and displays a detailed summary with the captured insights. At this point, those insights are proposed changes. Clicking Apply sends them to the agent. The website then receives the requested yellow background, red heading and changed emoji, and verification starts immediately.
The resulting report accepts the color scheme because it fits the demo’s approved rules. That detail matters: verification checks the specification it has been given. A rule set that disallows yellow backgrounds would instead flag the change. The human then has a judgment call—revise the specification because yellow is now desired, or ask the agent to fix the violation. The loop brings the disagreement into view without deciding the design preference on the user’s behalf.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Building the rest of the loop in the open
Annotation and verification are the first pieces of the harness, not a completed system for general knowledge work. Barth invites developers to try the open-source repositories, build on them, break things and report what is missing. Further development depends on discovering where these patterns help real collaboration and where they still fail. The goal is AI that makes people more effective together.
For the human-agent interaction questions behind this work, Barth recommends the podcast hosted by her colleague Danielle Perszyk, a cognitive scientist leading the lab’s HCI team. Perszyk’s Making a Mind explores these questions with industry experts. Barth also points conference attendees to expo talks, the lab’s booth and Gaurav Mishra’s From RL to IRL, scheduled for 1:30 in the computer use track. The closing invitation extends beyond the venue: continue the conversation through the lab’s GitHub projects and website as the remaining pieces take shape.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Chrome extension for marking page elements, drawing feedback and exporting structured instructions for coding agents.
Browser automation and UI verification skills, including design checks, user-flow execution and reports.
Danielle Perszyk hosts conversations about perception, reinforcement learning and useful general intelligence.
Wearable personal AI that turns conversations into transcripts, summaries and insights.
Further reading
Amazon AGI's launch explanation of annotation, verification and shared visual context.
Setup instructions and input/output formats for design.md specifications, user-flow scenarios and verification reports.
Developer commands for accessing conversation transcripts, summaries and personal context.
Full transcript
Loading transcript…