# Perception Agents — Antje Barth, Amazon AGI Lab

Antje Barth, Amazon AGI Lab

AI Engineer World's Fair 2026

[Talk page](https://ai.engineer/talks/2JX6JYyQG4Y-perception-agents) · [Video](https://www.youtube.com/watch?v=2JX6JYyQG4Y)

## Perception Agents

An agent that can click through an application still needs to understand what changed. Antje Barth explores shared context, visual annotation and verification as foundations for reliable work.

### The work between applications

[1:20](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=80s)

A new colleague starts on Monday. Someone needs to create their accounts, add them to a Slack channel, book introductions and order a laptop. An agent may be able to perform each operation, yet still struggle to complete the onboarding. The job crosses perhaps five systems, and nobody necessarily owns the whole process. **Competence at individual steps does not guarantee completion of the work between them.**

![Slide with that statement beside an illustration of overlapping application windows, with the speaker inset at left.](https://ai.engineer/api/talk-essay-media/65c5ddd6b6a92a813bee560ec8b7a9f66cfabb26863ff62ff63bea2fafe39478)

The agent can use every tool. It still can’t do the work.

That gap is the starting point for Antje Barth’s discussion of perception agents. Operating every tool in a workflow is useful, but the user delegates an outcome: a colleague ready to work on Monday. The seams between applications are where that outcome can fall apart.

### From tool capability to trustworthy delegation

[2:26](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=146s)

Computer use builds on concrete operations: clicking, scrolling, typing, calling an API and filling out a form. Those operations can be chained into workflows. This progress takes agents from finding controls on unfamiliar screens toward operating browsers and desktop applications—but it leaves a harder question: when can you hand over a job and walk away?

Adding tools, improving models and connecting agents expands **capability**. Delegation depends on **reliability**, because trust comes from expecting the whole job to succeed. Barth asks the audience to imagine end-to-end success rates of 60–80%; these are hypothetical guesses, not measured results. Her deliberately severe follow-up is an agent that deletes a database one time in four. A reassuring-looking aggregate success rate means little when a failure has that consequence. She calls for reliability “in the nines,” without specifying a measured target.

### Why coding moved faster

[4:36](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=276s)

Coding offers a useful progression: autocomplete became function generation, then agents began writing code and opening pull requests. Barth describes a corresponding change in review habits. Early outputs invited line-by-line inspection; the volume of generated code now makes that difficult, while confidence in coding agents has grown.

Her explanation is **verifiability**. Code can be run, tested and checked, giving the agent and its user feedback about whether a change worked. Those feedback mechanisms help turn a capability into something people can trust.

![Slide stating that code can be verified, beside a code-window illustration with a check mark.](https://ai.engineer/api/talk-essay-media/de8ca09047f2a2e845acf806b3ea1944481cd644c4f4ba7444a2485c149ed15c)

Code crossed it first because you can verify it.

Broader knowledge work often lacks such a direct check. Did the report land with its audience? Is the design on brand? Did the agent understand what the user meant? A unit test cannot settle those questions on its own. Across applications, success also depends on intent and judgment, making easy verification an open problem precisely where much everyday work happens.

### Shared context and a continuous feedback loop

[7:29](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=449s)

People already navigate this ambiguity together. You bring a colleague into a Zoom meeting, discuss the problem and point at the systems involved. In Barth’s example, a brief conversation resolves something that had no directly verifiable answer. Looking at the same screen lets both people refer to the same objects and notice the same state. The missing ingredient for an agent may therefore be shared context, rather than necessarily a bigger model: less of the situation has to be translated into a prompt.

That context must persist after an action. An agent that sees a screen, clicks and moves on can miss whether the operation succeeded or whether recovery is needed. Robotics supplies the relevant pattern: **perceive → plan → act**, then perceive again. On a computer, perception means attending to the rendered layout, the current state and what just changed. The screen is the grounding for the task, even when implementation tools also inspect the page’s underlying structure.

The interaction rhythm matters too. A chatbot encourages a sequence of prompt, wait, inspect and correct. If the result misses the user’s intent, another prompt sends the agent away again. Barth’s proposed alternative is an agent that can react and suggest changes while the user is still working. Simultaneous assistance without waiting is the desired interaction, not a latency result established by these demonstrations.

### What perception adds to computer use

[11:36](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=696s)

Barth calls this class of system a **perception agent**. Its first benefit is closing the action loop: after clicking, typing or scrolling, it reads the rendered result and checks whether its output actually appeared. Sending an action is no longer treated as sufficient evidence that the action worked.

The second benefit is access through an application’s visible interface without requiring that application to expose an API. Barth motivates this by saying that most everyday software does not expose APIs, although she supplies no measured population for that claim. The practical distinction is between needing a target application’s API and needing infrastructure to operate the agent: the released verifier’s [current setup documentation](https://github.com/amazon-agi-labs/nova-act-agent-skills/blob/main/docs/ui-verification.md) requires Nova Act authentication and an MCP server.

The third benefit runs in the other direction, from the person to the agent. Instead of describing a website heading through a long paragraph, the user can point at it and request a change. A marked heading or section supplies a precise target alongside the instruction, reducing the information lost when a visual intention is converted into text.

### Mark the target, then hand off the context

[13:23](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=803s)

The first two components of Amazon AGI Lab’s [perception agent harness](https://www.amazon.science/blog/introducing-the-perception-agent-harness-annotation-and-verification-open-source), released in May 2026, address opposite ends of the loop: annotation communicates what the user wants, and verification checks the agent’s work.

The annotation demonstration uses the [Nova Act Annotator](https://github.com/amazon-agi-labs/nova-act-browser-extensions), a Chrome extension. The workflow is concrete:

1. Draw around a heading or mark a section on the page.
2. Attach a request, such as changing the heading to red.
3. Alternatively, hover over an element, click to select it and request a change such as doubling its font size.
4. Hand the resulting summary to the implementation agent.

The summary captures the feedback, its location and style information. Rendered-screen grounding does not mean a pixels-only implementation: the launch description documents DOM selectors, bounding boxes and computed styles as part of the captured context.

The completed handoff appears beside the Some Podcast webpage in the demonstration. The useful output is not merely a sentence saying to change a heading; it ties the requested change to what the user selected. Barth presents that shared capture as a way to remove the clarification back-and-forth otherwise needed to establish which element the user meant.

![Annotation slide showing the Some Podcast webpage and a populated sidebar headed “Hand this off to your agent.”](https://ai.engineer/api/talk-essay-media/677916c041a10819f831b22e0d5a8ad487bedef6970d814f8d2ed7715bc89767)

The annotation demo displays its agent handoff summary.

### Check both the design and the experience

[15:07](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=907s)

The second demonstration uses verification from [Nova Act Agent Skills](https://github.com/amazon-agi-labs/nova-act-agent-skills). Its starting point is a `design.md` file describing the application’s design rules. Colors, components and layout expectations become criteria against which the agent checks its work; if those rules have not already been written down, the workflow can turn the design description into rules.

A small `design.md` example makes the role of that input concrete:

```markdown
# Design rules

## Colors
- Use a white page background.
- Use black text for the main heading.

## Layout
- Keep the main heading above the task list.

## Components
- Give each task a visible delete control.
```

These illustrative rules give verification explicit expectations to inspect. The visual check asks whether the result follows the intended brand and layout. In the released tool, that includes deterministic checks of live DOM computed styles, rather than relying solely on a model’s judgment of a screenshot.

The behavioral check then walks through the application like a user. Depending on the available tasks, it might add a task and delete one.

| Check | What it examines |
|---|---|
| Design | Colors, components and layout against the rules |
| User flow | Whether actions work through the interface |

These checks answer different questions: a page can look right while its controls fail, or function correctly while violating the design. The workflow produces a reviewable report identifying passing checks and failures, taking over the manual walkthrough that might otherwise happen at the end of a long day.

### When the shared context is a conversation

[16:48](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1008s)

Perception need not begin with a screen. For another experiment, Barth and her colleague Giovanni held a design meeting in a booth at the conference. Both wore [Bee devices](https://bee.computer/), supplied through Bee’s sponsorship, to transcribe the conversation. The experiment extends the same workflow to intentions expressed aloud during a meeting.

The demonstration imports the meeting transcript and displays a detailed summary with the captured insights. At this point, those insights are proposed changes. Clicking **Apply** sends them to the agent. The website then receives the requested yellow background, red heading and changed emoji, and verification starts immediately.

The resulting report accepts the color scheme because it fits the demo’s approved rules. That detail matters: verification checks the specification it has been given. A rule set that disallows yellow backgrounds would instead flag the change. The human then has a judgment call—revise the specification because yellow is now desired, or ask the agent to fix the violation. The loop brings the disagreement into view without deciding the design preference on the user’s behalf.

### Building the rest of the loop in the open

[19:30](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1170s)

Annotation and verification are the first pieces of the harness, not a completed system for general knowledge work. Barth invites developers to try the open-source repositories, build on them, break things and report what is missing. Further development depends on discovering where these patterns help real collaboration and where they still fail. The goal is AI that makes people more effective together.

For the human-agent interaction questions behind this work, Barth recommends the podcast hosted by her colleague Danielle Perszyk, a cognitive scientist leading the lab’s HCI team. Perszyk’s [Making a Mind](https://www.amazon.science/making-a-mind) explores these questions with industry experts. Barth also points conference attendees to expo talks, the lab’s booth and Gaurav Mishra’s *From RL to IRL*, scheduled for 1:30 in the computer use track. The closing invitation extends beyond the venue: continue the conversation through the lab’s GitHub projects and website as the remaining pieces take shape.

![“Come, find us” slide listing expo access, github.com/amazon-agi-labs, and labs.amazon.science beside a booth photograph.](https://ai.engineer/api/talk-essay-media/4b0ec91397efd5d2376ce232a52e9f7f69bfb36ad40c4415332f7975dc874ec1)

Find Amazon AGI Lab at the expo, on GitHub, or at labs.amazon.science.

## Resources

- [Introducing the perception agent harness](https://www.amazon.science/blog/introducing-the-perception-agent-harness-annotation-and-verification-open-source) — Amazon AGI's launch explanation of annotation, verification and shared visual context.
- [Nova Act Annotator](https://github.com/amazon-agi-labs/nova-act-browser-extensions) — Chrome extension for marking page elements, drawing feedback and exporting structured instructions for coding agents.
- [Nova Act Agent Skills](https://github.com/amazon-agi-labs/nova-act-agent-skills) — Browser automation and UI verification skills, including design checks, user-flow execution and reports.
- [UI Verification setup and workflow](https://github.com/amazon-agi-labs/nova-act-agent-skills/blob/main/docs/ui-verification.md) — Setup instructions and input/output formats for design.md specifications, user-flow scenarios and verification reports.
- [Making a Mind](https://www.amazon.science/making-a-mind) — Danielle Perszyk hosts conversations about perception, reinforcement learning and useful general intelligence.
- [Bee](https://bee.computer/) — Wearable personal AI that turns conversations into transcripts, summaries and insights.
- [Bee CLI](https://docs.bee.computer/docs/cli) — Developer commands for accessing conversation transcripts, summaries and personal context.

## Slides

### Clicking was the easy part

A year ago, finding and clicking buttons on unfamiliar screens was hard. Agents can now drive browsers and are starting to drive desktop apps; Barth calls clicking the easy part.

[0:44](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=44s) [The slide says clicking was the easy part beside a cursor icon.](https://assets.ai.engineer/sites/aiecode2025/sha256/3f8d6ae142d7a1ccf50ae82fb59f0e3585486b361ac96ce2f9824320c76ae297.webp)

Clicking was
the easy part.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### The agent can use every tool

Onboarding a new teammate can span several systems. Agents may be able to perform each step, yet still struggle to complete the workflow end to end across applications.

[1:13](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=73s) [The slide says the agent can use every tool but still cannot do the work, beside a stack of browser windows.](https://assets.ai.engineer/sites/aiecode2025/sha256/70c77c7eb12c85c898dc2d10025de802b0041ca200bc1ff12e391338dfc71047.webp)

The agent can use every tool.
It still can’t do the work.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### We taught computers to use computers

Barth describes teaching agents to click, scroll, type, call APIs and fill forms, then string those reliable steps into workflows.

[2:24](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=144s) [The slide says, “We taught computers to use computers.”](https://assets.ai.engineer/sites/aiecode2025/sha256/a2b928b6201f7493ccbaa14771f61cf905a9dd004088ad6ba40b2ad4c27dac74.webp)

We taught computers
to use computers.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Reliability is the hard part

Barth argues that agents need reliability to earn trust in end-to-end workflows. She illustrates the stakes with a hypothetical agent that deletes a database one in four times.

[3:12](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=192s) [The slide says capability is mostly figured out, but reliability is the hard part, beside a gauge icon.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d15632223bdf65564a744418b4a50dfa2420b8e68e79ad3379d59d659362736.webp)

Capability, we mostly figured out.

Reliability is the hard part.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Coding already made the jump

Barth traces coding’s evolution from autocomplete and function writing to agents that write code and open pull requests, describing a shift from capability to reliability and trust.

[4:33](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=273s) [The slide says coding already made the jump, beside rising bars and a check mark.](https://assets.ai.engineer/sites/aiecode2025/sha256/c95bfb765f49e1d0916e41dffe2584ad2eee70bfa7dfa1a61b54190f4bcf05e8.webp)

Coding already
made the jump.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Code crossed it first because you can verify it

Barth argues that code crossed first because it can be run, tested and checked, helping coding agents become reliable and trusted.

[5:57](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=357s) [The slide says code crossed it first because you can verify it, beside a code window and check mark.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8e044113479f4ea28335a6a1c9f88d53eff6b277714f4472072dd6a0c2bb9a3.webp)

Code crossed it first because you can verify it.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Most work isn’t verifiable

Barth asks whether a report landed or a design is on brand. No unit test can answer those questions, making agent reliability harder to verify.

[6:25](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=385s) [Knowledge work and the real world are messy, but most work isn’t verifiable.](https://assets.ai.engineer/sites/aiecode2025/sha256/b86aba169f16527eb8b1a48b1e1ae6356132d5ef76aac360b69330b67db46d6a.webp)

Knowledge work is messy. The real world is messy.

But most work
isn’t verifiable.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### How do humans handle messy work?

Barth asks how people handle messy work across systems, using onboarding a new colleague as an example.

[7:27](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=447s) [So how do humans handle the messy work?](https://assets.ai.engineer/sites/aiecode2025/sha256/2da4bc2814a147d1f99a30d8e9bcb71c456aee09eb15c86d5b4f392863de11f5.webp)

So how do humans
handle the messy work?

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### We figure things out together

Colleagues work through problems while looking at the same screen. Barth argues that shared context with an agent could reduce how much a person needs to explain.

[7:46](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=466s) [We figure things out together, beside an illustration of two people discussing a screen.](https://assets.ai.engineer/sites/aiecode2025/sha256/653f825a1fbd4d66134ed068d34b044b8059ca928aa91589351fbe2a4de6defe.webp)

We figure things
out together.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Perceive, plan, act

Barth proposes a perceive-plan-act loop for screen agents. It starts by observing the rendered screen and its changes, so agents can watch what happens and recover when a step fails.

[8:51](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=531s) [Perceive is emphasized alongside Plan and Act, with an eye, computer, and circular arrows.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ec77ce6c16bd213c249bec4ad02aa357d9d018d07b6b75c7d56afcc2b2544f8.webp)

Perceive
Act
Plan

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### It keeps up in real time

Barth contrasts today’s prompt-and-wait turns with an agent that could react and suggest ideas while a person is still working.

[10:17](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=617s) [A diagram contrasts staggered bars labeled Today: Take Turns with overlapping bars labeled In Sync: Overlap.](https://assets.ai.engineer/sites/aiecode2025/sha256/5ade2698b889e705feb77a53795124d44f00dffadfd7c7b9044dd46ba320f8b6.webp)

It keeps up in real time.

TODAY: TAKE TURNS
IN SYNC: OVERLAP

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Perception agents

Barth introduces perception agents as agents that perceive what you perceive and understand what you mean.

[11:34](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=694s) [Perception Agents appears in large, angular white lettering on a dark background.](https://assets.ai.engineer/sites/aiecode2025/sha256/c8131855229e6f1d5df223bd2b3f5664f3921c80b0ee84694d53745c1c9c6b45.webp)

PERCEPTION AGENTS

### Why perception agents?

Perception agents read the rendered interface to check their actions without API or back-end access. Users can point to a heading or section to show what they want changed.

[11:41](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=701s) [Slide lists three reasons for perception agents: closing the gap in computer use, working without back-end access, and showing what you mean.](https://assets.ai.engineer/sites/aiecode2025/sha256/b206ac3ba260a3e2338a938aabce7c19f1198c816c1a0e67f8924c8988f9a7f0.webp)

Why perception agents?

They close the gap on computer use.
They don’t need an API or back-end access.
You show what you mean, don’t describe it.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Perception agent harness

The open-source harness has two pieces: annotation lets users show an agent what they want, while verification lets the agent check its own work.

[13:21](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=801s) [Slide introduces a perception agent harness with annotation and verification.](https://assets.ai.engineer/sites/aiecode2025/sha256/3065ac683308adab93b12c6c038b931f88b2705f24d591c637311f011ddebefe.webp)

Perception agent harness with
annotation and verification.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Opening the annotator

A Chrome extension offers to annotate the sample podcast page and hand notes to a coding agent.

[13:55](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=835s) [An annotation extension prompt appears over a sample podcast webpage.](https://assets.ai.engineer/sites/aiecode2025/sha256/38630113a8b4597db339a7034a76267395061dfa8c3fcae7148d98378d3c8b70.webp)

Annotation

Some Labs

NEW PODCAST

Some Podcast

Where researchers discuss intelligence — artificial, biological, and occasionally accidental.

Machine Learning
Cognitive Science
Published July 11, 2026

We're thrilled to announce Some Podcast, a new podcast from Some Labs. Hosted by Dr. Mira Johansson, the show brings together researchers, engineers, and the occasional philosopher to explore the big questions about intelligence — what it is, how to build it, and whether it'll remember your name after a context window refresh. New episodes weekly, or whenever the inference costs clear.

1 What Even Is Intelligence? (Asking for a Friend's Neural Network)

“If my toaster could pass a Turing test, would I still be allowed to unplug it?”

Interface text:
Nova Act Annotator
Annotate this page and hand the notes to your coding agent.
Annotate Current Page

### Drawing an annotation

The demo draws around the podcast heading in Nova Act Annotator, marking it for a proposed change.

[14:13](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=853s) [The annotator sidebar is open and a line surrounds the podcast heading.](https://assets.ai.engineer/sites/aiecode2025/sha256/709e17b01c6c5bf111f77a1867007c4352856946d15317dc6354888153525d04.webp)

Annotation

NEW PODCAST
Some Podcast

Where researchers discuss intelligence — artificial, biological, and occasionally accidental.

Machine Learning · Cognitive Science · Published July 11, 2026

We're thrilled to announce Some Podcast, a new podcast from Some Labs. Hosted by Dr. Mira Johansson, the show brings together researchers, engineers, and the occasional philosopher to explore the big questions about intelligence — what it is, how to build it, and whether it'll remember your name after a context window refresh. New episodes weekly, or whenever the inference costs clear.

1 What Even Is Intelligence? (Asking for a Friend's Neural Network)

“If my toaster could pass a Turing test, would I still be allowed to unplug it?”

Interface text:
Some Labs
Blog
Nova Act Annotator
Draw
Element
Pin
NOTES · 0
Save & export
Apply changes

### Selecting a page element

The demo highlights an episode heading in Element mode while the sidebar retains an earlier note to make the podcast title red.

[14:23](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=863s) [The annotator shows a saved color-change note while a page heading is outlined for element selection.](https://assets.ai.engineer/sites/aiecode2025/sha256/e8a443612ddfee351f88d0c607c243a1441cf3d120cdfedd32f9efacb0218ac2.webp)

Annotation

NEW PODCAST
Some Podcast
Where researchers discuss intelligence — artificial, biological, and occasionally accidental.

Machine Learning    Cognitive Science    Published July 11, 2026

We're thrilled to announce Some Podcast, a new podcast from Some Labs. Hosted by Dr. Mira Johansson, the show brings together researchers, engineers, and the occasional philosopher to explore the big questions about intelligence — what it is, how to build it, and whether it'll remember your name after a context window refresh. New episodes weekly, or whenever the inference costs clear.

1 What Even Is Intelligence? (Asking for a Friend's Neural Network)

“If my toaster could pass a Turing test, would I still be allowed to unplug it?”

Interface text:
Some Labs
Blog
Nova Act Annotator
Draw    Element    Pin
NOTES · 1
1 DRAW · JUST NOW
Make it red
Save & export
Apply changes

### Annotation

The demo records a drawn color-change request and an element-specific font-size request. The annotator collects the feedback and its location for an agent to use.

[14:32](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=872s) [A note editor is open beside a selected page heading, with a font-size instruction partially entered.](https://assets.ai.engineer/sites/aiecode2025/sha256/a255c099e4ef6625ac6951a3b652dd0643c915a516df9f90a3fe788d144626e6.webp)

Annotation

NEW PODCAST
Some Podcast
Where researchers discuss intelligence — artificial, biological, and occasionally accidental.

Machine Learning    Cognitive Science    Published July 11, 2026

We're thrilled to announce Some Podcast, a new podcast from Some Labs. Hosted by Dr. Mira Johansson, the show brings together researchers, engineers, and the occasional philosopher to explore the big questions about intelligence — what it is, how to build it, and whether it'll remember your name after a context window refresh. New episodes weekly, or whenever the inference costs clear.

[illegible]ence? (Asking for a Friend's [illegible]

“If my toaster could pass a Turing test, would I still be allowed to unplug it?”

Interface text:
Some Labs    Blog
Nova Act Annotator
Draw    Element    Pin
NOTES · 1
1 DRAW · JUST NOW
Make it red
[illegible] · JUST NOW
[illegible]uble the font s
Esc to cancel    Save
Save & export
Apply changes

[14:36](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=876s) [A podcast page with two annotation notes in the Nova Act Annotator panel.](https://assets.ai.engineer/sites/aiecode2025/sha256/58ee428d4ac23f8e1c0699309fcad09b3479106c305d1006121115ba346c13c6.webp)

Annotation

NEW PODCAST
Some Podcast
Where researchers discuss intelligence — artificial, biological, and occasionally accidental.

Machine Learning    Cognitive Science    Published July 11, 2026

We're thrilled to announce Some Podcast, a new podcast from Some Labs. Hosted by Dr. Mira Johansson, the show brings together researchers, engineers, and the occasional philosopher to explore the big questions about intelligence — what it is, how to build it, and whether it'll remember your name after a context window refresh. New episodes weekly, or whenever the inference costs clear.

What Even Is Intelligence? (Asking for a Friend's Neural Network)

“If my toaster could pass a Turing test, would I still be allowed to unplug it?”

Interface text:
Some Labs    Blog
Nova Act Annotator
Draw    Element    Pin
NOTES · 2
1 DRAW · JUST NOW
Make it red
2 ELEMENT · JUST NOW
Double the font size
[illegible]
Save & export
Apply changes

### Visual Verification results

The demo reports 82 visual rules and four task flows passing, including adding, completing and deleting tasks. The report says its specs were derived from the live site and self-verified.

[15:47](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=947s) [An editor displays a verification report with task-flow checks and generated artifacts.](https://assets.ai.engineer/sites/aiecode2025/sha256/f5909273df74207175b002b37723441236f30e152c5263209fc5c3e0dd23a026.webp)

Visual Verification

• Typed "Buy groceries" into the task input
• Clicked "Add Task" button
• Verified: new task visible, total count updated to 4, pending updated to 3

Taskflow-2: Complete a task
• Clicked checkbox for "Review design specifications"
• Verified: task styled with line-through (confirmed via DOM), completed count updated to 2, pending updated to 1

Taskflow-3: Delete a task
• Clicked delete (×) button for "Write unit tests"
• Verified: task removed from list, total count updated to 2

Taskflow-4: Initial page state
• Verified heading, subtitle, 3 task items, stats (3 total, 1 completed, 2 pending), "Set up CI pipeline" shown as completed

Audit Findings
None. This is a generation run — specs were derived from the live site and self-verified.

Artifacts
• visual/design.md — human-readable design specification
• .ui-verification/specs/ — 5 compiled category files (82 rules)
• .ui-verification/flows/ — 4 flow scenarios
• .ui-verification/sessions/[illegible]/ — assertion JSON

Interface text:
Preview report.md
New Session
Verification complete. Here's the [illegible]
Verification Results for http:[illegible]
All checks pass — 82 visual ru[illegible]
Visual Verification (82 rules)
Category
Visual Style
Component Rules
Accessibility
Project Rules
Platform Conventions
Flow Verification (4 flows)
Flow | Description
taskflow-1 | Add a new ta[illegible]
taskflow-2 | Complete a t[illegible]
taskflow-3 | Delete a task
taskflow-4 | Initial page s[illegible]

### Design meeting

Barth and her colleague Giovanni wore Bee devices to transcribe their design meeting, capturing their discussion of website changes.

[17:10](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1030s) [Two people sit with laptops at a table beneath a screen reading Design Meeting.](https://assets.ai.engineer/sites/aiecode2025/sha256/d3d6990ff0e1b9861aaeb7ca458c19c157f174a108b7e067b6e712016bf8eeb5.webp)

Design Meeting!

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Podcast website before changes

The podcast page sits beside a connected Bee Conversations panel, which is waiting for the design discussion. Barth introduces the workflow for pulling the meeting transcript into the page.

[18:12](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1092s) [Podcast page with a Bee Conversations panel waiting for design conversations.](https://assets.ai.engineer/sites/aiecode2025/sha256/ae505a901928a3850283792d794687e5709597cd501a7337be5c5f3ba15b5f37.webp)

NEW PODCAST
Some Podcast
Where researchers discuss intelligence — artificial, biological, and occasionally accidental.

Machine Learning    Cognitive Science    Published July 11, 2026

We're thrilled to announce Some Podcast, a new podcast from Some Labs. Hosted by Dr. Mira Johansson, the show brings together researchers, engineers, and the occasional philosopher to explore the big questions about intelligence — what it is, how to build it, and whether it'll remember your name after a context window refresh. New episodes weekly, or whenever the inference costs clear.

1 What Even Is Intelligence? (Asking for a Friend's Neural Network)

“If my toaster could pass a Turing test, would I still be allowed to unplug it?”

Interface text:
Some Labs
Blo[illegible]
Bee Conversations
Connected
Waiting for design conversations...
When a Bee conversation about design, CSS, layout, or UI finishes, it will appear here with full context.
Retrieve Conversations

### Design meeting summary

Bee’s meeting summary captures three proposed edits: a red title, a yellow background, and a robot icon. The presenter says Apply sends the insights to an agent.

[18:20](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1100s) [A meeting summary lists proposed changes to the page title, background and icon.](https://assets.ai.engineer/sites/aiecode2025/sha256/9d0a1c7e8b28439743c51809e5db668192371d9c1661ff866061a4bba4d22334.webp)

Redesigning UI Elements

SUMMARY
Redesigning UI Elements

KEY TAKEAWAYS
• Discussed UI design changes for what appears to be an AI assistant interface
• Three modifications were proposed: red header title, yellow background, and robot emoji replacing brain icon
• Giovanni responded positively to all suggestions, particularly liking the bold red color choice
• The robot emoji was agreed to be more immediately recognizable than abstract brain imagery for AI functionality
• Giovanni offered to implement all changes within five minutes
• The yellow background was noted to provide better visual impact for stage presentation

FULL SUMMARY

# Summary

Giovanni participated in a brief design discussion about UI modifications for what appears to be an AI assistant interface. The conversation centered around three specific changes: making the main header title red (instead of blending in), changing the background to yellow, and replacing a brain icon with a robot emoji in the top left corner to make the AI nature more immediately recognizable. Giovanni responded positively to the proposals, calling the red color choice "bold" and noting that the robot emoji would be more intuitive than the abstract brain. He was enthusiastic about implementation, stating he could complete the changes in five minutes and commenting that the yellow background would "stand out on the stage."

## Atmosphere

The conversation had a collaborative, productive tone with Giovanni actively engaged and supportive of the proposed changes. There was a sense of urgency and efficiency, with Giovanni ready to implement [illegible]

Interface text:
Some Labs
Bee Conversations
Redesigning UI Elements
[illegible] Neural Network)
“If my toaster could pass a Turing test, would I still be allowed to unplug it?”
Retrieve Conversations

### Applied design changes

Applying the meeting’s proposed changes turns the podcast page yellow, makes its title red and replaces the brain icon with a robot. Verification starts immediately.

[18:39](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1119s) [The podcast page has a red title, robot icon and yellow content background; verification is running.](https://assets.ai.engineer/sites/aiecode2025/sha256/7e8df6c6e95f2f6325c1e9c688a7779cea545b077142a32853f526f79c1303e5.webp)

Some Labs

NEW PODCAST

Some Podcast

Where researchers discuss intelligence — artificial, biological, and occasionally accidental.

Machine Learning
Cognitive Science
Published July 11, 2026

We're thrilled to announce Some Podcast, a new podcast from Some Labs. Hosted by Dr. Mira Johansson, the show brings together researchers, engineers, and the occasional philosopher to explore the big questions about intelligence — what it is, how to build it, and whether it'll remember your name after a context window refresh. New episodes weekly, or whenever the inference costs clear.

1 What Even Is Intelligence? (Asking for a Friend's Neural Network)

“If my toaster could pass a Turing test, would I still be allowed to unplug it?”

Redesigning UI Elements

KEY TAKEAWAYS
Discussed UI design changes for what appears to be an AI assistant interface
Three modifications were proposed: red header title, yellow background, and robot emoji replacing brain icon
Giovanni responded positively to all suggestions, particularly liking the bold red color choice
The robot emoji was agreed to be more immediately recognizable than abstract brain imagery for AI functionality
Giovanni offered to implement all changes within five minutes
The yellow background was noted to provide better visual impact for stage presentation

Giovanni participated in a brief design discussion about UI modifications for what appears to be an AI assistant interface. The conversation centered around three specific changes: making the main header title red (instead of blending in), changing t...

Interface text:
Blo[illegible]
Bee Conversations
Connected
Redesigning UI Elements
just now
Verifying...
Details
Dismiss
Running verification...
Retrieve Conversations

### Verification report

The report records 111 checks and all three flows passing after the design changes. Barth says the yellow background met the current rules, which can be changed.

[18:54](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1134s) [A verification report shows passing visual and flow checks.](https://assets.ai.engineer/sites/aiecode2025/sha256/1f288b34d503c41343e1f1c5c1b554710c2152b0eb3322b083056b2c51123c16.webp)

| Accessibility | 8 | 8 | 0 |
| Project Rules | 17 | 17 | 0 |
| Platform Conventions | 14 | 14 | 0 |
| Total | 111 | 111 | 0 |

Result: ALL PASS

No visual failures detected. All rules match the design specification.

Flow Verification Summary

| Flow | Name | Type | Status |
| [tcap-1](flow-reports/tcap-1.report.md) | Podcast landing page displays all content sections | state_completene[illegible]
| [tcap-2](flow-reports/tcap-2.report.md) | Page scrolls to reveal all episodes and streaming links | happy_path[illegible]
| [tcap-3](flow-reports/tcap-3.report.md) | Episode cards display correct structure | state_completeness | PASS |

Result: ALL PASS (3/3)

Audit Findings

No audit required — integrity ledger hashes match all category files and design.md.

Screenshots

- [Landing page (top)](screenshots/landing-page.png)

Session

- Session ID: [illegible]
- Assertions: .ui-verification/session/[illegible]

### AI that makes us smarter together

Barth calls the two pieces a first step and invites people to try the open-source work, build on it, and share feedback.

[19:29](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1169s) [Slide reads: Just the first step. AI that makes all of us smarter, together.](https://assets.ai.engineer/sites/aiecode2025/sha256/33a85d7d8a4b16ed604f5492987f33a7d81987edc3c3fc869fbac044e2cf5d4b.webp)

Just the first step.

AI that makes all of us
smarter, together.

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Go deeper on human-agent interaction

Antje Barth recommends Making a Mind, hosted by cognitive scientist Danielle Perszyk, for discussions with industry experts about human-computer interaction patterns.

[20:13](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1213s) [Slide recommending the Making a Mind podcast, hosted by Dr. Danielle Perszyk.](https://assets.ai.engineer/sites/aiecode2025/sha256/e1012df914c61c18d4e10022dda3903ffb63916ae06b4670afbfe86dd034ba95.webp)

Go deeper on
human-agent interaction.

Making a Mind

MAKING A MIND · PODCAST
The cognitive science of building a mind
hosted by Dr. Danielle Perszyk

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

### Come, find us

Visit Amazon AGI Lab on the expo floor for demos and swag, or explore its code at github.com/amazon-agi-labs and website at labs.amazon.science.

[21:07](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1267s) [Slide inviting attendees to visit the expo booth or find Amazon AGI Lab online, alongside a booth photograph.](https://assets.ai.engineer/sites/aiecode2025/sha256/929191a67164cff8ea9d3cb3c86fef3fe9bbd07241906af3f2b31602863c525f.webp)

Come, find us.

IN PERSON
On the expo floor,
demos and swag

GITHUB
github.com/amazon-agi-labs

ONLINE
labs.amazon.science

Amazon AGI Lab
Want to learn more?
Visit Amazon AGI Lab
Welcome

Amazon AGI Lab
© 2026 Amazon.com, Inc. or its affiliates.

## Full transcript

[0:00](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=0s) [upbeat music] Joining us on stage is a member of technical staff at Amazon AGI Lab, Antje Barth. [upbeat music]

[0:36](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=36s) Good morning. It's so great to be back here at the AI Engineer World's Fair.

[0:43](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=43s) Just a year ago, the hard problem was getting an agent to find a button and click it on a screen, especially screens it had never seen before.

[0:57](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=57s) Now, agents can drive browsers, and they're starting to also drive desktop apps.

[1:06](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=66s) But what we figured out, clicking, clicking, was actually the easy part.

[1:16](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=76s) What we didn't solve is the actual work.

[1:20](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=80s) And what do I mean with this? Let's take a very simple example. A new team member starts on Monday, and maybe your job is to set up their accounts, add them to a Slack channel, book intros with colleagues, order the laptops, et cetera.

[1:40](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=100s) And nobody really owns this end-to-end process in the company,

[1:46](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=106s) and it might be also touching five different systems.

[1:51](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=111s) Now, agents can most likely perform each single individual, individual step of this workflow.

[1:59](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=119s) But agents still struggle to do this end to end because the real work lives within the seams of all of those different applications, of all of those different steps you have to take.

[2:12](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=132s) And this is mostly where it all falls apart.

[2:17](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=137s) The agent can use every single tool you give it, but it still can't do the full work.

[2:26](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=146s) So why do we see this gap? Think about for a minute what we actually built.

[2:34](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=154s) We taught computers to use computers. So what do I mean with this? We started building out the basics. We taught them clicking, scrolling, typing, calling an API, filling out a form, and we got those stops-- these steps really reliable, and you can string them together in a workflow.

[2:57](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=177s) And agents these days are fairly good at, like, operating those workflows.

[3:03](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=183s) So why can you not just hand them more of your work and then literally just walk away and trust it to be completed?

[3:14](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=194s) So all the things I talked about, like using a tool, models itself, tool use, stringing agents together, this is all capabilities.

[3:27](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=207s) And we mostly figured out how to add capabilities to models. Now, the next hard part is really reliability.

[3:38](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=218s) And without reliability, we cannot really build up trust in those systems.

[3:45](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=225s) So here's a quick gut check, and maybe all of you can just think about an agent doing work in an end-to-end workflow. How often do you think that actually succeeds these days?

[3:59](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=239s) Maybe sixty, maybe eighty percent of the time?

[4:03](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=243s) And it sounds really fine. But if you look into this,

[4:09](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=249s) if your agent one in four times deletes a database, you will never touch that agent again, right?

[4:19](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=259s) So when you need this reliability, you really need to be it in the nines. You need to have the trust that it actually can do the work successfully.

[4:36](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=276s) Now, there's actually one place where we made enormous progress on reliability and trust, and this is coding, right? Think about how fast coding evolved.

[4:52](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=292s) I still remember the first time when it started auto-completing for you, right? You just tapped auto-complete, amazing. Then short time later, it started to write functions, and we thought, "That is amazing."

[5:07](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=307s) And now look at these days, coding agents write the code, they open up the pull request themselves, and we heard it earlier this week, code keeps flying by.

[5:19](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=319s) So once in a time, we were able to just every single line that it generated, we felt like the urge we need to really read it and make sure it's correct, right?

[5:29](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=329s) I think most in the audience here can still relate to that.

[5:33](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=333s) These days, I think hardly anyone is still doing that. Like, we cannot even do that, right? Code is generated at such a pace.

[5:43](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=343s) At the same time, coding made the jump, so why is that? Because we were able to bring it from just being capable, the coding agents, to actually be reliable and then trusted.

[5:59](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=359s) So why is that? Why was coding first solved?

[6:04](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=364s) It's because code is verifiable. You can run it, you can test it, you can check it, and you can be for sure that it worked.

[6:16](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=376s) So reliability showed up in the first place. You can actually verify the answer.

[6:27](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=387s) But here's the catch. Most of the work we do, if you look at the broader knowledge work areas, it's not like that.

[6:37](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=397s) Knowledge work is messy. And heck, the whole real world is really messy.

[6:44](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=404s) Did the report I created land? Is the design on brand?

[6:50](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=410s) Did it get it what I actually meant? So there is no unit test that can answer those questions.

[6:59](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=419s) So verification really hits the wall right where most of our work lives.

[7:05](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=425s) It's living in the seams of all of those applications we're using on a day-by-day basis. And nobody really has cracked this part yet.

[7:15](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=435s) How do you make an agent reliable when there's no way to verify the answer that easily? And that's a field that is still wide open.

[7:29](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=449s) So how can we solve this? Well, so how do humans handle messy work? I mean, we're successful at it, right? Each of us, like, every day, we work across different systems, we manage out how to onboard a new colleague.

[7:44](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=464s) We do this. Well, we're doing it by figuring things out together. You grab a colleague, you jump on a Zoom meeting, you're discussing things, you're looking at the problem to solve, you're discussing pa-pa-pointing at systems, and maybe two minutes later, you solved it.

[8:06](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=486s) You're done. But none of this work is actually directly verifiable.

[8:12](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=492s) And we do this all day. So one of the things is we're looking mostly at the same screen, right? If you're jumping on a meeting with a colleague, you see the same screen, both of you, and you can actually, like, figure out really quickly what needs to be done.

[8:31](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=511s) So this is what the agent these days is missing. You don't necessarily need a bigger brain. What you need is this shared context, because if we're looking, the agent and myself, at the same screen, I probably have much less explaining to do.

[8:53](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=533s) So what kind of agent do we really need to build to achieve this?

[8:59](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=539s) And today's agent, as I said, they can already see a screen, right? And they can click and take actions in it. That part works. But if they fire off actions, what they usually do, they move on.

[9:13](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=553s) They don't watch what happens or recover if one step didn't succeed or something goes sideways.

[9:22](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=562s) And we need an agent that can actually work like you do, like humans work.

[9:28](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=568s) And one example is robotics. If you just look for a moment as how robotics do it, a robot perceives what's around it, and it plans what to do and then acts.

[9:42](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=582s) So this loop here from perceiving to planning to acting, this is actually what we also would need on a screen.

[9:53](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=593s) And it starts here really with the first word, which is perceive. The agent has to take in the screen the way you do.

[10:04](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=604s) Not scrape the code behind the page, but what's actually rendered. The layout, the state, what just changed, the work, what we're doing, and then do it.

[10:20](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=620s) And it would also have to keep up in real time. Think about how we as humans work together. You jump in, you react to build on top of what each other you say.

[10:35](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=635s) And today, agents can still don't do it. What we're doing is we're sending a prompt, we're waiting, it goes away, and at one point the agent come back, and we might have to take a couple of turns, right?

[10:47](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=647s) Because what the agent come back with is not exactly what we might wanna do. So we're sending another prompt, say, "Hey, go back, do this, do this differently." And we have this long back and forth, which we got so used to from our chatbot experience and from this rhythm taking turns.

[11:05](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=665s) But what we actually would need, think about it, is an agent that can react while you're still working. Wouldn't that be really cool, right? Like at the same time you're working, it can also come up with suggestions, can help you, and there is no waiting time.

[11:25](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=685s) So basically an agent that perceives what you perceive and understands what you mean.

[11:36](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=696s) We call them perception agents. So why perception agents? Why do they matter? So first, they complete the loop on computer use.

[11:52](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=712s) Today's agents, again, they can act. They can click, they can type, they can scroll, but what they can't do well is looking at the results and whether it actually worked.

[12:06](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=726s) A perception agent can read the rendered screen, so it can confirm its own output instead of just firing off those actions and then hoping.

[12:19](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=739s) Second, it doesn't need an API or backend process,

[12:24](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=744s) and that's important because it works off the rendered interface. It sees the same pixels and the structure you see.

[12:32](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=752s) And most of today's software people use every day don't expose APIs at all.

[12:41](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=761s) And then third, the input also goes the other way here. Instead of writing a long paragraph to describe what you wanna change, let's say you're working on a website, and you wanna describe all the changes you wanna apply.

[12:56](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=776s) Instead of writing this really long description, wouldn't it be great if you can just point to it and say, "Hey, here, this heading needs to change. Hey, can you update this section?"

[13:09](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=789s) This is a much more precise signal and less lossy than text,

[13:15](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=795s) and the agent can act exactly on what you marked.

[13:23](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=803s) So this is where we started, and I'm happy to share that we just recently launched the first two pieces of our perception agent harness,

[13:35](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=815s) open source. There's two pieces. There is annotation, which you can use to tell it what you want.

[13:45](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=825s) And then the second piece, the verification part, gives the agent the capability to check its own work.

[13:56](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=836s) So let me show you the first one. So here's a very quick demo on our annotation tool.

[14:05](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=845s) This one is a Chrome extension, so it's super easy to use, and I'm gonna play here this quick video demo.

[14:13](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=853s) So you have the extension installed, and then you can just select different elements on a screen. So this example, we're just drawing around the heading there, marking the section, and maybe you wanna change it.

[14:26](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=866s) Why not? Let's change it to red. You could also select the elements on this page. You see how if I hover over, it finds the right element. You click it, you select it, and say something, maybe double the font size.

[14:41](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=881s) And you see also how the agent here captures on the screen exactly the feedback, the location, the style elements, and it creates this complete summary, which you can then use and then give your agent to implement.

[14:56](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=896s) So there is no back and forth anymore because you captured exactly what you saw on screen, and the agent can see the same thing.

[15:07](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=907s) Now, let's have a very brief look at the second one, at verification.

[15:14](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=914s) So the idea of verification is that you can describe, let's stay in this case of the web development. You can describe in a design.md file what your design rules are for this.

[15:29](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=929s) And then what happens if I play this video here, the ac- the agent can actually check its own work against those design specs.

[15:38](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=938s) So it will take what you defined, the colors, the components, your layout, and it turns it into those rules if you don't have it written before yet. And it does two kinds of checks then.

[15:51](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=951s) It does a visual check, which is really cool. So everything is on brand, for example. It's the right layout.

[16:00](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=960s) The other part is also checking user flows. So what it does there, it actually walks through this experience, through the app, for example. Depending on the tasks available, it might add a task, it might delete a task like a real user would.

[16:17](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=977s) So it helps you walk through those user flows as well in an automated fashion. And then once it's done, it's writing a report which you can review, and it's gonna call out which tests passed, and it's gonna tell you anything that didn't.

[16:34](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=994s) So ultimately, you're the one that doesn't have to click through this at midnight at the end of the day because, great work, the agent already did this job for you.

[16:48](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1008s) Now, there might not always be a screen, right? So I talked a lot right now. I called it perception. I talked about the agent sees what you see on a screen.

[17:00](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1020s) But there are times in your day where you don't have a screen. Maybe you're in the office, you're walking into a meeting with a colleague.

[17:11](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1031s) So I did a fun experiment yesterday [chuckles] at the conference here. So I grabbed my colleague Giovanni, who's also here, and actually on the [REDACTED:location], there's a great, like, little meeting booth.

[17:22](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1042s) We found that by coincidence. So we went in there, and we had our design meeting. And the goal here is to really kind of show you how perception is so much more than just the visual part.

[17:35](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1055s) So in this example, what we wanna show you is perception can also be listening in the room to what you're discussing. And you can see here on the picture, both of us are wearing our Bee devices.

[17:48](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1068s) Big shout out to Bee for [chuckles] sponsoring these. Um, so we're sitting there. We have our Bee devices that can do a transcript. They're listening to what we're saying. And then we have this design meeting, and I had a couple of great ideas how to change this website.

[18:03](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1083s) Um, you will see them in a, in a second here. So let's have a quick look how this changed the same workflow on this website using this device.

[18:15](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1095s) So we had the discussion, the Bee did the transcript, and you can see here on the right, we're pulling this meeting transcript right in. There is a whole detailed summary of the meeting.

[18:29](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1109s) There is what we discussed, and then it basically captures those insights. We have them right here, and we can click Apply. So what this Apply button does is it sends it straight to the agent.

[18:44](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1124s) And you can see here my crazy idea is to turn the background to yellow, turn the heading to red, and also change the emoji directly applied. And it also straight kicks off the verification right away.

[18:56](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1136s) So it creates this report and, and luckily this color scheme was apparently into-- in the approved rules. Otherwise this would have flagged like you did some weird things here.

[19:07](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1147s) But again, you could change those rules if you don't wanna have yellow backgrounds, and it will make sure, um, that we still adhere to those guidelines. It would flag anything that's off.

[19:17](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1157s) So you have the judgment call if you wanna either update the design specs because you actually like yellow, or you take an action and say, "No, um, fix this violation."

[19:30](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1170s) But this is really the very first step. These two pieces are the very first beginning, and we're building out the rest in the open because these patterns can only get better if more people are using them, building on top of them, breaking things.

[19:48](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1188s) So my ask here to you is go and try them out. They're on our GitHub repos, open source. Tell us what we're missing. Give us the feedback, what you would like to see, where this should go next, because ultimately none of us get smart alone.

[20:06](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1206s) And that's the whole point. We wanna build AI that makes all of us smarter together.

[20:16](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1216s) Now, if you're interested in a little bit more on human-agent interactions and how we see those patterns changing, I would highly recommend this podcast by my colleague Danielle Perszyk.

[20:29](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1229s) She is a cognitive scientist and runs our AGI-- HCI team at the lab and discusses a lot about human-computer interaction patterns with experts in the industry. You can find the podcast on, on a popular podcast platform.

[20:46](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1246s) We also have more sessions this week, um, so check them out. We have a booth down there. We have expo talks. We also have another computer use track talk coming up with my colleague Gaurav Mishra at one thirty in the computer use track.

[21:02](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1262s) Highly recommend checking out his talk, From RL to IRL.

[21:08](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1268s) And then ultimately come find us. We have a huge presence down at the expo hall. We would love to continue the conversation with you all. If you're not here in person, you can also check out our code on our GitHub repo and check out our website.

[21:24](https://www.youtube.com/watch?v=2JX6JYyQG4Y&t=1284s) And with that, thank you very much. [clapping] [outro jingle]
