Teach Your Agent to Prove It's Done — Nick Nisi & Zack Proser, WorkOS
Read the talk
Teach Your Agent to Prove It's Done
Nick Nisi and Zack Proser use a broken waitlist to show how application-specific tools, repeatable checks and retained evidence can help coding agents recognize unfinished work.
From a talk by Nick Nisi and Zack Proser
At a glance
Ideas worth remembering
Define a checkable outcome: a signup displayed once is different from one that survives reload.
Give agents application-specific commands to interact with the system, inspect its state and rerun useful checks.
Evaluate the verifier too: stale receipts, misleading screenshots and model judgments can all misrepresent the result.
Use fast local feedback alongside enforced CI checks and production monitoring.
A successful signup that disappears
Midnight Snack Club appears to accept a signup: an email shows up on the page and the interface reports success. Reload the page, though, and the entry disappears. The app changed its visible state without saving the email anywhere. A test that only looks for the new entry immediately after submission would pass while missing the failure.
For WorkOS engineers Nick Nisi and Zack Proser, that gap explains why an agent saying “done” is not enough. Code can build, run and pass the tests that exist while still failing the intended task. The problem becomes harder to catch when a developer is supervising many parallel sessions and has less attention for each result.
Trust starts with defining the outcome and giving the agent a way to observe it. “Make no mistakes” leaves the agent to infer what success means; “the signup remains after reload” names a state it can check. Trust is a gradient built from checks and evidence, not a property conferred by a confident report.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give completion a verifier
Proser favors decomposing work into small tasks with clear completion criteria. Nisi asks for tangible evidence: test results, smoke tests, screenshots or recordings. His shorthand is “Status comes from records, not model prose.” A good implementation still needs the desired behavior; an elegant diff alone cannot establish that it delivered it.
Proser’s experimental Context, or CTX, service centralizes verification across agents. After an agent completes assigned work, it asks CTX over an MCP connection whether it is done. One reusable verifier queries the GitHub API to check that a pull request exists, is healthy and has passing merge checks. If the check fails, CTX returns the remaining failures; if it passes, the workflow node can be marked satisfied. A separate illustrated gate checks that pull requests have merged and reports which repository and PR were inspected.
Nisi’s Riker experiment addresses the work after implementation: follow reviews, respond to comments, rerun checks and retain evidence that the result still meets the goal. It alerts him when it needs attention instead of making him continually poll the PR. Both projects are experiments in where to put verification, not guarantees that every task can be checked automatically.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Give the agent a developer’s eyes and hands
A human debugging the waitlist might reload the page, inspect storage, check API response codes and the network tab, read console logs, and interact with the UI. An agent needs access to those observations too. A vague instruction to “verify in a browser” leaves room to skip the reload or persistence check.
Snack is the workshop’s application-specific CLI, invoked with pnpm snack. The workshop workbench provides the slides and step-by-step exercises. Snack is a template for building a verifier for an app, not a universal verifier. Its commands cover several parallel jobs:
- Health: start or stop the app and run a doctor check for its environment and services.
- Interaction: join the waitlist, reload the page and reset data.
- Observation: inspect page snapshots, storage, network activity and console logs, or capture a screenshot.
- Evaluation: run a deterministic proof or a model-based judge.
A repeatable command gives multiple agents the same way to exercise the application and returns output they can use to diagnose a failure. The script can make its checks more uniform than a long prompt, although the checks still need to represent the outcome the team cares about. Those outcomes differ across applications: backend and database behavior may matter as much as rendering.
In the demonstration, snack snapshot returns a JSON representation of the page. Nisi then adds an entry to the page to show how the snapshot can help verify its contents. A screenshot provides a different view for a human reviewer; a recording can show the path taken to reach the final state. Developer tools can also expose a rendering bottleneck through a flame graph instead of leaving the agent to guess.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn behavior into repeatable feedback
Some behavior is difficult for a model to inspect directly. Nisi describes a canvas game in which a ball moves inside a circle and can escape through a slit. As a simplified debugging aid, printing the ball’s coordinates on each iteration could reveal an unexpected jump across the screen. The useful observation is the change in position, made available in a form the agent can analyze.
For Midnight Snack, the illustrated Playwright proof starts with an empty waitlist, fills in an email, clicks the join button, and checks the status and displayed email. That establishes the immediate UI behavior. The earlier failure shows why a complete persistence proof also needs to inspect storage or reload and check the entry again. Once captured as a test, the same scenario can be rerun after later changes.
A screenshot alone is weaker evidence if it shows a state reached through an impossible path or hides side effects. Run the intended journey from start to finish, and make failures informative: an error that explains what failed and the state leading up to it gives the agent something to act on. “System error nine,” offered by an audience member, makes a good joke and a poor repair instruction.
The aim is to constrain the result without prescribing every implementation step. Proser suggests giving the agent a destination and a command that establishes when it has arrived. Within that boundary, the agent can explore a fix and rerun the check. Detailed failures can drive another attempt while the developer is occupied elsewhere.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The moment the agent knows it is not done
pnpm snack doctor checks prerequisites including Node, Chromium, the browser session, the running application and its build. These checks establish whether the environment is ready; they do not by themselves establish that signup data survives.
Next, snack join ada@example.test submits an entry. snack storage then reports that storage is empty. The visible join action has happened, but the desired saved state has not. Reloading exposes the same missing persistence. This is the point at which an agent with the right tools can see that it has more work to do.
Nisi proposes applying the persistence fix and honestCopy, then running the proof. The live proof fails because he has not actually applied the fix; the recording does not show a successful post-fix proof at that point. The failure prevents the attempted demonstration from being mistaken for a verified repair.
Proser calls the saved outputs of runs “receipts.” They can help an agent diagnose the present failure and, if retained, become examples for future evaluations. A support-ticket response, for example, could be preserved alongside a human standard for what a good response should accomplish.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Exact assertions, semantic judgments and their limits
After the persistence fix, the workshop introduces a wording problem: a check expects a waitlisted status, while the interface can say “You’re in. Your seat is confirmed.” That copy can overpromise: a saved waitlist entry does not establish that a seat is confirmed. Nisi runs the judge and reports a failure in this contrived example; the demonstration does not establish a reliable successful semantic judgment for that run.
The proposed complement to a strict assertion is a fast model judge that assesses meaning without requiring a one-to-one string match. That flexibility introduces judgment error of its own: the judge’s verdict needs evaluation against examples, including confirmations that overpromise what actually happened. Persistence remains a separate property to check.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Test the tester with a hidden bug
The Saboteur and Detective exercise makes verification itself the task. pnpm snack up --bug random starts the app with a hidden, randomly selected bug. The detective uses Snack to identify the bug and provide evidence, rather than merely naming something plausible.
The live run produces confusing output. The doctor shows a failed receipt from an earlier run for code that no longer exists; the presenters discuss a wrong port, then Nisi says he needs to debug the example’s output. The exercise does not demonstrate a clean, verified diagnosis. It does show why a previous receipt must be interpreted in the context of the code and conditions it checked.
An eval baseline can distinguish several outcomes: finding the right bug with evidence, reporting success when behavior is broken, blaming a bug that is not present, or correctly finding nothing wrong. Workshop cases include memory-only storage, a wrong email, lost confirmation and overpromising copy. Each reproducible miss can become another case the team reruns as the app changes.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep evidence across runs
The relevant observations depend on the product. For sign-on, Nisi suggests checking whether the correct cookie and expiration are set. An e-commerce journey might record whether a customer could see the catalog and add an item to a cart, along with errors and timing. Detailed outputs let an agent inspect a failure, try a repair and rerun the same journey.
Proser proposes aggregating receipts to identify recurring failures and guide future fixes. For human review, a recording of the actual flow can show the states traversed and the result reached. Retained unit tests, evals and smoke tests also form a growing behavioral specification: new features can be checked against what the application was already expected to do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Local feedback, CI and production
How should checks actually run? Proser prefers CI enforcement when a check must happen, because a model may choose not to follow a skill. Nisi also describes local hooks and an AGENTS.md instruction to run verification after significant changes or before opening a PR. Those approaches serve different moments: local checks provide quick feedback during development; CI can enforce checks across submissions.
The local benefit is fewer handoffs. Nisi wants an agent to discover a failed persistence check and retry while he works elsewhere, instead of returning later to run the app, find the failure and prompt it again. He reports relying on CI and automated review too, while noting that passing checks and a high review score could still accompany thousands of lines that accomplish nothing. The desired outcome needs its own verification.
The presenters say they use these processes in customer-facing production work. Proser describes an additional rollout pattern: verify in a preview environment, then monitor production metrics and feature flags, with an agent watching for conditions that warrant a revert. He presents this as an emerging approach, alongside the understandable unease of trusting a deployment.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
Nisi’s account of a coding-agent harness, including mechanical gates and evidence that tests ran.
A WorkOS example of agent-usable diagnostics, configuration and resource inspection.
Related talks
- Building AI Systems that Ship
Nisi develops mechanical verification gates and evidence-backed agent workflows.
- Your Attention Is the Bottleneck, Not Your Agents — Zack Proser, WorkOS
Proser connects verification loops to the human cost of supervising parallel agents.
- Skills at Scale
Nisi and Proser show how reusable procedures and scripts carry team knowledge into agent work.
Full transcript
Loading transcript…