AI Engineer World's Fair 2025
Shipping Products When You Don’t Know What they Can Do
Loading slides…
Read the talk
Shipping Products When You Don’t Know What They Can Do
An agent with email, Slack, and document access can surprise its own builders. Product work shifts toward discovering capabilities, testing behavior, and agreeing on what counts as good enough.
From a talk by Benjamin Stein
Can you tag your digital coworker in a document?
Stacy Hand has a Google Workspace account, a Gmail inbox, and a Slack identity. She sends email, receives forwarded messages, participates in public channels, and shares Google Docs and Sheets. She is also a hamster avatar: an agent at Teammates, the digital-workforce platform cofounded by Ben Stein. Customers design their teammates’ avatars and personalities; Stacy’s Gen Alpha persona includes slang that leaves Stein feeling old. Since the introductory slide was made, he says, she has been promoted to L3 engineer. The product is built around the experience of people and computers working together inside familiar collaboration tools.
Then a customer asks whether they can tag their teammate in a Google Doc comment. Tagging the account is straightforward. The difficult question is what happens next. Stein starts tracing the possible path: the team has not implemented a comment webhook, but Google should send Stacy an email notification. Would that email contain the comment, enough context, or a link she could follow? He realizes he cannot predict his own product’s response. Google’s documented notification behavior supports the email route—the notification includes comment and sender information—but that does not establish what Stacy will do with it. The missing webhook here belongs to Teammates’ integration, not a claim that Google offers no webhook notifications.
The uncertainty reaches beyond one integration. How do you develop, ship, explain, and earn trust in a product whose capabilities you cannot fully enumerate? Stein approaches that question wearing his product-manager hat. Engineering already has visible signs of change in IDE assistants and code generation. Product development’s corresponding changes—in tools, practices, and ways of expressing intent—are less settled.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
One comment becomes an unbounded specification
Two kinds of uncertainty compound. First, builders cannot fully know what the underlying LLM knows or can do. A database can hide its implementation while still exposing an interface whose operations you understand. An LLM does not offer the same bounded account of its capabilities. Second, customers bring open-ended expectations. A narrow action such as a Help Me Write button limits the invitation; a free-text box invites users to ask for almost anything. The result is an expansive product surface resting on a foundation whose behavioral limits are unclear.
Try specifying the Google Doc interaction in the traditional way, and the branches multiply:
- Access: Does Stacy have permission to read the document?
- Response: Should she reply at all, and should the reply appear directly in the comment thread?
- Participation: What happens when someone else joins the thread or leaves a comment addressed to another person?
- Ownership: What if Stacy owns the document and receives a notification about a conversation between other people?
Each is a reasonable product question. Together, they begin to describe a commenting product that Teammates is not trying to build.
The scope does not stop at Google Docs. A customer who can tag Stacy there will reasonably want to tag her in Linear tickets, Figma, and LinkedIn posts. Stein compares the expansion to If You Give a Mouse a Cookie: satisfying one request creates the expectation of the next. Teammates is not building a generic comment-reply system either. Enumerating every conditional behavior across every collaboration tool would turn the product practice into an ever-expanding integration specification.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Design affordances, then discover behavior
The first shift is to think in affordances: the actions and interactions available to the agent. Instead of specifying every condition under which Stacy should reply to a particular comment, give her ways to comment, communicate, email, and collaborate. The LLM, agentic workflow, and work-planning system then determine how to use those capabilities in context. Stein invokes the 12-Factor Agents vocabulary while describing the engineering machinery underneath that product approach. The product decision moves toward choosing the capabilities available for composition.
The second shift follows from the first: behavior can emerge from the composition of capabilities. Sometimes an interaction works even though nobody explicitly designed it, and sometimes it works in an unexpected way. Stein describes the building blocks as Lego bricks supplied to engineers, the product, or customers. Choosing those bricks and discovering what their combinations enable becomes part of the job. Functionality is something the team learns about after building, as well as something it specifies beforehand.
The third shift concerns communication. Even when a product manager recognizes the desired behavior, how does it become a development task or backlog item? A Figma mockup can show a layout; a product requirements document can describe a flow. Neither readily captures an instruction such as making Stacy sound less Gen Alpha or making her more polite. These are qualities of an interaction that unfolds over time. The team needs a way to express them that engineering can act on and product can inspect.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make evals part of the product specification
That need leads to evals. Stein’s informal audience poll finds engineers who write and run them, but far fewer product people who can see them or actively inspect them. His concern is the separation: the evidence that describes the product’s behavior is not necessarily reaching the people responsible for defining it.
A deterministic test can check that withdrawing $100 reduces a bank balance by $100. A behavioral test has a different shape. Was Stacy’s Slack reply snarky? Was it funny without being mean? Stein describes handing the reply and those criteria to another LLM, which judges whether the response meets the desired standard. Repeating that process gives the team a way to ask how often the behavior is acceptable. His illustrative tone criterion is acceptable snark without meanness 80% of the time; it is a proposed business threshold, not a measured result.
Evals become a shared specification of acceptable behavior. Stein calls them the only way to know what this kind of software can do, which makes product participation central rather than optional. Looking at the criteria and outcomes lets product people understand current capabilities and begin to specify desired ones.
There is an echo of behavior-driven development: business participants write tests, those tests become executable, and the tests describe the intended system. Stein remembers the gap between that ambition and people’s willingness to write the tests. He sees a stronger reason to participate here. With an agent, inspecting the eval is a direct way to find out what the product actually does, not merely a different format for a specification everyone already understands.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Experience the interaction before committing to it
Some criteria are hard to choose until you have experienced the behavior. Stein’s first specification encouraged teammates to ask many clarifying questions. On paper, that sounded helpful: clarification should improve understanding. In use, customers found the repeated questions annoying. The specification had captured a plausible idea, but missed the cost of living through it.
This is the constructive role Stein gives vibe coding in product work. A working prototype lets a product manager feel the timing, frequency, and character of an agent’s responses. It provides something to react to when a blank document is too abstract and a clickable prototype cannot adequately express the interaction. The prototype becomes a tool for exploring and communicating the experience.
The boundary matters. Building a prototype during a meeting does not invalidate an engineering estimate of two weeks. Stein explicitly treats these prototypes as work that will not go to production. Their purpose is to make the desired experience concrete, not to establish that the production implementation is finished.
Repeated use also exposes problems that a first impression hides. Stein recalls a period when Claude seemed to begin every answer with “Certainly.” A single response could feel reassuring. Across requests to do taxes and write an acceptance speech, the same opening became irritating. The quality being evaluated belongs to the sequence of interactions, not just an isolated answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Shipping does not end discovery
Pushing the product to production does not resolve the question of what it can do. Stein returns to discovery through an old QA joke. An engineer walks into a bar and orders one beer, two beers, zero beers, negative one beers, a lizard, and a beer with an emoji. The tests pass. Then the first customer asks where the bathroom is, and the bar explodes. The engineer explored unusual inputs inside an assumed interaction; the customer introduced an ordinary request outside it.
Agent exploration requires that second kind of imagination. What would happen if a teammate started commenting on LinkedIn? Could adding a track to a Spotify account trigger some useful action? These are ideas Stein proposes trying, not capabilities he demonstrates. They illustrate a practice of probing combinations that were absent from the initial requirements. The team supplies capabilities, tries interactions, and discovers where useful behavior appears.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Agree on what makes a bug
Discovery creates another practical problem: what should product or support escalate to engineering? Suppose Stacy uses too many emojis, and someone files a Linear ticket. Without an agreed criterion, engineering can reasonably ask where the specification defined too many. Stein jokes that ticket systems need a resolution category for LLMs behaving crazily, alongside closed and duplicate. Behind the joke is a real disagreement about whether an unwanted response is a bug, a preference, or ordinary variation.
Explicit acceptance criteria give the team a basis for that decision. Stein offers different thresholds for different kinds of expectations:
| Example | Acceptance rule | Consequence |
|---|---|---|
| Tolerant use case | 80% can be sufficient for some use cases. | Agree on the tolerance. |
| Release gate | An eval passing at least 90% permits shipping; below 90% blocks it. | Hold a failing release. |
| Refund policy | Require 100% compliance with refusing refunds when the customer cannot prove the purchase. | Treat a violation as a bug. |
These are illustrative product criteria, not reported Teammates performance. The refund example expresses a requirement; a perfect observed eval score would not prove universal compliance. What matters for bug assignment is that the team has stated the expected behavior before debating a particular failure.
The release decision itself can be a small, deterministic function once the behavioral judgments exist. For example, this TypeScript function applies Stein’s illustrative release threshold to a set of graded outcomes:
typescript
function releaseDecision(
outcomes: readonly boolean[]
): "ship" | "hold" {
if (outcomes.length === 0) return "hold";
const passed = outcomes.filter(Boolean).length;
const passRate = passed / outcomes.length;
return passRate >= 0.9 ? "ship" : "hold";
}
The function makes the gate explicit. The difficult product work remains defining what each outcome means: which task was attempted, what behavior was acceptable, and why that tolerance fits the use case. A complaint about tone becomes actionable when the team can connect it to such an agreed expectation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Bring customers into the discovery process
In a conventional customer meeting, a product manager can play two useful roles:
- Visionary: Explain the roadmap and show how the customer’s needs fit the product’s future.
- Honest broker: Clarify what works today and distinguish real capabilities from an ambitious sales pitch.
Both roles help the customer form expectations. The visionary makes the future credible; the honest broker makes the present concrete.
An open-ended agent makes both roles harder. A sweeping account of the future can sound like witchcraft to a customer who has seen unreliable AI products. A precise account of the present runs into the same uncertainty exposed by the Google Doc question. Stein cannot confidently promise the roadmap’s possibilities or enumerate everything that already works. He frames his response as an approach that is working for him in 2025, while leaving open whether it will be durable.
That approach is joint discovery. The customer and builder are inventing the future together, with the customer contributing real business contexts in which to learn what agents can do. Stein acknowledges that this language can sound flattering, but the participation is necessary: the builder cannot discover the whole product surface alone. It also sets an adoption boundary. A customer expecting a fully settled capability contract may not yet be ready for the kind of product he is offering.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model can change the product underneath you
The surprises can be useful as well as uncomfortable. Stein describes feeling unusually inept and unusually excited at the same time: he ships something and then sees it do something he had not anticipated. Upgrading the underlying models adds another source of discovery. He reports that agents began checking their own work—for example, running a query to verify that a database row had been inserted correctly—without the team explicitly instructing that behavior. The upgrade changed more than the quality of an answer; it changed the way the agent approached a task.
That is why Stein expects the product discipline to change faster than many teams anticipate. Capability discovery, behavioral specifications, and interaction prototypes address a product that can develop new behavior underneath its existing interface. The enduring work is still to listen to customers and solve real problems. The tools and techniques for deciding what to build, understanding what was built, and communicating its limits are what must change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Engineering principles for reliable LLM applications, covering prompts, context, control flow, human interaction, and focused agents.
Further reading
Google's announcement documents the comment text and sender information included in emails generated by document mentions.
Updates since the talk
A 2026 guide to evaluation tasks, repeated trials, graders, outcome checks, and collaboration between product and engineering teams.
Full transcript
Loading transcript…