AI Engineer World's Fair 2025
Building AI Products That Actually Work
Loading slides…
Read the talk
Building AI Products That Actually Work
Reliable AI products emerge from observing real user intent, detecting specific failures, and turning recurring tasks into workflows that teams can improve independently.
From a talk by Ben Hylak and Sid Bendre
What should you improve after the demo works?
How do you iterate on an AI product when another evaluation score cannot tell you what to fix? Before this talk, Ben Hylak asked his audience what he should cover. The response was emphatic: less about evals. The practical question underneath that request is how to make a product better after its initial promise is already visible. Iteration is the work that turns model capability into a useful product.
Hylak approaches that problem as CTO of Raindrop, which helps teams find and fix issues in AI products. His background spans robotics, SpaceX avionics, and nearly four years in engineering and design at Apple. Joining him is Sid Bendre, co-founder of Oleve; Hylak introduces Oleve as a four-person team whose suite of viral apps reached more than $6 million in annual recurring revenue. Their perspectives connect production monitoring with the practical constraints of a small consumer-product team.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A correct answer can still miss the task
Focused AI products show how much is possible when the task is well defined. Small models can become exceptionally capable at narrow use cases, and model providers increasingly compete by building those products themselves. Hylak points to ChatGPT deep research as an example of concentrating data collection and training on web research, producing an experience he considers one of OpenAI’s strongest releases.
But capability does not guarantee that the product understands the assignment. In Hylak’s experience with Codex at the time of the talk, asking it to write backend tests produced a correctly generated hash for hello. The calculation was right; the choice of what to test missed his intent. That distinction matters because a technically valid output can leave the user’s actual problem untouched.
Other examples expose different kinds of missing context. Hylak recounts a Virgin Money chatbot threatening to cut customers off for using the word virgin. When he asked Google Cloud where his credits were, its assistant asked whether he meant Azure or Roblox credits. Another user told him the same thing had happened to them. The surrounding product should have helped constrain the interpretation, but the assistant behaved as though that context were absent.
Grok supplied two further failures. In one public interaction Hylak describes, a question about enterprise software veered into claims of genocide in South Africa. Public conversations made that behavior visible. In his own interaction while preparing the talk, Grok initially said it could not search his tweets. After he challenged that denial, it acknowledged that it could search and asked for his username. The useful next step had been available, but the assistant first denied the capability needed to take it.
Raindrop’s observation base spans Oleve’s apps, Clay’s sales-outreach tooling, alien companion apps, and coding assistants. Hylak also describes an internal AI pipeline that analyzes and categorizes every event Raindrop receives. Its Deep Search feature explores production data and builds classifiers from a few examples. Those customer interactions and internal systems are the basis for his recommendations about discovering failures, rather than assuming all important failures are known in advance.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Better models still need context
Will building AI products get easier? Some parts already have. Hylak contrasts elaborate prompting to make GPT-4 emit JSON with specifying an output schema through an API parameter. Structured Outputs illustrates that historical shift, but the guarantee is structural: supported schemas, refusals, and incomplete generations still require handling, and valid JSON can contain incorrect values. Removing formatting friction does not resolve the user’s intent.
Communication remains difficult even between intelligent humans. Hylak disputes a claim he attributes to Paul Graham: that AGI would end prompt engineering because moderately intelligent people can infer what someone wants without elaborate instructions. Partners misunderstand each other; new hires complete assignments differently from what their managers intended. Intelligence does not automatically supply the missing context that makes an instruction unambiguous.
Additional capability also expands the space of behavior a team must understand. Integrations through MCP introduce more tools, data formats, and ways to act, along with edge cases the product team has not anticipated. A product requirements document can no longer enumerate the entire behavior space. The practical loop becomes shipping a product, observing what it actually does, and revising it in response.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluations cover the failures you know
Evaluations remain important in that loop, but their interpretation matters. Hylak invokes Goodhart’s Law to challenge the idea that an eval score tells you how good the whole product is. A collected evaluation set represents cases somebody already knew to include; repeated optimization can saturate it without exhausting the problems users encounter. He reports seeing model releases that performed worse on evals yet felt better in real use, without naming a benchmark or model comparison. His point is about the gap between the measured cases and the product’s actual use.
The same gap appears when a team asks a language model to score something subjective, such as how funny a generated joke is. A judge that accepts text and returns a number is convenient, but convenience does not establish that its score captures the experience people care about. Hylak recommends highly curated datasets and auto-gradable evaluations: checks with a deterministic way to establish whether the output passed. His observation that leading companies largely avoid generic model judges is qualified, with exceptions; it is a recommendation about what to reach for first.
Moving the same judges and scoring rules onto production traffic does not remove those limitations. A sufficiently capable judge may be expensive enough to force sampling a small fraction of traffic. Accurate setup is difficult, and a system designed around known failures can still miss new patterns. Hylak cites an OpenAI behavioral-issue postmortem to emphasize that real-world use reveals problems evaluations miss. That makes production observation complementary to evaluation, rather than a reason to abandon it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Combine signals with user intent
Traditional error monitoring gives you an issue, an occurrence count, and a count of affected users. An AI interaction can fail without throwing an exception at all. To build a comparable picture, Hylak proposes signals: indicators grounded in what happens in the product. An AI issue emerges from combining explicit or implicit signals with the user’s intent. The operating cycle is to define those signals, explore them, and refine them.
Explicit signals are events the application can send. Thumbs up and thumbs down are only the beginning. Hylak says ChatGPT also tracks which portion of an answer a user copies, and points to paired response preferences as another source of feedback. Different interactions expose different evidence:
| Signal | Observable event |
|---|---|
| Direct feedback | Thumbs up or down; preferred response |
| Retry behavior | Response regenerated |
| Technical failure | Error or coding syntax error |
| Reuse | Content copied or shared |
These events provide concrete behavior to inspect, even when none is a complete measure of answer quality.
Raindrop applies this idea to its own search experience. Users mark results correct or wrong, and Hylak describes those labels as feedback that can support reinforcement learning and improve search quality. Implicit signals come from detecting events within the interaction itself: refusals, task failure, or user frustration. Detecting a specific issue is a narrower question than judging the overall quality of a response.
Intent gives those detections a useful shape. A collection of frustrated conversations is difficult to act on; a cluster of frustrated users trying to search for tweets identifies a particular capability to investigate. That is how the earlier Grok example becomes an issue category instead of remaining an isolated anecdote.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make issue discovery a continuous practice
Exploration requires more than a feed of individual conversations. As with tags in Sentry, teams need to navigate properties and metadata: models, keywords, and especially intents. The intent changes what a signal means and therefore changes the issue being investigated. A refusal, for example, is not sufficiently specific without understanding what the user was trying to do.
Hylak recommends a continuous flow of application data into the team’s daily work. Raindrop uses Slack notifications; searching production data is another route. The mechanism is sustained exposure: inspect patterns, talk to users, define an unexpected issue precisely, and start tracking it. The definitions should evolve as the team learns, rather than freezing at the categories it anticipated before launch.
This part of the talk stops at discovery and refinement. Hylak sets aside a planned discussion of fixes and supervised fine-tuning, handing over to Bendre for a framework that organizes product improvement around the behaviors users actually want.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Guide the variability that makes the product compelling
At Oleve, the goal is to build consumer products that are fulfilling and productive. Bendre reports that the small New York team reached around $6 million in ARR profitably and generated about half a billion social views. He introduces its improvement framework as powered by Raindrop. Those are founder-reported business results, rather than a measurement of the framework’s causal effect.
A viral AI product needs both a wow factor that attracts attention and a reliable experience that keeps working. Nondeterministic behavior contributes to the appeal, so eliminating all variability would undermine part of the product. Trellis is Oleve’s approach to guiding that variability while making improvement systematic, designed around its virality engine.
Trellis rests on three operations:
- Discretize: Break the open-ended output space into specific buckets that a team can work on.
- Prioritize: Rank those buckets by their potential business impact.
- Recursively refine: Repeat the process inside each bucket, introducing more structure where the evidence warrants it.
The product becomes more understandable in layers, without requiring the team to specify every possible interaction at the outset.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn observed intent into dedicated workflows
Bendre translates those axioms into a six-step procedure:
- Launch an MVP agent. Start with product priors and expectations, but use the initial experience to collect real user data.
- Classify usage into intents. Identify what people actually use the product for and why they keep returning, especially in an open-ended conversational interface.
- Build dedicated workflows. Convert recurring intents into predefined steps. Keep each workflow broad enough to serve multiple possibilities and narrow enough to be reliable.
- Prioritize against company KPIs. Use a scoring mechanism connected to the outcomes the business needs.
- Analyze within each workflow. Inspect failure patterns and distinguish more specific sub-intents.
- Recurse. Apply the same process to those smaller spaces.
The key transition is from an unrestricted agent experience to semi-deterministic workflows. The sequence of steps supplies structure while leaving room for useful variation in the generated output.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Rank improvements by impact you can achieve
Volume is an obvious first ranking: improve the workflows people use most. But high usage alone does not tell you where satisfaction could improve most. Bendre recommends incorporating negative sentiment, then refining the score further with feasibility and strategic relevance.
| Ranking | Factors |
|---|---|
| Volume only | Usage volume |
| Dissatisfaction weighted | Volume × negative sentiment |
| Feasibility and strategy weighted | Volume × negative sentiment × achievable delta × strategic relevance |
The additional factors distinguish a common workflow from one where the team can deliver a meaningful improvement.
Estimated achievable delta means the improvement the team can realistically produce. A highly frustrating workflow may still be a poor use of effort if fixing it requires training a foundation model that the company cannot afford to build. In that situation, Bendre says the achievable delta may be near zero. The score is a prioritization heuristic, with estimates supplied by the team, rather than a measured forecast of uplift.
The multiplication is straightforward to express in TypeScript. This implementation makes the four inputs explicit and returns a newly ranked array, so the assumptions behind each workflow’s priority remain inspectable:
typescript
type WorkflowEstimate = {
id: string;
volume: number;
negativeSentiment: number;
achievableDelta: number;
strategicRelevance: number;
};
function rankWorkflows(workflows: readonly WorkflowEstimate[]) {
return workflows
.map((workflow) => ({
...workflow,
priority:
workflow.volume *
workflow.negativeSentiment *
workflow.achievableDelta *
workflow.strategicRelevance,
}))
.sort((a, b) => b.priority - a.priority);
}
Use consistent definitions and scales across the workflows being compared. The function calculates a ranking; it does not supply the judgment needed to estimate the inputs.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Keep changes accountable to one workflow
The payoff of dedicated workflows is containment. Bendre describes them as self-attributable, deterministic, and self-bound: improving one workflow should keep the change and its effects accountable to that workflow instead of spilling into unrelated experiences. In the earlier semi-deterministic framing, the useful boundary is the structured process around the generated output. It gives the team a smaller unit to change, inspect, and test.
Refinement can then continue deeper within each workflow as new sub-intents and failure patterns emerge. The desired result is an experience whose magic is engineered, repeatable, testable, and attributable—not accidental. Bendre closes by pointing viewers to a blog post on Trellis, leaving workflow isolation and recursive refinement as the mechanism for sustaining that result.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
The framework's creator explains intent segmentation, workflow prioritization, and recursive refinement, with a link to the Oleve playbook.
Historical API examples and an explanation of constrained decoding, schema support, and the limits of structural guarantees.
Updates since the talk
The current documentation entry point for adding Raindrop monitoring to an AI application.
Full transcript
Loading transcript…