AI Engineer World's Fair 2025
Stop Ordering AI Takeout: A Cookbook for Winning When You Build In-House
Loading slides…
Read the talk
Stop Ordering AI Takeout: A Cookbook for Winning When You Build In-House
An internal sales system shows how narrow scope, revenue instrumentation, proactive delivery and close user feedback can matter more than elaborate AI architecture.
From a talk by Jan Siml
The takeout trap
An AI strategy can resemble ordering takeout: choose something impressive online, pay a premium, and discover that the result falls short of the photographs. The engineering version starts with multi-agent graphs and the latest models because those are the recipes circulating on social media. An architecture designed to serve millions of customers, however, can be an expensive mismatch for one internal workflow.
Jan Siml’s team faced the build-or-buy decision in Q1 2024 and chose to build. Siml reports that two developers, with roughly ten-plus sprint weeks of effort, produced a system that resulted in several million dollars of annual recurring revenue and a group-level award. He withholds the exact financial figures. That is the business result behind the case study, rather than an independently established attribution of revenue to AI.
For this project, large evaluation efforts, multi-agent systems and reinforcement fine-tuned models threatened to increase cost and delay launch. The opening menu slide makes the temptation concrete: agentic RAG, synthetic-data evaluation, multi-agent systems and in-house models all look attractive as individual courses. The question is whether the internal job needs that entire menu.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Build when the workflow is yours
SaaS resembles a hotel buffet: broadly useful, predictable and designed for many customers. An internal tool can instead fit the particular way a company works. Neither option wins in every situation; the advantage depends on which knowledge and capabilities are already available.
| Advantage | SaaS | In-house |
|---|---|---|
| Data access | Vendor integrations | Data already owned |
| Task knowledge | Cross-industry practices | Exact deal-closing steps |
| Output checking | Less direct access to colleagues | Users involved in review |
The internal advantage is more than possession of a dataset. Colleagues know the keystrokes and decisions that turn information into a closed deal, and they can help double-check the system’s outputs.
Proximity also shortens the development loop: a colleague can describe a problem, a tweak can ship the same day, and the interface can use familiar business language. Compute running on infrastructure the company already pays for can reduce incremental costs, although the talk gives no unit-cost breakdown. Siml’s rule is conditional: buy to explore an unfamiliar workflow; build once you understand and own it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with one painful job
The foundation of the five lessons is depth. An internal tool does not need to expand its total addressable market; it can concentrate on one painful job. Choose a job with an identifiable value event: the dollar-based outcome that makes the work worthwhile. This gives the project a destination before its feature set starts growing.
The team began with sales alerts, then investigated what those alerts were supposed to accomplish. What happened after someone received an alert? What additional work was necessary? Asking users those questions let the team deepen the same use case rather than spread into unrelated features. A sufficiently focused workflow also kept the implementation simple enough to avoid agentic behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Instrument the path to revenue
F1 and nDCG can help assess system quality, but they do not establish whether a sales workflow produces revenue. Siml compares evaluations to smoke alarms: they remain useful, but their readings are not the financial outcome. Measure the path from the AI task to the value event, not just the quality of an intermediate output.
That requires instrumentation across the revenue funnel, from its beginning through the final dollar-based outcome. Siml’s example of connecting an AI task to twenty dollars is an illustration of the desired attribution, not a reported transaction. In this internal workflow, users also double-check outputs, providing guardrails for experiments without requiring the team to perfect a large offline evaluation program first.
Once work is connected to revenue, prioritization becomes a discussion about an idea’s expected sales impact. That visibility creates another demand: managers want team performance reports. Automating those reports and adding leaderboards can serve several purposes:
- Competition: Encourage teams to improve their results.
- Leadership involvement: Make the workflow’s contribution visible to managers.
- Champions: Identify people who use the system effectively.
- Support: Find colleagues quietly struggling to adopt the new workflow.
The reports therefore help manage adoption as well as demonstrate results.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deliver predictable work before users ask
Knowing the business workflow changes the interface requirement. If the system already knows what users are likely to need next, requiring a fresh question adds avoidable work. The team moved toward anticipating those needs and sending a daily digest containing what users needed to know that day.
| Interaction | Role |
|---|---|
| Daily digest | Deliver predictable information proactively |
| Chat | Handle unexpected and unplanned tasks |
Chat remained available, but became the fallback rather than the required starting point. This division reduces the effort of obtaining routine information while preserving flexibility for exceptions. The freed time still has to become useful action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn saved time into valuable action
Delivering information is only part of the job. The system must guide action, because saving thirty minutes accomplishes little if those minutes disappear into low-value email. As the revenue funnel becomes better understood, it reveals which activities deserve the user’s attention. That knowledge can shape what the system recommends next, converting time saved into time spent on higher-value work.
The proactive system surfaced things users would not have thought to do. Siml reports that it had twenty points higher NPS and an order of magnitude higher engagement than the chat app. The talk does not supply the measurement period, sample size or engagement definition, so this remains a comparison from the team’s deployment rather than a general benchmark for proactive interfaces.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend development effort on useful data and triggers
With revenue coming through the system, the next constraint is where to invest limited development effort. Siml favors better data over more capable models. In a sales-alert workflow, that means improving what the system can notice and how closely it fits the work users need to complete.
Siml describes o3 as sixty times more expensive and an order of magnitude slower than GPT-4.1 mini. The talk does not specify the workload, pricing basis, token usage or latency methodology. This should not be read as a current API price comparison: published token rates changed with the June 2025 o3 price reduction, and billed reasoning tokens mean a task’s total cost need not match its per-token price ratio. Those distinctions do not establish the cost of this particular deployment.
The team’s best results came from adding more alert triggers and going deeper into user needs. When they switched between standard and mini model series, Siml says the changes appeared in costs and evaluation results, rather than the business outcome. For this workflow, expanding useful coverage was therefore a more productive investment than changing models. The allocation principle is to build what users need, rather than what developers are curious to try.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Let adoption improve the next experiment
Close user contact then becomes a repeating development loop. When users see their feedback reflected in the product, they feel heard and contribute more ideas. Those suggestions inform weekly experiments; useful experiments increase adoption; increased adoption supplies more data for prioritization and more suggestions for the next change. Siml describes this as a revenue flywheel: the system becomes more useful through repeated contact with the people doing the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Apply the lessons as one operating sequence
The closing recipe connects these choices rather than treating them as independent optimizations. Start with one painful job whose dollar value is clear. Instrument its final outcome so that development decisions follow revenue, then push the predictable insights directly to users. Guide the time that delivery saves toward the highest-value activities. Keep investing in the underlying data and user experience as users reveal what they need next. A small starting scope supplies the focus; following the money and listening to users determine how it grows.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Original introduction to GPT-4.1, mini and nano, including launch pricing and evaluation results.
OpenAI's introduction to its reasoning models, capabilities and original API access.
Further reading
- Understanding token usage and billingDocumentation
Explains input, cached and reasoning tokens and how they affect total request cost.
The speaker's conference reflections expand on proactive recommendations and practical evaluation choices.
Updates since the talk
- June 2025 o3 price reductionArticle
Dated announcement of an 80% reduction in o3 API token prices.
Full transcript
Loading transcript…