AI Engineer World's Fair 2025

Stop Ordering AI Takeout: A Cookbook for Winning When You Build In-House

Jan Siml10:45

Loading slides…

Read the talk

Stop Ordering AI Takeout: A Cookbook for Winning When You Build In-House

An internal sales system shows how narrow scope, revenue instrumentation, proactive delivery and close user feedback can matter more than elaborate AI architecture.

From a talk by Jan Siml

The takeout trap

An AI strategy can resemble ordering takeout: choose something impressive online, pay a premium, and discover that the result falls short of the photographs. The engineering version starts with multi-agent graphs and the latest models because those are the recipes circulating on social media. An architecture designed to serve millions of customers, however, can be an expensive mismatch for one internal workflow.

Jan Siml’s team faced the build-or-buy decision in Q1 2024 and chose to build. Siml reports that two developers, with roughly ten-plus sprint weeks of effort, produced a system that resulted in several million dollars of annual recurring revenue and a group-level award. He withholds the exact financial figures. That is the business result behind the case study, rather than an independently established attribution of revenue to AI.

For this project, large evaluation efforts, multi-agent systems and reinforcement fine-tuned models threatened to increase cost and delay launch. The opening menu slide makes the temptation concrete: agentic RAG, synthetic-data evaluation, multi-agent systems and in-house models all look attractive as individual courses. The question is whether the internal job needs that entire menu.

Four menu courses beside a plated meal, with a yellow warning showing “12 months & $$$$ required!”Open full source frame
The Twitter AI Prix Fixe Menu lists agentic RAG, synthetic-data evaluation, multi-agent systems, and in-house models.
0:010:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:01 · section reference included

Build when the workflow is yours

SaaS resembles a hotel buffet: broadly useful, predictable and designed for many customers. An internal tool can instead fit the particular way a company works. Neither option wins in every situation; the advantage depends on which knowledge and capabilities are already available.

AdvantageSaaSIn-house
Data accessVendor integrationsData already owned
Task knowledgeCross-industry practicesExact deal-closing steps
Output checkingLess direct access to colleaguesUsers involved in review

The internal advantage is more than possession of a dataset. Colleagues know the keystrokes and decisions that turn information into a closed deal, and they can help double-check the system’s outputs.

Three comparison cards beneath a Swiss Army knife versus cutlery illustration contrast external SaaS with internal data, workflow knowledge, and colleague involvement.Open full source frame
SaaS and the home kitchen compared across data reach, task insight, and risk posture.

Proximity also shortens the development loop: a colleague can describe a problem, a tweak can ship the same day, and the interface can use familiar business language. Compute running on infrastructure the company already pays for can reduce incremental costs, although the talk gives no unit-cost breakdown. Siml’s rule is conditional: buy to explore an unfamiliar workflow; build once you understand and own it.

1:492:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:49 · section reference included

Start with one painful job

The foundation of the five lessons is depth. An internal tool does not need to expand its total addressable market; it can concentrate on one painful job. Choose a job with an identifiable value event: the dollar-based outcome that makes the work worthwhile. This gives the project a destination before its feature set starts growing.

The team began with sales alerts, then investigated what those alerts were supposed to accomplish. What happened after someone received an alert? What additional work was necessary? Asking users those questions let the team deepen the same use case rather than spread into unrelated features. A sufficiently focused workflow also kept the implementation simple enough to avoid agentic behavior.

3:123:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:12 · section reference included

Instrument the path to revenue

F1 and nDCG can help assess system quality, but they do not establish whether a sales workflow produces revenue. Siml compares evaluations to smoke alarms: they remain useful, but their readings are not the financial outcome. Measure the path from the AI task to the value event, not just the quality of an intermediate output.

That requires instrumentation across the revenue funnel, from its beginning through the final dollar-based outcome. Siml’s example of connecting an AI task to twenty dollars is an illustration of the desired attribution, not a reported transaction. In this internal workflow, users also double-check outputs, providing guardrails for experiments without requiring the team to perfect a large offline evaluation program first.

Once work is connected to revenue, prioritization becomes a discussion about an idea’s expected sales impact. That visibility creates another demand: managers want team performance reports. Automating those reports and adding leaderboards can serve several purposes:

  • Competition: Encourage teams to improve their results.
  • Leadership involvement: Make the workflow’s contribution visible to managers.
  • Champions: Identify people who use the system effectively.
  • Support: Find colleagues quietly struggling to adopt the new workflow.

The reports therefore help manage adoption as well as demonstrate results.

4:344:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:34 · section reference included

Deliver predictable work before users ask

Knowing the business workflow changes the interface requirement. If the system already knows what users are likely to need next, requiring a fresh question adds avoidable work. The team moved toward anticipating those needs and sending a daily digest containing what users needed to know that day.

InteractionRole
Daily digestDeliver predictable information proactively
ChatHandle unexpected and unplanned tasks

Chat remained available, but became the fallback rather than the required starting point. This division reduces the effort of obtaining routine information while preserving flexibility for exceptions. The freed time still has to become useful action.

6:106:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:10 · section reference included

Turn saved time into valuable action

Delivering information is only part of the job. The system must guide action, because saving thirty minutes accomplishes little if those minutes disappear into low-value email. As the revenue funnel becomes better understood, it reveals which activities deserve the user’s attention. That knowledge can shape what the system recommends next, converting time saved into time spent on higher-value work.

“Turn Time-Saved into Revenue” slide with two numbered principles beside a clock decorated with coins.Open full source frame
Guide action, not just share information; convert time-saved into time-well-spent.

The proactive system surfaced things users would not have thought to do. Siml reports that it had twenty points higher NPS and an order of magnitude higher engagement than the chat app. The talk does not supply the measurement period, sample size or engagement definition, so this remains a comparison from the team’s deployment rather than a general benchmark for proactive interfaces.

7:067:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:06 · section reference included

Spend development effort on useful data and triggers

With revenue coming through the system, the next constraint is where to invest limited development effort. Siml favors better data over more capable models. In a sales-alert workflow, that means improving what the system can notice and how closely it fits the work users need to complete.

Siml describes o3 as sixty times more expensive and an order of magnitude slower than GPT-4.1 mini. The talk does not specify the workload, pricing basis, token usage or latency methodology. This should not be read as a current API price comparison: published token rates changed with the June 2025 o3 price reduction, and billed reasoning tokens mean a task’s total cost need not match its per-token price ratio. Those distinctions do not establish the cost of this particular deployment.

The team’s best results came from adding more alert triggers and going deeper into user needs. When they switched between standard and mini model series, Siml says the changes appeared in costs and evaluation results, rather than the business outcome. For this workflow, expanding useful coverage was therefore a more productive investment than changing models. The allocation principle is to build what users need, rather than what developers are curious to try.

7:598:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:59 · section reference included

Let adoption improve the next experiment

Close user contact then becomes a repeating development loop. When users see their feedback reflected in the product, they feel heard and contribute more ideas. Those suggestions inform weekly experiments; useful experiments increase adoption; increased adoption supplies more data for prioritization and more suggestions for the next change. Siml describes this as a revenue flywheel: the system becomes more useful through repeated contact with the people doing the work.

9:039:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:03 · section reference included

Apply the lessons as one operating sequence

The closing recipe connects these choices rather than treating them as independent optimizations. Start with one painful job whose dollar value is clear. Instrument its final outcome so that development decisions follow revenue, then push the predictable insights directly to users. Guide the time that delivery saves toward the highest-value activities. Keep investing in the underlying data and user experience as users reveal what they need next. A small starting scope supplies the focus; following the money and listening to users determine how it grows.

“The Home Kitchen Advantage — Your Recipe for Success” recap with five labeled principles and explanatory text; Data+UX > Models is emphasized.Open full source frame
Five lessons: nail one high-value task, chase dollars, be the chef, turn time-saved into revenue, and prioritize data and UX over models.
9:4410:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:44 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…