AI Engineer World's Fair 2026

Why Off-the-Shelf AI Doesn't Understand Money

Udi Menkes19:50

Loading slides…

Read the talk

Why Off-the-Shelf AI Doesn't Understand Money

Financial advice can sound convincing while ignoring the constraints that keep a business alive. Udi Menkes shows how observed actions and outcomes can ground better recommendations.

From a talk by Udi Menkes

Before you start: Familiarity with LLMs and basic business concepts such as profit, revenue, and cash flow is helpful; the causal-inference concepts are introduced as they arise.

Just buy another car

When Udi Menkes scratched his car while parking, his daughter asked what had happened. He explained the problem. Her solution was immediate: “Just buy another one.” It solved the visible problem without considering what replacing a car would cost.

Financial advice from an LLM can have the same appealing simplicity. Menkes asks the audience who has used one for financial advice; many hands go up. When he asks who trusted the answer enough to follow it, many come down. His own experience involved choosing between a real-estate investment and a stock-market niche. He supplied his finances, relevant books, extensive context, and the latest models. The recommendation came back with convincing reasoning—until a small change in an assumption flipped it from A to B. Another small change flipped it back. The explanations remained persuasive while the judgment became difficult to trust.

0:000:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

Fix the first property's economics before buying another

Menkes introduces an Intuit study covering thousands of businesses and roughly 100,000 situations and time frames. Its first example is a new landlord with one rental property, negative cash flow, an outstanding loan, and profit trending into the red. Asked how to improve profit, a frontier model recommends acquiring a second property: more rental income would compensate for the existing deficit.

The model already had the business's data. The missing ingredient was therefore more specific than another financial statement in the prompt. Menkes contrasts that recommendation with one grounded in similar businesses, the actions they took, and what happened afterward: raise the existing tenant's rent by 5–10% before renewal. He says that change would move this particular case into profitability. The distinction is between adding another commitment while the business is under strain and improving the economics of the commitment it already has.

Slide contrasts a frontier model’s recommendation to acquire a second rental property with an outcome-grounded recommendation to raise rent 5–10% before renewal.Open full source frame
Two recommendations for a cash-flow-negative landlord: acquire another property or raise the existing tenant’s rent.
2:322:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:32 · section reference included

A price increase can put the whole business at risk

The next business is an egg supplier. One customer accounts for 70% of its revenue, and one vendor accounts for almost all its costs. The question is again how to improve profit, but the concentration changes which action is sensible.

Recommendation sourceProposed actionConstraint it addresses or exposes
Frontier modelRaise the main customer's prices 15–20%Risks losing the dominant revenue source
Model grounded in outcomesNegotiate vendor pricing down 5–10%Targets the dominant cost source

The second recommendation shifts attention from revenue to costs. It does not simply produce a more detailed version of generic growth advice; it selects a different action because losing that customer could damage the whole business. Both examples concern frontier models that, Menkes says, had the business context available.

4:104:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:10 · section reference included

Confident recommendations and the long game

Menkes calls this failure mode the fluent bluff: generic, confident financial advice learned from books, blogs, and advice columns without grounding in what happened when businesses acted. He extends that criticism broadly across financial answers. In the Intuit research he describes as forthcoming, 40% of recommendations centered on acquiring customers and another 14% on increasing product revenue. Together, those categories account for more than half the advice—a narrow repertoire for businesses facing very different constraints.

The problem becomes more visible when decisions compound over time. Menkes describes a recent Princeton study in which models received tools and business data, $1 million in starting capital, and a 500-day simulation horizon. He reports that most drove their companies bankrupt before the horizon ended, that a simple rules-based system beat almost all of them, and that even models with profitable runs could fail on reruns.

The study appears to be CEO-Bench: Can Agents Play the Long Game?, which operates a fictional subscription-software company and scores ending cash. That makes two distinctions essential: finishing below the starting balance is different from bankruptcy, and the original paper reports each model's best of three runs rather than a typical run. Later simulator changes also mean the current leaderboard is not the historical comparison described here. The narrower engineering question survives those distinctions: can an agent preserve sound business judgment across a sequence of consequential decisions, rather than produce one convincing answer?

5:145:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:14 · section reference included

A complete company snapshot is still not experience

Company context describes a business; experience connects situations, actions, and consequences. Even a complete QuickBooks record—general ledger, profit-and-loss statement, cash flows, and invoices—remains a collection of data points about one company. Reading those records is not the same as observing how comparable businesses responded to comparable decisions.

This distinction shapes Menkes's product mandate. He describes 15 years in AI and finance, moving from AI science and team leadership into product management, then joining Intuit about four years before the talk. As a principal product manager leading financial intelligence and advisory systems, his recurring question is what Intuit possesses that access to the latest model cannot replicate—and how to turn it into useful customer experiences.

An apparel company supplies another example. One vendor represents 80% of costs while enabling 97% of revenue. A textbook instruction to cut the biggest cost overlooks the dependency: removing that supplier could remove almost all the business it supports. An expense is not necessarily waste merely because it dominates the ledger.

The human analogy is choosing an advisor. One candidate has spent years guiding businesses through real decisions; another is exceptionally smart and has read all the textbooks but has no practical experience. The value of the first advisor lies in recognizing how an apparently sensible action can unfold under actual business constraints.

7:558:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:55 · section reference included

What did raising prices actually cause?

Experience is difficult to measure. Suppose a restaurant raises prices and becomes more profitable six months later. Did the price increase cause the improvement, or was the restaurant already on a successful trajectory? You cannot run the same business twice under identical conditions to observe both alternatives. Menkes describes comparing groups of similar businesses with similar propensities to raise prices, separating those that did from those that did not.

In the restaurant illustration, daily profit gains are $4,200 for businesses that raised prices and $2,800 for those that did not. The raw difference is $1,400 per day. That arithmetic is straightforward:

python

profit_gain_per_day = {
    "raised_prices": 4200,
    "did_not_raise_prices": 2800,
}

raw_gap = (
    profit_gain_per_day["raised_prices"]
    - profit_gain_per_day["did_not_raise_prices"]
)
print(f"Unadjusted difference: ${raw_gap:,}/day")

But the difference does not by itself isolate the action's effect. Businesses able to raise prices may already be more successful; part of their subsequent improvement may reflect that advantage.

Menkes gives about $1,150 per day as the illustrative adjusted effect, rather than the raw $1,400 difference. He calls the relevant quantity CATE, the conditional average treatment effect: the expected effect of an action for businesses with specified characteristics. The talk does not supply the estimator or adjustment needed to reproduce $1,150. Interpreting observational comparisons causally also requires assumptions about which factors influence both the action and the outcome; finding similar businesses alone does not establish causation.

Restaurant comparison shows +$4,200 per day profit for those raising prices and +$2,800 for similar restaurants that did not, with a naive difference of +$1,400 and an illustrative true effect of +$1,150.Open full source frame
The slide separates a naive profit difference from an adjusted action effect.
10:5911:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:59 · section reference included

Generate candidates, then evaluate them against outcomes

The useful data asset is a repeated pattern across entities: a situation, an action taken, and a subsequent outcome that can be verified. Menkes does not discard frontier models. He gives them a specific role: generate hypotheses and candidate actions. A model trained using reinforcement learning then evaluates which candidates are promising and which could damage the business. Generating an option and judging its consequences are separate jobs.

The training material comes from business trajectories. Menkes describes millions of them, built from Intuit data across QuickBooks, TurboTax, Credit Karma, and Mailchimp. The construction proceeds through three kinds of information:

  1. Build the state. Summarize a business at a particular point in time using its general ledger, profit-and-loss statement, cash-flow statements, and invoices.
  2. Derive the action. Infer what the business did from its records. A ledger may reveal investment in a marketing campaign, payments, payroll, or hiring costs.
  3. Measure the outcome. Observe subsequent profit, revenue, and cash flow over different time frames, including combinations of those measures.

These records become state–action–outcome vectors. The RL model learns which actions lead to better outcomes in situations involving similar businesses.

A subsequent training step teaches an LLM to generate the improved advice. The accompanying slide makes the sequence explicit: generate candidates, score them using an RL model, and GRPO-train the advisory model. The financial records therefore serve a different purpose from documents retrieved into a prompt: they supply evidence for evaluating candidate actions and training the system that produces recommendations.

Slide titled “The LLM proposes. Verified outcomes decide.” shows three panels: generate candidates, score them using an RL model, and GRPO-train the advisory model.Open full source frame
The LLM proposes candidates, an RL model scores outcomes, and GRPO trains the advisory model.
12:3312:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:33 · section reference included

From a trained model to an inspectable recommendation

The outcome-grounded recommendations in the opening examples came from a model developed with Intuit researchers using data across millions of small and medium businesses. Menkes reports that a cheaper, mid-sized model outperformed leading frontier models in head-to-head tests. He does not provide model identities, evaluation metrics, or performance margins, so the result is a reported finding rather than a reproducible comparison. Its product implication is that proprietary outcome data can matter more than access to a larger model.

At the time of the talk, Intuit had turned this work into an AI business advisor in a customer research preview. The advisor proactively surfaces opportunities, then explains why it recommends them: which businesses resemble the customer, what those businesses did, and how that experience informs the proposed action. Users can drill into the rationale and create action plans. The interface makes the reasoning available for inspection before a recommendation becomes a business decision.

15:0215:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:02 · section reference included

Ground the decision—and include the person making it

The same design question applies beyond finance: how can AI be steered toward the outcomes a domain actually values? Menkes points to anti-fraud systems, healthcare, logistics, and developer tools. He expects the advantage to accrue to organizations with strong systems of record that can derive distinctive datasets and train models against domain outcomes. Model selection remains part of the system, but the records determine which experiences it can learn from.

Grounded science is only part of a trusted advisory experience. A good advisor also understands a person's preferences and dislikes, and involves that person in the decision. An action can be well supported by comparable business outcomes without being the action a particular owner wants to take. The product must connect its evidence to the user's priorities and preserve their participation in choosing what happens next.

Coding provides a useful qualification to the claim that models lack experience: coding agents already make substantial use of verified outcomes. Menkes describes finance and other domains as comparatively unexplored. His proposed direction is to embed what worked at scale into training, rather than expect a larger model alone to close the gap.

The practical starting point is inside the existing records. After a brief invitation to enjoy the Fourth of July weekend, Menkes asks builders to look for situations repeated across entities, identify the actions those entities took, and find outcomes they can verify. That is the material from which an advisory system can begin learning what follows a decision—not merely how advice about that decision is usually phrased.

16:3116:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:31 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…