AI Engineer World's Fair 2025

How Intuit uses LLMs to explain taxes to millions of taxpayers

Loading slides…

Read the talk

How Intuit uses LLMs to explain taxes to millions of taxpayers

TurboTax combines prepared prompts, retrieval, tax experts and layered evaluations to explain personal tax outcomes while keeping calculations in its tax engine.

From a talk by Jaspreet Singh

Explain the tax outcome

How do you help someone understand their taxes well enough to trust the outcome? For Jaspreet Singh, a senior staff engineer working on generative AI for TurboTax, that question comes with substantial scale. Intuit reports that TurboTax processed 44 million tax returns for tax year 2023. That is filing volume, not a count of users receiving LLM explanations.

Slide stating that Intuit TurboTax successfully processed 44 million tax returns last year, with the presenter inset below.Open full source frame
Intuit TurboTax processed 44 million tax returns last year.

The experience begins with information the taxpayer enters into TurboTax. As users review which credits they qualify for, explanations expand on the tax breaks they are receiving. A second experience explains the overall refund. The product goal is to make those results understandable and give taxpayers confidence that they are receiving the deductions available to them.

0:160:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Shared infrastructure for a regulated product

Intuit builds these experiences on GenOS, its internal generative AI platform. Off-the-shelf tooling did not cover all of its use cases, particularly the safety and security requirements of tax software. The platform also had to support teams across a company operating at Intuit’s scale.

GenUX supplies UI capabilities. The Orchestrator connects solutions built by different teams so the system can find an appropriate way to answer a question. Intuit Assist is the customer-facing experience powered by these capabilities; the TurboTax explanations are one application of that shared infrastructure.

Intuit GenOS diagram showing Products, GenUX, GenStudio and GenRuntime, including GenOrchestrator with Planner, Agents & Tools, and Memory.Open full source frame
Intuit GenOS platform architecture.
1:371:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:37 · section reference included

Prepared explanations and open-ended questions

The first implementation used prompt tooling. A refund explanation has to account for several constituents: deductions, credits, the standard deduction and W-2 withholding. The team built a prompt-based solution to explain that information and selected Claude for the production use case. Singh reports a multimillion-dollar Claude contract for that year, underscoring the commercial scale behind the model choice.

Two kinds of request call for different approaches:

RequestWhat is known in advanceWhat arrives from the user
Static explanationThe task, such as explaining a refund summaryPersonal tax information
Dynamic questionThe broad question-answering capabilityThe question and its relevant context

A static prompt resembles a prepared statement: the application already knows what explanation it needs, but fills in the taxpayer’s information when they arrive. Dynamic questions are less predictable. Singh’s dog-deduction example illustrates that range of input rather than establishing a general tax-law rule. GPT-4o mini had been the choice for dynamic question answering until a few months before the talk; the team was already evaluating newer versions.

Retrieval-augmented generation, including GraphRAG, brings in tax information that cannot be treated as fixed model knowledge. IRS forms change annually, and Intuit has proprietary tax information and tax engines to incorporate into answers.

The team also piloted fine-tuning Claude for static queries. Singh describes good response quality, but the work required effort and produced a model that seemed too specialized for the particular use case. Whether changing prompts or adapting a model, the team evaluates both development results and behavior in production.

3:033:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:03 · section reference included

Tax experts write prompts and establish the baseline

Tax analysts form the foundation of the development process. They interpret annual IRS changes, supply domain knowledge and perform the initial manual evaluations. Intuit also uses these analysts as prompt engineers, putting the people who understand the tax rules directly in charge of expressing the explanation requirements.

That division of work lets data science and ML teams concentrate on quality metrics and reusable test datasets. Those datasets support iteration as both models and tax rules change—for example, when moving from tax year 2023 to 2024. Expert-written prompts and initial human judgments then become the basis for automated evaluation with an LLM judge.

Pillars of Development diagram with LLM fine-tuning, context store and retrieval, and prompt iterations above automated evaluation and a green Human Domain Expert-in-the-Loop foundation.Open full source frame
Human domain experts underpin the development and evaluation pillars.
5:526:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:52 · section reference included

Fine-tuning to reduce the instruction burden

As GenOS added fine-tuning tooling, the TurboTax experiment used Claude 3 Haiku through Amazon Bedrock. The intended benefit was fewer instructions at inference time: a model adapted to the task might need a smaller prompt while preserving response quality. Latency motivated that reduction, although Singh does not give a measured latency improvement from the pilot.

Platform teams provided separate test AWS accounts and environments. For the data used in this work, Singh says the team used only consented user data, citing Section 7216. Consent was therefore a constraint on the experiment’s inputs, alongside the technical work of preparing and evaluating the model.

7:257:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:25 · section reference included

Evaluate the answer and the evaluator

The evaluation pillars are accuracy, relevancy and coherence. Manual and automated evaluations cover development, while automated production monitoring samples responses given to real users. This makes evaluation an ongoing check on the deployed experience, not just a test before launch.

The automated judge itself runs on a prompt. Intuit’s in-house automated prompt-engineering tools help update that judge prompt using manually evaluated samples—the golden dataset. Singh identifies AWS Ground Truth as the tool used for those human labels.

Clear evaluations supported the move from Claude Instant to Claude Haiku for tax year 2024. Even an upgrade within the same vendor required substantial testing; a new model was not a transparent replacement. Across those checks, tax accuracy remained the central target.

8:298:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:29 · section reference included

Contracts and latency shape the product

Large model contracts are expensive, and longer commitments can lower costs while increasing dependence on the vendor. Strong vendor partnerships help the team iterate, but the contract is only one switching cost. Prompts also create lock-in: instructions developed for one model may need significant work even when upgrading to another model from the same provider.

Latency creates a separate product constraint. Singh contrasts illustrative LLM waits of 3, 5 or 10 seconds with backend expectations of 100–200 milliseconds. These are examples of the responsiveness gap, not benchmark percentiles or service guarantees. A taxpayer with a home, stock holdings and jobs for both spouses brings a larger tax context, which can make the explanation prompt balloon. Filing-day demand around April 15 compounds the problem.

The product has to accommodate that behavior through fallback mechanisms and interaction design, although Singh does not specify the fallback implementations. The goal is an experience that remains useful while delivering helpful explanations. Clear evaluation guidelines and a golden dataset are launch requirements, rather than work to defer until after the feature ships.

10:0910:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:09 · section reference included

How much evaluation does a prompt change need?

The first audience question turns the requirement to evaluate everything into an operational problem: if small prompt changes trigger expensive reviews, how can development keep moving? Singh describes a phased approach rather than sending every iteration through the same process.

  1. Establish a baseline manually. During initial development, tax experts evaluate responses and define what acceptable behavior looks like.
  2. Use expert judgments to refine the judge prompt. Minor prompt iterations can then use automated evaluation. This is refinement of the evaluator’s instructions, not necessarily training its model weights. The judge had used GPT-4-series models until recently and remained a meaningful cost.
  3. Return to manual evaluation for major changes. Product and engineering align on quality expectations; tax-year transitions and substantial prompt revisions warrant renewed expert review.

This changes the evaluation method as the work changes. It does not establish a numerical release threshold or an exemption from evaluation for small edits.

12:3012:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:30 · section reference included

Route questions by what the user needs

The next question asks where these interactions fit into the product. Question answering spans both help using TurboTax and questions about a personal tax situation. A user might need instructions for completing an action in the software, or ask whether tuition paid for a grandchild can be claimed on their taxes. The latter is an example of the question category; Singh does not answer that tax question here.

Different teams build solutions for those different needs. The Planner interprets what an incoming question is asking and selects a suitable solution. That routing role connects the broad ambition to answer user questions with the specialized components available inside the platform.

13:5714:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:57 · section reference included

Keep calculated numbers separate from generated language

An audience member raises a more concrete failure case: an authoritative tax amount could appear in an answer as $11,100. The difference may be easy to overlook in otherwise plausible prose. Singh first identifies the source of the numbers: TurboTax’s proprietary Tax Knowledge Engine, maintained and developed in-house.

The LLM does not perform the tax calculations in this workflow. It receives numbers and tax-profile information already established in Intuit’s systems. Safety guardrails then inspect the raw generated response before it reaches the user, looking for hallucinated numbers.

The follow-up distinguishes having correct inputs from preserving them in the final explanation. Singh says ML models within the security controls check for numerical hallucinations in that output. He does not describe the comparison algorithm or provide a detection rate, so this is a description of an additional checking layer, not a guarantee that every numerical error is caught.

15:1315:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:13 · section reference included

Personalization matters beyond retrieval choice

Asked how traditional RAG and GraphRAG work together, and whether Claude 4 might change the need for fine-tuning, Singh starts with answer quality. In the team’s experience, he reports better response quality with GraphRAG than with regular RAG, while emphasizing personalization as an even greater contributor to user helpfulness. The comparison is qualitative: he supplies neither a hybrid retrieval architecture nor comparative measurements and evaluation conditions.

The next model choice remained open. With the April filing season behind them, the team was evaluating newer models as well as models trained and developed internally at Intuit. Singh did not announce a choice for the following tax year or conclude that the newer Claude release would make fine-tuning easier.

16:4216:53
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:42 · section reference included

Assurance depends on the scope of the explanation

The final question asks how the system handles a complicated situation: does the answer come from the LLM or the tax engine, how can its derivation be explained, and what happens when a wrong answer creates legal risk? Singh emphasizes Intuit’s legal and privacy controls, then narrows his response to the static explanation solution presented in the talk.

For that solution, the Tax Knowledge Engine supplies the underlying numbers, tax experts craft the prompts, and evaluations test the specific explanation components. The assurance rests on that bounded workflow and its tests. It does not extend into a universal correctness or legal-liability guarantee for every open-ended tax question.

17:4918:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:49 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…