AI Engineer World's Fair 2025

Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier - Andrew Thompson

Loading slides…

Read the talk

Buy Now, Maybe Pay Later: Managing the Prompt Tax of Model Upgrades

New models can unlock better agentic products while breaking prompts that already work. Orbital’s experience shows how prompt design, domain expertise, and progressive delivery shape that tradeoff.

From a talk by Andrew Thompson

Before you start: Familiarity with LLM prompts, tool-using agents, and basic software release practices will help you follow the migration and deployment examples.

A new model is a new bike

How do you adopt a more capable model without breaking the agentic product your users already depend on? Andrew Thompson calls the work and uncertainty surrounding that decision prompt tax. He opens with Peter Gostev of Moonpig’s animated chart of model-company Elo scores: successive releases keep moving the frontier. From a developer’s perspective, the stream of releases from Anthropic, Google’s Gemini, and OpenAI can feel like having a birthday every month.

Getting a new model is like getting a new bicycle. First you imagine speed, freedom, and trips to a friend’s house. Then you try riding it, fall off, and discover an unexpected cost. A model offers new capabilities you can incorporate into an application, but its probabilistic behavior can also produce surprises. The opportunity to improve a product arrives together with the risk of regressions in behavior that already works.

Illustration of a crying child beside a fallen bicycle, with the text “It’s amazing but you also encounter unexpected things” and a presenter inset.Open full source frame
New capabilities can bring unexpected problems.
0:000:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

From legal paperwork to a reviewable lease report

At Orbital, where Thompson introduces himself as CTO, the customer problem is real-estate due diligence. Before homes, offices, and development sites can change hands, lawyers search substantial document collections for red flags affecting their clients. Orbital, with offices in New York and London, builds agentic software to find that information and assemble the paperwork needed to progress a transaction.

Thompson reports roughly 80 employees, with about half in product engineering. The team photograph happens to come from an off-site in Mallorca rather than either office. Delivery teams combine a product manager and designer with embedded domain experts, software engineers, and AI engineers, under a tech lead. Legal expertise sits inside the team building the product, rather than appearing only at the end to review it.

By January 2024, the company had developed Orbital Copilot, its first agentic product, and started selling it to law firms and real-estate professionals. The demonstration follows an occupational lease report from document intake to review. It runs at accelerated speed, so the recording shows the workflow rather than real-time processing latency.

  1. Choose the report and upload the evidence. The selected occupational lease report receives a deed and a lease. Thompson estimates their lengths at about 30 and 70 pages, approximately 100 pages in total.
  2. Prepare the documents. OCR extracts handwritten and typed text, which must be structured before the legal analysis begins.
  3. Decompose the work. The system creates a plan and breaks it into subtasks. Each subtask is itself an agentic system involving multiple LLM calls.
  4. Extract the required facts. Objectives include finding the lease date and current annual rent. The system reads the documents shown on the right of the interface, and completed tasks turn green on the left.
  5. Review against the source. A lawyer reviews the assembled report. Clickable citations let a licensed real-estate lawyer return to the underlying documents and check the extracted information.
Review interface with lease fields on the left, extracted landlord and tenant details in the center, and a highlighted section of the source lease on the right.Open full source frame
Lease report review alongside the source document.

The final step downloads a Word report for storage and eventual delivery to the client. The workflow automates document preparation and extraction while retaining a review step in which the lawyer can inspect the evidence behind the report.

1:522:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:52 · section reference included

Growing through successive model generations

Thompson reports that, over roughly 18 months after commercialization, Orbital’s monthly token consumption grew from under one billion to nearly 20 billion. He also reports annual recurring revenue growing from zero to multiple seven figures over that period. These are company usage and commercial figures; token volume does not establish an equivalent quantity of human labor saved.

That growth involved repeated model migrations. Orbital began with GPT-3.5, then adopted GPT-4 32K, GPT-4 Turbo, GPT-4o, and GPT-4.1. GPT-4 32K was particularly consequential: Thompson remembers it as extremely expensive, but its larger context window made their agentic system possible. Alongside these models, the team adopted reasoning models from o1-preview through o4-mini, including intervening releases. He calls the two groups system-one and system-two models, respectively.

Three operating choices shaped how the team moved through those releases:

  • Prompting over fine-tuning. The priority was development speed. User feedback could become a prompt edit, and the running agentic system could pull in the revised prompt in real time. That shortened iteration while searching for product-market fit and remained useful for shipping features afterward.
  • Embedded legal expertise. Former private-practice real-estate lawyers wrote many of the prompts. Their job was to turn decades of professional experience into instructions the system could use.
  • “Vibes over evals.” Orbital wanted a comprehensive evaluation system, but it remained on the roadmap. Release assessment depended on people—usually domain experts—trying the system, judging its behavior, and sometimes recording possible regressions in spreadsheets. This was testing, but it was subjective and incomplete rather than a rigorous, objective evaluation system.
5:275:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:27 · section reference included

A growing prompt portfolio makes upgrades consequential

The prompt collection has two distinct owners and purposes:

Prompt categoryPrimary ownerResponsibility
Agentic promptsAI engineersSystem behavior, tool selection, and when to use tools
Domain-specific promptsReal-estate lawyersLegal expertise and task-specific instructions

Thompson reports that the domain-specific portfolio grew from near zero to well over 1,000 prompts. Each new prompt expands the behavior that may need attention during a migration: more prompts mean more prompt tax.

When a new model arrives, AI engineers and domain experts experiment with it during the release week. They first ask whether it can now support features they had already envisioned, or whether its behavior suggests new product ideas. Then they estimate the work needed to migrate the existing prompts. Finally, they examine the fear surrounding the release: what might change, which concerns are rational, and which risks can be mitigated? The decision still has to be made without knowing every possible outcome.

Thompson distinguishes that situation from technical debt:

Technical debtPrompt tax
Immediate motivationShip an implementation quicklyAdopt valuable new model capabilities
Future workRepair or replace deferred engineeringDiscover and repair changed behavior
UncertaintyWhether the feature survives or needs rebuildingWhat the upgrade improves or breaks

A prototype may disappear if it never finds product-market fit, or require rebuilding if it becomes central to the product. Prompt tax arises when an upgrade is desirable now, but its effects on the existing application are not fully known.

The practical response is to release the new model to a subset of users, learn what works and what fails, repair problems quickly, and then expand access. That sequence attempts to capture the new model’s benefits while protecting functionality users already rely on. It also avoids assuming that every feared regression will actually occur.

8:539:09
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:53 · section reference included

Specify the objective and remove obsolete scaffolding

The migration from GPT-4o to o1-preview changed how Orbital wrote instructions. With the earlier models, the team often specified exactly how to accomplish a task. With reasoning models, they found it useful to strip away that procedural scaffolding and focus on what the model should accomplish. Repeated instructions that had previously helped enforce compliance could also be removed.

The resulting pattern was a clear objective, relevant context, and enough freedom for the model to work out a plan. For the lease-report task, an illustrative objective-focused prompt would look like this:

Using the supplied deed and lease, identify the lease date and
current annual rent for the occupational lease report.

Cite the supporting passage for each answer so that a lawyer
can verify it against the source documents.

If the documents do not establish an answer, or contain conflicting
information, state the uncertainty rather than guessing.

The prompt retains the required outcome and evidence standard without prescribing a reading order, a fixed sequence of reasoning steps, or repeated reminders. Removing unnecessary procedure is different from removing the constraints that define a correct result.

Slide listing “Specified what, not how,” “Made prompts leaner,” and “Unblocked the model,” beside a Prompt-tax illustration with a GPT-4o robot and a presenter inset.Open full source frame
Prompt changes: specify what, make prompts leaner, and unblock the model.
11:5012:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:50 · section reference included

Choose models and manage the experience of change

Reasoning models did not eliminate the usefulness of system-one models. Thompson describes the latter as often cheaper and faster. He also describes using visible model output, which he calls “thought tokens,” for two purposes: giving lawyers explanations of complex legal matters and helping engineers investigate failures found through testing or user feedback. The output format is unspecified here; it should not be interpreted as access to raw internal reasoning, which o1-preview did not expose.

Feature flags apply to model upgrades as well as conventional software features. They let a team introduce a model progressively instead of changing every user’s experience at once. Thompson pairs that engineering mechanism with a behavioral observation: familiar shortcomings can feel safer than unknown ones. Users know the weaknesses of the current system; a replacement creates uncertainty even if it improves the overall experience.

Announcing the switch can also heighten scrutiny. Thompson describes this as “poking the bear”: once people know a model has changed, they may actively search for problems, and the problems they find can dominate their perception of the benefits. This is his interpretation of change aversion, rather than a measured comparison of announced and unannounced rollouts. It matters when interpreting feedback because heightened attention and actual regressions can arrive together.

13:0513:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:05 · section reference included

Build for the model you expect to have

Orbital’s team uses a simple mantra: bet on the model. Product planning should consider what models might do in three, six, or twelve months, rather than treating today’s limitations as permanent. If models become smarter, cheaper, faster, and more capable, a well-chosen feature can improve as the underlying models improve. The payoff depends on making the right bet.

The mantra serves a practical purpose in meetings. It gives the team a short way to interrupt a design discussion and ask whether they are building around a temporary limitation. Could the feature be designed differently if the team considered future capabilities instead of only the current model?

The newer models can also help with the migration itself. Orbital feeds domain-specific prompts written for earlier models into a newer reasoning model and asks it to help adapt them. Thompson reports that this substantially reduces manual migration effort. His rationale is that the newer model can help write instructions suited to itself; the useful operation is model-assisted rewriting, not an established guarantee that a model understands its own behavior.

14:5415:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:54 · section reference included

Give someone the release decision—and a fast repair loop

A model upgrade encounters both the inputs users have always supplied and new use cases they discover afterward. That makes uncertainty unavoidable. Thompson argues that an identifiable person or team must own the residual risk, weigh the expected benefits against the downsides, and make the release decision after reasonable mitigation. Otherwise, anxiety can indefinitely postpone the benefits of the upgrade.

That willingness to ship depends on an effective feedback loop. Feedback may arrive manually or through simple thumbs-up and thumbs-down controls in the product, but it needs to reach AI engineers and domain experts quickly. Orbital’s process is concrete:

  1. A user submits feedback through the product.
  2. The feedback immediately reaches a domain expert.
  3. The expert evaluates the issue and identifies the prompt that needs changing.
  4. The expert edits the prompt and ships the correction to production.
  5. The revised behavior becomes available to the reporting user and other users of the system.

Thompson reports that some feedback-driven prompt repairs reach production in minutes or hours, rather than the days or weeks he associates with more traditional bug-fix cycles. The mechanism is the short path from a legal judgment about the output to a prompt change that the product can consume.

16:1116:26
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:11 · section reference included

Product judgment on a moving technical foundation

An embedded Hard Fork podcast clip gives the product-design problem a broader frame. Demis Hassabis, introduced as Google DeepMind’s chief executive, contrasts AI with the internet and mobile eras: eventually, those technical stacks stabilized enough for attention to concentrate on the products built on top. In AI, the underlying stack itself keeps changing rapidly.

Hassabis asks how a team should place a product bet when the underlying technology might become twice as good within a year. That is an illustration of the planning problem, not a benchmark with a defined task or metric. His answer emphasizes deeply technical product designers and managers who can anticipate which capabilities a product launching a year later may be able to rely on. Teams need to try different approaches and double down quickly when one works.

For Thompson, this creates an opportunity for product AI engineers. A product engineer connects implementation knowledge to the customer’s problem; a product AI engineer also understands model capabilities from the bottom up and can translate them into useful features. Keeping those connections within one engineer or a small team reduces the distance between discovering what a model can do and deciding how that capability could solve a real user problem.

17:5818:15
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

17:58 · section reference included

Will the confidence mechanism scale?

The remaining question is what gives a team enough confidence to keep shipping as both models and product surface area expand. Orbital’s combination of domain experts, real-time user feedback, and tools for acting quickly had worked well by Thompson’s account. But success so far does not settle whether that approach will become harder to sustain as the system grows.

An evaluation system might reduce that burden. The difficulty is deciding what it needs to measure. For a legal report, answer correctness alone is insufficient:

DimensionWhat needs assessment
Answer correctnessWhether the legal or factual answer is right
StyleWhether the answer is expressed appropriately
ConcisenessWhether it communicates without unnecessary detail
Citation correctnessWhether the cited evidence supports the answer

Those dimensions must be assessed across the prompts and use cases lawyers bring to the system, including variation in probabilistic outputs.

The coverage problem extends beyond happy paths. Thompson worries that attempting to evaluate every edge case could become prohibitively expensive or slow, particularly while new features and prompts continue to expand the surface that needs evaluation. He leaves evals as a plausible way to improve confidence, not a demonstrated resolution of the scaling problem.

20:2820:37
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

20:28 · section reference included

Expand exposure at the pace you can repair

Progressive delivery offers another possible source of confidence. Thompson proposes starting internally, moving to a limited group of users willing to try new behavior, and then expanding toward half and eventually all users. Rather than advancing only on a fixed schedule, the team could calibrate exposure to the feedback it can handle:

  1. Release internally to find problems before customer exposure.
  2. Introduce the model to a limited user group and act on the feedback.
  3. Increase exposure while the team can keep up with investigation and repair.
  4. Dial exposure back if feedback overwhelms the team, then continue expanding as issues are resolved.
  5. Reach full rollout when little remaining feedback requires product changes.

This is a proposed strategy for controlling rollout, not a claim that Orbital has implemented an automated controller. Its key idea is to make the rate of exposure responsive to the organization’s ability to repair what it discovers.

Thompson invites other AI engineers to contribute tactics from their own deployments and potentially develop a future community presentation together. His own preference remains to put frontier capabilities into users’ hands. Some feared failures will never happen; others may be caught and fixed after feedback from one or two users, before exposure grows to tens or hundreds of thousands. Those larger numbers describe the rollout logic, not Orbital’s reported user count.

Adopt the capability now; determine the actual repair cost case by case. That is the meaning of buying now and maybe paying later. It is a commitment to shipping with a way to learn and respond, while accepting that the eventual prompt tax is not fully knowable in advance.

The evaluation question remains open at the end. Thompson invites continued feedback by email or LinkedIn and suggests returning after Orbital has built and rolled out its eval system. The useful follow-up would examine both the difficulties of implementing it and whether it actually solved the confidence and scaling problems raised here.

22:3022:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:30 · section reference included

Resources

From the talk

  • Andrew Thompson's January 2024 account of building an AI assistant for real-estate legal work.

  • OpenAI's guidance on choosing reasoning models and writing direct prompts with explicit goals and constraints.

  • The original o1-preview research announcement, including the distinction between hidden reasoning and user-visible summaries.

Updates since the talk

Transcript download
Full transcript

Loading transcript…