AI Engineer World's Fair 2026

Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards

Dan Bjornn· Senior Data Scientist, Lease End16:39

Loading slides…

Read the talk

Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards

Lease End’s SMS application delivered strong commercial results, but repairing its intent classifier exposed costs that cheaper inference could not offset.

From a talk by Dan Bjornn

A message decides when the phone rings

Should a customer get a call tomorrow or right now? At Lease End, which helps customers finance the purchase of their leased cars, that decision became part of an LLM application built in late 2024. Customers could text questions, schedule calls and receive reminders. Dan Bjornn, the company’s senior data scientist, describes an initial workflow built on retrieval-augmented generation: search a vector database of previously seen messages labeled with customer intent, then use that classification to route the conversation.

The straightforward examples made the approach look manageable. “Call me tomorrow” meant the customer wanted to talk later; “I've got time now” meant they wanted to talk immediately. But real conversations contained nuance that the retrieval approach did not reliably capture. The system worked, yet its most consequential decision—what the customer wanted to happen next—remained fragile.

Slide titled “First Solution” showing colored clusters of connected nodes, a blue diamond, and a translucent circle, with the speaker inset at left.Open full source frame
The first solution, illustrated with clusters of connected points and a highlighted circular region.
0:160:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

A textbook case for supervised fine-tuning

Bjornn’s next move was supervised fine-tuning. Accuracy came first because every downstream action depended on intent: call now, schedule a call or honor an opt-out. Smaller models also promised cheaper inference and lower latency for thousands of messages answered in real time each day. The task was narrow and structured, with conversations assigned to one of six categories.

There was a portability argument, too. Lease End owned its training data. In principle, the team could pass those examples to another model, repeat the fine-tuning process and recover similar behavior. Owning the dataset appeared to offer control over the choice of model provider.

The training pipeline followed a familiar sequence:

  1. Collect conversation examples.
  2. Use LLM-as-judge classification to label them.
  3. Manually review the labels.
  4. Create holdout sets.
  5. Fine-tune and check the resulting metrics.

This gave the team both a way to train the classifier and a separate set of examples against which to evaluate it.

Bjornn reports that within a year, the application helped bring in $12 million in revenue at 50× ROI. Those are application-level results; he does not provide the accounting denominator or isolate fine-tuning’s contribution. The commercial success was real in the team’s experience, but it did not reveal how difficult the system would become to maintain.

1:462:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:46 · section reference included

When acknowledgment becomes permission to call

The first failure, which Bjornn calls the confused confirmer, began with an appointment already arranged. Lease End told Tracy that her advisor would call on Thursday at 2 PM. Tracy replied, “Sounds good.” The application answered, “Great. I'm calling you right now.” An acknowledgment of a future appointment had become an immediate-call action. The result was frustrated customers and missed opportunities.

The second failure, the overeager puppy, needed even less encouragement. Alex from Lease End contacted James about his upcoming lease maturity. James replied, “Hi, good morning.” The application responded, “Good morning, I'm giving you a call.” Bjornn confirms that this happened in production. A greeting had been treated as a request to speak immediately.

Both examples expose the same operational problem: a conversational response is not necessarily authorization for the next action. Positive language can confirm an existing plan or simply acknowledge another person. The application could produce strong revenue results while still making conspicuous mistakes at that boundary.

4:274:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:27 · section reference included

An hour of training inside a week of repair

The difficult question was how to make repairs manageable. Each newly observed failure started another data preparation cycle:

  1. Gather examples of the problem.
  2. Decide whether there are enough examples to retrain.
  3. If necessary, generate additional examples with an LLM.
  4. Manually validate the generated conversations.
  5. Label the examples with the classification categories and manually review those labels.

The manual work protected training-data quality, but it also meant that correcting a seemingly small behavioral mistake required much more than launching a training job.

Bjornn reports that fine-tuning itself normally took about an hour, depending on dataset size. It was the shortest part of the process. Evaluation then exposed a second problem: fixing the latest failure could introduce regressions elsewhere. The team would repair one behavior only to see an older issue return, forcing another iteration.

The complete repair process—from gathering data through labeling, training iterations and deployment—took about a week, according to Bjornn. That cost changed which bugs the team could afford to address.

6:176:34
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:17 · section reference included

How much customer pain can wait?

Before retraining, the team asked how often an issue occurred. An everyday failure had a different priority from a one-off. Severity could override frequency, however: a customer repeatedly stating a preferred call time while the model ignored it deserved urgent attention. Another serious failure occurred when the application told a customer that an appointment was scheduled but returned a malformed payload. The promised appointment never existed, so the follow-up never happened.

The final question was whether a workaround could avoid retraining. Could a temporary fix let the issue wait until the next run, rather than trigger a week of work for one or two problems? Together, frequency, customer harm and the possibility of postponement became the gate before anyone acted.

Slide titled “3 QUESTIONS before anyone could act:” asks “How frequent is it?”, “Is it hurting CX too much?”, and “Can it wait until the next retraining run?”Open full source frame
Three questions before acting: frequency, customer experience, and whether the issue can wait until retraining.

This was the practical cost of the repair loop: the team ranked bugs by how much customer pain it could tolerate in the meantime. Maintenance expense had become part of the customer experience.

8:048:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:04 · section reference included

The calcification tax

Bjornn calls the growing rigidity the calcification tax. The first form was model lock-in—the opposite of the portability he had expected. Even versions from the same provider differed in what training data they needed. Across providers, the differences extended to data formats, the volume of examples required and the interfaces used to run training.

Owning the examples did not make those differences disappear. Switching was too costly, so the team kept the same model for consistency. Routine retraining already consumed the capacity that might otherwise have gone toward evaluating and adopting a better model.

The second form was architectural lock-in. Bjornn describes predefined workflows as the production gold standard when the application was built in late 2024. As newer approaches became available, keeping the existing system running left little room to adopt them. The accumulated cost constrained both the model and the architecture around it.

9:239:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:23 · section reference included

Moving behavior into context

The alternative became apparent through the team’s use of Claude Code. Different coding tasks did not require a different model each time. The team changed the skills, resources and context supplied to it. Better context changed the resulting behavior. Bjornn began asking whether the messaging application could work the same way.

Slide titled “The AHA moment” reads “Drop in some context, behavior reliably changes. No fine-tuning. No retraining.” beside an angled terminal image.Open full source frame
The AHA moment: change behavior through context, without fine-tuning or retraining.

Reconsidering fine-tuning meant reversing an approach Bjornn had championed, but an existing project gave the team a route forward. They migrated the workflow into skills, tools and resources that the skills could load for context. The messaging application became one of the first production tests of an agentic framework already under development. This was Lease End’s own rebuild; the Claude Code inspiration does not establish that it adopted Claude Code’s particular skill format or loader.

11:1111:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:11 · section reference included

Repairing instructions, then validating behavior

Previously, the team waited for a critical mass of problems before entering the training cycle. After the rebuild, a discovered problem could lead directly to an edit of the affected system prompt or skill. The unit of repair had changed from a training dataset and model update to an instruction change.

For the earlier Tracy failure, a proposed instruction could make the distinction explicit in Markdown:

markdown

# Appointment acknowledgments

When the customer acknowledges an already scheduled call,
preserve the scheduled time. Do not treat agreement as a request
to call immediately.

Example:
- Existing appointment: Thursday at 2 PM.
- Customer reply: "Sounds good."
- Required behavior: Keep the Thursday appointment;
  do not initiate an immediate call.

The edit expresses the intended behavior; validation still has to establish whether the model follows it. Lease End checked changes against a curated set collected during production, iterated a few times, then deployed by uploading Markdown files to an S3 bucket.

Bjornn reports that after the rebuild, the complete process from discovering a problem to deploying its fix took less than an hour. This measures the team’s repair turnaround, including validation and iteration, rather than training speed or inference latency.

12:1812:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:18 · section reference included

More expensive messages, a cheaper system

The rebuild increased API cost per message because the team used better models. Bjornn reports substantially better accuracy than the fine-tuned system, but supplies no numerical accuracy results or evaluation-set size. Faster repairs also let the team respond to problems without waiting for another batch of failures to justify retraining.

The new framework also restored the model choice the training dataset had failed to provide. Bjornn describes it as model-agnostic, supporting OpenAI, Anthropic and other providers, with task behavior supplied through context. He reports that total cost fell despite higher per-message API charges, because the team spent much less time maintaining and retraining the system. Inference cost was only one part of the operating cost.

13:3413:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:34 · section reference included

What still justifies fine-tuning?

The original reasons for fine-tuning look different when tested against the operating history:

Original reasonLease End’s reported outcome
Better accuracyThe rebuild outperformed the fine-tuned model.
Lower cost at volumeHigher message costs, lower total cost.
Lower latencySmaller models delivered marginal gains with no practical difference.
Narrow, structured taskA textbook fit still accumulated technical debt.
Vendor controlReusing data alone did not make switching easy.

The comparison changes the decision from whether a model can learn the task to whether the team can afford to keep that learned behavior correct as requirements and models change.

Privacy, data control and offline operation remain possible reasons to fine-tune. Bjornn treats them cautiously: other solutions may exist, and the long-term consequences still need to be assessed. His closing rule is deliberately demanding: fine-tune only when calling a frontier model is not an option—and even then, the decision must justify the continuing cost of changing, validating and maintaining the system.

14:4915:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:49 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…