AI Engineer World's Fair 2026
Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards
Loading slides…
Read the talk
Your Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards
Lease End’s SMS application delivered strong commercial results, but repairing its intent classifier exposed costs that cheaper inference could not offset.
From a talk by Dan Bjornn
A message decides when the phone rings
Should a customer get a call tomorrow or right now? At Lease End, which helps customers finance the purchase of their leased cars, that decision became part of an LLM application built in late 2024. Customers could text questions, schedule calls and receive reminders. Dan Bjornn, the company’s senior data scientist, describes an initial workflow built on retrieval-augmented generation: search a vector database of previously seen messages labeled with customer intent, then use that classification to route the conversation.
The straightforward examples made the approach look manageable. “Call me tomorrow” meant the customer wanted to talk later; “I've got time now” meant they wanted to talk immediately. But real conversations contained nuance that the retrieval approach did not reliably capture. The system worked, yet its most consequential decision—what the customer wanted to happen next—remained fragile.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A textbook case for supervised fine-tuning
Bjornn’s next move was supervised fine-tuning. Accuracy came first because every downstream action depended on intent: call now, schedule a call or honor an opt-out. Smaller models also promised cheaper inference and lower latency for thousands of messages answered in real time each day. The task was narrow and structured, with conversations assigned to one of six categories.
There was a portability argument, too. Lease End owned its training data. In principle, the team could pass those examples to another model, repeat the fine-tuning process and recover similar behavior. Owning the dataset appeared to offer control over the choice of model provider.
The training pipeline followed a familiar sequence:
- Collect conversation examples.
- Use LLM-as-judge classification to label them.
- Manually review the labels.
- Create holdout sets.
- Fine-tune and check the resulting metrics.
This gave the team both a way to train the classifier and a separate set of examples against which to evaluate it.
Bjornn reports that within a year, the application helped bring in $12 million in revenue at 50× ROI. Those are application-level results; he does not provide the accounting denominator or isolate fine-tuning’s contribution. The commercial success was real in the team’s experience, but it did not reveal how difficult the system would become to maintain.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
When acknowledgment becomes permission to call
The first failure, which Bjornn calls the confused confirmer, began with an appointment already arranged. Lease End told Tracy that her advisor would call on Thursday at 2 PM. Tracy replied, “Sounds good.” The application answered, “Great. I'm calling you right now.” An acknowledgment of a future appointment had become an immediate-call action. The result was frustrated customers and missed opportunities.
The second failure, the overeager puppy, needed even less encouragement. Alex from Lease End contacted James about his upcoming lease maturity. James replied, “Hi, good morning.” The application responded, “Good morning, I'm giving you a call.” Bjornn confirms that this happened in production. A greeting had been treated as a request to speak immediately.
Both examples expose the same operational problem: a conversational response is not necessarily authorization for the next action. Positive language can confirm an existing plan or simply acknowledge another person. The application could produce strong revenue results while still making conspicuous mistakes at that boundary.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An hour of training inside a week of repair
The difficult question was how to make repairs manageable. Each newly observed failure started another data preparation cycle:
- Gather examples of the problem.
- Decide whether there are enough examples to retrain.
- If necessary, generate additional examples with an LLM.
- Manually validate the generated conversations.
- Label the examples with the classification categories and manually review those labels.
The manual work protected training-data quality, but it also meant that correcting a seemingly small behavioral mistake required much more than launching a training job.
Bjornn reports that fine-tuning itself normally took about an hour, depending on dataset size. It was the shortest part of the process. Evaluation then exposed a second problem: fixing the latest failure could introduce regressions elsewhere. The team would repair one behavior only to see an older issue return, forcing another iteration.
The complete repair process—from gathering data through labeling, training iterations and deployment—took about a week, according to Bjornn. That cost changed which bugs the team could afford to address.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
How much customer pain can wait?
Before retraining, the team asked how often an issue occurred. An everyday failure had a different priority from a one-off. Severity could override frequency, however: a customer repeatedly stating a preferred call time while the model ignored it deserved urgent attention. Another serious failure occurred when the application told a customer that an appointment was scheduled but returned a malformed payload. The promised appointment never existed, so the follow-up never happened.
The final question was whether a workaround could avoid retraining. Could a temporary fix let the issue wait until the next run, rather than trigger a week of work for one or two problems? Together, frequency, customer harm and the possibility of postponement became the gate before anyone acted.
This was the practical cost of the repair loop: the team ranked bugs by how much customer pain it could tolerate in the meantime. Maintenance expense had become part of the customer experience.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The calcification tax
Bjornn calls the growing rigidity the calcification tax. The first form was model lock-in—the opposite of the portability he had expected. Even versions from the same provider differed in what training data they needed. Across providers, the differences extended to data formats, the volume of examples required and the interfaces used to run training.
Owning the examples did not make those differences disappear. Switching was too costly, so the team kept the same model for consistency. Routine retraining already consumed the capacity that might otherwise have gone toward evaluating and adopting a better model.
The second form was architectural lock-in. Bjornn describes predefined workflows as the production gold standard when the application was built in late 2024. As newer approaches became available, keeping the existing system running left little room to adopt them. The accumulated cost constrained both the model and the architecture around it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Moving behavior into context
The alternative became apparent through the team’s use of Claude Code. Different coding tasks did not require a different model each time. The team changed the skills, resources and context supplied to it. Better context changed the resulting behavior. Bjornn began asking whether the messaging application could work the same way.
Reconsidering fine-tuning meant reversing an approach Bjornn had championed, but an existing project gave the team a route forward. They migrated the workflow into skills, tools and resources that the skills could load for context. The messaging application became one of the first production tests of an agentic framework already under development. This was Lease End’s own rebuild; the Claude Code inspiration does not establish that it adopted Claude Code’s particular skill format or loader.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Repairing instructions, then validating behavior
Previously, the team waited for a critical mass of problems before entering the training cycle. After the rebuild, a discovered problem could lead directly to an edit of the affected system prompt or skill. The unit of repair had changed from a training dataset and model update to an instruction change.
For the earlier Tracy failure, a proposed instruction could make the distinction explicit in Markdown:
markdown
# Appointment acknowledgments
When the customer acknowledges an already scheduled call,
preserve the scheduled time. Do not treat agreement as a request
to call immediately.
Example:
- Existing appointment: Thursday at 2 PM.
- Customer reply: "Sounds good."
- Required behavior: Keep the Thursday appointment;
do not initiate an immediate call.
The edit expresses the intended behavior; validation still has to establish whether the model follows it. Lease End checked changes against a curated set collected during production, iterated a few times, then deployed by uploading Markdown files to an S3 bucket.
Bjornn reports that after the rebuild, the complete process from discovering a problem to deploying its fix took less than an hour. This measures the team’s repair turnaround, including validation and iteration, rather than training speed or inference latency.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
More expensive messages, a cheaper system
The rebuild increased API cost per message because the team used better models. Bjornn reports substantially better accuracy than the fine-tuned system, but supplies no numerical accuracy results or evaluation-set size. Faster repairs also let the team respond to problems without waiting for another batch of failures to justify retraining.
The new framework also restored the model choice the training dataset had failed to provide. Bjornn describes it as model-agnostic, supporting OpenAI, Anthropic and other providers, with task behavior supplied through context. He reports that total cost fell despite higher per-message API charges, because the team spent much less time maintaining and retraining the system. Inference cost was only one part of the operating cost.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What still justifies fine-tuning?
The original reasons for fine-tuning look different when tested against the operating history:
| Original reason | Lease End’s reported outcome |
|---|---|
| Better accuracy | The rebuild outperformed the fine-tuned model. |
| Lower cost at volume | Higher message costs, lower total cost. |
| Lower latency | Smaller models delivered marginal gains with no practical difference. |
| Narrow, structured task | A textbook fit still accumulated technical debt. |
| Vendor control | Reusing data alone did not make switching easy. |
The comparison changes the decision from whether a model can learn the task to whether the team can afford to keep that learned behavior correct as requirements and models change.
Privacy, data control and offline operation remain possible reasons to fine-tune. Bjornn treats them cautiously: other solutions may exist, and the long-term consequences still need to be assessed. His closing rule is deliberately demanding: fine-tune only when calling a frontier model is not an option—and even then, the decision must justify the continuing cost of changing, validating and maintaining the system.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Practical documentation for creating Claude Code skills with Markdown instructions and supporting resources.
Further reading
Anthropic explains how skills package procedural knowledge and load instructions and resources only when needed.
- Building effective agentsArticle
Architectural guidance covering routing workflows, tool-using agents, evaluation and complexity tradeoffs.
Full transcript
Loading transcript…