AI Engineer World's Fair 2024

The GenAI Maturity Curve (or: You Probably Don’t Need Fine-Tuning)

Kyle Corbitt· Founder & CEO, OpenPipe18:03

Loading slides…

Read the talk

The GenAI Maturity Curve (or: You Probably Don’t Need Fine-Tuning)

Start with prompting to discover the task, then use real inputs and better training outputs to decide whether specialization can improve cost, latency, and consistency.

From a talk by Kyle Corbitt

Before you start: Familiarity with prompting an LLM and the basic distinction between inference and training will help.

Do you need fine-tuning yet?

Do you need fine-tuning—or do you need to understand your application better first? Kyle Corbitt opens with that tension: he works on OpenPipe, a fine-tuning platform, yet suspects most people in the audience do not need its core capability at their current stage. The title moves from a declaration that you probably do not need fine-tuning to a more useful question: whether you need it at all.

Slide asking “Do You Even Need Fine-Tuning?” with Kyle Corbitt and OpenPipe credited below and a speaker inset.Open full source frame
Do You Even Need Fine-Tuning?

The decision has three parts. First, what can fine-tuning improve? Second, when do those improvements justify the investment? Third, what does implementation require? The goal is not to reach fine-tuning as a milestone. It is to recognize when a specialized model will help deliver the product outcome you want—and when the flexibility of prompting remains more valuable.

0:160:27
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Choose a model on the quality–cost frontier

Begin with the model choice you already face. Corbitt plots output quality vertically and tokens per dollar horizontally: higher means better answers, while farther right means more output for the same spend. His graph averages quality across tasks, but the practical comparison is how each candidate performs on your application. Evaluating several models on the actual task is more useful than assuming their general reputation determines the result.

Among frontier models, quality and expense typically rise together. Larger models, and models with more training behind them, often produce stronger zero-shot answers; smaller models may be cheaper but miss the quality threshold. The Pareto frontier marks the best available trade-offs: a model belongs on that boundary when another available choice does not beat it on both quality and cost-effectiveness.

Starting constraintSelection rule
Minimum acceptable qualityChoose the cheapest model that meets it.
Maximum inference spendChoose the highest-quality model within budget.

These are two directions through the same decision. A quality requirement tells you how far down in price you can go; a unit-economics constraint tells you how much quality you can afford.

2:192:30
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:19 · section reference included

Spend more inference compute to improve the answer

That frontier is not fixed. One way to move beyond the quality of a single model call is to spend more compute at inference time. Corbitt uses Mixture-of-Agents, citing Together AI’s research and work at OpenPipe. The mechanism is successive layers of prompting: later stages use earlier outputs to improve the answer before returning a final response.

In Corbitt’s plotted example, Mixture-of-Agents produces higher-quality answers than GPT-4 alone. This is the talk’s comparison, separate from Together’s published benchmark. The point moves above the single-call frontier, but it also sits far to the left: multiple calls make the answer expensive, and successive stages add latency. A technique can therefore improve quality while remaining unsuitable for serving every live request.

Output Quality versus Tokens per Dollar graph showing MoA (GPT 4 Turbo) at the upper left, prompted models, and a red Pareto Frontier line.Open full source frame
MoA extends the graph toward higher output quality and fewer tokens per dollar.
4:114:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:11 · section reference included

Turn better answers into training data

The expensive answer-generation process becomes more useful when its outputs become training data. Once you have a way to produce unusually good answers, fine-tuning gives a smaller model examples of the behavior you want. The opportunity is to improve the model’s task performance without repeating the entire multi-stage generation process for every request. This is Mixture-of-Agents feeding fine-tuning, not a Mixture-of-Experts model architecture.

Corbitt then adds three fine-tuned models to the graph. These points come from OpenPipe’s internal evaluations on four real customer datasets, averaged together. Customers defined the evaluations, LLM-as-a-Judge supplied the quality assessment, and the cost axis used then-standard model pricing. That makes the graph an empirical result for those tasks, not a universal ranking of model capability.

On the four-task average, fine-tuned Llama 3 70B and Llama 3 8B scored above their prompted counterparts and every compared base prompted model. The upward arrows make the mechanism visible: the same model families occupy a different quality position after specialization.

Graph with green fine-tuned model points, orange prompted points, a red MoA point, and dashed upward arrows from Llama 3 70B and Llama 3 8B to their fine-tuned counterparts.Open full source frame
Arrows connect prompted Llama models to their higher fine-tuned results.

Redrawing the frontier around those results changes the purchasing decision. Instead of selecting only among general models, you can invest in creating a model that delivers more task quality at a given inference budget. The graph makes this look like an improvement without a sacrifice—but its axes count output quality and inference economics, not all the work required to obtain that model.

5:255:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:25 · section reference included

Wait until specialization solves a real constraint

The missing costs are preparation and flexibility. You must gather a dataset, train the model, and check that it works. You are also specializing it for a particular task. Inputs outside that domain should not be expected to receive the same performance boost. Task clarity is a prerequisite for useful specialization: you need to know what the model should become good at before narrowing its behavior around examples.

Once that task is understood, three pressures can justify the investment:

  • Cost: the prompted model works, but inference spend strains the product’s unit economics.
  • Latency: a smaller model performs less computation per generation and can lower the latency floor.
  • Consistency: the model usually solves the task but intermittently breaks the format, misses an instruction, or makes another recurring error.

Corbitt illustrates the consistency problem with a hypothetical GPT-4 success rate of 80–90%, leaving 10–20% of requests with errors. These are illustrative proportions, not measured benchmark results. Fine-tuning is attractive here because the desired behavior is already identifiable; the problem is reproducing it reliably.

For most applications where prompting works, Corbitt recommends starting there. Early usage is usually limited enough to absorb higher inference costs, while changing a prompt lets you change behavior quickly. Once the product shape and the task you intend to serve at scale become clear, specialization is easier to justify. Corbitt estimates a couple of hours to make that transition in suitable cases; the estimate is not a guarantee of total project or training time. The diagram’s progression from GPT-4 to a fine-tuned model represents this shift from discovering the product to serving its recurring task efficiently.

Slide titled “When?” showing an orange arrow labeled GPT-4 from 0 to 1, followed by an arrow labeled Fine-tuned model from 1 to 100.Open full source frame
GPT-4 from 0 to 1; a fine-tuned model from 1 to 100.
8:008:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:00 · section reference included

Begin with the requests users actually send

The implementation is less intimidating when separated into concrete steps. Corbitt reports that engineers without machine-learning or data-science experience have produced fine-tunes that outperform their starting model with less than an hour of work. That is an experience-based anecdote, not a general time requirement. His workflow is straightforward:

  1. Prepare the data.
  2. Train the model.
  3. Evaluate its behavior.
  4. Deploy it.

He jokes that evaluation is technically optional, but endorses it as part of the process.

Starting with a prompted production model helps with the first and often hardest step: discovering the input distribution. Users reveal what they actually want, including uses you did not anticipate. Logging their prompts gives you a record of what the specialized model must handle, instead of forcing you to guess before training begins.

This addresses a longstanding difficulty in machine learning: determining what representative inputs and good outputs should look like. A prompted model does not make every application feasible, but where it works, it lets the application collect realistic examples before committing to specialization. The dataset can then reflect the task people are actually asking the product to perform.

The capture mechanism can be simple. Corbitt uses OpenPipe, but an existing application database or observability service can retain the same essential information. For example, a Python application can append each completed interaction to a JSONL file before deciding which outputs deserve to become training targets:

python

import json
from pathlib import Path


def record_interaction(
    log_path: Path,
    messages: list[dict[str, str]],
    response: str,
) -> None:
    record = {
        "messages": messages,
        "response": response,
    }
    with log_path.open("a", encoding="utf-8") as log:
        log.write(json.dumps(record, ensure_ascii=False) + "\n")


record_interaction(
    Path("interactions.jsonl"),
    messages=[
        {"role": "system", "content": "Classify sentiment as positive or negative."},
        {"role": "user", "content": "The delivery arrived early. Thank you!"},
    ],
    response="positive",
)

The important property is preservation of the interaction. A logging product is a choice; retaining the data is the requirement.

11:2611:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:26 · section reference included

Make the outputs worth learning from

Captured traffic tells you what the model needs to handle. It does not automatically tell you which answers the model should learn to reproduce. The next step is to improve dataset quality, and the amount of work depends on how well the current model already performs:

  • Reuse good production outputs: if GPT-4 already handles the task well and the goal is lower cost or latency, those existing input–output pairs can supply the training data.
  • Review incorrect outputs: when the model makes mistakes, human review can correct the examples before training.
  • Relabel with stronger generation: a stronger model or a multi-iteration process can produce better targets for the same inputs.

This separates a cost-reduction project from a quality-repair project. The latter needs better answers in its dataset, not merely a copy of current behavior.

Mixture-of-Agents returns here as an offline labeling technique. Several rounds of generation can address some reliability problems while keeping the process automated. It is still slower and more expensive than a single call, but that expense applies to the training examples rather than every live request. The distinction is what makes an otherwise impractical serving strategy useful: spend extra effort producing targets, then train a model to perform the recurring task.

13:4914:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:49 · section reference included

Use established training infrastructure

Training itself often looks like the intimidating part: selecting hyperparameters, obtaining infrastructure, and figuring out how to run the job. Corbitt’s assessment is that tooling has made much of this work routine. OpenPipe offers a one-click workflow, other hosted providers offer fine-tuning, and open-source libraries provide another route.

The choice is therefore between using a managed training service and operating a training stack yourself, rather than between inventing the infrastructure and abandoning the project. Standardized architectures and more mature tools reduce the amount of custom work required. That shifts attention back to the application-specific questions: whether the examples represent the task and whether the trained model performs it well.

14:5215:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:52 · section reference included

Evaluate in two loops

A trained model needs two kinds of feedback. The inner loop should run quickly enough to guide iteration. OpenPipe uses LLM-as-a-Judge evaluations with user-defined criteria, comparing fine-tuned outputs against the dataset to ask whether the model appears to be doing the right thing.

The outer loop asks whether that behavior achieves the outcome the business or application actually needs. First define what matters, then determine how to measure it. That usually requires deployment or a production A/B test; a judge score alone does not establish the product result.

Evaluation loopMain questionEvidence
InnerDoes the model meet our criteria?Fast judgments of model outputs
OuterDoes it achieve the intended outcome?Use-case metrics in deployment or A/B tests

OpenPipe integrates the inner loop so teams can compare fine-tunes and decide which candidates to promote for production testing. The loops connect, but they are not interchangeable: fast evaluation selects a promising candidate; the outer loop tests whether that promise holds in the application.

15:4515:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:45 · section reference included

Deploy the specialization where it pays off

Deployment has two main routes. You can self-host on your own GPUs, or use a provider to host the model. Corbitt names vLLM and a project he calls TRTLM as self-hosting options with OpenAI-compatible APIs. A familiar interface can reduce application changes, though compatibility does not imply full API parity: the versioned vLLM 0.4.1 documentation excludes chat tools and tool_choice, as well as the completions suffix parameter.

OpenPipe and other inference providers offer the hosted route. The relevant improvement is that mature serving tools give teams practical deployment choices; fine-tuning no longer necessarily entails building a bespoke inference system. You still choose who operates the model and validate its behavior in the deployed application.

The workflow returns to the same sequence: prepare data, train, evaluate, deploy. Corbitt closes with the claim that a fine-tuned model can deliver better performance than GPT-4 at one twenty-fifth the cost. That claim belongs to the talk’s task-specific comparison, not a universal price promise; the closing slide does not supply a separate model-specific calculation for the ratio. The decision is whether your application has reached the point where a well-understood task, representative data, and a real cost, latency, or consistency constraint make specialization worth the investment.

16:4716:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:47 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…