AI Engineer World's Fair 2024

Everything you need to know about Finetuning and Merging LLMs

Loading slides…

Read the talk

Everything you need to know about Finetuning and Merging LLMs

A base model can continue a question without answering it. Fine-tuning changes that behavior; model merging offers another way to reuse capabilities already learned by other models.

From a talk by Maxime Labonne

Before you start: Familiarity with language-model tokens, training loss and GPU memory will help; no prior fine-tuning or model-merging experience is required.

From completing questions to answering them

Why might a language model autocomplete your question instead of answering it? Pretraining teaches next-token prediction: given large amounts of raw text, the model learns to continue that text. The resulting base model has learned useful patterns, but an instruction is still text to continue. Supervised fine-tuning, or SFT, supplies question–answer pairs so that a similar prediction objective teaches the model to respond to instructions.

Preference alignment adds another stage. Instead of supplying only examples of answers, it supplies preferences about how the model should behave. In the lifecycle Maxime Labonne presents, raw-text pretraining produces a base model, SFT produces an instruction-following model, and preference alignment produces what is commonly called a chat model.

Three connected stages map raw text to a base model, instructions and answers to an instruct model, and AI or human preferences to a chat model.Open full source frame
The LLM training lifecycle: pre-training, supervised fine-tuning, and preference alignment.
1:001:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:00 · section reference included

Establish whether prompting is enough

Fine-tuning starts with an evaluation problem. Begin with prompt engineering and establish what success means. Accuracy may require a custom benchmark for a niche application, or an existing open benchmark where one fits. Cost and latency belong in the same evaluation: a correct answer that is too expensive or too slow may still miss the requirement. If prompting meets those requirements, the problem is already solved.

If prompting falls short, ask whether you can create an instruction dataset: pairs of questions and desired answers. If you cannot, Labonne treats that as a reason to reconsider the project’s scope. If you can, fine-tuning becomes an option, and the evaluation framework you already built lets you test whether it helped.

The motivation is not exclusively technical. Labonne cites a16z’s 2024 enterprise report, whose interviews and survey emphasize control and customization as reasons to use open models. Fine-tuning is one way to exercise that customization. Those findings describe the enterprises surveyed, rather than a universal ordering of priorities.

1:542:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:54 · section reference included

Choose a training interface

After acknowledging Unsloth, covered in the preceding session, Labonne distinguishes three other training libraries by how you work with them.

LibraryInterface emphasized in the talk
TRLHugging Face library built on Transformers
AxolotlVersatile training through YAML configurations
LLaMA-FactoryBuilt-in graphical interface

These offer different entry points into training: a library integrated with the Transformers ecosystem, a configuration-driven workflow, or a graphical interface.

Four library logos and descriptions highlight Transformers integration, reusable YAML configurations, a graphical interface, and efficient single-GPU fine-tuning.Open full source frame
Fine-tuning libraries: TRL, Axolotl, LLaMA-Factory, and Unsloth.
3:223:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:22 · section reference included

Separate context, answers and preferences

An SFT example separates the instruction from the answer. The system prompt steers behavior—for example, by asking for an explanation suitable for a five-year-old—while the user prompt supplies the task. Labonne’s task example asks the model to remove spaces from a sentence. In the training setup he describes, the system and user tokens remain available as context, but are masked out of the loss. The answer tokens receive the supervision. Masking therefore does not mean hiding the instruction from the model; it means not training the model to reproduce that part of the example.

Those answers need not all be written by people. Labonne describes synthetic data as common in SFT datasets and endorses generation with frontier models as a way to build high-quality examples. The generated answer is still the training target, so its quality matters just as much as a human-written answer’s quality.

Preference training changes the example format. Labonne names PPO, Direct Preference Optimization, KTO and IPO, describing DPO as the most popular in practice at the time. A preference example contains an instruction, a chosen answer and a rejected answer. The pair gives the model a positive and a negative example for the same request.

Labonne’s high-level intuition for DPO is to increase the probability of chosen answers relative to a reference version of the model that has not undergone this preference-training step. That is an intuition rather than a complete specification of the objective. His behavioral example prefers a refusal to a request for bomb-making instructions; preference training can also target better general performance, rather than refusal behavior alone.

4:064:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:06 · section reference included

Build data that teaches the intended behavior

What makes an instruction example useful? Labonne questions the reliability of human review alone and proposes three dimensions for judging samples:

  • Accuracy: Answers should be factually correct and cleanly written. Training on false information risks damaging knowledge the model already has.
  • Diversity: Cover the topics and writing styles relevant to the application. A specialized summarizer does not need the breadth of a general-purpose assistant, but it still benefits from variety within its scope.
  • Complexity: Include tasks that demand more than retrieving a fact. Labonne points to reasoning-bearing answers, summarization and explanations for a five-year-old as examples that require the model to work with the prompt and transform information.

The preparation process is iterative:

  1. Start with existing open datasets where possible, combining sources that fit the intended application.
  2. Deduplicate with exact matching and fuzzy matching to remove repeated or near-repeated material.
  3. Filter for quality. Rules can remove unwanted response patterns; reward models or an LLM judge can score samples for filtering. Labonne’s rule-based example removes formulaic refusals, which is a choice about desired behavior rather than a universal quality rule.
  4. Inspect coverage. Lilac, Nomic Atlas and topic clustering can reveal how the dataset is distributed and suggest improvements.
  5. Generate additional data from those findings, then repeat the process.

Inspection feeds the next round of generation instead of serving only as a final check.

5:475:59
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:47 · section reference included

Decide which weights to train

Full fine-tuning takes the model and trains its weights on the instruction dataset. Labonne presents it as the strongest-performing but least efficient option in his comparison. LoRA reduces the trainable portion: it freezes the pretrained weights and adds trainable adapter matrices, A and B, at targeted layers. The adapters learn the update while the base weights remain fixed.

MethodTrainable parametersBase-weight storage described
Full fine-tuningModel weightsUnquantized model
LoRAAdapter matrices A and BFrozen 16-bit weights
QLoRAAdapter matricesFrozen four-bit weights

LoRA still requires loading the base model. QLoRA reduces that memory burden by quantizing the frozen base weights while retaining trainable adapters. Labonne warns about a performance trade-off, but degradation is not inevitable: the original QLoRA paper reports preserving 16-bit fine-tuning performance in its experiments. Neither that result nor the talk’s ordering guarantees the outcome for a particular task; the evaluation framework must settle it.

8:198:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:19 · section reference included

Find a stable learning rate and a workable memory budget

Learning rate is Labonne’s first tuning priority. It depends on the model and requires experiments. His suggested exploration is to try higher rates, observe where the loss becomes unstable, and reduce the rate from there. The exploding-loss graph illustrates a boundary to avoid, not a desirable training outcome.

The remaining choices interact:

  • Epoch count: Choose how many passes to make with the size of the dataset in mind.
  • Sequence length: Longer sequences consume more VRAM and constrain the batch size. Training does not have to use the pretrained model’s full context window.
  • Batch size: Within memory limits, larger batches can improve GPU utilization.
  • LoRA rank: This controls another part of the adapter setup, although Labonne names it without developing a detailed tuning procedure.

Sequence length and batch size therefore need to be chosen together: allocating more memory to each sequence leaves less room for sequences in a batch.

A Hyperparameters slide lists dataset-dependent epoch counts, sequence length and batch size trade-offs in VRAM usage, gradient accumulation, and LoRA ranks of 16–64.Open full source frame
Hyperparameter guidance for epochs, sequence length, batch size, and LoRA rank.
9:309:40
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:30 · section reference included

Reuse training already embodied in model weights

Model merging combines weights from existing fine-tuned models. Instead of creating every capability through your own training run, you can draw on models released by the open-source community, including those on Hugging Face Hub. The merging operation can run without a GPU; that does not eliminate the compute needed to run the resulting model.

Labonne reports that merged models occupy roughly the top eight to ten positions among 7B models in the Open LLM Leaderboard v1 snapshot shown in the talk. This is historical evidence: he says version two had launched that morning, while his slide still showed version one. The displayed ranking is a motivation to investigate merging, not a claim about current leaderboard standings.

His recommended tool is MergeKit, which implements multiple merging techniques. A source model can itself be a merge, so the ancestry quickly branches: the family-tree illustration connects models that were combined into other models, which were then merged again. Even the tree that fits on the slide is only a small example of that reuse.

10:2610:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:26 · section reference included

Interpolate two models or combine sparse updates

SLERP, spherical linear interpolation, interpolates between two models’ weights. It merges two models at a time, but the interpolation factor can vary by layer, allowing different parts of the result to favor different sources. Labonne’s example is NeuralBeagle14-7B. Its ancestry matters: the model card identifies NeuralBeagle as a DPO fine-tune of Beagle14-7B, which was itself a merge. Its results therefore cannot be attributed to interpolation alone.

DARE takes a pruning-and-rescaling approach to reducing redundancy. Labonne describes this informally as retaining significant parameters. More precisely, the original DARE method randomly drops entries from the fine-tuning deltas—the changes relative to a base model—and rescales the retained deltas by 1 / (1 − p), where p is the drop probability. It does not select the largest raw model weights. This approach can be used to merge more than two source models.

For multi-model experiments, Labonne recommends trying seven or eight sources with DARE, based on his experience. That is practical guidance rather than a measured optimum: the useful distinction is that SLERP is pairwise, while the DARE-based approach he recommends can combine contributions from a larger group.

12:0112:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:01 · section reference included

Repeat layers without another training run

Passthrough changes the construction more directly: concatenate layers from different LLMs, or reuse layers from the same model. The latter is a self-merge. Labonne describes creating Meta-Llama-3-120B-Instruct from Llama 3 70B Instruct by repeating a group of ten layers six times. This is layer stacking rather than interpolation between corresponding weights.

The counterintuitive part is that he did not train the enlarged model afterward. Labonne reports enthusiastic online reception, particularly for creative writing, while also describing the model as erratic. That is a reported use-case assessment, not evidence of general superiority. When a questioner asks whether there was any fine-tuning at all, he explicitly confirms there was none.

13:1713:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:17 · section reference included

Assemble experts and initialize a router from prompts

A conventional mixture of experts, or MoE, is pretrained with a router and multiple feedforward networks. A FrankenMoE instead extracts feedforward layers from existing fine-tuned models, combines those layers as experts, and adds a router. The construction reuses already-trained components rather than pretraining the whole expert system from scratch.

For Beyonder, Labonne selects four fine-tuned sources with different specialties: chat, code, role-play and mathematics. The MergeKit configuration shown associates positive prompts with the experts. These prompts are not additional SFT examples in this construction; they help initialize routing.

The router’s decision happens for each token at each layer: which expert feedforward networks should process this token here? The described setup generally selects two experts. To initialize that behavior without a new fine-tuning run, the construction calculates embeddings from the positive prompts and uses those embeddings to initialize the routers. Further fine-tuning is possible, but it is not required for the assembly process Labonne demonstrates.

14:2614:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:26 · section reference included

Evaluate the assembly against its sources

Labonne also describes Phixtral, for which he had to modify the implementation for compatibility with Phi-2. He reports that Phixtral outperformed its base model on multiple tasks. Even so, when performance is the priority, he recommends SLERP or DARE ahead of FrankenMoE, which he considers more experimental and less effective in his experience.

The final Beyonder comparison uses its own source models as baselines. Labonne reports that Beyonder outperforms those source models on multiple benchmarks. That makes the comparison particularly useful: it asks whether assembling the experts improves on the components already available, rather than comparing only with an unrelated model. The reported gains coexist with his preference for simpler merging techniques when seeking performance.

For hands-on continuation, Labonne points to his Large Language Model Course and its GitHub notebooks. The closing slide brings together entry points for merging, evaluation, SFT, DPO, Axolotl and quantization, so readers can move from these mechanisms to a concrete experiment.

Six boxes list LazyMergekit, LLM AutoEval, Mistral-7b fine-tuning with SFT+TRL and DPO+TRL, LazyAxolotl, and AutoQuant.Open full source frame
Large Language Model Course resources with Open in Colab buttons.
16:1116:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:11 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…