AI Engineer World's Fair 2024

Fixing bugs in Gemma, Llama & Phi-3

Loading slides…

Read the talk

Fixing bugs in Gemma, Llama & Phi-3

From Gemma’s activation function to Llama 3’s special tokens, reliable fine-tuning depends on preserving the model’s assumptions through training, conversion and inference.

From a talk by Daniel Han

Before you start: Familiarity with tokenization, supervised fine-tuning and LoRA will help; the code examples use Python and PyTorch.

The activation function is part of the model

What happens when a model expects approximate GELU, but its implementation uses exact GELU? The mathematically more exact function is still the wrong function for those weights. Gemma provides Daniel Han’s opening example: one of Unsloth’s fixes was to use the approximate activation the model expected. The opening slides show GELU code comparisons alongside other implementation and precision differences.

Two posts show GELU code comparisons, a list of Gemma bugs, and a graph comparing precision fixes across layers.Open full source frame
Gemma implementation differences and Unsloth precision fixes.

Tokenization creates another class of mismatches. Han points to the Gemma bug report and its accompanying Colab, then briefly revisits Phi-3: he reports a sliding-window correction from 2,047 to 2,048 and recommends separating its fused QKV matrices for LoRA fine-tuning. These are the background to the main walkthrough, Llama 3, which he introduces as eight findings, including some not yet publicly announced.

0:380:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:38 · section reference included

Insert the beginning-of-sequence token once

A fine-tune can train successfully and still encounter a different input format at inference. Han’s first Llama 3 example is duplicate beginning-of-sequence tokens: training examples start with two BOS tokens, while inference prompts start with one. He reports lower inference accuracy from this mismatch. Unsloth checks for the extra token and removes it automatically; Han notes that the same issue also affects Mistral and Gemma.

Llama 3 needs a BOS token, but only one component should insert it. The distinction is between rendering a chat template and subsequently tokenizing that rendered text. A template can already contain the special tokens, so adding them again during tokenization duplicates them. Current Hugging Face chat-template guidance makes the two safe paths explicit: tokenize inside apply_chat_template, or render first and then tokenize with add_special_tokens=False.

python

def tokenize_conversation(tokenizer, messages):
    return tokenizer.apply_chat_template(
        messages,
        tokenize=True,
        add_generation_prompt=False,
    )


def tokenize_rendered_conversation(tokenizer, messages):
    rendered = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=False,
    )
    return tokenizer(
        rendered,
        add_special_tokens=False,
    )["input_ids"]

Both functions assume a model-appropriate template that supplies its required special tokens. Inspect the resulting token sequence before training; checking only the human-readable prompt can miss an extra insertion by the tokenizer.

2:152:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:15 · section reference included

A chat template can reference untrained embeddings

The next mismatch is between the checkpoint and its template. Llama 3 Base and Llama 3 Instruct do not have the same training history for special tokens. Han warns that applying the instruct template to the base model can introduce untrained tokens and produce NaN gradients during fine-tuning. His list includes reserved special tokens numbered 0–250, the end-of-turn token, and the start-header and end-header tokens. The embedding-mean graph shows some entries at zero: tokens deliberately left unused in the base model.

Slide showing an embedding-mean graph beside reserved, end-of-turn, start-header, and end-header token labels.Open full source frame
Untrained special tokens in Llama-3 Base.

The proposed repair initializes untrained rows from the mean of the trained rows. Unsloth performs this automatically. Exclude untrained rows from the mean itself, including its denominator. Otherwise, zero rows dilute the initialization. Han’s example of 10,000 untrained tokens illustrates this arithmetic; it is not a count of the affected Llama 3 tokens. The Llama 3 companion article describes the same masked-mean repair.

python

import torch


@torch.no_grad()
def initialize_untrained_rows(weight, untrained_mask):
    # weight: [vocabulary_size, embedding_dimension]
    # untrained_mask: one Boolean per vocabulary row
    untrained_mask = untrained_mask.to(
        device=weight.device, dtype=torch.bool
    )
    trained_mask = ~untrained_mask
    if not trained_mask.any():
        raise ValueError("At least one trained row is required")
    if not untrained_mask.any():
        return

    mean_row = weight[trained_mask].float().mean(dim=0)
    weight[untrained_mask] = mean_row.to(weight.dtype)

The mask must identify the untrained rows for the checkpoint being repaired. The function leaves trained rows unchanged and computes the average only over them.

For the ordinary fine-tuning path, Han’s recommendation is to use the instruct template with the instruct checkpoint. If the task requires training the base model with these tokens, another remedy is to train the token embeddings and language-model head rather than leaving those parameters frozen. That lets the newly used token representations learn alongside the rest of the adaptation.

3:243:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:24 · section reference included

Do not mask away the instruction to stop

Padding creates a subtler failure. In a training pipeline that masks labels by the padding token’s identity, assigning PAD and EOS the same token ID removes both from the cross-entropy loss. The real end-of-sequence token then provides no stopping supervision. Han describes the resulting symptom as infinite generation and points to Phi-3’s matching PAD and EOS IDs. The mechanism depends on how labels are masked: sharing an ID is dangerous here because the masking rule cannot distinguish genuine endings from padding.

Consider a teaching sequence with a content token 42, an EOS token 2, and two padding positions. If PAD is also 2, masking every occurrence of the pad ID turns [42, 2, 2, 2] into labels [42, -100, -100, -100]. Assigning padding a distinct ID, here 0, lets the same rule preserve EOS: [42, 2, 0, 0] becomes [42, 2, -100, -100]. The -100 labels are ignored by the loss.

Han describes Unsloth’s repair as selecting an available untrained token for padding, or adding a new token when necessary. Adding one requires checking the vocabulary for collisions; his implementation appends extra hash characters until the proposed token string is unique.

A distinct padding ID preserves EOS supervision

Constructed example: Token IDs 42, 2 and 0, the four-position sequence, and the use of ignore label -100 are teaching values, not token IDs or a sequence shown in the recording. The comparison illustrates the described repair, not an executed training result.

One sequence with two padding positions — unchanged
Content token 42 → EOS token 2 → padding → padding

Operation: Assign padding the distinct token ID 0 instead of EOS ID 2, then rebuild labels by masking positions whose token ID equals PAD.

Position 1: content

Before: PAD shares EOS ID 2
Token 42; label 42
After: PAD has distinct ID 0 · Unchanged
Token 42; label 42

Position 2: actual ending

Before: PAD shares EOS ID 2
Token 2; label -100 (ignored)
After: PAD has distinct ID 0 · Changed
Token 2; label 2 (supervised)

Position 3: padding

Before: PAD shares EOS ID 2
Token 2; label -100 (ignored)
After: PAD has distinct ID 0 · Changed
Token 0; label -100 (ignored)

Position 4: padding

Before: PAD shares EOS ID 2
Token 2; label -100 (ignored)
After: PAD has distinct ID 0 · Changed
Token 0; label -100 (ignored)
Padding remains excluded from the loss while the real end-of-sequence token becomes a training target.
5:315:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:31 · section reference included

Carry the training template into deployment

The same formatting contract must survive export. For Ollama, the chat template must match the one used during fine-tuning. Han introduces automatic generation of the Ollama Modelfile to avoid reconstructing that template by hand. His two Colab workflows cover an Alpaca dataset and a user-uploaded CSV, both ending in an Ollama deployment.

Three community findings extend the checks into conversion and inference:

  • GGUF conversion: Han reports that the affected conversion workflow required CPU conversion rather than GPU conversion. He tentatively attributes the difference to float16 conversion precision. This is a historical workaround for the workflow he describes, not a general prohibition on GPU conversion.
  • Duplicate-BOS warnings: A community contribution added a warning in llama.cpp when BOS tokens are duplicated. The warning identifies the same training-and-serving mismatch discussed earlier.
  • System prompts: Some users found that including a system prompt improved results with Llama 3 Instruct. Han suggests trying it, especially when a prompt may have been omitted; he gives no measured comparison.
6:507:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:50 · section reference included

Train sensitive parameters carefully, and offload asynchronously

Han directs readers to the open-source Unsloth repository, where the Start Free Fine-Tune entry opens a corrected Colab notebook. By this point in the presentation, he says the Llama 3 fixes have been pushed to the repository. He also points to Discord support and blog posts covering Gemma, Llama 3, Phi-3 and continued pretraining.

Continued pretraining brings the embedding and output layers back into the optimization problem. Han recommends training the token embeddings and language-model head with learning rates roughly 5–10 times lower than the other trained parameters. The intent is to adapt these layers without updating them as aggressively as the rest of the fine-tune.

Han reports four times longer context with approximately 1–2% slower training completion using Unsloth’s offloaded gradient checkpointing. The talk does not specify the benchmark configuration. He describes moving gradients into system RAM; the accompanying engineering explanation clarifies that the checkpointing mechanism offloads activations. RAM is preferred before disk because disk transfers can make the fine-tune much slower. Even RAM offloading can lose its benefit if transfers block computation: the crucial implementation detail is non-blocking transfers, allowing data movement and GPU work to overlap.

8:388:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:38 · section reference included

Choose sequence length and memory settings from the data

The live walkthrough opens the Ollama Colab notebook, installs Unsloth and starts with max_seq_length. This setting should reflect the long sequences actually available for training. Setting it to one million or ten million does not create long-context training data; Han’s deliberately oversized example shows why the configuration must match the dataset’s content.

Next comes load_in_4bit. Han describes four-bit loading as reducing memory use by four times and warns that disabling it may push usage to 16 GB in the demonstrated setup. He recommends leaving it enabled on the free Colab Tesla T4 and reserving higher-precision loading for larger GPUs. The talk does not separate weight storage from total training memory, so the fourfold figure should not be used as a total-memory sizing rule.

The model-name field takes a Hugging Face model identifier. Han names Llama, Mistral, Gemma and Phi-3, then encourages trying models beyond the examples listed in the notebook. Compatibility still depends on the architecture and the installed Unsloth version; an identifier by itself does not establish support. The notebook frame shows model-loading settings alongside completed download bars before the walkthrough moves to adapter configuration.

Colab notebook displaying model-loading settings, Unsloth runtime information, and green download bars at 100 percent.Open full source frame
Model loading code and completed download output in the Unsloth notebook.
10:3410:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:34 · section reference included

Give LoRA enough capacity without exhausting memory

get_peft_model adds the PEFT LoRA adapters. The notebook starts with rank r=16; Han suggests 16, 32, 64 or 128 as practical choices. Powers of two are a preference, not a requirement. Increasing rank gives the adaptation more capacity, but also increases memory use and the risk of overfitting. He gives the model dimension as an upper-bound intuition, using 4,096 as his example, rather than recommending that rank as a starting point.

Han recommends targeting all seven named attention and feed-forward projections rather than adapting only a subset:

Part of the modelLoRA target modules
Attentionq_proj, k_proj, v_proj, o_proj
Feed-forward networkgate_proj, up_proj, down_proj

For lora_alpha, he recommends at least the rank and reports better results with twice the rank, illustrated by r=16 and lora_alpha=32. These settings control different things: rank sets the adapter’s capacity, while alpha participates in scaling its contribution.

One API distinction matters when following the recording. Han describes use_rslora=True as choosing alpha automatically. In current PEFT LoRA documentation, it instead changes the scaling rule; rank and alpha remain configured values.

Standard LoRA scale=αrRank-stabilized LoRA scale=αr\begin{aligned} \text{Standard LoRA scale} &= \frac{\alpha}{r} \\ \text{Rank-stabilized LoRA scale} &= \frac{\alpha}{\sqrt{r}} \end{aligned}

Finally, the notebook offers Unsloth’s gradient-checkpointing mode for long-context fine-tuning. Han contrasts it with setting the option to ordinary True, which he describes as using more memory in this workflow.

12:1512:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:15 · section reference included

Turn a CSV row into a repeatable conversation

For data preparation, Han uploads a Titanic CSV. The task is to predict whether a passenger survived from attributes such as age, fare and embarkation location. Those feature columns must become the model’s input, with survival as the output.

The demonstrated notebook expects an instruction/output pair. When a CSV has many feature columns, merge them into one instruction instead of discarding information. For example, this Python function formats a row while keeping the target out of the prompt:

python

def passenger_example(row):
    instruction = (
        "Predict whether this Titanic passenger survived.\n"
        f"Age: {row['Age']}\n"
        f"Fare: {row['Fare']}\n"
        f"Embarked: {row['Embarked']}"
    )
    output = "survived" if int(row["Survived"]) == 1 else "died"
    return {"instruction": instruction, "output": output}

Han contrasts this with Alpaca-style data, which separates instruction, input and response. Combining the instruction and auxiliary input gives the notebook a single user-side field and one response field. This is the interface of the export workflow he demonstrates, not a general two-column restriction imposed by Ollama or GGUF.

The custom-template step then requires two input/output repetitions. One exchange does not fully specify how the next exchange joins it: trailing newlines and separators can become ambiguous. Han says the second repetition resolves these dangling-newline boundaries. His Llama 3 template example also contains two iterations, and he warns that omitting the second causes this notebook’s template processing to error. The repeated example defines the conversation boundary that must later be reproduced during inference.

13:4413:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:44 · section reference included

Accumulate gradients, then check the inference prompt

Han’s suggested training configuration uses a per-device batch size of 2 and gradient_accumulation_steps=4. For the single-device notebook, these give an effective batch size of eight:

Beffective=Bmicrobatch×Naccumulation=2×4=8\begin{aligned} B_{\text{effective}} &= B_{\text{microbatch}} \times N_{\text{accumulation}} \\ &= 2 \times 4 = 8 \end{aligned}

A larger microbatch requires more batch-related memory at once. Increasing accumulation instead lets multiple forward/backward passes contribute to one optimizer update while retaining the smaller microbatch. Other choices, including sequence length and model configuration, still affect overall memory use.

For the learning rate, Han suggests 2e-4 or smaller, with 2e-5 as another starting value. After training, the notebook runs inference using apply_chat_template. This returns to the first failure mode: the inference prompt must preserve the training format, including exactly one BOS token where required. Han notes that Unsloth handles the duplicate-BOS correction in this path.

15:2915:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:29 · section reference included

Export the model and the configuration that makes it usable

The final notebook steps carry the fine-tune into Ollama:

  1. Install Ollama for the deployment environment.
  2. Save the model as GGUF, specifying multiple output formats if needed. The notebook can generate multiple GGUF outputs automatically instead of requiring a separate save workflow for each one.
  3. Inspect the generated Modelfile and copy it into a custom Ollama setup when appropriate.

Han says the earlier workflow for producing multiple GGUF outputs required an extra ten minutes; he does not specify the model, hardware or formats for that timing. The generated Modelfile is the more consequential artifact: it carries the prompt template and stop parameters into the serving environment. Han identifies its automatic generation as the complicated part of the integration.

Notebook showing code to start Ollama and printed Modelfile text containing a prompt template and stop parameters.Open full source frame
The generated Ollama Modelfile displayed in Colab.

The walkthrough ends with Ollama inference. The exported weights now have a corresponding serving configuration, so the model can receive the same conversation structure it learned during fine-tuning. Han points back to the Ollama chat-template notebook in the talk slides as the end-to-end path from data preparation to that final inference step.

16:1116:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:11 · section reference included

Resources

From the talk

  • The April 2024 explanation of activation offloading, with maximum-context estimates and benchmark conditions.

Updates since the talk

Transcript download
Full transcript

Loading transcript…