AI Engineer Summit 2023
Domain adaptation and fine-tuning for domain-specific LLMs
Loading slides…
Read the talk
Domain adaptation and fine-tuning for domain-specific LLMs
Adapting a language model requires more than choosing a fine-tuning method: the domain, training data, memory budget, evaluation, and operating pipeline all shape the result.
From a talk by Abi Aryan
Before you start: Familiarity with neural-network weights, attention, and the distinction between training and inference will help with the adaptation mechanisms.
When a general model does not know your domain
How do you make a general-purpose language model useful in a domain that its training data barely represents? Some specialist knowledge is scarce; some cannot enter public training corpora because of compliance restrictions. Abi Aryan begins with this practical gap: a broadly capable model does not automatically cover every enterprise or hobby use case. Fine-tuning is one response, but knowledge bases, retrieval augmented generation (RAG), and prompting also belong in the decision.
A second motivation is avoiding a fresh data-collection effort for every domain. Aryan uses language transfer to illustrate the attraction of reusable representations: embeddings can capture structure that helps a model transfer between related settings. She tentatively offers English and Latin as an example of similarity. That analogy motivates transfer, but shared structure alone does not establish that a model can reliably handle an entirely unseen language without examples. The broader goal is access: adaptation can also personalize a model for users whose needs the general model does not adequately serve.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Locate the parameters before choosing an update
Fine-tuning changes learned parameters to improve a pretrained model for a new requirement. The initial neural-network picture is familiar: inputs pass through hidden layers, whose weights and biases determine the outputs. Aryan then moves to an encoder–decoder transformer diagram, with attention and feed-forward networks on both sides. This is the architectural reference for the methods that follow, rather than a claim that every language model has both an encoder and a decoder. The useful question is where adaptation enters that computation and which parameters remain fixed.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Full updates versus selective adaptation
The first choice is whether to update all model weights or a much smaller set. Aryan connects full updates to earlier transfer-learning and teacher–student distillation work from the 2016–2018 period: a student learns from a teacher, with training changing the student's parameters. Distillation describes the source of supervision, however, rather than requiring a particular parameter-update scope. The practical pressure is compute and storage. As models grow, repeatedly training and storing complete task-specific copies becomes expensive. Selective adaptation aims to retain useful performance with less trainable state; matching ChatGPT is not a guaranteed consequence.
The talk organizes the next methods as adapter tuning, prefix tuning, and parameter-efficient tuning. More precisely, parameter-efficient fine-tuning (PEFT) is the umbrella that includes adapters and learned prefixes as well as low-rank updates. Aryan also mentions instruction tuning through examples and sets aside reinforcement learning from human feedback because collecting human judgments is expensive. Examples used for instruction tuning become training data; examples placed only in a prompt do not update weights. That distinction separates training from in-context learning.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Add small adapter modules
In adapter tuning, small trainable components are inserted into the existing network. Aryan's illustration adds two components containing extra weights while leaving the original transformer intact. These modules provide a place to learn domain-specific changes without updating the entire pretrained model. This is the central mechanism of adapter-based parameter-efficient transfer learning.
Aryan reports matching model performance with only 0.15% of the parameters, but does not specify the task, benchmark, or parameter denominator. Treat that number as an attributed result, not a capacity-planning target. Her application example is biochemical engineering: a substantially different domain whose terminology and patterns may require adaptation. She uses “adapter” and “adaptive” fine-tuning interchangeably here; in implementation, keep the distinction between an adapter module, which is a mechanism, and domain adaptation, which is a goal.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Steer attention with a learned prefix
Prefix tuning keeps the base model frozen and learns continuous prefix vectors that subsequent tokens can attend to. Aryan describes this as adding an embedding layer in front of attention. Unlike an ordinary text instruction, the prefix consists of trainable values: optimization changes those values while preserving the pretrained parameters.
The analogy is a water tank and a tap. The tank supplies the water; the tap shapes how it comes out. Similarly, a learned prefix steers how the model uses its existing capabilities. Aryan also calls this a masking layer, but the literal mechanism is conditioning attention on learned vectors, not masking or replacing the base weights. The model's output behavior changes even though its pretrained parameters do not.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn a low-rank update, not a smaller base model
LoRA, or Low-Rank Adaptation, and QLoRA enter the discussion through resource constraints. Aryan points to laptops and even Arduino-class devices as motivations for efficiency. Those examples express the desire to work with limited resources; reducing fine-tuning memory does not by itself make a large language model deployable on a microcontroller. Training memory, adapter storage, and inference requirements are separate budgets.
To explain rank, Aryan asks the reader to imagine matrix rows or columns that repeat information. If one is a scalar multiple of another, it does not add an independent direction. That intuition helps explain why a change to a large matrix might be represented with fewer degrees of freedom. LoRA applies that low-rank constraint to the learned update. It does not inspect the pretrained matrix and delete redundant rows or columns. The base weight matrix stays frozen, while two smaller matrices are trained.
For a base matrix W₀ with output dimension d and input dimension k, the forward computation is:
Here, r is the chosen rank and α controls the update's scale. The following PyTorch module makes the frozen base and trainable update explicit:
python
import torch
from torch import nn
class LoRALinear(nn.Module):
def __init__(self, base: nn.Linear, rank: int, alpha: float):
super().__init__()
if rank <= 0:
raise ValueError("rank must be positive")
self.base = base.requires_grad_(False)
self.scale = alpha / rank
self.A = nn.Parameter(
base.weight.new_empty(rank, base.in_features)
)
self.B = nn.Parameter(
base.weight.new_zeros(base.out_features, rank)
)
nn.init.normal_(self.A, std=0.02)
def forward(self, x: torch.Tensor) -> torch.Tensor:
update = (x @ self.A.T) @ self.B.T
return self.base(x) + self.scale * update
Initializing B to zero makes the initial update zero, so the layer initially preserves the base output. Training changes A and B, not base.weight. This reduces trainable parameters and their associated gradient and optimizer state; it does not automatically shrink the frozen base model. That is the precise memory benefit behind the talk's broader model-size discussion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Quantize the frozen weights with QLoRA
QLoRA adds another memory-saving choice: the frozen base weights use four-bit storage, and gradients pass through that base into trainable LoRA adapters. Aryan describes a workflow of starting with a pretrained model, collecting labeled target data, and training adaptation matrices. The low-rank matrices multiply each other to form an update whose contribution is added to the base computation; they are not simply multiplied into the main weight matrix. Likewise, reducing a distance between source-domain and target-domain predictions is her description of the adaptation goal, not the defining QLoRA objective. The training loss still depends on the task and dataset.
The implementation tradeoff is support. Aryan flags dependencies such as bitsandbytes, warns that device availability and cross-model efficiency testing are limited, and favors optimizing LoRA at the time of this October 2023 talk. That is historical implementation advice: current library support covers additional hardware configurations, so today's choice requires checking the actual backend and device rather than carrying forward a blanket preference.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompting, retrieval, and training can work together
The discussion now returns to the larger choice among prompting, RAG, and fine-tuning. Prompting can provide no demonstrations, one demonstration, or several: zero-shot, one-shot, and few-shot use. Aryan suggests about ten few-shot examples, citing ChatGPT rather than a task-specific experiment. She presents additional examples as helpful and prompting as cheaper and less data-intensive, while characterizing it as more suited to general demonstrations than demanding applications. Neither a fixed example count nor a blanket performance ranking against fine-tuning follows from that discussion; the examples and task determine whether the extra context helps.
These are composable choices. Fine-tuning can shape a model's behavior, prompting can specify the current request, and retrieval can supply relevant information at inference time. Aryan explicitly allows prompt engineering and RAG alongside fine-tuning, as well as combining domain-focused and behavior-focused training. Choosing one does not require abandoning the others.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Separate the domain, the task, and the update budget
Consider Aryan's legal-company example. A company may want a model that handles five or ten tasks across legal work, or it may want to optimize one specific task. She calls the first goal adaptive fine-tuning and the second behavioral fine-tuning. For the latter, the focus narrows to the task's label space and prior probability distribution: what outputs are possible, and how frequently they occur. She suggests inference and reasoning as potential uses and offers LangChain functions as an analogy for task-focused behavior, rather than as a training mechanism.
| Scope | Question it answers | Example or mechanism |
|---|---|---|
| Domain adaptation | What subject area should improve? | Several tasks within legal work |
| Behavioral tuning | What specific behavior should improve? | One task and its output space |
| Parameter efficiency | How much trainable state can we afford? | Freeze the base; train a small update |
The first two rows describe the goal; the third describes how to pursue it. LoRA or QLoRA can therefore support either domain-focused or task-focused training. They are not competing objectives on the same axis.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The adaptation method depends on the data
Once the method is chosen, data quality becomes the limiting factor. Aryan's preparation sequence covers collection, tokenization, cleaning, normalization, and noise removal. These choices determine what information the model actually receives—not merely whether the training job can load the dataset.
Deduplication is especially consequential. Repeated records expose the model to the same material again and again, increasing its influence on training and encouraging memorized outputs. Aryan attributes much memorization to duplication; the narrower supported conclusion is that duplication contributes to it, not that removing duplicates eliminates memorization. After deduplication, she adds data augmentation to the preparation toolkit. Augmentation should expand useful variation rather than recreate the repeated exposure that cleaning was meant to reduce.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Specialization does not guarantee broad generalization
Even with a suitable base model and carefully prepared data, specialization has limits. Aryan warns against expecting a fine-tuned model to match GPT-4 on broad, complex tasks merely because it learned the nuances of one dataset. Her reference to GPT-5 also occurs in this 2023 discussion, not as a measured comparison with that later model. The actionable distinction is between performance inside the target domain and generalization to new data or entirely different domains.
For changing inputs, Aryan proposes in-context learning with dynamically loaded examples. The examples can change with the request or current data, providing an inference-time way to address drift while managing costs. She then recommends decomposing broad objectives into smaller tasks: instead of attempting to train for an entire language at once, identify the particular operations the application needs. A narrower task makes both adaptation and evaluation more concrete.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trade recomputation for activation memory
Gradient checkpointing addresses a different part of the training-memory budget. Normally, training retains intermediate activations from the forward pass so the backward pass can calculate gradients. Checkpointing retains fewer intermediates and recomputes the missing ones when needed. Aryan describes this as recomputing weights or retraining; the objects being recomputed are intermediate activations, not a second set of learned weights. The result is additional forward computation in exchange for less activation storage. Whether that trade is worthwhile depends on whether memory or computation is the tighter constraint.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Treat training settings as hypotheses
Aryan suggests batch sizes of 32 or 64 without specifying a model, dataset, or memory budget. She suggests one epoch for a casual Google Colab test and 100 epochs as a starting point for domain optimization. These are the talk's proposed settings, not validated defaults: a useful training budget depends on dataset size, repeated exposure, and held-out performance. A smoke test and a domain-training run answer different questions, so success at the former does not establish the duration of the latter.
Adam is her general-purpose optimizer recommendation. She then turns to regularization and early stopping, arguing that increasing parameter counts can distract from extracting better performance from existing models. Early stopping means monitoring a validation criterion and stopping when further training no longer improves it. If performance declines, Aryan recommends inspecting the batches and embeddings instead of assuming that more training or a larger model is the remedy.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate from four perspectives
After fine-tuning comes the harder question: did the application improve? Aryan mentions Aviary by Ray and NVIDIA libraries among the available tooling, but emphasizes that loss, accuracy, and perplexity do not provide a complete picture. Evaluation must reflect the business and use case. The historical Aviary repository now leads to the archived RayLLM repository, so that reference should not be treated as a recommendation for a currently maintained evaluation service.
- Metric-based evaluation: BLEU and ROUGE provide quantitative comparisons against reference outputs. They offer one perspective on quality rather than a complete judgment of usefulness.
- Tool-based evaluation: Aryan mentions Weights & Biases debugging tools and an Auto Evaluator, associating automated checks with catching compilation errors. The precise products and integrations are not specified, so the actionable category is automated detection of concrete failures.
- Model-based evaluation: Another model, potentially a smaller one, judges the model being tested. Aryan sees potential for automation and standardization but reports limited success in her own experience.
- Human-in-the-loop evaluation: Human review remains a common option, although Aryan considers it inefficient and does not explore it further in this session.
These perspectives expose different kinds of failure. A strong aggregate metric cannot substitute for checking whether the system produces a valid, useful result for its intended users.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The unit of reliability is the whole pipeline
Fine-tuning is only one part of the operating system around a model. Data collection, storage management, and base-model selection constrain what training can achieve. The closing pipeline diagram extends that scope through pretraining, fine-tuning and prompt engineering, evaluation, orchestration, deployment and monitoring, and security and reliability engineering. A one-off demonstration can succeed while leaving several of those responsibilities unresolved.
A durable application must keep working as its data and prompts change. Aryan does not prescribe a fixed testing duration; she focuses instead on detecting problems early enough to avoid exposing an enterprise to reputational or compliance risk. The adaptation decision therefore includes how the system will be monitored, how drift will be recognized, and how failures will be caught—not just which parameters the next training run will update.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Learned continuous prefixes steer frozen language models for generation and summarization.
The original method for adapting frozen models with trainable low-rank matrices.
Four-bit frozen model weights combined with LoRA adapters to reduce fine-tuning memory.
Further reading
Original BERT adapter research, with task-specific parameter and GLUE performance results.
Experiments connecting training-data duplication, memorized output and evaluation contamination.
Foundational research on trading extra forward computation for lower training memory.
Updates since the talk
Quantization library with installation guidance and a current hardware support matrix.
Full transcript
Loading transcript…