Fine-Tuning with LoRA and Its Relatives: A Practical Guide to PEFT

LoRA has become the default way to adapt an open-weight model: it trains a fraction of a percent of the parameters, fits on one GPU and produces a file small enough to email. Running Models Locally introduces it with a QLoRA setup. This article goes deeper: the wider family of methods, how rank and scaling behave, which variants matter, what careful experiments say, and how to serve adapters in production.

What fine-tuning changes

Fine-tuning continues training a pretrained model on a narrower dataset. Three kinds of training signal are common, and all of them work with the methods below:

• Supervised fine-tuning. SFT trains on example conversations, teaching format, style and narrow skills.
• Preference tuning. Direct preference optimization (DPO) and its relatives train on pairs of better and worse answers, without a separate reward model.
• Reinforcement learning. A reward function or verifier scores the model's answers, and the update favours high-reward outputs. This is how reasoning models learn to think before answering.

Full fine-tuning updates every weight, and most of its cost lies outside the weights themselves: each parameter also needs a gradient and, with Adam, two optimiser statistics, so training takes several times the memory of inference (the numbers are worked out here). Every fully fine-tuned variant is also a complete copy of the model to store and serve.

A 2020 study by Aghajanyan and colleagues explains why a cheaper approach can work. Pretrained language models turned out to have a low intrinsic dimension for fine-tuning: optimising just 200 parameters, randomly projected back into the full weight space, brought RoBERTa to 90% of full fine-tuning performance on the MRPC paraphrase task, and larger models had even lower intrinsic dimension. The change that adaptation needs lives in a small subspace. Parameter-efficient fine-tuning (PEFT) exploits this by restricting the update to a small structure while the original weights stay frozen.

The PEFT family

PEFT methods differ in where they attach to the transformer and in what they learn:

• Adapters. Houlsby and colleagues (2019) inserted small bottleneck networks after the attention and MLP sublayers. Only the adapters train, but they are extra layers on every token and add latency.
• Soft prompts. Prompt tuning learns "virtual token" embeddings prepended to the input; prefix tuning learns key and value vectors prepended inside every attention layer. Prompt tuning catches up with full fine-tuning only at multi-billion-parameter scale, and the prefix occupies context on every call.
• Rescaling. IA³ learns vectors that multiply the attention keys, values and the MLP's inner activations. It trains even fewer parameters than LoRA and can be folded into the weights.
• Selective tuning. BitFit trains only bias terms. Most current LLMs have no biases in their linear layers, so little is left to train.
• Reparameterisation. LoRA and its variants express the change to each weight matrix as a low-rank product beside the frozen matrix, mergeable after training. This is the default for SFT, preference tuning and RL.

GaLore competes for the same hardware budget from another direction: it trains all the weights but projects gradients into a low-rank subspace, shrinking the optimiser state enough that its authors pretrained a 7B model on a 24 GB consumer GPU. It suits updates that genuinely need full rank, such as continued pretraining on a large corpus.

LoRA mechanics in depth

For a frozen weight matrix W₀ with d_out rows and d_in columns, LoRA learns a down-projection A of shape r × d_in and an up-projection B of shape d_out × r, and the layer computes h = W0·x + (α / r)·B·A·x. A starts random and B starts at zero, so training begins from the base model's exact behaviour. The factor α/r sets how strongly the update counts.

Each adapted matrix adds r × (d_in + d_out) parameters. For a 4096 × 4096 attention projection at rank 16 that is 16 × 8,192 = 131,072 parameters, against 16.8 million in the matrix itself: under 0.8%. Rank 16 on all seven linear projections of Llama 3.1 8B (four in attention, three in the MLP, in each of 32 layers) gives about 42 million trainable parameters, roughly half a percent of the model. About two thirds of them sit in the MLP, whose matrices are 4096 × 14,336.

Rank and alpha

Rank caps how many independent directions the update can use: a capacity setting, not a quality dial. A low rank is enough to teach a format or a tone; datasets carrying much new information need more. Alpha acts like part of the learning rate; the LoRA paper notes that with Adam, tuning α is roughly equivalent to tuning the learning rate. Some practitioners tie α to the rank (α = r or 2r); others keep α fixed, which makes the best learning rate nearly rank-independent. Pick one convention and tune the learning rate within it.

Which matrices to adapt

The original paper adapted only the attention query and value projections. Later work, including the study discussed below, found that adapting every linear layer, and the MLP above all, matters more than raising the rank. In Hugging Face PEFT, target_modules="all-linear" does this for most architectures. Embedding and output layers normally stay frozen unless you add new tokens.

Merging

After training, W₀ + (α/r)BA is an ordinary matrix of the original shape. Merge it and the model runs at exactly the base model's speed. Keep it separate and you can swap adapters at run time, at the cost of a small extra matrix multiplication per adapted layer.

LoRA variants worth knowing

Dozens of variants have been published. Those below are implemented in Hugging Face PEFT or in widely used training tools, and each addresses a specific weakness.

Two of these are worth understanding. rsLoRA starts from Kalajdzievski's observation that with the standard α/r factor the update shrinks as rank grows, so rank 256 learns little more than rank 16. Dividing by √r keeps the update's size stable, so a higher rank buys real capacity. DoRA lets the magnitude of each weight train separately from its direction, which plain LoRA struggles to imitate, and still merges into a single matrix. In PEFT both are single flags on LoraConfig.

What careful experiments say

LoRA learns less and forgets less. Biderman and colleagues (2024) compared LoRA with full fine-tuning on code and mathematics, for instruction tuning on about 100,000 examples and for continued pretraining on 20 billion tokens. In standard low-rank settings LoRA learned the target domain substantially worse, but preserved the base model's abilities outside it better than full fine-tuning, weight decay or dropout. Full fine-tuning learned weight changes with a rank 10 to 100 times higher than typical LoRA configurations. So LoRA is a partial defence against catastrophic forgetting, not a cure. Shuttleworth and colleagues (2024) traced its forgetting to intruder dimensions: new high-ranking singular directions in LoRA-trained matrices that full fine-tuning does not create. Scaling them down after training restored much of the pretraining behaviour, and they accumulate when adapters are trained one after another.

LoRA without regret. In September 2025 John Schulman and colleagues at Thinking Machines compared the two methods while sweeping the learning rate separately for every configuration, so that no method lost through a poor setting. They found that:

• for supervised fine-tuning on small and medium datasets, LoRA on all layers matched full fine-tuning. It fell behind only when the data carried more information than the adapter could hold, and then gradually;
• attention-only LoRA underperformed MLP-only LoRA with a comparable parameter count, and adding attention to the MLP brought no further gain;
• LoRA tolerated large batch sizes worse than full fine-tuning, whatever the rank;
• for reinforcement learning even rank 1 matched full fine-tuning in their runs, which they explain by policy-gradient RL extracting only on the order of one bit of information per episode;
• the best LoRA learning rate was about ten times the full fine-tuning one and close to rank-independent, and LoRA needed a little over two thirds of the compute per step.

The two studies fit together once data volume is counted: 20 billion tokens of continued pretraining is the kind of load under which an adapter runs out of capacity. For a few thousand to a few hundred thousand examples of a behaviour, well-tuned LoRA on all layers gives up little or nothing.

A practical training recipe

Most failed fine-tunes fail on data, not hyperparameters. Before training:

• Use the model's chat template and the exact conversation format, system prompt included, that you will use at inference. TRL applies the template automatically to data in the standard messages format.
• Prefer clean to large. A few hundred to a few thousand reviewed, consistent examples usually beat a large noisy set.
• Hold out two test sets first: one for the target task and one of general tasks that must not get worse. Rerun both as evals after every training run.

Reasonable starting points for supervised training of a 7–8B model: all linear layers, rank 16 to 64, α equal to r or 2r, a learning rate around 1–2 × 10⁻⁴ with a short warm-up, an effective batch of 8 to 32 sequences, and one to three epochs. If held-out loss rises while training loss falls, stop earlier or lower the learning rate. Switching on rsLoRA changes the effective scale, so search the learning rate again.

# In a virtual environment: pip install trl peft datasets
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

# One conversation per line:
# {"messages": [{"role": "system", "content": "..."},
#               {"role": "user", "content": "..."},
#               {"role": "assistant", "content": "..."}]}
data = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,                  # update scaled by lora_alpha / r
    lora_dropout=0.05,
    target_modules="all-linear",    # attention and MLP projections
    task_type="CAUSAL_LM",
    # use_dora=True,                # DoRA: separate magnitude and direction
    # use_rslora=True,              # rsLoRA: lora_alpha / sqrt(r); retune the LR
)

args = SFTConfig(
    output_dir="ticket-triage-lora",
    learning_rate=2e-4,             # roughly 10x a typical full fine-tuning LR
    num_train_epochs=2,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,  # effective batch: 16 conversations
    bf16=True,
    logging_steps=10,
)

trainer = SFTTrainer(
    model="meta-llama/Llama-3.1-8B-Instruct",
    args=args,
    train_dataset=data["train"],
    eval_dataset=data["test"],
    peft_config=peft_config,
)
trainer.train()
trainer.save_model()                # saves the adapter, not a copy of the base model

The same peft_config works with TRL's DPOTrainer for preference data and GRPOTrainer for reinforcement learning. To fit a larger model, load the base in 4-bit for QLoRA as shown in Running Models Locally.

Serving, merging and combining adapters

Merging folds an adapter into the base weights and produces an ordinary model that any runtime can serve, including llama.cpp and Ollama after conversion to GGUF. Keeping adapters separate lets one copy of the base model serve many of them: ten fine-tunes of an 8B model are either ten 16 GB copies or one 16 GB base plus ten small files.

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("ticket-triage-lora")
merged = model.merge_and_unload()    # W0 + (alpha / r)·BA; adapter layers removed
merged.save_pretrained("triage-merged")
AutoTokenizer.from_pretrained("ticket-triage-lora").save_pretrained("triage-merged")

# Then, in a llama.cpp checkout:
#   python convert_hf_to_gguf.py triage-merged --outfile triage-f16.gguf
#   llama-quantize triage-f16.gguf triage-q4_k_m.gguf Q4_K_M

Requests for different adapters arrive mixed together, and switching adapters between batches would leave the GPU idle. S-LoRA (2023) showed how to serve thousands of adapters from one GPU: the base model runs once for the whole batch, custom kernels apply each request's own low-rank product, and adapters move between CPU and GPU memory on demand. vLLM uses the same approach:

# One base model, three adapters, all trained on this exact checkpoint
vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --enable-lora \
    --lora-modules support=./support-lora sql=./sql-lora triage=./ticket-triage-lora \
    --max-lora-rank 64 \
    --max-loras 3

# The adapter name goes in the "model" field
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "triage",
       "messages": [{"role": "user", "content": "Printer on floor 3 shows error 0x61"}]}'

--max-lora-rank must cover the largest adapter's rank, and --max-loras caps how many different adapters share one batch. Serve every adapter on the base model it was trained on: one trained on another checkpoint, or with QLoRA on a 4-bit base, may load without error and still behave differently.

Adapters can also be combined. Task arithmetic treats a fine-tune's weight difference as a vector to add to or subtract from the base model, and TIES-merging improves on plain addition by trimming small changes and resolving sign conflicts first; PEFT supports both for LoRA adapters. Merged skills interfere, so evaluate the combination. For distinct tasks, routing each request to its own adapter is usually safer.

Choosing a method, and what goes wrong

Fine-tuning is the tool for behaviour: format, tone, a narrow task done cheaply by a small model, or a policy learned from preferences or rewards. For knowledge that changes, retrieval is cheaper and easier to audit (see Embeddings, Vector Databases and RAG). Among training methods, start with LoRA on all layers; use QLoRA if memory is short; try DoRA, or a higher rank with rsLoRA, if evaluation shows a gap to a full fine-tuning baseline; and keep full fine-tuning or GaLore for large data volumes.

All of these methods find the update with gradient descent. A November 2026 study explores a different route, in which a frontier LLM edits weights directly by reasoning about what they compute; see Weight-Space Programming. As of November 2026 that is a research result, and LoRA remains the practical tool.