LoRA, QLoRA and parameter-efficient fine-tuning

Arjun’s new plan, “Shanti in a box”, had good data: 2,000 of Crumb’s best support replies, scrubbed and split. What it did not have was hardware. Full fine-tuning of the 8B model needed about 128 GB of GPU memory, and the weekend rental that had produced CrumbGPT had cost Crumb ₹38,000.

“What do we have for free?” Meera asked.

“Google Colab gives a free T4 GPU with 16 GB,” Arjun said. “The model’s weights alone are 16 GB. It’s not even close.”

He tried anyway. The Colab notebook crashed with CUDA out of memory before the first training step. So he tried something cleverer: freeze the whole model except the very last layer, and train only that. It fitted. It trained. And after an hour, the replies sounded exactly as generic as before.

“You tried to change how the whole kitchen cooks by retraining only the person at the pass,” said Priya. “Style lives everywhere in the network, not in one layer. But you were right about one thing: you don’t need to change all the weights. You need to change all the layers, a tiny bit, in a clever way.”

Why full fine-tuning is so heavy

Recall last lesson’s memory table. Full fine-tuning with the Adam optimiser needs about 16 bytes per parameter: the weights, their gradients, a 32-bit master copy, and Adam’s two running averages. Notice that only the first 2 bytes are the model itself. The other 14 bytes per parameter exist because that parameter is being trained.

So the cost is not “having a big model”. It is “training every number in a big model”. If most of the 8 billion numbers were frozen, they would need only their 2 bytes each, and the 14-byte training overhead would apply only to the small set being trained.

Arjun’s last-layer experiment used exactly that logic. It failed for a different reason: he froze the wrong things. Tone, length and phrasing are shaped by every layer’s attention and feed-forward weights. Changing only the top layer is like trying to change a cake’s texture by adjusting only the icing.

Priya’s sticky notes

“Imagine Crumb’s master recipe book,” Priya said. “Hundreds of recipes, perfected over years. Meera wants every recipe adjusted for the new ‘less sweet’ range. Do you rewrite the book?”

“No. Sticky notes. ‘Reduce sugar 20%, add a pinch of salt.’”

“Do you need a different, unique note for every recipe?”

Arjun thought. “Not really. It’s the same few ideas applied everywhere: less sugar, a bit more salt, maybe more vanilla to compensate. Each recipe gets some mix of those.”

“That’s the key. The changes you want are simple, even though the book is huge. A few underlying adjustments, mixed in different amounts per recipe. Mathematicians would say the change has low rank. LoRA bets that adapting a model works the same way: the change you want, ‘sound like Shanti’, is a handful of directions, applied in different amounts across the network.”

LoRA stands for low-rank adaptation, introduced by Microsoft researchers in 2021. It keeps the original weights frozen and, next to each chosen weight matrix, learns a small correction described by a few “themes”.

Where the analogy breaks: sticky notes say “less sugar” in words you chose. LoRA’s themes are learned numbers that no one picks; you only choose how many there are.

Following one weight matrix through LoRA

Take a weight matrix $W$ from one layer. In a real model it might be $4096\times 4096$; we will use a $4\times 4$ one. Full fine-tuning would learn a change $\Delta W$ of the same size: 16 numbers here, 16.8 million in the real one.

LoRA instead writes the change as the product of two thin matrices: $B$, which is $4\times r$, and $A$, which is $r\times 4$. The rank $r$ is the number of themes. With $r=1$:

$$B=\left[\begin{array}{c}1.0\\ 0.5\\ 0.0\\ -1.0\end{array}\right],A=\left[\begin{array}{cccc}0.2& 0.0& -0.4& 0.1\end{array}\right]$$

Multiply them (each entry is a row of $B$ times a column of $A$):

$$\Delta W=BA=\left[\begin{array}{cccc}0.2& 0& -0.4& 0.1\\ 0.1& 0& -0.2& 0.05\\ 0& 0& 0& 0\\ -0.2& 0& 0.4& -0.1\end{array}\right]$$

Read it like the recipe book. $A$ is the one theme: “+0.2 of input 1, -0.4 of input 3, +0.1 of input 4”. $B$ says how much of that theme each output row gets: all of it for row 1, half for row 2, none for row 3, the opposite for row 4. A full $4\times 4$ change, described by 8 numbers instead of 16.

The saving grows dramatically with size. For a $4096\times 4096$ matrix:

• Full change: $4096\times 4096=16,777,216$ numbers.
• LoRA with $r=8$: $4096\times 8+8\times 4096=65,536$ numbers. 0.39% of the full change.

Now the whole model. Apply LoRA with $r=16$ to all seven weight matrices in each of the 32 layers of Crumb’s Llama-3-style 8B model (its attention matrices are $4096\times 4096$ for queries and outputs and $4096\times 1024$ for keys and values; its feed-forward matrices are $4096\times 14336$). The adapters add up to about 42 million trainable parameters: roughly 0.5% of the model. The training overhead of 14 bytes per parameter now applies to 42 million numbers, about 0.6 GB, instead of 8 billion.

Opening the hood: the LoRA formula

For a frozen weight matrix $W_0$ and input $x$, a LoRA layer computes:

$$h=W_{0}x+\frac{\alpha }{r}BAx$$

• $W_0$ is frozen: no gradients, no optimiser state.
• $A$ ($r\times d_{in}$) and $B$ ($d_{out}\times r$) are the only trained numbers.
• $\alpha$ (alpha) is a scaling constant. Dividing by $r$ means that if you change the rank, you do not have to retune the learning rate as much. Common settings are $r=16$ with $\alpha =16$ or $32$.

Why should a low-rank change be enough? The LoRA authors’ hypothesis, supported by their experiments, is that the update needed to adapt a pretrained model has low “intrinsic rank”: the model already knows how to write; you are only nudging which of its existing skills it leans on. On GPT-3 (175 billion parameters), they reported reducing trainable parameters by about 10,000 times and GPU memory by about 3 times, with quality on par with full fine-tuning on their tasks.

Two practical bonuses follow from the formula.

No slowdown when serving. After training, you can merge the adapter into the weights once: $W=W_0+\frac{\alpha }{r}BA$. The merged model has exactly the original shape and speed.

Tiny, swappable files. Unmerged, an adapter is a small file: 42 million parameters in 16-bit is about 84 MB, against 16 GB for the model. You could keep one adapter for support replies, another for writing product descriptions, and load whichever you need on top of the same base model.

A LoRA layer from scratch

Before using a library, build one. This wraps an ordinary PyTorch linear layer.

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    """A frozen linear layer plus a small trainable low-rank correction."""
    def __init__(self, base: nn.Linear, r: int = 8, alpha: int = 16):
        super().__init__()
        self.base = base
        for p in self.base.parameters():
            p.requires_grad = False                                    # freeze the original
        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.01)   # small random
        self.B = nn.Parameter(torch.zeros(base.out_features, r))         # zeros: no change at start
        self.scale = alpha / r

    def forward(self, x):
        return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale

torch.manual_seed(0)
base = nn.Linear(4096, 4096, bias=False)
lora = LoRALinear(base, r=8)
x = torch.randn(2, 4096)
print("same output at start:", torch.allclose(base(x), lora(x)))
trainable = sum(p.numel() for p in lora.parameters() if p.requires_grad)
frozen = sum(p.numel() for p in lora.parameters() if not p.requires_grad)
print(f"trainable: {trainable:,}  frozen: {frozen:,}  ({100 * trainable / frozen:.2f}%)")

same output at start: True
trainable: 65,536  frozen: 16,777,216  (0.39%)

requires_grad = False is how you freeze weights in PyTorch: the optimiser never touches them and no gradients are stored. Notice the order of operations in forward: x @ A.T first squeezes the 4,096 inputs down to 8 numbers, then @ B.T expands them back to 4,096. The full $4096\times 4096$ correction is never built, which saves memory and time.

QLoRA: freeze it smaller

LoRA solved the training overhead, but the frozen 8B model still needs 16 GB in 16-bit, plus room for activations. On a 16 GB T4, that still does not fit.

QLoRA (2023) combines LoRA with the quantisation you met in Module 32: store the frozen base model in 4 bits, keep the small LoRA adapters in 16-bit, and train only the adapters. When a layer runs, its 4-bit weights are converted back to 16-bit on the fly for the arithmetic; gradients flow through them into the adapters, but the frozen weights never change.

Three ideas from the QLoRA paper make this work well:

• NF4 (4-bit NormalFloat). Pretrained weights roughly follow a bell curve, so NF4 places its 16 levels where weights are dense instead of spacing them evenly. Less rounding error than plain 4-bit integers.
• Double quantisation. The per-group scales themselves are quantised, saving roughly another third of a bit per parameter.
• Paged optimisers. When memory briefly spikes, optimiser state is moved to ordinary CPU memory and back, avoiding out-of-memory crashes.

The headline result: the authors fine-tuned a 65-billion-parameter model on a single 48 GB GPU, and reported matching 16-bit fine-tuning quality on their benchmarks.

For Crumb’s 8B model, the rough budget becomes:

Item	Memory
Frozen base weights in 4-bit (with scales)	about 5 GB
LoRA adapters, gradients and Adam state (42M parameters, 16 bytes each)	about 0.7 GB
Activations during training (with gradient checkpointing, short sequences)	a few GB
Total	comfortably under 16 GB

Gradient checkpointing is a standard trick that saves memory by not storing every intermediate result during the forward pass and recomputing them during the backward pass instead, trading time for memory.

Using the PEFT library

Hugging Face’s PEFT library (parameter-efficient fine-tuning) wraps all of this. Here it is on a small model you can run on a laptop, SmolLM2-135M, which has the same layer names as Llama.

# uv add transformers peft torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")
config = LoraConfig(
    r=16,                        # rank: how many "themes" per matrix
    lora_alpha=32,               # scaling; the update is multiplied by alpha / r
    lora_dropout=0.05,           # light regularisation on the adapter path
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",      # attention matrices
                    "gate_proj", "up_proj", "down_proj"],        # feed-forward matrices
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, config)
model.print_trainable_parameters()

trainable params: 4,884,480 || all params: 139,399,488 || trainable%: 3.5039

On this tiny model the adapters are 3.5% of the total; on an 8B model with the same settings it is about 0.5%, because big matrices shrink relatively more. The target_modules names depend on the model family; look at print(model) to see a model’s layer names.

From here, the model trains with any ordinary training loop or trainer. Afterwards:

model.save_pretrained("crumb-support-lora")   # saves only the adapter: megabytes, not gigabytes
merged = model.merge_and_unload()             # fold B·A into the weights for serving
merged.save_pretrained("crumb-support-merged")

For QLoRA, you load the base model in 4-bit with a BitsAndBytesConfig before adding the adapters. This needs an NVIDIA GPU, such as Colab’s T4:

# uv add bitsandbytes   (NVIDIA GPU required)
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",              # NormalFloat 4-bit
    bnb_4bit_use_double_quant=True,         # quantise the scales too
    bnb_4bit_compute_dtype=torch.bfloat16,  # do the arithmetic in 16-bit
)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct",
                                             quantization_config=bnb, device_map="auto")
model = prepare_model_for_kbit_training(model)   # enables gradient checkpointing and other fixes
model = get_peft_model(model, config)            # same LoraConfig as before

Llama models on Hugging Face require accepting Meta’s licence on the model page first. Library APIs change quickly. If a name has moved, check the docs linked below.

Other parameter-efficient methods

LoRA is the most popular member of a family called parameter-efficient fine-tuning (PEFT). Others you may meet:

• Prompt tuning and prefix tuning: learn a few “virtual tokens” that are prepended to every input, leaving all weights frozen. Very small, but usually weaker than LoRA on harder tasks.
• DoRA: a LoRA variant that separates the size and direction of each weight change, sometimes giving better results at the same rank (use_dora=True in PEFT’s LoraConfig).
• Adapters (the original kind): small extra layers inserted into each block. They add a little latency, which LoRA’s merge avoids.

Where it breaks

Rank too small for big changes. “Sound like Shanti” is a small shift that fits in a low rank. Teaching a model a language it barely knows is a big shift; a small rank may not hold it, and continued pretraining or a different base model may be needed.

Learning rate surprises. LoRA usually wants a higher learning rate than full fine-tuning (around 1e-4 to 2e-4 is a common start, versus 1e-5 or lower for full fine-tuning). Copying a full fine-tuning recipe often makes LoRA learn almost nothing.

Which matrices to target. The original paper adapted only the attention query and value matrices. Later experience, including the QLoRA paper’s, suggests adapting all linear layers works noticeably better for chat models, for a modest increase in adapter size.

Merging into a quantised model. If you trained on a 4-bit base, merge the adapter into a 16-bit copy of the base, then quantise again for serving. Merging directly into the 4-bit weights adds rounding error.

Adapters are tied to their base. An adapter trained on Llama-3.1-8B-Instruct will not work on a different model, or even reliably on a later revision of the same one. Record the exact base model and version with every adapter.

Try it yourself

1. A layer’s weight matrix is $4096\times 14336$. How many trainable parameters does a LoRA adapter with $r=16$ add, and what percentage of the matrix is that?
2. Change r=16 to r=4 in the PEFT example and rerun print_trainable_parameters(). Roughly what happens to the trainable count?

Show the answer

1. $16\times 4096+14336\times 16=65,536+229,376=294,912$ parameters, against $4096\times 14336=58,720,256$ in the matrix: about 0.5%.
2. It falls to about a quarter (roughly 1.2 million), because adapter size grows in direct proportion to $r$. The total parameter count barely moves.

Common mistakes

• Freezing everything except the last layer. Style and behaviour live throughout the network. Adapt every layer a little, not one layer a lot.
• Initialising both LoRA matrices randomly. The model is damaged before training starts. $B$ starts at zero.
• Using full fine-tuning learning rates. LoRA usually needs higher ones.
• Losing track of the base model. Save the exact base model name and revision with each adapter.
• Merging into 4-bit weights. Merge into 16-bit, then quantise for serving.

Recap

• Full fine-tuning’s memory cost comes mostly from training every parameter: about 14 of the 16 bytes per parameter are training overhead.
• LoRA freezes the model and learns a low-rank correction $\frac{\alpha }{r}BA$ for each chosen matrix, training well under 1% of the parameters.
• $B$ starts at zero so training begins from the unchanged model; after training, the adapter can be merged with no slowdown, or kept as a small swappable file.
• QLoRA stores the frozen base in 4-bit NF4 with double quantisation and paged optimisers, fitting an 8B fine-tune on a single 16 GB GPU.
• The PEFT library provides LoraConfig, get_peft_model, save_pretrained and merge_and_unload; target all linear layers and record the base model.

Arjun’s message to Meera that evening was short. “Good news: we don’t need the big rental. We freeze the whole model, shrink it to 4-bit so it fits on Google’s free GPU, and train only a set of tiny correction notes clipped onto every layer: half a percent of the model. The notes come out as an 84 MB file.”

“So it costs nothing?”

“The GPU is free. The work is in the data and the testing.” Priya had already sent him a Colab link. “And there’s a library called Unsloth that makes the whole thing faster and simpler. Tomorrow we fine-tune on the real support replies, end to end, and run the result in Ollama on our own server.”

Go deeper

• LoRA: Low-Rank Adaptation of Large Language Models
• QLoRA: Efficient Finetuning of Quantized LLMs
• Hugging Face docs, see the PEFT and bitsandbytes libraries.
• Unsloth documentation