Quantisation and distillation

Meera had the GPU invoice in her hand. “We’re paying about ₹95,000 a month for that 24 GB server. The provider has a 16 GB card for less than half that. And the Camp store manager asked if the assistant could run on the back-office PC when the internet drops. Can it?”

“Not as it is,” said Arjun. “The model’s weights alone are 16 GB. They won’t fit on a 16 GB card with room for conversations, and they certainly won’t fit on a shop PC.”

“Then make the model smaller.”

“It has eight billion parameters. I can’t delete half of them.”

Priya pulled up a chair. “You don’t need fewer numbers. You need smaller numbers. And for one of your other jobs, you might want a smaller model that learned from the big one. Two tools today: quantisation and distillation.”

Where the 16 GB comes from

A model’s parameters (also called weights) are the numbers it learned during training. Crumb’s model has about 8 billion of them. Each is stored as a floating-point number, a format that can represent tiny fractions and huge values alike, using a fixed number of bits (binary digits).

Format	Bits per weight	Bytes per weight	8 billion weights
FP32 (32-bit float)	32	4	32 GB
BF16 / FP16 (16-bit float)	16	2	16 GB
INT8 (8-bit whole number)	8	1	8 GB
INT4 (4-bit whole number)	4	0.5	4 GB

Models are usually shipped in 16-bit, hence 16 GB. If each weight could be stored in 4 bits, the model would be about 4 GB, small enough for the cheaper card with plenty of room for the KV cache from last lesson, and small enough for a shop PC.

There is a speed bonus too. Remember that writing each token (the decode phase) is limited by how fast the GPU can read the weights from memory, not by arithmetic. Read a quarter of the bytes, and each token can come out noticeably faster.

Arjun’s first attempt: convert the numbers directly

“So convert the weights to 8-bit integers,” Arjun said, and tried exactly that:

import torch
w = torch.tensor([0.12, -0.50, 0.03, 0.31])   # four typical weights
print(w.to(torch.int8))                        # tensor([0, 0, 0, 0], dtype=torch.int8)

Every weight became zero. Real model weights are mostly small fractions, around 0.01 to 0.5. An 8-bit integer can only hold whole numbers from -128 to 127, and converting a fraction to an integer throws away everything after the decimal point. He had built an 8-billion-parameter model that knew nothing.

“You turned the cake prices into crore,” said Priya. “Whole numbers are fine. The unit is wrong.”

Priya’s measuring jug

“Picture a measuring jug with only 255 marked lines,” Priya said. “If you use a jug marked in litres to measure a pinch of saffron, every pinch reads zero. So what do you do?”

“Use a smaller jug for the saffron. Marked in grams.”

“And for the flour?”

“A big jug, marked in kilos.”

“So each ingredient gets a jug sized to it, and you write down which jug you used. That’s quantisation. For each group of weights, pick a scale so the biggest weight in the group lands on the top mark. Store each weight as ‘which mark’, a small whole number, plus one scale per group. To use a weight, multiply its mark by the scale.”

“With only 255 marks, some weights land between lines.”

“So they get rounded. That rounding error is the price. The question is whether the model notices.”

Following four weights through quantisation

Take the same four weights: [0.12, -0.50, 0.03, 0.31].

8-bit: 127 marks each side of zero

Step 1: find the scale. The largest weight by size is 0.50. To put it on the top mark, 127:

$$s=\frac{0.50}{127}=0.003937$$

Step 2: convert each weight to a mark by dividing by the scale and rounding:

• $0.12/0.003937=30.5\to 30$
• $-0.50/0.003937=-127$
• $0.03/0.003937=7.6\to 8$
• $0.31/0.003937=78.7\to 79$

Stored: [30, -127, 8, 79], each fitting in one byte, plus the one scale.

Step 3: when the model runs, convert back by multiplying by the scale: [0.1181, -0.5000, 0.0315, 0.3110]. The errors are at most about 0.002. The model will barely notice.

4-bit: only 7 marks each side

Now the scale is $0.50/7=0.0714$. The marks become $[1.7\to 2,-7,0.4\to 0,4.3\to 4]$, which convert back to [0.143, -0.500, 0.000, 0.286]. The errors are ten times bigger, and the small weight 0.03 has vanished entirely.

One outlier ruins the group

Now add a fifth weight to the same group: an unusually large 4.0. Models really do have a few weights and activations far larger than the rest, called outliers.

• 8-bit: scale $4.0/127=0.0315$. The small weights become [0.126, -0.504, 0.0315, 0.315]. Coarser, but alive.
• 4-bit: scale $4.0/7=0.571$. The small weights become [0.0, -0.571, 0.0, 0.571]. Every carefully learned small value is flattened to zero or plus-minus 0.57.

One big ingredient forced a giant jug on everyone in its group.

Proving it on a real model

Arjun wanted numbers, not intuition. He took GPT-2 and quantised every weight matrix inside its 12 transformer blocks, then measured perplexity on a typical Crumb message. Perplexity is a standard measure of how surprised a language model is by a text: $e$ raised to its average next-token loss. Lower is better; it roughly means “how many tokens the model was choosing between, on average”.

The code simulates quantisation: round to the marks, then immediately convert back, so you can measure the damage on any computer.

import math, torch
from transformers import AutoTokenizer, AutoModelForCausalLM

def quantize(w, bits, group_size=None):
    """Symmetric absmax quantisation, then dequantise back to floats (simulated)."""
    qmax = 2 ** (bits - 1) - 1                     # 127 for 8-bit, 7 for 4-bit
    shape = w.shape
    w = w.reshape(-1, group_size) if group_size else w.reshape(1, -1)
    scale = w.abs().amax(dim=1, keepdim=True) / qmax              # one scale per group
    q = torch.clamp(torch.round(w / scale), -qmax, qmax)          # the marks
    return (q * scale).reshape(shape)                             # back to floats

print(quantize(torch.tensor([0.12, -0.50, 0.03, 0.31]), bits=8))  # the worked example

reshape(-1, group_size) cuts the weights into rows of group_size, and amax(dim=1) finds the largest absolute value in each row, giving one scale per group. Without a group size, the whole matrix shares one scale.

Now apply it to GPT-2 and measure:

tok = AutoTokenizer.from_pretrained("gpt2")
text = ("Thank you for ordering from Crumb and Co. Your eggless chocolate cake is out for delivery "
        "and should reach you within forty minutes. If anything is wrong with your order, reply to "
        "this message and our support team will help you with a replacement or a refund. We hope "
        "you enjoy your cake, and we look forward to baking for you again this Diwali.")
ids = tok(text, return_tensors="pt")["input_ids"]

def perplexity(model):
    with torch.no_grad():
        return math.exp(model(ids, labels=ids).loss.item())

def load():
    return AutoModelForCausalLM.from_pretrained("gpt2").eval()

print("original 32-bit:", round(perplexity(load()), 2))
for bits, group in [(8, None), (4, None), (4, 64), (3, 64)]:
    model = load()
    with torch.no_grad():
        for name, p in model.named_parameters():
            if p.dim() == 2 and ".h." in name:     # weight matrices inside the blocks
                p.copy_(quantize(p, bits, group))
    print(f"{bits}-bit, group={group}:", round(perplexity(model), 2))

original 32-bit: 25.03
8-bit, group=None: 26.18
4-bit, group=None: 2588.25
4-bit, group=64: 29.9
3-bit, group=64: 113.65

Read it slowly, because this table is the whole lesson:

• 8-bit, one scale per matrix: perplexity 25.0 to 26.2. Half the memory, barely any damage.
• 4-bit, one scale per matrix: 2,588. The outlier effect at full scale. The model is destroyed.
• 4-bit, one scale per 64 weights: 29.9. A quarter of the memory, a modest, measurable loss. Grouping rescued it.
• 3-bit, groups of 64: 113. Below 4 bits, simple rounding falls apart.

GPT-2 is small, and small models suffer more from quantisation than large ones; an 8-billion-parameter model usually loses less at 4 bits. But the shape of the result holds widely: 8-bit is close to free, 4-bit with care costs a little, and below that you need cleverer methods or should expect real damage.

Opening the hood: the formula and the real methods

The scheme you built is symmetric absmax quantisation. For $b$ bits and a group of weights $w$:

$$s=\frac{\max |w|}{2^{b-1}-1},q=\text{round}\left(\frac{w}{s}\right),\hat{w}=s\cdot q$$

$s$ is the scale (the jug), $q$ the stored whole number (the mark), and $\hat{w}$ the value the model actually uses. The rounding error $w-\hat{w}$ is at most $s/2$ per weight, which is why smaller groups (smaller $s$) help.

Production methods add cleverness on top, mostly applied after training, which is called post-training quantisation:

• GPTQ quantises weights one at a time and adjusts the remaining ones to compensate for each rounding error, using a little sample text to see which errors matter.
• AWQ (activation-aware weight quantisation) notices that a small fraction of weights matter most, because the inputs they multiply are large, and protects them.
• NF4 (4-bit NormalFloat, from the bitsandbytes library) spaces its 16 levels to match the bell-curve shape of typical weights instead of spacing them evenly. You will meet it in QLoRA in Module 33.
• GGUF is the file format used by llama.cpp and Ollama, which you used in Module 20. Names like Q8_0, Q5_K_M and Q4_K_M describe its quantisation recipes: roughly 8, 5 and 4 bits per weight with grouped scales. A 4-bit Q4_K_M file of an 8-billion-parameter model is about 5 GB, slightly more than 4 GB because of scales and a few layers kept at higher precision.

The alternative, quantisation-aware training, simulates rounding during training so the model learns to tolerate it. It gives better results at very low bit counts but needs a training run.

You can quantise more than weights. Many servers store the KV cache in 8-bit to fit more conversations, the lever from last lesson.

Distillation: a different kind of smaller

Quantisation keeps the same model and shrinks its numbers. Arjun had a second problem that needed a different tool.

Since Module 30, his ticket router had been a small BERT-style classifier trained on 1,500 hand-labelled emails. It was fast but missed subtle cases. The big chat model handled subtle cases beautifully but cost real money per email and took seconds. Could the small model learn the big model’s judgement?

Knowledge distillation (popularised by Geoffrey Hinton and colleagues in 2015) trains a small student model to imitate a large teacher. The key insight is about what the student copies.

Following one email through distillation

Take an email: “Is your eggless cake safe for my son? He’s allergic to eggs.” The three queues are product question, allergy and refund. A human labelled it product question. That hard label says: 100% product question, 0% anything else.

The teacher sees more. Its raw output scores (logits) are [4.0, 2.0, 0.5], which softmax turns into probabilities:

	Product question	Allergy	Refund
Hard label	1.00	0.00	0.00
Teacher, $T=1$	0.858	0.116	0.026
Teacher, $T=2$	0.649	0.239	0.113

The teacher’s 0.116 on allergy is information the hard label throws away: “this is a product question, but it smells strongly of allergy”. Hinton called this dark knowledge. A student trained on these soft probabilities learns not only the answer but which answers are near-misses, which helps it generalise from fewer examples.

The temperature $T$ (the same knob you met for sampling in Module 01) softens the probabilities further by dividing the logits before the softmax. At $T=2$, the allergy signal rises to 0.239, easier for the student to learn from.

The distillation loss

The student is trained on a blend of two goals:

$$\mathcal{L}=\alpha \cdot T^2\cdot \text{KL}(p_T^{\text{teacher}}\Vert p_T^{\text{student}})+(1-\alpha )\cdot \text{CE}(y,p^{\text{student}})$$

The first term is the KL divergence (a measure of how different two probability distributions are) between the teacher’s and the student’s softened probabilities. The second is ordinary cross-entropy against the true label $y$. $\alpha$ balances them. The $T^2$ factor keeps the soft term’s training signal at a sensible size, because dividing logits by $T$ shrinks its gradients by about $T^2$.

import torch
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, labels, T=2.0, alpha=0.7):
    """Blend 'match the teacher's soft opinions' with 'get the true label right'."""
    soft_teacher = F.softmax(teacher_logits / T, dim=-1)
    log_soft_student = F.log_softmax(student_logits / T, dim=-1)
    soft = F.kl_div(log_soft_student, soft_teacher, reduction="batchmean") * T * T
    hard = F.cross_entropy(student_logits, labels)
    return alpha * soft + (1 - alpha) * hard

teacher = torch.tensor([[4.0, 2.0, 0.5]])     # product question, allergy, refund
label = torch.tensor([0])                     # the human label: product question
print(distillation_loss(torch.tensor([[6.0, -2.0, -2.0]]), teacher, label))  # about 1.60
print(distillation_loss(torch.tensor([[4.0, 2.0, 0.5]]), teacher, label))    # about 0.05

PyTorch’s kl_div expects the student’s log-probabilities and the teacher’s plain probabilities, which is why one side uses log_softmax. Getting that backwards is a common silent bug.

In a real training loop, you would run each batch of emails through the frozen teacher (or load its saved outputs) and the student, and update only the student with this loss.

Distillation in the LLM era

For classifiers, the student copies probabilities. For language models there is a simpler, very common form: have the big model write answers to thousands of prompts, then fine-tune a small model on those answers as if they were human-written examples. This is sometimes called sequence-level distillation. It is how many small open models gained surprising skill, including the small “distilled” reasoning models you will meet in the DeepSeek case study.

Famous results show what is possible: DistilBERT’s authors reported a model 40% smaller and 60% faster than BERT that kept about 97% of its language-understanding performance.

Where it breaks

Quantisation hurts some skills more than others. Average scores can hide damage to arithmetic, code, rare languages or long-context recall. Arjun’s 4-bit Crumb model answered menu questions perfectly but got noticeably worse at Marathi. Always evaluate the quantised model on your tasks, with the evaluation habits from Module 23, not only on a benchmark.

Small models are more fragile. The same 4-bit recipe that costs an 8-billion-parameter model a little can cost a 1-billion-parameter model a lot.

Speed gains depend on hardware and kernels. Simulated quantisation, like our experiment, saves nothing; real speedups need kernels that compute directly on the small integers. Some formats are fast on some GPUs and slow on others. Measure tokens per second on the actual machine.

A student cannot exceed its teacher’s knowledge. Distillation transfers skill, mistakes included. If the teacher misroutes a kind of email, the student will too, confidently.

Check the licence. Many API providers’ terms forbid using their models’ outputs to train competing models. Before distilling from a commercial API, read the terms; distilling from an open-weights model with a permissive licence avoids the question.

Try it yourself

1. Quantise [0.8, -0.2, 0.05, -0.6] to 4 bits by hand. What are the scale, the stored marks and the dequantised values?
2. Rerun the GPT-2 experiment with (4, 32) and (4, 128). Which group size gives lower perplexity, and what does it cost?

Show the answer

1. Scale $=0.8/7=0.1143$. Marks: $0.8/0.1143=7$; $-0.2/0.1143=-1.75\to -2$; $0.05/0.1143=0.44\to 0$; $-0.6/0.1143=-5.25\to -5$. Dequantised: [0.800, -0.229, 0.000, -0.571]. The 0.05 disappears.
2. Smaller groups (32) usually give lower perplexity than larger ones (128), because each scale fits its weights more tightly. The cost is more scales to store: one 16-bit scale per 32 weights adds half a bit per weight, against an eighth of a bit for groups of 128.

Common mistakes

• Casting weights straight to integers. Without a scale, small weights become zero. Quantisation always means “whole numbers times a scale”.
• Using one scale for a whole matrix at 4 bits. One outlier wrecks everything. Use grouped scales or a proper method like GPTQ, AWQ or a GGUF K quant.
• Trusting “4-bit is fine” without testing. Measure on your own prompts and languages.
• Forgetting the KV cache. A 4 GB model with 40 long conversations can still run out of memory. Budget both.
• Swapping kl_div’s arguments. It wants student log-probabilities first, teacher probabilities second.

Recap

• Memory per model is parameters times bytes per parameter: 8 billion weights are 16 GB in 16-bit and about 4 to 5 GB in 4-bit.
• Quantisation stores each weight as a small whole number times a per-group scale; rounding error is the price.
• 8-bit is nearly lossless; 4-bit needs small groups or smarter methods (GPTQ, AWQ, NF4, GGUF K-quants); below 4 bits quality drops sharply.
• Distillation trains a small student to match a large teacher’s soft probabilities, which carry “dark knowledge” about near-misses, blended with the true labels.
• For LLMs, distillation often means fine-tuning a small model on a large model’s generated answers; check licences, and evaluate everything on your own tasks.

Arjun’s summary for Meera fitted on one line of the whiteboard: “Same model, smaller numbers: 16 GB becomes 5 GB. Small model, big teacher: the router learns the big model’s judgement.”

“And the quality?”

“Menu and order questions, no difference we could measure on our test set of 300 real chats. Marathi got slightly worse, so Marathi chats still go to the big server. The 4-bit model now runs on the cheaper card, and on the Camp store PC when the internet drops.”

Meera did the arithmetic on the invoice. Then she asked the obvious next question. “If small numbers make it cheaper, could a model with more knowledge cost the same to run?”

“Strangely, yes,” said Priya. “There’s a design where a model holds a huge amount of knowledge but only uses a small slice of itself for each word. And a trick where a small model guesses ahead and the big one checks its work. Next lesson.”

Go deeper

• Hugging Face docs, see the Transformers quantisation guides (bitsandbytes, GPTQ, AWQ).
• Ollama, which serves GGUF-quantised models.
• vLLM documentation, for quantised serving and KV cache quantisation.
• QLoRA, which introduced the NF4 format.