Pretraining vs fine-tuning, and when not to

Meera’s one-line question from the DeepSeek lesson, “Could we teach a small model our own homework?”, had turned into a plan by Monday. Arjun had a slide titled “CrumbGPT”. The idea: take an open 8-billion-parameter model and train it on Crumb’s 200-page store operations handbook. “Then it will know our policies,” he said. “No more retrieval, no more long prompts. Ask anything.”

Priya asked one question. “When the Christmas menu changes on 1 December, what happens to CrumbGPT?”

“We… retrain it?”

“Every time a price changes? And how will you know it learned the refund policy rather than something that sounds like the refund policy?”

Arjun went ahead anyway, on a rented GPU, over a weekend. On Monday he had numbers, and they were not what he expected.

Two very different kinds of training

You have met both kinds of training already in this course. Here they are side by side.

Pretraining is the first, enormous phase. A model starts with random numbers and learns to predict the next token (a chunk of text, roughly three quarters of a word) across a huge slice of the internet, books and code. Meta reported training its Llama 3 models on over 15 trillion tokens, using millions of GPU-hours. A common rule of thumb says training takes about $6\times \text{parameters}\times \text{tokens}$ arithmetic operations. For an 8-billion-parameter model on 15 trillion tokens:

$$6\times (8\times 10^9)\times (15\times 10^{12})\approx 7.2\times 10^{23}\text{ operations}$$

That is far beyond any bakery’s budget, or most companies’.

Fine-tuning starts from a pretrained model and continues training on a small, focused dataset. Suppose Crumb fine-tunes on 2,000 support conversations of about 400 tokens each, passing over them 3 times (3 epochs). That is 2.4 million tokens:

$$6\times (8\times 10^9)\times (2.4\times 10^6)\approx 1.2\times 10^{17}\text{ operations}$$

About six million times less work than pretraining. Hours on one GPU, not months on thousands.

There are a few flavours of fine-tuning you will hear about:

• Supervised fine-tuning (SFT), also called instruction tuning when the examples are instructions and answers: show the model input-output pairs and train it to produce the outputs. This is what most people mean by “fine-tuning”.
• Preference tuning: RLHF or DPO from Module 32, using “this answer, not that one” pairs.
• Continued pretraining: more next-token training on raw domain text (legal documents, medical notes) without question-answer structure. This is what Arjun did with the handbook.

What Arjun’s weekend actually produced

Arjun fine-tuned the 8B model on the handbook’s raw text, then tested it on the 300-chat evaluation set Crumb had been building since Module 23. He also ran the alternatives for comparison. “Correct” means the answer matched current policy; “on-brand” means a reviewer judged the tone right for Crumb.

Approach	Correct	On-brand	Notes
Base instruct model, short system prompt	61%	55%	Polite but generic, often guessed policy
Same, with 5 example replies in the prompt	64%	78%	Tone improved a lot
Retrieval (RAG) over the handbook	89%	60%	Facts right, tone stiff and long
Fine-tuned on handbook text (CrumbGPT)	58%	52%	Worse than doing nothing

“It got worse?” said Meera.

Arjun had looked at the failures. CrumbGPT now confidently quoted refund windows that did not exist (“refunds within 14 days of delivery”; the real policy is 48 hours). It mixed the Christmas 2023 prices with the current menu. And it had become oddly worse at ordinary conversation, sometimes answering a customer’s question with a paragraph that read like a handbook section.

“So what happened?” Arjun asked.

Priya’s new hire

“Think about hiring a new counter assistant at Camp,” Priya said. “You have two ways to prepare them. One: make them memorise the 200-page handbook. Two: put the handbook in a binder under the counter, and spend their first month with Shanti, watching how she talks to customers. Which works?”

“The binder, for facts. Shanti, for… how to be.”

“Right. What does the memoriser do when a customer asks something they half-remember?”

“Makes up something that sounds like the handbook.”

“And when the handbook changes?”

“They keep quoting the old one.”

“Fine-tuning is the month with Shanti. It is very good at teaching habits: tone, format, what to include, what to leave out, how to handle a type of request. It is a poor way to install facts, especially facts that change. For facts, you hand the model the binder at the moment it needs it. That’s retrieval, from Module 09.”

Where the analogy breaks: a person who memorises a handbook either remembers a fact or knows they have forgotten it. A fine-tuned model blurs what it saw into its weights and cannot tell you which “memories” are reliable. It produces the half-remembered version with full confidence.

Following one customer message through all four approaches

Take one real message from the test set:

“Hi, my son has a nut allergy. Is the Diwali dry-fruit cake safe? We need it by Saturday.”

Base model, short prompt. “Thank you for reaching out! Many dry-fruit cakes contain nuts, so please check the label…” Safe-ish, generic, does not know Crumb’s products, and ignores the Saturday deadline.

Few-shot prompt. The five examples taught it Crumb’s warm, short style. It now writes like Shanti, but still guesses the facts: “Our dry-fruit cake is made in a nut-free kitchen” (false).

RAG. The retriever finds the handbook’s allergy section and the product sheet: contains almonds, cashews and pistachios; made in a facility handling nuts; order cut-off for Saturday delivery is Thursday 6 p.m. The answer is correct and complete, but reads like a policy document: three paragraphs, no warmth.

CrumbGPT (fine-tuned on handbook text). “Our dry-fruit cakes are nut-free on request with 24 hours’ notice.” There is no such option. Confident, on-brand-ish, invented.

The pattern is the whole lesson. RAG supplied the facts. Few-shot examples supplied the style. Neither alone was enough, and fine-tuning on raw facts made things worse.

Opening the hood: what fine-tuning changes, and what it can’t

Fine-tuning runs the same training loop as pretraining: predict tokens, measure the loss, nudge the weights. The difference is the data and the starting point. Starting from a pretrained model means almost everything the model knows is already in place; a few thousand examples only steer it.

Steering works well on things that show up consistently in every example: a tone, a length, a structure, always asking for a photo on damage complaints, always replying in the customer’s language. Each example reinforces the same pattern hundreds of times.

Specific facts are different. “Refund window: 48 hours” appears once. One example is not enough for the model to store it reliably, and nothing in training teaches it to say “I’m not sure” when it hasn’t. Researchers have even found that fine-tuning on many new facts can increase hallucination, because the model learns to answer confidently beyond what it actually knows.

There is a second risk: catastrophic forgetting. Training hard on a narrow dataset can wear away general abilities the model had before. CrumbGPT’s handbook-style answers to simple questions were a mild case. The usual defences are small learning rates, few epochs, mixing in some general conversation data, and, as you will see next lesson, training only a small add-on rather than the whole model.

The decision ladder

Experienced teams climb this ladder and stop at the first rung that meets the bar on their evaluation set:

flowchart TD
    A["1. Better prompt and system prompt"] --> B["2. Few-shot examples in the prompt"]
    B --> C["3. RAG for facts that live in documents"]
    C --> D["4. Fine-tune for behaviour the prompt can't hold"]
    D --> E["5. Continued pretraining (rarely worth it)"]
    E --> F["6. Pretraining from scratch (almost never)"]

Each rung costs more to build and maintain than the one above it. Fine-tuning makes sense when:

• Style or format must be consistent and a prompt does not hold it reliably across thousands of replies.
• A narrow task runs at high volume, and a small fine-tuned model can replace a big general one: cheaper, faster, private.
• The prompt is getting huge. Ten pages of instructions and examples on every call cost tokens and time; a fine-tune can bake them in.
• The behaviour is hard to describe but easy to show, like “sound like Shanti”.

Fine-tuning is usually the wrong tool when:

• The knowledge changes (prices, menus, stock, policies). Use RAG or tools.
• You need citations to show where an answer came from.
• You have fewer than about a hundred good examples, or examples of mixed quality.
• You have no evaluation set. Without one you cannot tell whether fine-tuning helped, as Arjun discovered.

What good fine-tuning data looks like

Priya reframed the project. “Don’t teach it the handbook. Teach it Shanti.” The plan: fine-tune on the best of Crumb’s real support conversations to fix tone and format, and keep RAG for the facts.

Fine-tuning data for a chat model is a list of conversations in the same messages format you have used with every API since Module 01: a system message, a user message and the ideal assistant reply. One example per line in a JSONL file (JSON Lines: one JSON object per line).

{"messages": [{"role": "system", "content": "You are Crumb & Co.'s friendly support assistant. Be warm, brief and accurate."}, {"role": "user", "content": "My cake arrived squashed. What can you do?"}, {"role": "assistant", "content": "Oh no, I'm sorry! Could you send a quick photo? We'll send a fresh one today or refund you in full, whichever you prefer."}]}

Quality beats quantity. Researchers have shown that a strong pretrained model can be steered remarkably well with around a thousand carefully chosen examples. Five hundred excellent replies beat five thousand mixed ones, because the model learns the mistakes as faithfully as the good habits.

Here is the script Arjun used to turn Crumb’s support export into training data. It keeps only replies rated “good” by the team lead, removes duplicates, scrubs personal details and holds back 50 examples for evaluation.

import csv, json, random, re

PHONE = re.compile(r"(\+91[\s-]?)?[6-9]\d{4}[\s-]?\d{5}")      # Indian mobile numbers
EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
ORDER = re.compile(r"\bCR-\d{6}\b")                              # Crumb order ids

def scrub(text: str) -> str:
    """Replace personal details with placeholders before they reach a training set."""
    text = PHONE.sub("[PHONE]", text)
    text = EMAIL.sub("[EMAIL]", text)
    return ORDER.sub("[ORDER_ID]", text)

SYSTEM = "You are Crumb & Co.'s friendly support assistant. Be warm, brief and accurate."

def to_example(customer: str, agent: str) -> dict:
    return {"messages": [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": scrub(customer)},
        {"role": "assistant", "content": scrub(agent)},
    ]}

with open("support_export.csv", newline="", encoding="utf-8") as f:
    rows = [r for r in csv.DictReader(f) if r["agent_rating"] == "good"]   # keep the best replies

seen, examples = set(), []
for r in rows:
    key = r["customer_message"].strip().lower()
    if key not in seen:                                                    # drop exact duplicates
        seen.add(key)
        examples.append(to_example(r["customer_message"], r["agent_reply"]))

random.seed(42)
random.shuffle(examples)
test, train = examples[:50], examples[50:]                                 # hold 50 back for evaluation
for name, data in [("train.jsonl", train), ("test.jsonl", test)]:
    with open(name, "w", encoding="utf-8") as f:
        for ex in data:
            f.write(json.dumps(ex, ensure_ascii=False) + "\n")
print(len(train), "train,", len(test), "test")
print(scrub("Call me on +91 98765 43210 about CR-204518, or mail asha.k@gmail.com"))

The last line should print Call me on [PHONE] about [ORDER_ID], or mail [EMAIL]. ensure_ascii=False keeps Hindi and Marathi readable in the file instead of turning them into escape codes. The fixed random.seed(42) makes the split repeatable, so your test set stays the same between experiments.

The memory problem waiting for you

Arjun’s weekend run had one more surprise: the bill. To fine-tune every weight of the 8B model, a GPU must hold far more than the weights. With the standard Adam optimiser and mixed precision, a common estimate is about 16 bytes per parameter:

What	Bytes per parameter
Weights in 16-bit	2
Gradients in 16-bit	2
A 32-bit master copy of the weights	4
Adam’s two running averages, 32-bit each	8
Total	16

$8\times 10^9\times 16$ bytes is 128 GB, before counting the activations stored during training. That needs two 80 GB GPUs, or more. He had rented them for a weekend to train the model that made things worse.

def full_finetune_gb(params_billion: float, bytes_per_param: int = 16) -> float:
    """Rough GPU memory for full fine-tuning with Adam, excluding activations."""
    return params_billion * bytes_per_param

for size in (1, 8, 70):
    print(f"{size}B model: about {full_finetune_gb(size):.0f} GB")

A 70B model needs over a terabyte for full fine-tuning. Next lesson’s tools, LoRA and QLoRA, bring an 8B fine-tune down to a single free Colab GPU.

Where it breaks

No evaluation, no fine-tuning. Arjun’s 300-chat test set is what revealed that CrumbGPT was worse. Build the evaluation first, record a baseline, then fine-tune (Module 23 covers how).

Overfitting small datasets. Too many epochs on a few hundred examples and the model starts reciting training replies word for word. Watch the loss on the held-out set, not only the training loss.

Licences. Open models come with licences that may restrict use or require attribution, and some API providers’ terms forbid training on their outputs. Check both before you start.

Maintenance. A fine-tuned model is a new artefact to version, evaluate and redeploy whenever the base model or your data changes. Budget for that, not only the first training run.

Try it yourself

For each Crumb request, pick the lowest rung on the ladder likely to meet the bar, and say why:

1. “Always reply in the same language the customer wrote in.”
2. “Answer questions about today’s stock at each store.”
3. “Classify 50,000 emails a month into 12 queues, cheaply, on our own server.”
4. “Make the assistant sound like Shanti across thousands of replies; the prompt version drifts.”

Show the answer

1. Prompt. A clear system-prompt instruction usually holds this; add a couple of examples if needed.
2. Tools or RAG. Stock changes by the hour; call the inventory system (Module 06) or retrieve fresh data. Never fine-tune it in.
3. Fine-tune a small model. High volume, narrow task, cost and privacy matter: a fine-tuned small classifier or small LLM is the classic fit.
4. Fine-tune for style, keep RAG for facts. Consistent tone across thousands of replies is exactly what fine-tuning is good at.

Common mistakes

• Fine-tuning to add facts. It teaches the style of facts better than the facts themselves. Use retrieval.
• Skipping the baseline. Measure prompt-only, few-shot and RAG first; fine-tuning must beat them to be worth maintaining.
• Training on everything you have. Filter for your best examples; the model copies mistakes too.
• Too many epochs on small data. Leads to memorised replies and forgotten general skills.
• Forgetting personal data. Scrub before training, every time.

Recap

• Pretraining builds broad knowledge from trillions of tokens at enormous cost; fine-tuning steers an existing model with hundreds to thousands of examples at roughly a millionth of the compute.
• Fine-tuning reliably teaches habits (tone, format, narrow tasks) and unreliably teaches facts, especially changing ones.
• Climb the ladder: prompt, few-shot, RAG, then fine-tune; stop at the first rung that passes your evaluation.
• Good data is a JSONL file of high-quality, scrubbed conversations in the messages format, with a held-out test split.
• Full fine-tuning needs about 16 bytes per parameter (128 GB for an 8B model), which is why parameter-efficient methods exist.

Arjun’s second slide was titled “Shanti in a box”. He explained it to Meera without any charts.

“The first version tried to make the model memorise the handbook, like a new hire cramming the binder. It got the tone and invented the facts. The new plan is the other way round: the model reads the handbook at the moment it needs it, through search, and we train it only on our best real replies so it talks like Shanti. Facts from the binder, manners from training.”

“And it won’t need two giant computers this time?”

“That’s the next problem,” Priya said. “There’s a way to fine-tune by changing less than one percent of the model. It fits on a free GPU.”

Go deeper

• Hugging Face LLM course, including its chapters on fine-tuning.
• Hugging Face docs, see the TRL library for supervised fine-tuning.
• Training language models to follow instructions with human feedback, for the SFT stage in context.
• Unsloth documentation, which includes guidance on preparing datasets.