Hands-on fine-tuning with Unsloth

Arjun had everything except a trained model. He had 1,900 scrubbed, high-quality Crumb support conversations in train.jsonl and 50 held back in test.jsonl, from the first lesson of this module. He understood LoRA and QLoRA from the second. He had a free Colab GPU.

“Today it’s real,” Priya said. “Fine-tune a small open model on our support replies, prove it’s better than the original on the test set, and run it in Ollama on our own server. If you can do that, you can fine-tune anything.”

“How long?”

“The training is minutes. The parts that go wrong take longer. They always do.”

They did. This lab follows Arjun’s actual afternoon, mistakes included.

Before you start

You need:

• A Google account for Colab (a free hosted notebook with a GPU). Open a new notebook, then choose Runtime → Change runtime type → T4 GPU.
• Your train.jsonl and test.jsonl from the first lesson of this module. If you skipped it, write 100 to 200 example conversations by hand in the same format; the lab works the same way, with weaker results.
• Ollama installed on your own computer (Module 20).
• Optionally, an Anthropic API key for the automatic evaluation in Step 7.

We will use Unsloth, an open-source library that wraps Hugging Face’s training tools with faster kernels and lower memory use, so a small model fine-tunes comfortably on a free T4. The base model is Llama 3.2 1B Instruct, small enough to train in minutes and to run on an ordinary shop PC afterwards. Its licence (Meta’s Llama licence) allows commercial use with conditions; read it before shipping anything.

Step 1: Set up the GPU and install Unsloth

In the first notebook cell:

!nvidia-smi                 # confirm the GPU: you want to see "Tesla T4" and about 15 GB
!pip install unsloth        # installs Unsloth plus compatible transformers, TRL and PEFT

In Colab, a leading ! runs a shell command. (On your own machine you would use uv add unsloth, but Unsloth needs an NVIDIA GPU, which is why we use Colab.) Installation takes a few minutes.

You should now see a table from nvidia-smi naming a Tesla T4 with about 15 GB of memory, and the install finishing without red errors. If Colab offers to restart the session after installing, accept and rerun nothing but the next cell.

Step 2: Load and inspect the data

Upload train.jsonl and test.jsonl with the folder icon in Colab’s left sidebar, then:

from datasets import load_dataset

data = load_dataset("json", data_files={"train": "train.jsonl", "test": "test.jsonl"})
print(data)
print(data["train"][0]["messages"])

load_dataset("json", ...) from Hugging Face’s datasets library reads JSONL files into a table you can index like a list.

You should now see two splits (Arjun’s had 1,900 train and 50 test rows) and one conversation as a list of three messages: system, user and assistant.

Arjun skimmed twenty examples at random before moving on, and found a problem he would only understand later: several assistant replies contained [ORDER_ID], the placeholder his scrubbing script had inserted. Keep that in mind.

Step 3: Load the base model in 4-bit

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-1B-Instruct",   # Unsloth's copy of Meta's model
    max_seq_length=2048,                           # longest conversation we will train on
    load_in_4bit=True,                             # QLoRA: store the frozen base in 4-bit
)

max_seq_length caps how many tokens per example the trainer handles; Crumb’s conversations are far shorter than 2,048. load_in_4bit=True is the QLoRA idea from the previous lesson: the frozen model sits in memory at about a quarter of its 16-bit size.

You should now see Unsloth’s banner with the GPU name and version numbers, and the model loading in under a minute. Run !nvidia-smi again: the model uses only a gigabyte or two of the T4’s memory.

Step 4: Record a baseline before training

This is the step everyone skips and everyone regrets skipping. Without a “before”, you cannot prove an “after”.

FastLanguageModel.for_inference(model)      # switch Unsloth into fast generation mode

def reply(model, messages, max_new_tokens=200):
    """Generate one assistant reply for a conversation (system + user messages)."""
    inputs = tokenizer.apply_chat_template(
        messages, add_generation_prompt=True, return_tensors="pt").to("cuda")
    output = model.generate(input_ids=inputs, max_new_tokens=max_new_tokens,
                            temperature=0.7, do_sample=True)
    return tokenizer.decode(output[0][inputs.shape[1]:], skip_special_tokens=True)

test_prompts = [ex["messages"][:2] for ex in data["test"]]   # system + user, no answer
baseline = [reply(model, m) for m in test_prompts]
print(test_prompts[0][1]["content"])
print("BASELINE:", baseline[0])

apply_chat_template wraps the messages in the model’s special formatting (more on this in Step 5), and add_generation_prompt=True ends the text with “assistant’s turn”, so the model writes the reply. Slicing output[0][inputs.shape[1]:] keeps only the newly generated tokens.

You should now see the first test customer message and the base model’s reply. Arjun’s first test message was “my cake came squashed, what now”. The baseline replied with four paragraphs: a general apology, advice to “contact the bakery directly”, tips on transporting cakes, and a reminder to check the bakery’s refund policy. Polite, long, and not Crumb at all. Generating all 50 takes a few minutes.

Step 5: Apply the chat template

Every chat model is trained to see conversations in an exact text format, with special tokens marking who is speaking. This is the model’s chat template. Train with the wrong format, or none, and the model learns a pattern it will never see at inference time.

from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(tokenizer, chat_template="llama-3.1")  # Llama 3.x format

def to_text(batch):
    texts = [tokenizer.apply_chat_template(m, tokenize=False, add_generation_prompt=False)
             for m in batch["messages"]]
    return {"text": texts}

train_data = data["train"].map(to_text, batched=True)
print(train_data[0]["text"])

tokenize=False returns the formatted text instead of token ids, so you can read it. add_generation_prompt=False because training examples already contain the assistant’s reply.

You should now see something like this (the template may also add lines such as a knowledge-cutoff date to the system section; that is normal):

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are Crumb & Co.'s friendly support assistant. Be warm, brief and accurate.<|eot_id|><|start_header_id|>user<|end_header_id|>

My cake arrived squashed. What can you do?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Oh no, I'm sorry! Could you send a quick photo? We'll send a fresh one today or refund you in full, whichever you prefer.<|eot_id|>

Those <|...|> markers are single special tokens. <|eot_id|> (“end of turn”) is how the model learns when to stop talking. Remember it; it comes back in Step 8.

Step 6: Add LoRA adapters and train

First, attach the adapters from the previous lesson:

model = FastLanguageModel.get_peft_model(
    model,
    r=16,                                   # rank of each adapter
    lora_alpha=16,                          # scaling (alpha / r = 1)
    lora_dropout=0,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],   # every linear layer
    use_gradient_checkpointing="unsloth",   # trade a little time for a lot of memory
    random_state=3407,
)

Then the trainer. Unsloth uses the SFTTrainer from Hugging Face’s TRL library (supervised fine-tuning):

from trl import SFTTrainer, SFTConfig
from unsloth.chat_templates import train_on_responses_only

trainer = SFTTrainer(
    model=model,
    processing_class=tokenizer,             # older notebooks call this argument tokenizer=
    train_dataset=train_data,
    args=SFTConfig(
        dataset_text_field="text",
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,      # effective batch of 8 examples per update
        num_train_epochs=1,                 # one pass over the data
        learning_rate=2e-4,                 # typical for LoRA
        warmup_steps=5,
        logging_steps=10,
        optim="adamw_8bit",                 # 8-bit optimiser state saves memory
        seed=3407,
        output_dir="outputs",
        report_to="none",
    ),
)
# Only learn from the assistant's words, not from predicting the customer's message
trainer = train_on_responses_only(
    trainer,
    instruction_part="<|start_header_id|>user<|end_header_id|>\n\n",
    response_part="<|start_header_id|>assistant<|end_header_id|>\n\n",
)
trainer.train()

Two settings deserve a sentence each. Gradient accumulation adds up gradients over 4 small batches before each update, so you get the stability of a batch of 8 without the memory of one. train_on_responses_only masks the system and customer text out of the loss, so the model spends its learning on how to reply, not on how customers write.

You should now see a training log every 10 steps, with the loss falling from around 2 towards somewhere near 1 over a single epoch, finishing in roughly 10 to 20 minutes for about 1,900 examples. Your exact numbers will differ.

What went wrong the first time

Arjun’s first run used num_train_epochs=3. The loss dropped beautifully, to about 0.3. Then he tried it on the test set. Replies were short and Crumb-like, but several were word-for-word copies of training replies that did not fit the question. Worse, four replies told customers “We’ve located your order [ORDER_ID] and…”.

Two problems at once. Three epochs on under 2,000 examples had made the model memorise rather than generalise: classic overfitting, which a very low training loss on a small dataset often signals. And the placeholders his scrubbing script inserted had become part of “how Crumb talks”. The model had learned them as faithfully as the warmth.

The fixes: one epoch instead of three, and a cleaning pass that dropped the 140 training examples whose assistant reply contained a placeholder. The second run’s training loss ended higher, around 1.1, and its replies were better. A lower training loss is not the goal; better answers on unseen questions are.

Step 7: Evaluate against the baseline

Generate replies for the same 50 test prompts with the fine-tuned model:

FastLanguageModel.for_inference(model)
tuned = [reply(model, m) for m in test_prompts]
for i in range(3):
    print("CUSTOMER:", test_prompts[i][1]["content"])
    print("BEFORE:  ", baseline[i][:200])
    print("AFTER:   ", tuned[i][:200], "\n")

You should now see the same customer messages with two replies side by side. For “my cake came squashed, what now”, Arjun’s fine-tuned model wrote: “Oh no, I’m so sorry! Could you send us a photo of the cake? We’ll arrange a fresh one or a full refund, whichever you’d prefer.” Two sentences, warm, and following Crumb’s process.

Reading 50 pairs is worth doing, but a consistent score is better. Arjun used Claude as a judge with a fixed rubric, following the evaluation approach from Module 23:

# !pip install anthropic   and set ANTHROPIC_API_KEY in Colab's secrets
import anthropic
from pydantic import BaseModel

class Score(BaseModel):
    on_brand: bool          # warm, brief, sounds like a friendly local bakery
    invents_policy: bool    # promises something not in the reference reply

client = anthropic.Anthropic()

def judge(customer, reference, candidate):
    response = client.messages.parse(
        model="claude-opus-5-5", max_tokens=500, output_format=Score,
        messages=[{"role": "user", "content":
            f"Customer: {customer}\nReference reply from our best agent: {reference}\n"
            f"Candidate reply: {candidate}\n"
            "Is the candidate on-brand for a warm, brief bakery support team? "
            "Does it promise anything the reference does not support?"}])
    return response.parsed_output

refs = [ex["messages"][2]["content"] for ex in data["test"]]
for name, replies in [("baseline", baseline), ("fine-tuned", tuned)]:
    scores = [judge(p[1]["content"], r, c) for p, r, c in zip(test_prompts, refs, replies)]
    print(name, "on-brand:", sum(s.on_brand for s in scores), "/ 50,",
          "invents policy:", sum(s.invents_policy for s in scores), "/ 50")

messages.parse with a Pydantic model returns a typed object, so every judgement has the same shape. Judging 100 replies takes a minute or two and costs a few rupees.

You should now see two lines of scores. Arjun’s:

baseline on-brand: 21 / 50, invents policy: 11 / 50
fine-tuned on-brand: 44 / 50, invents policy: 6 / 50

On-brand replies more than doubled. But six of fifty still invented something: a delivery slot, a discount. That is the first lesson’s warning made concrete. Fine-tuning fixed the manner; the facts still need retrieval from the handbook in production. And an LLM judge is itself fallible, so Arjun hand-checked ten of its verdicts and agreed with nine.

Step 8: Export to GGUF and run in Ollama

First, save the small adapter, so you can always reload or re-export without retraining:

model.save_pretrained("crumb-lora")         # adapter only: tens of megabytes
tokenizer.save_pretrained("crumb-lora")

Then export a merged, 4-bit model in the GGUF format that Ollama uses (Module 32 explained Q4_K_M):

model.save_pretrained_gguf("crumb-support", tokenizer, quantization_method="q4_k_m")
!ls -lh crumb-support*                      # find the .gguf file and its size

This merges the adapter into the base weights, converts them with llama.cpp’s tools and quantises them. It takes several minutes.

You should now see a .gguf file under a gigabyte for this 1B model (the exact file name and folder vary by Unsloth version). Recent versions also write an Ollama Modelfile alongside it. Download the .gguf file (and the Modelfile, if present) to your computer from Colab’s file browser.

On your computer, in the folder with the GGUF file, create a file called Modelfile (or open the one Unsloth wrote) containing at least:

FROM ./crumb-support.Q4_K_M.gguf
SYSTEM "You are Crumb & Co.'s friendly support assistant. Be warm, brief and accurate."
PARAMETER temperature 0.7

Use your actual GGUF file name on the FROM line. Then:

ollama create crumb-support -f Modelfile
ollama run crumb-support "my cake came squashed, what now"

You should now see a short, warm, Crumb-style reply, generated entirely on your own machine.

What went wrong the second time

Arjun’s first Ollama test produced the right first sentence, then kept going: it wrote “user” on a new line, invented the customer’s next message, and answered that too, until it hit the length limit.

“It doesn’t know when to stop,” said Priya. “What token means ‘end of turn’?”

“<|eot_id|>. From Step 5.”

His hand-written Modelfile had no chat template, so Ollama was feeding the model plain text and not treating <|eot_id|> as a stop signal. The fix was to use the Modelfile that Unsloth had generated, which includes the Llama 3 TEMPLATE and a PARAMETER stop "<|eot_id|>" line. If your Unsloth version does not write one, copy the template and stop lines from the Modelfile of the base model in Ollama (ollama show llama3.2:1b --modelfile) into your own. The broader lesson: the chat template used in training, in evaluation and in serving must match.

Common mistakes

• Skipping the baseline. Without Step 4 you cannot show the fine-tune helped.
• Training for too many epochs on a small dataset. A very low training loss often means memorisation. Start with one epoch and compare on the test set.
• Leaving placeholders or artefacts in replies. The model learns them. Clean assistant replies, not only customer messages.
• Mismatched chat templates. Train, evaluate and serve with the same template and stop token, or the model rambles past its turn.
• Losing the adapter. Save crumb-lora before exporting; Colab sessions are wiped when they disconnect.

Recap

• Unsloth loads a 4-bit base model, attaches LoRA adapters and trains with TRL’s SFTTrainer, fitting comfortably on a free Colab T4.
• The data is JSONL conversations in the messages format, formatted with the model’s own chat template; train_on_responses_only focuses learning on the assistant’s replies.
• Always record a baseline and evaluate both models on the same held-out examples, with a fixed rubric and spot checks by hand.
• One epoch and clean replies beat a low training loss: overfitting shows up as copied replies and memorised artefacts.
• save_pretrained_gguf produces a quantised GGUF file that Ollama can run locally, as long as the Modelfile’s template and stop token match training.

That evening Arjun ran a live demo for Meera on the back-office PC at the Camp store, with the Wi-Fi switched off. She typed “do u have eggless cakes for tmrw” and got back, in about a second: “Yes, we do! Our eggless chocolate truffle and pineapple cakes are available for tomorrow. Would you like me to note an order?”

“That sounds like Shanti,” she said.

“It learned from Shanti’s best replies. It costs nothing per message, and nothing leaves this computer. It still makes up the odd detail, so in the real assistant it reads the menu and handbook through search before it answers. Fine-tuning taught it the manners. The facts come from the binder.”

Meera nodded slowly. “And the Christmas posters? Can a model like this make pictures?”

“Not this one,” said Priya. “But you’ve now seen every piece needed to understand the ones that can. Last lesson of the module: diffusion models, and the transformers that see.”

Go deeper

• Unsloth documentation, including its maintained Colab notebooks.
• Hugging Face docs, see TRL (SFTTrainer), PEFT and Datasets.
• Ollama, including how to import GGUF models with a Modelfile.
• QLoRA