Instruction-tuned open models like Gemma

The email sorter was a hit. Each morning the support team opened the inbox to find every message already tagged: late delivery, allergy question, refund request. Meera was not satisfied for long.

“Sorting is nice,” she said. “But my team still types the same twelve replies all day. ‘We’re sorry your order is late, here’s the tracking link.’ ‘Our walnut brownies are made in a kitchen that handles peanuts.’ Can a free model write the first draft? On our machine, like the sorter?”

Arjun had seen people chatting with open models online. How hard could it be? He found Google’s Gemma family on the Hub and wrote his first script before lunch.

Arjun’s first attempt, and two surprises

His script was four lines:

from transformers import pipeline

gen = pipeline("text-generation", model="google/gemma-3-1b-pt")
print(gen("Do you have eggless cakes?", max_new_tokens=60)[0]["generated_text"])

Surprise one arrived before the model did. The download stopped with an error mentioning a gated repo and a 401 code: access to the model was restricted. Arjun had a valid token. His code was fine. What was wrong?

He opened the model page in his browser and found a box he had scrolled past: “To access Gemma on Hugging Face, you’re required to review and agree to Google’s usage license.” He read it, clicked to accept, ran the script again, and the 2 GB download started.

Surprise two was the answer:

Do you have eggless cakes? Do you have sugar-free cakes? Do you have cakes for diabetics? Our Frequently Asked Questions page lists all the answers. Q: How long do your cakes stay fresh? A: …

It never answered. It wrote more questions, then an FAQ page, until it ran out of its 60-token allowance.

“It’s broken,” Arjun told Priya.

“Why did you pick the -pt model?”

“PT. PyTorch. We use PyTorch.”

Priya laughed, kindly. “Everyone thinks that. On Gemma, pt means pretrained. You downloaded the version that has read the whole internet but has never worked a shift at the counter.”

Two apprentices

Priya drew two stick figures on the whiteboard.

“Two new hires. The first has read every cookbook, food blog and bakery website ever written. You hand her a slip of paper that says ‘Do you have eggless cakes?’ What does she do?”

Arjun thought. “If all she’s ever done is read pages… she’d guess what comes next on the page?”

“Exactly. And on the internet, what usually comes after one customer question?”

“More questions. An FAQ.” He looked at his screen. “That’s what it did.”

“Now the second hire. She read all the same books, and then spent weeks at the counter with a trainer, looking at thousands of examples: a customer asks something, a good staff member answers it. Same slip of paper. What does she do?”

“She answers it.”

That is the whole difference between the two versions of most open model families:

• A base model (also called pretrained) has learned one thing: predict the next token (a small chunk of text, often part of a word) over trillions of tokens of text. It is an extraordinarily well-read autocomplete. It has no idea it is supposed to help you.
• An instruction-tuned model is that same base model after extra training on conversations, so that it has learned the role of an assistant.

You can usually tell them apart by name. Look for -it, -Instruct or -chat. On Gemma, -pt is the base model and -it is instruction-tuned.

Where the analogy breaks: the second apprentice did not learn many new recipes during her counter training. Instruction tuning mostly changes behaviour (answer the question, stay polite, stop when done), not knowledge. That is why base models still matter. They are the starting point when you fine-tune (train a model further on your own examples) for a narrow job, which you will do in Module 33.

Show the answer

Because the base model is not trying to answer. It is doing exactly what it was trained to do, continuing the most likely document, and the most likely document that starts with a customer question is an FAQ page. The knowledge to answer is there; the habit of answering is not. Instruction tuning teaches the habit.

How an apprentice gets trained

Opening the hood one level, instruction tuning usually happens in two stages after pretraining.

1. Supervised fine-tuning. The model trains on tens of thousands to millions of examples written as conversations: an instruction, then a high-quality response. It learns the shape of a helpful answer.
2. Preference tuning. People (or another model) compare pairs of answers and mark the better one. The model is nudged towards the preferred style: clearer, safer, more on-topic. You may see this called RLHF (reinforcement learning from human feedback) or DPO; the InstructGPT paper linked at the end made the approach famous.

All of those training conversations were written in one exact format, with special markers for who is speaking. Remember that. It is about to matter a lot.

Gemma, and what “open” really means

Gemma is Google’s family of open-weights models. Open weights means you can download the trained parameters and run the model yourself, unlike closed models such as Claude or GPT that you can only reach through their APIs. Gemma comes in several sizes. In the Gemma 3 generation, for example, the 1B model handles text only and reads up to about 32,000 tokens at once (its context window, the amount of text it can consider in one go, which you met in Module 01), while the 4B, 12B and 27B sizes read about 128,000 tokens and also accept images. Newer generations arrive regularly.

Open weights is not the same as “anything goes”. Every model has a licence, and they come in three broad kinds:

Kind	Examples	What it means for Crumb
Standard open-source licence	Apache 2.0, MIT	Use it commercially; keep the notice
Company-written licence	Gemma Terms of Use, Llama community licence	Usually commercial use is allowed, with conditions: a list of prohibited uses, rules you must pass on to your users, sometimes size limits on who may use it
Non-commercial	CC-BY-NC	Prototypes and research only

Earlier Gemma generations ship under Google’s Gemma Terms of Use: commercial use is allowed, a prohibited-use policy applies, and you must pass those restrictions on to anyone you let use the model. Licences can change between generations of the same family, so read the one on the exact model page you use.

Gated models add one more step. The files are public, but the Hub will not serve them to your account until you accept the terms on the website. Approval is often instant; for some models a person reviews each request, which can take hours. Three things must line up:

1. You accepted the licence on huggingface.co.
2. You did it while logged in as the same account your token belongs to.
3. Your code is using that token (hf auth login, or HF_TOKEN in .env).

A 401 or 403 error that mentions a gated repo almost always means one of those three is missing. It is a permissions problem, not a code problem.

Following one question through Gemma

Arjun switched to google/gemma-3-1b-it. This time Priya made him trace a single question all the way through, slowly.

1. The messages. Instead of a bare string, he wrote a conversation in the same shape you used with Claude and OpenAI in Module 02:

messages = [
    {"role": "system", "content": "You are Crumb's friendly bakery assistant. Answer in two sentences."},
    {"role": "user", "content": "Do you have eggless cakes?"},
]

2. The chat template. The tokenizer turned that list into the exact text format Gemma was trained on. Printed out, it looked like this:

<bos><start_of_turn>user
You are Crumb's friendly bakery assistant. Answer in two sentences.

Do you have eggless cakes?<end_of_turn>
<start_of_turn>model

Three things to notice. <bos> (“beginning of sequence”) marks the start. <start_of_turn> and <end_of_turn> wrap each speaker. And Gemma has no separate system role: the template quietly glued the system message onto the front of the first user turn. The last line, <start_of_turn>model, is the cue: “your turn to speak”. This text format is called a chat template, and every model family has a different one.

3. Tokens. The tokenizer split that text into roughly 30 tokens. Each special marker is a single token with its own ID; Gemma’s <bos>, for example, is ID 2.

4. Generation. The model looked at all 30 tokens and produced scores for every token in its vocabulary (about 262,000 possibilities in Gemma 3), picked one, appended it, and repeated. “Yes” … “,” … “we” … “do” … Each step is one full pass through the model’s billion parameters.

5. Stopping. After about 40 tokens, the most likely next token was <end_of_turn>. The model had learned, during instruction tuning, that a good assistant stops. Generation ended there.

6. The answer.

Yes, we do! We bake a range of eggless cakes, including chocolate truffle and pineapple, and you can order them on our website.

On Arjun’s laptop the whole thing took a couple of seconds. The model weighed about 2 GB in memory. Cost per reply: electricity. For comparison, the same short exchange through Claude would cost around $0.0016, about ₹0.14. Cheap either way at this size; the bigger difference is privacy and control.

One thing should bother you about that answer. Pause before reading on.

Show the answer

Gemma has no idea what Crumb sells. “Chocolate truffle and pineapple” are plausible guesses, not facts. Instruction tuning made the model helpful-sounding, not informed. It will confidently invent a menu, a price or a policy. You fix this the way you did in Module 09: give it the real facts (retrieved from Crumb’s menu or handbook) in the prompt, and have a person approve drafts before they go to customers.

Will it fit? The back-of-the-envelope sum

Before downloading any model, do this sum. Each parameter is a number, and each number takes memory:

Precision	Bytes per parameter	1B model	4B model	12B model	27B model
32-bit float	4	~4 GB	~16 GB	~48 GB	~108 GB
16-bit (bf16)	2	~2 GB	~8 GB	~24 GB	~54 GB
4-bit quantized	~0.5	~0.6 GB	~2.5 GB	~7 GB	~15 GB

Then add some headroom for the conversation itself; long prompts need more. Quantization means storing each number with fewer bits: a little quality traded for a much smaller, faster model. You will use it properly in Module 20, with tools like Ollama.

Show the answer

The 1B model in 16-bit (about 2 GB) runs comfortably. The 4B in 16-bit (about 8 GB) fits but leaves little room for everything else and runs slowly on a processor. The 4B quantized to 4 bits (about 2.5 GB) is the sweet spot for that machine. The 12B only fits quantized, and would be slow. The 27B does not fit. Size is the first filter, before quality.

The real code, step by step

Step 1: accept the licence and log in. Open google/gemma-3-1b-it on the Hub, accept the terms, then:

uv add transformers torch accelerate python-dotenv
uv run hf auth login

Step 2: chat through a pipeline.

from dotenv import load_dotenv
from transformers import pipeline

load_dotenv()   # HF_TOKEN, needed because Gemma is gated

# Newer Gemma versions exist; any "-it" model you have accepted works the same way
MODEL_ID = "google/gemma-3-1b-it"
chat = pipeline("text-generation", model=MODEL_ID, device_map="auto")

messages = [
    {"role": "system", "content": "You are Crumb's friendly bakery assistant. Answer in two sentences."},
    {"role": "user", "content": "Do you have eggless cakes?"},
]
out = chat(messages, max_new_tokens=120)

# The pipeline returns the whole conversation; the last message is the reply
print(out[0]["generated_text"][-1]["content"])

Line by line: passing a list of messages (not a string) tells the pipeline to apply the chat template for you. device_map="auto" (which needs the accelerate package) places the model on a GPU if you have one, Apple Silicon if you have that, or the processor otherwise. max_new_tokens caps the reply length. The result contains the full conversation with the model’s reply appended at the end.

Step 3: look at what the model really receives. Whenever a prompt misbehaves, print the template:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained(MODEL_ID)
print(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))

tokenize=False gives you text instead of numbers. add_generation_prompt=True appends the <start_of_turn>model cue. Without it, the model is not told it is its turn, and replies get strange.

Step 4: control the style. Open models let you set sampling options that Claude’s newest models do not expose. temperature controls randomness: low values pick the most likely token almost every time (consistent, a bit flat), higher values allow more variety:

out = chat(messages, max_new_tokens=120, do_sample=True, temperature=0.4)

For customer replies, keep it low. Same question, similar answer every time.

Step 5: a draft-reply helper. Putting it together for the support team:

FACTS = """Crumb bakes eggless versions of: chocolate truffle, black forest, fresh fruit.
Eggless cakes need 24 hours' notice. Order online or at any of our six Pune stores."""

def draft_reply(email: str) -> str:
    messages = [
        {"role": "system", "content": (
            "You draft replies for Crumb's support team. Use ONLY the facts below. "
            "If the facts do not cover the question, say a team member will reply.\n\n" + FACTS)},
        {"role": "user", "content": email},
    ]
    return chat(messages, max_new_tokens=150, do_sample=False)[0]["generated_text"][-1]["content"]

print(draft_reply("Hi! Can I get an eggless black forest for tomorrow evening?"))

do_sample=False means always take the single most likely token: the most repeatable output.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks

The drafts went to the support team on Monday. On Wednesday, a customer replied, delighted: “So I get free delivery on orders over ₹500?” No such offer existed. One of Gemma’s drafts had added a friendly line about free delivery, and a tired staff member had clicked send.

The FACTS block helped, but a 1B model does not follow instructions as reliably as a large one. Here is what experienced engineers watch for with small open models:

• Invention. Small models fill gaps with plausible fiction. Keep a human approving every draft, give the model the real facts, and test with questions the facts do not cover.
• Weak instruction following. Long, many-rule system prompts get partly ignored. Keep instructions short and test each rule.
• Wrong template, silent damage. Hand-written prompts like "User: ... Assistant:" or a missing add_generation_prompt make the model behave half like a base model. Always use the tokenizer’s template.
• Double start tokens. If you turn the template text into tokens yourself, pass add_special_tokens=False, because the template already includes <bos>. Two <bos> tokens degrade answers subtly, without any error.
• Truncated replies. If max_new_tokens is too small, answers stop mid-sentence. Check whether the reply ended naturally.
• Language gaps. Small models are noticeably weaker in Hindi and Marathi than in English. Measure on real customer messages.
• Licence drift. When you upgrade to a newer generation, read its licence again.

Crumb’s fix was simple and effective: every draft is shown to staff with a yellow “AI draft, check before sending” banner, and any draft mentioning money, discounts or delivery promises is flagged for a second look.

Try it yourself

Accept the licences for both google/gemma-3-1b-pt and google/gemma-3-1b-it. Then run three experiments:

from transformers import pipeline

question = "Do you have eggless cakes?"
for model_id in ["google/gemma-3-1b-pt", "google/gemma-3-1b-it"]:
    gen = pipeline("text-generation", model=model_id, device_map="auto")
    print(model_id, "| plain string ->", gen(question, max_new_tokens=60)[0]["generated_text"], "\n")

it = pipeline("text-generation", model="google/gemma-3-1b-it", device_map="auto")
reply = it([{"role": "user", "content": question}], max_new_tokens=60)
print("it | messages ->", reply[0]["generated_text"][-1]["content"])

Before you run it, predict what each of the three will do.

Show the answer

The base model with a plain string continues the text like a web page: more questions, an FAQ, sometimes a recipe. The instruction-tuned model with a plain string usually does better but may ramble, because no chat template was applied, so it never saw its “your turn” cue. The instruction-tuned model with a message list answers like an assistant and stops. Same company, same size, same knowledge: the differences come from training and template.

Common mistakes

• Reading -pt as “PyTorch”. It means pretrained, the base model. For chat, pick -it, -Instruct or -chat.
• Hand-writing chat markers. Pass messages and let apply_chat_template or the pipeline do it.
• Treating a gated-repo error as a bug. Accept the licence with the same account your token belongs to.
• Expecting a 1B model to behave like Claude. Small open models are fast, cheap and private, and they make more mistakes. Use them for narrow, well-tested jobs with a human in the loop.
• Skipping the size sum. Parameters × bytes per parameter, before you download.

Recap

• Base models continue text; instruction-tuned models follow requests and know when to stop.
• Instruction tuning is extra training on conversations (and preferences); it changes behaviour more than knowledge.
• Open weights means you can run the model yourself, not that every use is allowed. Gated models need you to accept the licence first.
• Chat templates turn a list of messages into the exact format the model was trained on. Let the library apply them.
• Memory needed is roughly parameters × bytes per parameter; quantization shrinks it.

Arjun walked Meera through it with the two stick figures still on the whiteboard. “There are two versions of each free model. One has read everything but just keeps writing. The other has been trained to answer, like a new hire after counter training. We use the trained one, on our own laptop, and it drafts replies in two seconds. But it doesn’t actually know our menu, so we give it the facts, and a person checks every draft.”

“Good,” said Meera. “One more question. When it breaks at midnight, can you fix it? What is that pipeline thing actually doing?”

Arjun opened his mouth, then closed it. Priya grinned. “Tomorrow we take the vending machine apart.”

Go deeper

• Hugging Face documentation (Transformers, chat templates, gated models): https://huggingface.co/docs
• Hugging Face LLM course: https://huggingface.co/learn/llm-course
• Training language models to follow instructions (InstructGPT): https://arxiv.org/abs/2203.02155