Prompt formats: Alpaca, ChatML and INST

Crumb’s review tagging was now accurate, and Arjun had become slightly obsessed with its cost. Nine thousand reviews a month was small money with a small hosted model. But Meera wanted to start tagging every WhatsApp message too, about 60,000 a month, and the recipe team wanted to summarise thirty years of handwritten recipe notes once they were scanned.

There was a spare gaming PC in the office with a decent GPU, left over from a short-lived plan to make a Crumb mobile game. Arjun had an idea: run an open-weights model on it (a model whose trained numbers anyone can download and run on their own hardware), and tag everything for the price of electricity.

He installed Hugging Face’s transformers library, downloaded a small instruction-tuned model, and wrote the same prompt that worked so well with the hosted API:

from transformers import AutoTokenizer, AutoModelForCausalLM

name = "Qwen/Qwen2.5-0.5B-Instruct"          # a small open model, about 1 GB
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)

prompt = 'Classify this review as positive or negative: "The plum cake was dry."'
inputs = tok(prompt, return_tensors="pt")     # text to token IDs
out = model.generate(**inputs, max_new_tokens=60)
print(tok.decode(out[0], skip_special_tokens=True))

The output:

Classify this review as positive or negative: "The plum cake was dry."
Classify this review as positive or negative: "The brownies were amazing!"
Classify this review as positive or negative: "Delivery was late again."
Classify this review as

It never answered. It wrote more questions, like a student who misread the exam and started writing new exam questions. On another try it wrote Answer: negative and then User: Thanks! Now classify..., playing both sides of a conversation that did not exist.

“It’s broken,” Arjun said. “Or it’s too small to be useful.”

Priya looked at his code for about five seconds. “It’s fine. It’s a chat model, and you didn’t talk to it in its chat format. You handed a letter to someone who only reads forms.”

Arjun patches the symptoms

Arjun’s first fixes were the ones most people try. He added Answer: to the end of the prompt and cut max_new_tokens to 5, so it could not ramble far:

Classify this review as positive or negative: "The plum cake was dry."
Answer:

Now it usually produced one word. He ran it on Kavita’s 50 hand-labelled reviews from the few-shot lesson (converted to positive/negative). It scored 61%. The hosted model had scored 96% on the same set. Sometimes the answer was negative, sometimes Negative., sometimes The review.

Next he invented a chat layout himself, the way chat transcripts look in his head:

User: Classify this review as positive or negative: "The plum cake was dry."
Assistant:

80%. Better. But about one reply in ten ended with User: and a new invented question, because nothing told the model where its turn ended.

“I’m guessing at its language,” Arjun admitted.

“You are,” Priya said. “So stop guessing. Let me ask you something first. When you call the OpenAI API with a list of messages, a system message, a user message, what does the model actually receive?”

“The list?”

“A neural network can’t receive a Python list. It receives one thing. What?”

Arjun remembered Module 01. “A sequence of token IDs. Numbers.”

“So somewhere between your list of messages and that sequence of numbers, someone turned the list into one long string. Who?”

“The API provider. On their servers.” He paused. “And here, nobody did.”

Scripts, forms, and the cue to speak

Priya drew a page from a theatre script on the whiteboard.

MEERA:  (entering) Is the oven on?
ARJUN:  Since six.

“An actor only ever gets pages of text. How do they know when to speak?”

“Their name in capitals before their line.”

“And when to stop?”

“When the next name comes.”

“That’s a chat template. It’s the script format a model was trained on. During training, the model read hundreds of thousands of conversations, every single one laid out in one exact format. It learned: when I see the marker that means it’s my turn, I answer; when I’ve finished, I write the end marker. Your plain-text prompt had no ‘your turn’ marker, so it did what an untrained reader does with a page of text: it continued the page.”

That reveals something important about how these models are built, in two big stages:

1. A base model is trained on a huge pile of text from the internet and books to predict the next token. It is a superb autocomplete, but it does not follow instructions. Given “Classify this review…”, it continues the document, and on the internet, a list of classification questions is usually followed by more classification questions.
2. An instruction-tuned model is that base model trained further on many example conversations, all written in one template. This is where it learns to answer, follow a system prompt, and stop.

Arjun’s model was instruction-tuned. But because his input was not in the template, it never saw the “assistant’s turn” marker, so it fell back to base-model behaviour: continue the text. And it never stopped, because it was waiting to write an end marker in a format it had never been put into.

Following one review through the real template

Priya rewrote the code to use the template that ships with the model. Then they traced a single review, “The plum cake was dry.”, from Python list to answer.

Step 1: the messages, as you would send them to any chat API.

messages = [
    {"role": "system", "content": "You classify Crumb bakery reviews. Reply with one word: positive or negative."},
    {"role": "user", "content": '"The plum cake was dry."'},
]

Step 2: the tokenizer applies the template. The tokenizer (the component that turns text into token IDs and back) knows its model’s template. apply_chat_template uses it:

text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(text)

The output is the flattened script:

<|im_start|>system
You classify Crumb bakery reviews. Reply with one word: positive or negative.<|im_end|>
<|im_start|>user
"The plum cake was dry."<|im_end|>
<|im_start|>assistant

This is ChatML, the format this model family uses. Each turn opens with <|im_start|> and a role name, and closes with <|im_end|>. The final line, an assistant turn opened but not closed, is the cue: your line, please. That line is what add_generation_prompt=True adds.

Step 3: the string becomes token IDs. The interesting part is how the markers are tokenized. <|im_start|> looks like twelve characters, but it is a single special token: a reserved entry in the vocabulary that exists only to mark structure. In Qwen2.5’s vocabulary, <|im_start|> is ID 151644 and <|im_end|> is ID 151645. Ordinary text can never accidentally produce them by being split into pieces. The whole prompt comes to about 40 tokens.

Step 4: the model generates. It produces negative (one token), then immediately produces ID 151645, <|im_end|>. The generate function is configured to stop at that token. Total new tokens: 2. Time on the office GPU: well under a second.

Step 5: decode the answer. Slice off the prompt tokens, decode the rest, and skip the special tokens: negative.

The result. Rerun on all 50 labelled reviews: 47 correct, 94%, against 61% for plain text. No rambling, no fake turns, no Answer: The review. Same model, same hardware, same review. The only change was speaking its language.

Opening the hood: three formats you will meet

Every model family chose its own script format. You do not need to memorise them, but you should recognise them, because you will see them in model cards, datasets and error messages.

Alpaca: plain text with headings

In 2023, Stanford’s Alpaca project fine-tuned an early Llama model on 52,000 instruction-and-answer pairs laid out like this:

Below is an instruction that describes a task. Write a response that
appropriately completes the request.

### Instruction:
Classify this review as positive or negative: "The plum cake was dry."

### Response:

The model learns to answer after ### Response:. An optional ### Input: section holds extra data. The markers are ordinary text, which makes Alpaca easy to read and write, but it has no proper system prompt and no real multi-turn structure. It is still everywhere in fine-tuning tutorials and older datasets because of that simplicity.

ChatML: role tags with special tokens

ChatML (Chat Markup Language) came from OpenAI and was adopted by many open families, including Qwen. You traced it above: <|im_start|>role, content, <|im_end|>, repeated for every turn, ending with an open assistant turn. It handles system prompts and long multi-turn chats cleanly.

INST: the Llama 2 and Mistral style

Meta’s Llama 2 chat models wrap each user turn in [INST] and [/INST], with an optional system block inside the first one:

<s>[INST] <<SYS>>
You classify Crumb bakery reviews. Reply with one word.
<</SYS>>

"The plum cake was dry." [/INST]

<s> and </s> are special tokens meaning “beginning of sequence” and “end of sequence”. The model’s answer follows [/INST] and ends with </s>. A second user turn starts another <s>[INST] ... [/INST] block. Mistral’s models use the same [INST] idea but no <<SYS>> block, so system instructions are folded into the first user message.

Newer families keep inventing their own. Llama 3 uses <|start_header_id|>user<|end_header_id|> and <|eot_id|>. Gemma uses <start_of_turn>user and <end_of_turn>, and calls the assistant model. The principle never changes: one family, one script, and the model only performs well with its own.

Where the template lives

Here is the neat part, tying back to the templates lesson: on the Hugging Face Hub, each model’s chat template is stored as a Jinja template inside its tokenizer configuration file. When you call apply_chat_template, the tokenizer renders that Jinja template with your messages, exactly like you rendered Crumb’s prompts. You can print it with print(tok.chat_template). It is dense, but recognisable: loops over messages, if blocks for roles.

When you use OpenAI, Anthropic or Gemini, the same thing happens on their servers with their own formats, which they do not publish. That is why you never had to think about it until now.

Building it properly, step by step

Step 1: look before you generate. Install with uv add transformers torch. Then print the template’s output for your messages, every time you try a new model:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
print(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))

You should see the ChatML block from above. Swap in a Mistral or Llama model name (some require accepting a licence on the Hub first) and the same code prints a completely different layout.

Step 2: generate from the templated text.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")

def classify_local(review: str) -> str:
    msgs = [
        {"role": "system", "content": "You classify Crumb bakery reviews. "
                                      "Reply with one word: positive or negative."},
        {"role": "user", "content": review},
    ]
    text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
    inputs = tok(text, return_tensors="pt", add_special_tokens=False)  # template already added them
    out = model.generate(**inputs, max_new_tokens=5)
    new_tokens = out[0][inputs["input_ids"].shape[-1]:]                # drop the prompt part
    return tok.decode(new_tokens, skip_special_tokens=True).strip().lower()

print(classify_local('"The plum cake was dry."'))   # negative

Two details in there deserve a line each. add_special_tokens=False stops the tokenizer adding its own start-of-sequence token on top of the one the template may already include; with some model families, a doubled start token quietly hurts quality. Slicing off inputs["input_ids"].shape[-1] tokens removes the prompt from the output, so you decode only what the model wrote. Library APIs change quickly. If a name has moved, check the docs linked below.

Step 3: or let a runner handle it. Local runners like Ollama read the template from the model file and apply it for you. When you call ollama.chat(model="llama3.2", messages=[...]), Ollama builds the string. You can see the template it uses with ollama show llama3.2 --template. You will use Ollama properly in Module 20.

Where it breaks

The template fixed Arjun’s tagger. Over the next weeks he and the team found the other ways templates go wrong, most of them silent.

The wrong template looks like a weak model. When Arjun tried a Mistral model by copying the Qwen code and hand-editing a ChatML string, it ran without errors and scored 82%. With its own [INST] template, 93%. Nothing crashed; it was quietly worse. Many “this open model is terrible” opinions online are template bugs.

The forgotten cue. With add_generation_prompt=False, the string ends after the user’s turn. The model may continue the user’s message instead of answering it.

The fine-tune that learned nothing. In Module 33, you will fine-tune open models on your own examples. Your training data must use the same template you will use when you chat with the result. Training in Alpaca format and then chatting in ChatML is one of the most common reasons a freshly fine-tuned model seems to have learned nothing.

The customer who typed a special token.

Try it yourself

1. Print the template output for Qwen/Qwen2.5-0.5B-Instruct with the messages above.
2. Print it again with add_generation_prompt=False. What disappears from the end?
3. Add an assistant reply ({"role": "assistant", "content": "negative"}) and a second user turn ("And this one: 'Best brownies in Pune!'"), and print again. Where does each turn start and end?
4. Bonus: print tok.chat_template and find the loop over messages.

Show the answer

1. You see <|im_start|>system ... <|im_end|>, then <|im_start|>user ... <|im_end|>, then a final <|im_start|>assistant line with nothing after it.
2. The final <|im_start|>assistant line disappears. The string ends after the user’s <|im_end|>, so the model has no cue that it is its turn.
3. Every turn is wrapped the same way. The earlier “negative” sits inside its own <|im_start|>assistant ... <|im_end|> block, the second user turn follows, and a new open assistant turn sits at the very end. This is how multi-turn history reaches the model: one long string, rebuilt and re-sent in full on every call, which you met in Module 02.
4. You will find something like {% for message in messages %} with {% if message['role'] == 'system' %} branches: the same Jinja you used for Crumb’s prompts.

Common mistakes

• Sending raw text to a chat model. Use apply_chat_template or a chat endpoint that applies the template for you.
• Forgetting the generation prompt. The open assistant turn is the model’s cue to answer.
• Hand-writing templates from blog posts. Templates change between model versions. Use the one shipped with the exact model you downloaded.
• Doubling special tokens. If the template adds a start token, do not let the tokenizer add another.
• Mismatched training and chat templates. Fine-tune and serve with the same format.
• Using a base model for chat. Base models continue text. Pick the “-Instruct” or “-Chat” version for conversations.

Recap

• A model sees one long token sequence; a chat template turns your list of messages into that sequence.
• Base models continue text; instruction-tuned models learned to answer and stop, but only inside their own template.
• Alpaca uses plain ### Instruction: / ### Response: headings; ChatML uses <|im_start|>role ... <|im_end|>; Llama 2 and Mistral use [INST] ... [/INST].
• Special tokens are reserved IDs that mark turns, which is what makes stopping reliable.
• Hosted APIs apply templates for you; with open models, use apply_chat_template or a runner like Ollama, and always print the result once.

Meera stopped by to see the office GPU earning its keep. “So the free model was rubbish, and now it’s good?”

“It was never rubbish,” Arjun said. “Each model learned to chat using one particular form, like a script with special markers for ‘your turn’ and ‘I’m done’. I was handing it plain text, so it didn’t know it was supposed to answer. Now the library fills in the form the model was trained on. Same model, same computer: 61% before, 94% now.”

“And it costs?”

“Electricity. A few hundred rupees a month at our volume.”

Meera nodded and moved on to the thing that was actually bothering her. She had a printout of a support chat. Forty messages in, the chatbot had recommended a sesame cake to a customer who had said, near the start, that her granddaughter was allergic to sesame. “Can you explain this?” she asked. Arjun could. He did not like the explanation.

Go deeper

• Hugging Face documentation (search for “chat templates”): https://huggingface.co/docs
• Hugging Face LLM course: https://huggingface.co/learn/llm-course
• Ollama: https://ollama.com