Encoders, decoders, GPT vs BERT

Meera dropped three printed proposals on the meeting table. The first agency wanted to rebuild Crumb’s website search “with BERT”. The second said BERT was old news: “Use a GPT-style model for everything: search, chat, tagging, the lot.” The third pitched “a T5 model to summarise customer reviews in Marathi into English”.

“They all say transformer,” Meera said. “They all sound confident. They cannot all be right. Do I need one AI or three?”

Priya slid the proposals to Arjun. “You built a transformer from scratch this week. You tell her.”

Arjun opened his mouth, closed it, and said, “Give me a day.”

Arjun tries “one model for everything”

The second agency’s pitch was tempting. Arjun already had a chat model API. Why not use it for everything?

He started with a real job: Crumb wanted its 20,000 past support emails tagged by topic (refund, late delivery, allergy, other) to find patterns. He wrote a loop that sent each email to a chat model with the instruction “Reply with one word: the topic.”

Then he did the arithmetic. The average email plus instructions came to about 400 tokens (the small chunks of text that models read and providers bill by). That is $20,000\times 400=8$ million input tokens. At roughly 1.5 seconds per call, one after another, it would take over eight hours. And the website search would need the same kind of call on every keystroke a customer typed.

So he tried the opposite. The first agency loved BERT, so he loaded BERT and asked it to write a support reply. It could not. All it could do was fill in a blank, and when he pushed it to keep producing words, it gave him repetitive nonsense.

“So the giant chat model is too slow and expensive for tagging,” he told Priya, “and BERT can’t write a sentence. They’re both transformers. What’s the difference?”

“Which half of the transformer they kept,” said Priya, “and which game they played while they learned.”

Two halves of one machine

Recall the original 2017 transformer, a neural network built from stacked blocks of attention and feed-forward layers. It was built for translation and had two stacks:

• An encoder, which reads the whole input with no mask. Every token can look at every other token, left and right. This is bidirectional attention.
• A decoder, which writes the output one token at a time with masked attention: each token sees only the tokens before it, as you built last lesson. Each decoder block also has cross-attention, which lets it look at what the encoder read.

Researchers soon asked: what if we keep only one half?

flowchart LR
    T["Original transformer (2017)"] --> E["Encoder-only: BERT, RoBERTa"]
    T --> D["Decoder-only: GPT, Llama, Mistral"]
    T --> ED["Encoder-decoder: T5, BART, Whisper"]
    E --> E2["Understand: classify, search, tag"]
    D --> D2["Generate: chat, write, code"]
    ED --> ED2["Transform: translate, summarise, transcribe"]

Priya’s three people at the bakery

“Picture three people,” Priya said.

“The first is a crossword solver. Give them ‘Our ___ cake is eggless’ and they use the words on both sides of the gap. Brilliant at understanding a sentence. But they have never written a paragraph in their life.”

“The second is a storyteller. They write one word, then the next, never peeking ahead. Every reply your chatbot sends is written this way.”

“The third is a translator. They read the customer’s whole Marathi letter first, then write an English summary from scratch. The output can be a different length, even a different language.”

“Which of them,” she asked, “would you hire to sort 20,000 emails into four piles?”

“The crossword solver. Sorting is understanding, not writing.”

“And to answer customers on WhatsApp?”

“The storyteller.”

“Now you know what BERT and GPT are.”

Where the analogy breaks: these are not separate species. They share the same building blocks, and a large storyteller can also sort, translate and summarise, at a price. The split is about what each design does naturally and cheaply.

The games they played: following one sentence through both

Pretraining is the first, huge training phase in which a model learns from raw text with no human labels. The text supplies its own answers through a fill-in-the-blank game. The game a model plays decides what it becomes good at. Let us run one sentence through both games: “Our eggless cake is soft” (five tokens, one per word, to keep it simple).

BERT’s game: masked language modelling

Masked language modelling (MLM) hides some tokens and asks the model to recover them using context on both sides. BERT hid about 15% of tokens. In a five-word sentence, that is usually one:

Our [MASK] cake is soft

The model sees Our on the left and cake is soft on the right, and must put a probability on every word in its vocabulary for the blank. Suppose it gives eggless a probability of 0.30. The loss for this sentence is the “surprise” at the right answer, measured as $-\log p$:

$$-\log (0.30)\approx 1.20$$

One sentence, one graded prediction. The other four positions produced no training signal at all.

GPT’s game: causal language modelling

Causal language modelling (CLM) predicts each next token from everything before it. With the causal mask, all of these happen in one pass:

Sees	Must predict	Model’s probability for the right word	Loss $-\log p$
Our	eggless	0.02	3.91
Our eggless	cake	0.60	0.51
Our eggless cake	is	0.70	0.36
Our eggless cake is	soft	0.10	2.30

The sentence’s loss is the average: $(3.91+0.51+0.36+2.30)/4\approx 1.77$.

Look at what each game teaches. GPT finds eggless hard to guess from Our alone (only 0.02): it has no right-hand context. BERT, seeing cake is soft, can guess the blank much better. That is why BERT-style models are so good at understanding a sentence. But GPT gets four graded predictions from the same five words, and every one of them is practice at the exact skill of writing the next word. That is why GPT-style models are good at writing, and why they squeeze more learning out of the same text.

Opening the hood: the two objectives in maths

For CLM, the loss over a sentence of $T$ tokens is the average surprise at each true next token:

$${\mathcal{L}}_{\text{CLM}}=-\frac{1}{T}\sum _{t=1}^T\log p_{\theta }(x_t\mid x_{<t})$$

$x_t$ is the token at position $t$, $x_{<t}$ is everything before it, and $p_{\theta }$ is the model’s probability, with $\theta$ standing for all of the model’s learned numbers, its parameters. A perfect prediction (probability 1) has loss 0; a probability of 0.01 has loss about 4.6.

For MLM, with $M$ the set of masked positions and $x_{\setminus M}$ the sentence with those positions hidden:

$${\mathcal{L}}_{\text{MLM}}=-\frac{1}{|M|}\sum _{i\in M}\log p_{\theta }(x_i\mid x_{\setminus M})$$

Three details from BERT’s 2018 recipe are worth knowing:

• Of the 15% of tokens chosen, 80% became [MASK], 10% became a random token and 10% stayed unchanged. Real text never contains [MASK], so this stopped the model relying on seeing it.
• BERT also played a second game, next sentence prediction: were these two sentences neighbours in the original text? Later work (RoBERTa) found it did not help much and dropped it.
• An encoder outputs one vector per token. For classification, BERT adds a special [CLS] token at the start and feeds its final vector to a small classifier. For search, models usually average all token vectors into one embedding (a list of numbers capturing the meaning of the whole text).

Encoder-decoder models add a twist. T5 used span corruption: blank out whole runs of words, and have the decoder write the missing pieces. And the decoder’s cross-attention is ordinary attention with one change: queries come from the decoder, while keys and values come from the encoder’s output:

$$\text{CrossAttention}=\text{softmax}\left(\frac{Q_{\text{dec}}{K}_{\text{enc}}^{\top }}{\sqrt{d_k}}\right)V_{\text{enc}}$$

The translator, mid-sentence in English, glancing back at the Marathi letter.

GPT vs BERT, side by side

	BERT (encoder-only)	GPT (decoder-only)
Attention	Bidirectional: sees left and right	Causal: sees only the left
Pretraining game	Masked language modelling	Next-token (causal) language modelling
Graded predictions	About 15% of tokens	Every token
Natural output	A rich vector per token	The next token, again and again
Typical use	Classification, search embeddings, tagging names and places, reranking	Chat, writing, code, agents
Typical size	100 to 400 million parameters	From a few hundred million to hundreds of billions

Why did decoder-only models win chat? Generation is built into their game. Grading every token squeezes more from the same data. And as GPT-style models grew, they showed in-context learning: picking up a new task from a few examples in the prompt, with no retraining (you used this in Module 03 as few-shot prompting). One model became useful for almost everything.

But encoders never went away. The embedding model you used for Crumb’s semantic search in Module 07, all-MiniLM-L6-v2, is a small BERT-style encoder with about 22 million parameters. So are the cross-encoder rerankers from the RAG modules. They are fast, cheap and excellent at “understand this text”, which is exactly what search and tagging need.

Seeing all three in code

Hugging Face’s pipeline and Sentence Transformers let you try each family in a few lines. The first run downloads small models, a few hundred megabytes in total.

Step 1: BERT plays its blank-filling game.

# uv add transformers torch sentence-transformers
from transformers import pipeline

fill = pipeline("fill-mask", model="bert-base-uncased")
for guess in fill("Our [MASK] cake is completely eggless.")[:3]:
    print(guess["token_str"], round(guess["score"], 3))

You will see three plausible words for the blank with their probabilities. BERT used both sides: “cake” to its right and “eggless” further on.

Step 2: GPT-2 plays its next-token game.

gen = pipeline("text-generation", model="gpt2")
out = gen("Your order from the bakery is", max_new_tokens=15)
print(out[0]["generated_text"])

GPT-2 (2019, 124 million parameters in this smallest version) rambles forward. It writes fluently because it practised exactly this, though with no instructions it has no idea it is meant to be a helpful bakery.

Step 3: an encoder making search vectors.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")      # a small BERT-style encoder
vecs = model.encode([
    "eggless chocolate cake",
    "cake made without eggs",
    "late delivery refund",
])
print(vecs.shape)                                    # (3, 384): one 384-number vector each
print(model.similarity(vecs[0], vecs[1]), model.similarity(vecs[0], vecs[2]))

The first pair, same meaning in different words, scores much higher than the second. That is the job the first agency wanted BERT for, and it runs in milliseconds on a laptop.

Step 4: the CLM loss in PyTorch. Under the hood, every GPT-style training step computes this. The trick is shifting labels by one, exactly as in last lesson’s training loop.

import torch
import torch.nn.functional as F

vocab, T = 1000, 6
ids = torch.randint(0, vocab, (1, T))         # a sentence of 6 token ids
logits = torch.randn(1, T, vocab)             # pretend model output

# Position t predicts token t+1: drop the last prediction and the first label.
loss = F.cross_entropy(logits[:, :-1].reshape(-1, vocab), ids[:, 1:].reshape(-1))
print(loss)   # around 7 for random guesses (log of 1000 is 6.9)

Where the families break

Encoders have short attention spans. Classic BERT models accept at most 512 tokens. Feed in a long email and the rest is cut off, silently in some tools. Newer encoders accept more, but check the limit before you rely on it.

Generators are expensive for bulk understanding. Arjun’s eight-hour, eight-million-token tagging run was the wrong tool. He fine-tuned a small BERT-style classifier on 1,500 hand-labelled emails instead (Module 33 covers fine-tuning), and it tagged all 20,000 in under two minutes on a single GPU. The chat model would still win on the weird emails that fit no category, so he sent only the low-confidence ones to it: a few hundred calls instead of 20,000.

The lines are blurring. Many of today’s strongest embedding models are actually built from decoder LLMs and then trained for embeddings. And decoder-only LLMs now translate and summarise so well that new encoder-decoder language models are rarer. Encoder-decoder designs are still thriving where the input is a different kind of thing from the output, such as Whisper turning audio into text.

Loss numbers are not comparable across games. An MLM loss of 1.2 and a CLM loss of 1.8 measure different tasks. Do not use them to argue one model is better.

Try it yourself

Meera’s three proposals were: (a) website search over product descriptions, (b) a chat assistant on WhatsApp, © turning long Marathi reviews into short English summaries. Match each to the most natural family, and give one reason each.

Show the answer

• (a) Encoder-only. Search needs a good meaning vector for every product and query, fast and cheap, on every keystroke. A BERT-style embedding model fits.
• (b) Decoder-only. A chat assistant generates replies token by token. Today that is almost always a GPT-style model.
• © Encoder-decoder is the textbook fit: read one language fully, write another. In practice a decoder-only LLM with a good prompt also does this well, and you may already be paying for one. Either answer is defensible if you can say why.

Common mistakes

• Using BERT to generate text. It was never trained to write left to right. It fills blanks; it does not compose replies.
• Using a giant chat model for a job a small encoder does better. Bulk classification and search are usually faster and far cheaper with an encoder.
• Thinking “decoder-only” means “doesn’t understand”. Decoder models understand prompts well; they only read left to right, and the reply comes after the whole prompt.
• Comparing MLM and CLM loss numbers. Different games, different scales.
• Believing a vendor’s family name is a strategy. “BERT” or “GPT” says how a model was built, not whether it fits your job, data, budget or speed needs. Ask what it will cost per thousand requests and how they will evaluate it.

Recap

• Encoder-only (BERT): bidirectional, trained by masked language modelling, great for understanding, search and classification.
• Decoder-only (GPT, Llama): causal, trained by next-token prediction, great for generation; the basis of today’s chat models.
• Encoder-decoder (T5, BART, Whisper): read the whole input, then write a new output, with cross-attention linking the two halves.
• CLM grades every token; MLM grades only the roughly 15% that were hidden, but uses context from both sides.
• Pick the family by the job: understanding in bulk is cheap with encoders; writing needs a decoder.

The next morning Arjun brought Meera a single page. “You need two, maybe three, and they’re cousins. One reads in both directions, like someone solving a crossword. That’s the search box: fast and cheap. One writes word by word, like a storyteller. That’s the WhatsApp assistant. The third reads a whole letter and then writes a new one. That’s the summaries, though our chat model can probably do that job too.”

“So the agency that said ‘GPT for everything’?”

“Would work, slowly and expensively. I tried it. Eight hours and eight million tokens to tag our old emails, against two minutes with the small reader.”

Meera underlined two minutes. Priya, reading over her shoulder, had a different question. “Every one of these models starts by chopping text into tokens. You’ve used tokenizers for months, Arjun. Tomorrow you build one, and you’ll find out why our Hindi messages cost more.”

Go deeper

• Hugging Face LLM course, with chapters on encoder, decoder and sequence-to-sequence models.
• Hugging Face docs
• Attention Is All You Need