Module 30
Transformers from Scratch
Open the box you have been calling through an API. You build self-attention by hand, add positions, heads and masks, stack them into a working GPT-style model in PyTorch, and see why BERT, GPT and T5 each kept a different half. You finish by writing a byte-pair encoding tokenizer in plain Python.
By the end of this module you can
- compute self-attention by hand from queries, keys and values
- explain why transformers need positional encoding and what multi-head attention adds
- build a masked transformer block and a tiny GPT in PyTorch
- choose between encoder-only, decoder-only and encoder-decoder models for a job
- explain the MLM and CLM pretraining objectives and how GPT and BERT differ
- implement BPE training, encoding and decoding from scratch
Lessons
5 in this module- 01Self-attentionCompute self-attention by hand with queries, keys and values, then write it as a PyTorch layer.
- 02Positional encoding and multi-head attentionExplain why attention ignores word order, add sinusoidal positions to fix it, and build multi-head attention in PyTorch.
- 03Masked attention and the full transformerAdd a causal mask so a model cannot cheat, assemble a full transformer block, and run a tiny GPT in PyTorch.
- 04Encoders, decoders, GPT vs BERTTell encoder-only, decoder-only and encoder-decoder models apart, explain MLM and CLM pretraining, and pick the right family for a job.
- 05Build a BPE tokenizerImplement byte-pair encoding training, encoding and decoding in plain Python, and measure why some languages cost more tokens.