Module 33
Fine-Tuning, LoRA/QLoRA and Trending Topics
Learn when fine-tuning is the right tool and when prompting or retrieval is better, then fine-tune for real: LoRA and QLoRA by hand and with PEFT, and an end-to-end Unsloth run on Crumb support replies that ends in Ollama. The module closes with the ideas behind modern image generation and vision: diffusion models, vision transformers and CLIP.
By the end of this module you can
- decide between prompting, RAG and fine-tuning for a task, and explain why fine-tuning teaches habits better than facts
- prepare a clean, scrubbed fine-tuning dataset in the messages format with a held-out test split
- explain LoRA and QLoRA with worked numbers and apply them with the PEFT library
- fine-tune a small open model with Unsloth in Colab, evaluate it against a baseline and run it in Ollama as GGUF
- explain how diffusion models, vision transformers and CLIP work and where each one breaks
Lessons
4 in this module- 01Pretraining vs fine-tuning, and when not toChoose between prompting, RAG and fine-tuning for a real task, and prepare a clean fine-tuning dataset with a held-out test split.
- 02LoRA, QLoRA and parameter-efficient fine-tuningExplain and implement LoRA from scratch, apply LoRA and QLoRA with the PEFT library, and estimate the memory a fine-tune needs.
- 03Hands-on fine-tuning with UnslothFine-tune a small open model on Crumb-style support replies in Colab with Unsloth, prove it beats the baseline, and run it locally in Ollama as a GGUF file.
- 04Diffusion, Vision Transformers and CLIPExplain how diffusion models generate images, how vision transformers read images as patch tokens, and how CLIP links images and text, and use each in code.