Module 23
Evaluations: The Underrated Skill
Most AI teams tweak prompts on gut feel. This module teaches the loop that replaces guessing: read real traces, bucket the failures, turn each one into a pass/fail check, and trust an LLM judge only after it agrees with human labels. You also diagnose RAG systems with RAGAS and package everything as pytest tests that run in CI.
By the end of this module you can
- run error analysis on 50 to 100 traces with open and axial coding, and prioritise the failure modes you find
- write binary pass/fail evals with clear definitions, using code checks first and model judges where needed
- build an LLM-as-judge and measure its alignment with human labels using TPR, TNR and a hold-out set
- sort any failure into the gulfs of comprehension, specification or generalisation and choose the right fix
- use RAGAS metrics to tell retrieval failures from generation failures in a RAG system
- test LLM apps with pytest: unit tests, a mocked model client, prompt snapshots and threshold-based eval tests in CI
Lessons
6 in this module- 01Reading traces by hand and bucketing failuresRead 50 to 100 real traces, write a specific note on the first failure in each, group the notes into failure modes, and decide what to fix first.
- 02Binary pass/fail evaluationsTurn each failure mode into a yes-or-no eval with a written definition, prefer code checks, and track one pass rate per failure mode instead of a vague average score.
- 03LLM-as-judge and judge alignmentWrite a pass/fail LLM judge for one failure mode, align it against human labels split into training, development and hold-out sets, and report its TPR and TNR.
- 04The three gulfs: comprehension, specification, generalisationDiagnose any LLM failure as a gap of comprehension, specification or generalisation, and apply the cheapest matching fix in the right order.
- 05RAGAS metrics in practiceScore a RAG system with faithfulness, answer relevancy, context precision and context recall, read them together to find whether retrieval or generation failed, and validate the scores against your own labels.
- 06Testing AI code with pytestTest an LLM app in layers: fast pytest unit tests for plain code, a mocked model client, prompt snapshots, and a threshold-based eval test that runs in CI without flaky failures.