Module 22
Observability and Tracing
AI apps fail softly: a wrong answer still returns 200 OK in 3.2 seconds. This module follows Crumb's support bot from a dangerous allergy answer to its root cause, teaching traces and spans, the main tracing tools (Langfuse, LangSmith, Braintrust, OpenLLMetry), the OpenTelemetry GenAI conventions, and how to instrument a full agent so every request records what it read, what it cost and where its time went.
By the end of this module you can
- explain why logs and metrics miss the soft, multi-step failures of AI apps, and what a trace adds
- describe a trace as a tree of spans with names, timings, parents, attributes and status, grouped into sessions
- compare Langfuse, LangSmith, Braintrust and OpenLLMetry on licence, hosting, strengths and where your data lives
- create OpenTelemetry spans that follow the GenAI semantic conventions for model, agent and tool calls
- instrument a RAG agent end to end with Langfuse, read its trace tree to find a failing step, and compute cost and latency per request
- protect personal data in traces with masking, retention limits and access control, and choose a sensible sampling strategy
Lessons
4 in this module- 01Why logs are not enough for agentsYou can explain why a 200 OK log line hides AI failures, and describe a request as a trace: a tree of timed spans with attributes, grouped into sessions.
- 02Langfuse, LangSmith, Braintrust and OpenLLMetryYou can compare the main LLM tracing tools on licence, hosting and data location, and add basic tracing with Langfuse, OpenLLMetry or LangSmith.
- 03OpenTelemetry GenAI conventionsYou can create OpenTelemetry spans for model, agent and tool calls using the GenAI semantic conventions, and send them to any OTLP backend.
- 04Tracing model calls, tools and retrievalYou can instrument a RAG agent end to end with Langfuse, read its trace tree to find a failing step, compute cost and latency per request, and attach user feedback.