Module 26
Production Best Practices
An AI app that works in a demo still needs layered guardrails, a habit of attacking itself, and a plan for the night everything goes wrong. In this module Crumb’s WhatsApp bot gets moderation and output validation, a red-team regression suite, kill switches and fallbacks, a cloud home in a container and a CI/CD pipeline with an eval gate, and then you build a trading research agent that applies every one of these lessons.
By the end of this module you can
- wrap a model in moderation, injection defences, least-privilege tools and code-enforced output validation
- run a red-team session and turn every successful attack into a repeated regression test
- shrink the blast radius of failures with fallback ladders, kill switches, rate limits and staged rollouts
- containerise a FastAPI AI app and deploy it to a managed cloud platform with secrets and health checks
- build a GitHub Actions pipeline that lints, tests, runs an eval gate on every pull request and deploys from main
- build an agent whose tools prevent look-ahead bias and whose risky decisions are made by code and humans
Lessons
6 in this module- 01Moderation APIs and output validationYou can layer input moderation, a custom safety classifier, prompt-injection defences and Pydantic-plus-business-rule output validation around a model, so a fooled model cannot turn into a harmful reply or action.
- 02Adversarial testingYou can plan and run a red-team session, store attacks with variants in an attack library, and turn every successful attack into a regression test that runs on every change, repeated for critical cases.
- 03Minimising model risk in productionYou can shrink the blast radius of AI failures with a time-boxed fallback ladder, shared feature flags and kill switches, rate limits, an automatic spend breaker, staged rollouts and owned alerts.
- 04Deploying your AI app to the cloudYou can containerise a FastAPI AI app with Docker and uv, run it locally, and deploy it to Google Cloud Run with secrets from a secrets manager, a cheap health check and sensible scaling limits.
- 05CI/CD for AI appsYou can build a GitHub Actions pipeline that lints, runs model-free unit tests and a per-category eval gate on every pull request, requires review for prompt changes, and deploys from main with a smoke test.
- 06Bonus project: AI trading research agentYou build an educational, paper-trading-only research agent that reads date-locked market data and news through agentic RAG, writes cited TradeIdea objects with invalidation levels, passes them through code-enforced risk checks and human review, and is evaluated with an honest backtest.