Module 19
Streaming, Retries, Fallbacks and Rate Limits
Real traffic is messy: providers slow down, rate-limit you and sometimes go dark, and customers do things nobody planned for. Over one Diwali rush at Crumb, you make the support bot feel fast with streaming, ride out rate limits with backoff, stop duplicate refunds with idempotency keys, cap costs with token budgets and stay online through outages with a multi-provider gateway.
By the end of this module you can
- stream model replies from FastAPI to a browser with Server-Sent Events
- sort retryable errors from permanent ones and retry with exponential backoff, jitter and retry-after
- make side-effecting tools safe to call twice with idempotency keys
- enforce per-reply, per-request and per-user token budgets
- build an LLM gateway with automatic fallback and circuit breakers, and judge when LiteLLM fits better
Lessons
5 in this module- 01Streaming with Server-Sent EventsStream a model reply from a FastAPI endpoint to the browser with Server-Sent Events, and keep it streaming through proxies and errors.
- 02Exponential backoff and retry strategiesDecide which API errors deserve a retry and retry them with exponential backoff, jitter and the retry-after header, without stacking retry layers.
- 03Idempotency keysMake refunds, messages and other side-effecting tools safe to retry by sending idempotency keys and storing results on the server.
- 04Token budgetsCap what a reply, a request and a customer can cost with max_tokens, token counting before sending, a usage ledger and agent step limits.
- 05A multi-provider LLM gateway with fallbackBuild a gateway that hides OpenAI, Claude and Gemini behind one interface, falls back automatically during outages and uses circuit breakers to skip a failing provider.