Module 10
Scalable RAG with Async Queues
A RAG answer takes seconds, and a burst of users can turn seconds into timeouts, duplicates and lost questions. You will move the slow work onto a job queue with Redis (or Valkey) and RQ, expose it through a FastAPI enqueue-and-poll API, and size a pool of workers with simple arithmetic so the system stays calm under load.
By the end of this module you can
- explain when a synchronous RAG endpoint breaks under load and when it is perfectly fine
- run Redis or Valkey in Docker and use Python RQ to enqueue jobs, run workers and inspect failures
- build a FastAPI chat API that returns a job ID with 202 Accepted, reports status and handles retries idempotently
- calculate how many workers you need and check that memory and LLM rate limits can support them
- scale workers horizontally with Docker Compose, with retries, timeouts, priorities and backpressure
Lessons
4 in this module- 01Sync vs async RAG architecturesDiagnose why a synchronous RAG endpoint melts down in a burst, and design the queue-and-worker architecture that keeps it orderly.
- 02Python RQ with Redis on DockerRun Valkey (or Redis) in Docker, enqueue jobs with RQ, run workers that preload your models, and inspect queued, finished and failed jobs.
- 03A FastAPI chat queue: enqueue, poll, dequeueBuild a FastAPI chat API that enqueues questions with 202 Accepted, reports job status and progress, polls with backoff, and handles double submissions safely.
- 04Workers and horizontal scalingSize a worker pool with Little's law, check it against memory and rate limits, and scale it with Docker Compose using retries, timeouts, priorities and backpressure.