Python RQ with Redis on Docker

Arjun was keen to fix the WhatsApp line before the next broadcast, and he had a plan by lunchtime. Python already has a queue built in. He put a queue.Queue() inside the FastAPI app, started a background thread that took questions off it and ran the RAG pipeline, and changed the webhook to drop each message on the queue and return straight away.

import queue, threading

jobs = queue.Queue()                      # lives inside this one server process

def worker_loop():
    while True:
        message = jobs.get()              # wait for the next job
        result = ask(message.text, role="customer")
        send_whatsapp_reply(message.sender, result["answer"])

threading.Thread(target=worker_loop, daemon=True).start()

@app.post("/whatsapp/webhook")
def whatsapp_webhook(message: IncomingMessage):
    jobs.put(message)                     # enqueue, then acknowledge immediately
    return {"ok": True}

He tested it on his laptop. Webhooks were acknowledged in milliseconds. Answers arrived a few seconds later. He deployed it on Thursday afternoon.

On Thursday evening he deployed a small prompt fix. The server restarted. Twenty-three questions that were sitting in jobs disappeared, because jobs was just a Python object in the memory of a process that no longer existed. Then he noticed a second, quieter problem in the logs: production runs two server processes, so there were two separate queues and two worker threads, and neither knew about the other. And a third: when he tried to add a status page (“how many questions are waiting?”), each process could only count its own queue.

“The idea was right,” Priya said. “You built a rail for the order slips. But you nailed it inside one cook’s pocket. When he goes home, the slips go with him.”

Why the queue cannot live inside your app

“List everyone who needs to see the queue,” Priya said.

Arjun counted on his fingers. “Both web server processes, to add jobs. Every worker, to take jobs. Any status page, to count them. And it has to survive restarts.”

“So where can it live?”

“Not in any of them. Somewhere separate that they all connect to.”

Priya drew a big noticeboard in the middle of a kitchen. “Every bakery kitchen has one of these. Orders get pinned up by whoever takes them, at the front counter or on the phone. Any baker can walk over, take the next slip, and pin it to the ‘in progress’ section. The manager can glance at it and see how far behind they are. If a baker goes home, the board is still there.”

Redis is that noticeboard, as a server. It is an in-memory data store: a program that keeps data in RAM, so reads and writes take well under a millisecond, and lets any number of programs connect to it over the network and share that data. It is not a general database like PostgreSQL. It stores simple structures: strings, lists, hashes (small dictionaries), sets. That turns out to be exactly what a queue needs. A queue is a list where you push jobs on one end and pop them off the other.

Where the analogy breaks: a real noticeboard does not vanish in a power cut. Redis keeps everything in RAM, so by default a crash can lose recent data. You can tell it to also write every change to disk (you will turn that on below). It is still fast, and now it survives a restart.

Valkey needs a sentence of history. In 2024, Redis changed its licence away from a traditional open-source licence. Many of its original contributors, backed by large cloud companies, forked it (took the last open-source version and continued it as a separate project) under the Linux Foundation, and named the fork Valkey. Valkey speaks the same protocol as Redis, so the same Python client library and every tool in this module work with either. Crumb uses Valkey because it is open source and identical for this purpose. If your company already runs Redis, use that. Everything below says “Redis” for the concept and shows Valkey commands; swap the names freely.

Docker: a shipping container for software

Arjun’s next question was practical. “Do I install Valkey on my Mac, and then again on the server, and hope they match?”

“That’s what Docker is for.”

Docker runs software in containers: sealed packages that include a program and everything it needs (libraries, settings, a minimal operating system), so it runs the same on your laptop as on any server. The name comes from shipping containers: a ship does not care whether a container holds mangoes or motorbikes, because all containers have the same shape. Docker does not care what is inside an image, because all containers are started, stopped and connected the same way.

Two words to keep straight:

• An image is the packed container, ready to run: for example valkey/valkey:8, the official Valkey version 8 image from Docker Hub.
• A container is a running copy of an image. You can run several containers from one image.

Install Docker Desktop (Mac or Windows) or Docker Engine (Linux) from Docker’s website, then start Valkey with one command:

docker run -d --name valkey -p 6379:6379 -v valkey-data:/data \
  valkey/valkey:8 valkey-server --appendonly yes

Piece by piece:

• -d runs it in the background (detached).
• --name valkey gives the container a name you can refer to.
• -p 6379:6379 connects port 6379 on your machine to port 6379 inside the container. 6379 is the standard Redis port.
• -v valkey-data:/data stores Valkey’s files in a Docker volume (a storage area that outlives the container), so deleting and recreating the container keeps the data.
• valkey-server --appendonly yes starts the server with append-only file persistence: every change is also written to disk, so a restart does not lose the queue.

Check it is alive:

docker exec -it valkey valkey-cli ping

You should see PONG. docker exec runs a command inside the running container, here the Valkey command-line client.

A queue, by hand, in four commands

Before any library, Priya made Arjun build a queue with his bare hands in valkey-cli, so the library would never feel like magic.

docker exec -it valkey valkey-cli
127.0.0.1:6379> LPUSH orders "order-101"
(integer) 1
127.0.0.1:6379> LPUSH orders "order-102"
(integer) 2
127.0.0.1:6379> RPOP orders
"order-101"
127.0.0.1:6379> LLEN orders
(integer) 1

LPUSH pushes onto the left end of a list called orders. RPOP pops from the right end. Push on one side, pop from the other: first in, first out, a queue. LLEN counts what is left. Now open a second terminal, connect again, and RPOP orders. You get order-102. Two separate programs, one shared queue. That is the whole trick.

“So why do I need a library at all?” Arjun asked.

“Because a job is more than a string. You need to store which function to run and its arguments, track whether it’s queued, running, finished or failed, keep the result, notice when a worker dies halfway through, retry failures, and clean all of it up later. You could write that. People have. It’s called RQ.”

RQ: jobs, queues and workers in Python

RQ (Redis Queue) is a small, popular Python library that turns Redis into a job queue. It has three ideas, which map directly onto the restaurant:

• A Queue is a named rail, like chat. You put jobs on it from anywhere.
• A Job is one function call to be run later, with its arguments, status and result.
• A Worker is a separate process that watches one or more queues, takes jobs off them and runs them.

The one rule that trips everyone up at first: the worker runs your function by importing it, so the function must live in a module that the worker can import, not inside the script that enqueues it.

Step 1: the task module.

# tasks.py
import time

def slow_answer(question: str) -> str:
    """Pretend to be the RAG pipeline: slow, then returns a string."""
    time.sleep(3)
    return f"Answer to: {question}"

Step 2: enqueue a job.

# enqueue.py   (uv add rq redis)
import os
from redis import Redis
from rq import Queue
from tasks import slow_answer

redis = Redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379"))
q = Queue("chat", connection=redis)

job = q.enqueue(slow_answer, "How long can cream cakes stay out?",
                job_timeout=120,      # kill the job if it runs longer than 120 s
                result_ttl=3600)      # keep the result for one hour
print(job.id, job.get_status())

Run uv run python enqueue.py. You should see something like fca581b1-0f43-4475-bd78-a55641dfb831 JobStatus.QUEUED. The job is now sitting in Valkey. Nothing is running it yet, because there is no worker. Run the script a few more times; the jobs pile up patiently.

Step 3: start a worker in another terminal, in the same folder (so it can import tasks):

uv run rq worker chat --url redis://localhost:6379

You should see it start, report *** Listening on chat..., and then work through the waiting jobs one by one, logging a line like chat: tasks.slow_answer('How long can cream cakes stay out?') (fca581b1-...) followed, about three seconds later, by Job OK. Leave it running; enqueue another job from the first terminal and watch it get picked up almost instantly.

Step 4: check the result from anywhere.

from rq.job import Job
job = Job.fetch("fca581b1-0f43-4475-bd78-a55641dfb831", connection=redis)
print(job.get_status(), job.return_value())

You should see JobStatus.FINISHED and Answer to: How long can cream cakes stay out?. This lookup could run in a web server process on a different machine. All it needs is the job ID and a connection to Valkey. That is the token-number system from the previous lesson, working.

Following one job through Valkey

Priya wanted Arjun to see exactly what RQ wrote to the noticeboard, so they traced one job with valkey-cli before and after the worker ran. (These key names come from RQ 3; older versions are very similar.)

Right after enqueue, before any worker:

127.0.0.1:6379> KEYS rq:*
1) "rq:queues"
2) "rq:queue:chat"
3) "rq:job:fca581b1-0f43-4475-bd78-a55641dfb831"
127.0.0.1:6379> LRANGE rq:queue:chat 0 -1
1) "fca581b1-0f43-4475-bd78-a55641dfb831"
127.0.0.1:6379> HGET rq:job:fca581b1-0f43-4475-bd78-a55641dfb831 status
"queued"

Three keys:

• rq:queues is a set listing every queue name RQ knows about.
• rq:queue:chat is the queue itself: a plain Redis list holding only job IDs. It is the same kind of list you built by hand with LPUSH.
• rq:job:<id> is a hash (a small dictionary) with everything about the job. Its fields include status (queued), origin (chat), description (tasks.slow_answer('How long can cream cakes stay out?')), timeout (120), result_ttl (3600), timestamps like created_at and enqueued_at, and data: the function reference and arguments, compressed and pickled (Python’s built-in format for turning objects into bytes).

While the worker runs it: the worker takes the ID off rq:queue:chat, sets the job’s status to started, records which worker has it, and adds it to a registry of in-progress jobs (rq:wip:chat). That registry is how RQ notices a worker that died mid-job: a job that sits “in progress” with no live worker heartbeat is eventually marked as failed instead of being stuck forever.

After it finishes:

127.0.0.1:6379> LLEN rq:queue:chat
(integer) 0
127.0.0.1:6379> HGET rq:job:fca581b1-0f43-4475-bd78-a55641dfb831 status
"finished"
127.0.0.1:6379> TTL rq:job:fca581b1-0f43-4475-bd78-a55641dfb831
(integer) 3597

The queue is empty, the job’s status is finished, the return value is stored under its own key (rq:results:<id>), and the job ID now sits in a finished registry (rq:finished:chat). The TTL (time to live) of 3,597 seconds is result_ttl counting down: in an hour, Valkey deletes the job and its result automatically, so the noticeboard does not fill up forever.

“So it’s only lists and dictionaries,” Arjun said.

“It’s lists and dictionaries, plus a lot of careful bookkeeping that you didn’t have to write.”

When jobs fail, and how you find out

Priya made him break it on purpose. He added a task that raises an error:

# tasks.py (add this)
def broken_answer(question: str) -> str:
    raise ValueError("Chroma index not found")

After enqueuing it with a worker running, the worker logged the full traceback and moved on to the next job. Nothing crashed. The failed job went into a failed registry (rq:failed:chat) with its error stored on it.

Three tools for failures:

• job.is_failed and job.exc_info in Python show whether a job failed and the error text, so your status endpoint can report it.
• uv run rq info --url redis://localhost:6379 prints every queue with its job count, every worker and what it is doing, and failure counts. Add -i 1 to refresh every second, which is a surprisingly good live dashboard during a rush.
• uv run rq requeue --all --queue chat --url redis://localhost:6379 puts every failed job back on the queue, once you have fixed the cause.

There is also a time limit. If a job runs longer than its job_timeout (120 seconds above), the worker kills it and marks it failed. Choose a timeout comfortably above your slowest normal job (Crumb’s slowest RAG answers take about 15 seconds) but low enough that a hung call to an LLM provider does not tie up a worker for an hour.

Plugging in the real RAG pipeline

The real task wraps the ask() function from the Module 09 project. One detail matters a lot for speed: where the models get loaded.

By default, an RQ worker on Linux and macOS forks (makes a copy of itself, as a short-lived child process) for every job, runs the job in the child, and lets the child exit. That protects the worker from a job that crashes or leaks memory. But if your task module loads the embedding model and the cross-encoder when it is first imported, and the parent never imported it, every child loads the models again: two or three seconds wasted on every job.

The fix is a small custom worker script that imports your task module, and therefore loads the models, before the worker starts. Each forked child then starts with the models already in memory.

# rag_tasks.py
from pipeline import ask            # importing this loads Chroma, BM25 and the reranker

def answer_question(question: str, role: str) -> dict:
    return ask(question, role)      # the full guarded RAG pipeline from Module 09

# worker.py
import os
from redis import Redis
from rq import Queue, Worker

import rag_tasks                    # preload models once, in the parent process

redis = Redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379"))
Worker([Queue("chat", connection=redis)], connection=redis).work()

Run it with uv run python worker.py. The first job should take about as long as the pipeline itself (around 3 seconds), not 3 seconds plus model loading.

Two platform notes. RQ’s normal worker relies on fork, which Windows does not have, so on Windows run workers inside WSL or in a Docker container. On macOS, some libraries complain or crash when used in a forked child; if you hit that during development, use RQ’s SimpleWorker class, which runs jobs in the worker process itself without forking (from the command line: --worker-class rq.worker.SimpleWorker).

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks

• Valkey has no password by default. Exposing port 6379 to the internet is a well-known way to get a server hijacked. On a real server, keep Valkey on a private network, or set a password and use a redis://:password@host:6379 URL.
• Pickled data trusts its source. RQ stores arguments with Python’s pickle by default, and unpickling data from an untrusted writer can run code. Only your own services should be able to write to the queue.
• Memory is finite. Valkey keeps everything in RAM. Large results (say, whole retrieved documents) multiplied by thousands of jobs and a long result_ttl add up. Store answers and IDs, not documents, and keep TTLs short.
• Code drift. Workers run whatever version of tasks.py they started with. Deploy workers along with the web server, or old workers will run old code on new jobs.

Try it yourself

With Valkey running in Docker and one RQ worker:

1. Enqueue 10 slow_answer jobs in a loop and time how long until all are finished.
2. Start a second worker in another terminal and repeat.
3. While jobs are running, run rq info -i 1 in a third terminal and watch the queue drain.
4. Stop the Valkey container with docker stop valkey, start it again with docker start valkey, and check whether a finished job’s result is still there.

Show the answer

With one worker, 10 jobs of 3 seconds take about 30 seconds: jobs run one at a time per worker. With two workers, about 15 seconds. rq info shows the queue count dropping and both workers flipping between busy and idle. After restarting the container, finished results within their result_ttl are still there, because --appendonly yes wrote them to the valkey-data volume. If you had started Valkey without the volume and the append-only flag, recreating the container would have wiped everything. You have now measured horizontal scaling, which is the subject of the last lesson in this module.

Common mistakes

• Keeping the queue in your app’s memory. It cannot be shared between processes and disappears on restart.
• Defining task functions in the enqueuing script. Workers import tasks by name; put them in a module both sides can import.
• Loading models inside the task function. Every job reloads them. Preload in a custom worker script.
• Running Valkey without persistence or a volume. A container restart wipes the queue.
• Opening Valkey to the internet. Keep it private, or at least password-protected.
• Timeouts that are too long or missing. A hung LLM call can hold a worker hostage. Set job_timeout deliberately.

Recap

Meera came down to see “the new noticeboard” and Arjun explained it with a real one, the order board in the Baner kitchen.

• The list of waiting questions now lives in a small, very fast program called Valkey, a free twin of Redis. Every part of the system connects to it, like bakers walking up to the board.
• It runs inside a Docker container, which packages it so it works the same on my laptop and on the server, and it saves to disk so a restart doesn’t wipe it.
• A Python library called RQ handles the bookkeeping: it pins up each question as a job, lets separate worker programs take jobs, and records whether each one is waiting, running, finished or failed.
• If we restart the website, nothing is lost. If a job fails, it goes to a “failed” pile we can inspect and retry.
• Adding a second worker halved the time for ten questions.

“So customers ask, the board fills up, and the workers empty it,” Meera said. “But how does the customer’s phone know when their answer is ready?”

“For WhatsApp, the worker sends it,” Arjun said. “For the website chat, the browser has to ask.” He looked at Priya. “Which means I need to build the asking part.”

Go deeper

• RQ documentation: https://python-rq.org
• Valkey documentation: https://valkey.io
• Docker documentation: https://docs.docker.com
• redis-py, the Python client used here: https://github.com/redis/redis-py