Workers and horizontal scaling

Eleven days before Diwali, the marketing team shared their plan: a WhatsApp broadcast to all 18,000 customers at 7:00 p.m. on Diwali eve, announcing last-minute hamper slots. Priya pulled up the numbers from the pilot Saturday, when a broadcast to the same list produced about 600 questions in five minutes. “Diwali eve is the busiest evening of our year,” Meera said. “Assume worse.”

Priya’s estimate: about 2,000 questions in the first ten minutes. That is 3.3 questions every second.

Arjun had one worker running. He knew from the previous lessons that more workers meant faster draining, so on the Saturday before Diwali he ran a load test with what felt like a safe margin: he started 100 workers on Crumb’s 8 GB server with one command, and fired 2,000 test questions at the queue.

It went badly in two different ways at once. Within a minute, the server’s memory was full. Linux’s out-of-memory killer started shooting worker processes at random, and their half-finished jobs landed in the failed registry. And the workers that survived fired so many Claude calls at the same moment that the provider began answering with HTTP 429, “too many requests”. After four minutes, 300 test questions had failed and the queue was longer than with one worker.

Meera read the summary over his shoulder. “So more workers made it worse?”

“Too many did,” Priya said. “There’s a right number, and we can calculate it. Let’s do it on paper before we do it on the server.”

Why “more workers” has a ceiling

Priya drew the Crumb central kitchen.

“Diwali orders are piling up. What are your options?”

“Hire more bakers,” Arjun said.

“That’s horizontal scaling: more of the same unit, side by side. What else?”

“Buy a bigger oven? Faster mixers?”

“That’s vertical scaling: making one unit more powerful. A faster server, a bigger machine, a quicker model. Now, what stops you hiring 100 bakers?”

He thought about it. “The kitchen is only so big. They’d be falling over each other.”

“That’s your server’s memory. And?”

“The gas supply? If every oven is on, the pressure drops.”

“That’s your LLM provider’s rate limit. Every worker draws from the same supply. Past a point, adding bakers only means more people standing around waiting for gas.” She circled the kitchen. “So the job isn’t ‘add workers’. It’s: how many do we need, how many fit, and how many can the supply feed? The answer is the smallest of the three.”

Where the analogy breaks: bakers get tired and slow down over a long shift; workers do not. But workers have their own failure that bakers rarely do: they can die instantly and silently mid-task (crash, out of memory, a server restart), so the system must be designed to notice and recover. That is the second half of this lesson.

How many workers Crumb needs: the arithmetic

The rule from the first lesson of this module is Little’s law:

average jobs in progress = arrival rate × time per job

Diwali eve: 3.3 questions per second arrive. Each takes about 3.1 seconds of a worker’s time (the median from the Module 09 project; scope check, retrieval, reranking and the answer). So on average 3.3 × 3.1 ≈ 10.3 workers must be busy at every moment to keep up at all.

Here is what different worker counts would do during a 10-minute burst of 2,000 questions. Each worker finishes 1 ÷ 3.1 ≈ 0.32 jobs per second.

Workers	Jobs finished per second	Backlog after 10 minutes	Worst wait for a customer
1	0.32	about 1,800	over 90 minutes
4	1.29	about 1,230	about 16 minutes
8	2.58	about 450	about 3 minutes
12	3.87	none (it keeps up)	a few seconds, plus the 3 s answer
16	5.16	none	the same as 12

How the 4-worker row works: in 600 seconds, 4 workers finish 1.29 × 600 ≈ 774 questions. The other 2,000 − 774 ≈ 1,230 are still waiting when the burst ends. The last customer to ask has 1,230 jobs ahead of them, which takes 1,230 ÷ 1.29 ≈ 950 seconds, about 16 minutes, to clear.

Two lessons hide in this table. First, the relationship is sharp: 8 workers mean a 3-minute wait, 12 mean almost none. Below the Little’s-law number, queues grow without limit for as long as the burst lasts; above it, they stay short. Second, beyond the right number, extra workers buy nothing for customers. Sixteen workers are no faster for Mrs Kulkarni than twelve; they sit idle and cost memory.

Arrivals are lumpy, not perfectly even, so add headroom: Crumb planned for 12 workers, about 20% above the 10.3 average.

How many workers fit: memory

Arjun measured one worker process on his laptop after it had loaded the embedding model and the cross-encoder reranker and answered one question: about 580 MB. Almost all of that is the PyTorch library and the two models; the RAG code itself is tiny.

Workers	Memory for workers	Fits on Crumb’s 8 GB server?
4	about 2.3 GB	Yes, easily
12	about 7.0 GB	Barely, with nothing left for Valkey, the API or the operating system
100	about 58 GB	Not remotely: this was the out-of-memory massacre

So twelve workers needed more than one 8 GB server. Crumb’s options were a bigger server (vertical) or a second one (horizontal). They chose two servers of 8 GB, six workers each, because two servers also mean one can fail without taking the whole service down.

This is where horizontal scaling with a queue shows its real strength. Workers do not need to be on the same machine as anything else. A worker on the second server connects to the same Valkey URL, pulls from the same chat queue, and the web server neither knows nor cares. Adding capacity is: start more workers anywhere that can reach Valkey.

One trap Arjun nearly fell into: the web server must not load the models. In the previous lesson, main.py imported answer_question from rag_tasks to enqueue it, and rag_tasks imports the pipeline, which loads the models. That quietly added 580 MB to every API process for nothing. RQ lets you enqueue a function by its import path as a string, so the API never imports the heavy module:

job = chat_queue.enqueue("rag_tasks.answer_question", req.question, "customer",
                         job_timeout=60, result_ttl=600)

The worker resolves the string and imports the function on its side, where the models belong.

How many workers the supply can feed: rate limits

Every LLM provider limits how much each account may use per minute, usually counted in both requests per minute and tokens per minute. Your limits are shown in your provider’s console and grow with your usage tier.

Crumb’s demand on Diwali eve: each question makes two Claude calls (the scope check and the answer). At 3.3 questions per second, that is about 3.3 × 2 × 60 ≈ 400 requests per minute. Each question sends roughly 1,650 input tokens in total, so about 3.3 × 1,650 × 60 ≈ 330,000 input tokens per minute. Arjun compared those two numbers with the limits in the console, and asked the provider for a higher tier a week in advance, because limit increases are not always instant.

Here is the insight that made Priya smile: the number of workers is your rate limiter. Twelve workers can never have more than about twelve calls in flight at once, however many thousands of questions are queued. In the synchronous design from the start of this module, every customer meant a simultaneous call; here, the queue absorbs the burst and the worker count sets the pace. If you ever get 429s, you have a dial to turn down.

And the money, for Meera: at an illustrative $3permillioninputtokensand$15 per million output tokens (check current pricing; it changes), each question costs about 1,650 × $3/M+250\times$15/M ≈ $0.009,underonerupee.TheDiwalieveburstof2,000questionscostsabout$17, roughly ₹1,500, plus two small servers for the evening.

Following the Diwali eve burst minute by minute

Priya and Arjun simulated the evening with 12 workers across two servers before the real thing. Here is the plan they wrote down, the way it actually ran on the night.

Time	Queue depth	Workers busy	What happened
6:45 p.m.	0	2 of 12	Arjun scaled from 2 to 12 workers, fifteen minutes before the broadcast
7:00 p.m.	0	2	Broadcast sent
7:01 p.m.	9	12	Messages arriving at about 3.5 per second; a short queue forms in the first rush
7:03 p.m.	4	12	Workers slightly faster than arrivals; queue stays tiny
7:05 p.m.	22	12	A second wave (people forwarding the message); worst wait about 9 seconds
7:07 p.m.	3	12	Wave absorbed
7:12 p.m.	0	5	Arrivals falling off
7:40 p.m.	0	1	Arjun scaled back down to 2 workers

Total questions: 2,140. Failed jobs: 6, all automatically retried and answered (more on that below). Duplicate replies: 0. Lost messages: 0. The worst wait anyone experienced was about 12 seconds.

Notice 6:45 p.m. The cheapest autoscaler in the world is a calendar. When you know a burst is coming, because you are the one sending the broadcast, scale up before it, not after.

Opening the hood: workers that recover from failure

Running twelve workers on two servers means things will go wrong somewhere. These four settings turn “something failed” into “something was retried”.

Retries with backoff. A Claude call can fail for a passing reason: a network blip, a brief 429, a provider hiccup. RQ can retry a failed job automatically, waiting longer before each attempt (backoff), so you do not hammer a struggling service:

from rq import Retry

job = chat_queue.enqueue("rag_tasks.answer_question", question, "customer",
                         job_timeout=60, result_ttl=600,
                         retry=Retry(max=3, interval=[5, 15, 45]))   # wait 5 s, 15 s, 45 s

Retries that wait between attempts are scheduled for later, so at least one worker must run RQ’s scheduler (start workers with --with-scheduler, or call .work(with_scheduler=True) in a custom worker script). Without it, retries with intervals never fire. This is the most common “retries don’t work” bug in RQ.

Retrying is only safe if the job is idempotent: running it twice must not cause harm. Answering a question twice is harmless (a few paise). Sending the customer’s WhatsApp reply twice is not. So Crumb’s task checks whether a reply was already sent for this message ID before sending.

Timeouts. job_timeout=60 kills any job that runs more than a minute, freeing the worker. A hung network call should never hold one of your twelve bakers hostage all evening.

Graceful shutdown. When you scale down or deploy, workers receive a stop signal. An RQ worker reacts with a warm shutdown: it finishes the job it is running, then exits. But Docker, by default, waits only 10 seconds before force-killing a container, and Crumb’s slowest answers take about 15. Setting a longer grace period (90 seconds, below) lets every in-flight answer finish.

Priorities. Not all jobs are equally urgent. A customer waiting on WhatsApp is urgent; re-indexing the handbook after an update, or running the nightly evaluation, is not. RQ workers can listen to several queues in priority order:

uv run rq worker chat-high chat-low --with-scheduler --url redis://valkey:6379

A worker always takes the next job from chat-high if there is one, and only touches chat-low when the high queue is empty. On Diwali eve, nightly jobs waited until the rush was over.

Crumb renamed its customer queue to chat-high (so the API now enqueues with Queue("chat-high", ...)) and put re-indexing and evaluation jobs on chat-low. The custom worker script from the Redis lesson grows by two lines, to listen to both queues and run the scheduler that retries need:

# worker.py
import os
from redis import Redis
from rq import Queue, Worker

import rag_tasks                                   # preload models once, before forking

redis = Redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379"))
queues = [Queue(name, connection=redis) for name in ("chat-high", "chat-low")]
Worker(queues, connection=redis).work(with_scheduler=True)

Backpressure in the submit endpoint takes four lines:

from fastapi import HTTPException

MAX_QUEUE = 230                      # about 60 s of work for 12 workers

@app.post("/chat", status_code=202)
def submit(req: ChatRequest, response: Response):
    if chat_queue.count > MAX_QUEUE:                 # jobs waiting right now
        raise HTTPException(status_code=503, headers={"Retry-After": "60"},
                            detail="We're very busy. Please try again in a minute.")
    ...                                              # enqueue as in the previous lesson

Running it all with Docker Compose

Docker Compose runs several containers together from one file: here Valkey, the API and the workers. It also makes horizontal scaling a single command.

The image. One Dockerfile serves both the API and the workers; they differ only in the command they run.

FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen --no-install-project      # install dependencies (cached layer)
COPY . .
RUN uv sync --frozen

The services.

# compose.yaml
services:
  valkey:
    image: valkey/valkey:8
    command: valkey-server --appendonly yes
    volumes:
      - valkey-data:/data

  api:
    build: .
    command: uv run fastapi run main.py --host 0.0.0.0 --port 8000
    environment:
      - REDIS_URL=redis://valkey:6379
    ports:
      - "8000:8000"
    depends_on:
      - valkey

  worker:
    build: .
    command: uv run python worker.py           # preloads models, runs with the scheduler
    environment:
      - REDIS_URL=redis://valkey:6379
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    depends_on:
      - valkey
    stop_grace_period: 90s                      # let in-flight answers finish

volumes:
  valkey-data:

Inside Compose, containers reach each other by service name, so the URL is redis://valkey:6379, not localhost. Valkey has no ports: entry, so it is reachable only by the other containers, not from the internet. Only the worker gets the Anthropic key; the API never calls Claude.

Scale with one flag:

docker compose up -d --scale worker=6

You should see one Valkey, one API and six worker containers. docker compose up -d --scale worker=2 scales back down, with each stopping worker finishing its current job first. On the second server, run only the worker service, pointed at the first server’s Valkey over a private network.

RQ also ships a rq worker-pool command (rq worker-pool chat-high chat-low -n 6) that starts several workers from one command inside a single container, which can be handy on one machine.

Library APIs change quickly. If a name has moved, check the docs linked below.

Watching it run

You cannot scale what you cannot see. During the burst, Arjun kept three numbers on screen:

• Queue depth (chat_queue.count, or rq info -i 1): how many jobs are waiting. Rising for more than a minute means too few workers.
• Age of the oldest waiting job: the wait the unluckiest customer is experiencing right now. This is the number customers feel.
• Failed jobs per minute (the size of the failed registry): a jump means something downstream broke, often the LLM provider.

On bigger platforms, these numbers drive automatic scaling. On Kubernetes, for example, a tool called KEDA can add and remove worker containers based on the length of a Redis list. For Crumb’s two servers, a calendar reminder and one --scale command were enough. Do the simple thing until it hurts.

Where it breaks

• More workers than memory. Each worker carries its models. Measure one, multiply, and leave room for everything else.
• More workers than the provider allows. You will see 429s. The worker count is your concurrency cap; set it to what your rate limit can feed.
• Retries that never run. Retry intervals need a worker with the scheduler enabled.
• Retries that double side effects. Make tasks idempotent before turning on retries: check before sending, charging or writing.
• Default shutdown grace periods. Docker’s 10 seconds will kill long answers mid-flight. Set it above your slowest job.
• Many processes for waiting work. Each RQ worker runs one job at a time, and an LLM job spends most of its 3 seconds waiting for the provider. At very large scale, teams move the models into a shared embedding-and-reranking service and use lighter workers that juggle many waiting jobs each (with async code or threads). At Crumb’s scale, a dozen simple workers is cheaper than that complexity.

Try it yourself

Measure horizontal scaling yourself with the fake 3-second task from the Redis lesson.

# load_test.py: enqueue N fake jobs and time how long until all are finished
import os, sys, time
from redis import Redis
from rq import Queue

redis = Redis.from_url(os.getenv("REDIS_URL", "redis://localhost:6379"))
q = Queue("chat", connection=redis)
n = int(sys.argv[1])

start = time.perf_counter()
jobs = [q.enqueue("tasks.slow_answer", f"question {i}") for i in range(n)]
while not all(j.is_finished for j in jobs):
    time.sleep(0.5)
    for j in jobs:
        j.refresh()                                    # re-read status from Valkey
print(f"{n} jobs finished in {time.perf_counter() - start:.0f} s")

Start 1 worker and run uv run python load_test.py 60. Then repeat with 2, 4 and 8 workers (start extra workers in more terminals, or with docker compose up -d --scale worker=N). Predict each time before you run it.

Show the answer

Each job sleeps 3 seconds, so 60 jobs take about 180 seconds with 1 worker, 90 with 2, 45 with 4 and 23 with 8, plus a little overhead. Time halves every time workers double, because nothing else is a bottleneck: sleeping uses no memory, no CPU and no rate limit. With the real RAG task, the curve flattens as soon as you hit one of the three ceilings from this lesson: memory (workers crash or the machine slows down), CPU (reranking on a small server), or the provider’s rate limit (429s and retries). Finding where your curve bends is exactly how you find your real capacity.

Common mistakes

• Guessing the worker count. Calculate it: arrival rate × time per job, plus headroom.
• Scaling after the rush starts. If you know when the burst is coming, scale before it.
• Loading models in the API. Enqueue by string path so only workers import the heavy code.
• Turning on retries without idempotency or a scheduler. You get either duplicated side effects or retries that never happen.
• Accepting unlimited work. Without backpressure, a long queue means answering customers who already left.
• Watching only CPU. Queue depth and the age of the oldest job are what customers feel.

Recap

On Diwali morning, Meera asked Arjun to explain the plan for the evening to the store managers in the family WhatsApp group. He wrote it in five lines.

• At 6:45 we go from 2 “cooks” to 12, spread over two servers, because the maths says about 10 must be busy every second to keep up with the broadcast.
• We can’t run 100: each cook needs about half a gigabyte of memory, and our AI provider only lets us make so many calls a minute. Twelve is enough, and it fits.
• If a call fails, the job is retried automatically after a short wait, and we never send the same reply twice.
• If the website queue gets too long, it politely asks people to try again in a minute instead of making them wait five.
• At 7:40 we go back down to 2.

On the night, the queue never went above 22. Meera watched the dashboard for the first ten minutes, then went downstairs to help pack hampers. The next morning she forwarded one customer message to the whole team: “Your WhatsApp bot answered my allergy question faster than my own family answers my messages.”

Priya read it and turned to Arjun. “Every worker last night loaded two models from Hugging Face, the embedding model and the reranker. You’ve been using them like magic boxes. Next module, we open those boxes: what Hugging Face actually is, and how to run pre-trained models properly in production.”

Go deeper

• RQ documentation (workers, retries, scheduling): https://python-rq.org
• Docker documentation (Compose and scaling): https://docs.docker.com
• Valkey documentation: https://valkey.io
• FastAPI documentation: https://fastapi.tiangolo.com