Sync vs async RAG architectures

The handbook assistant had been such a hit at the managers’ meeting that Meera made a bigger decision: put the same RAG engine behind Crumb’s customer WhatsApp line for the two pilot stores, Koregaon Park and Baner. Customers could ask about hampers, allergens, delivery times and refunds, and get an answer with the policy behind it.

For four days it was lovely. Twenty, thirty questions an hour, each answered in about three seconds.

On Saturday at 7:00 p.m., the marketing team sent a WhatsApp broadcast to 18,000 customers: “Diwali Delight hampers are back in stock!” In the next five minutes, about 600 messages arrived. By 7:04, customers were waiting four minutes for a reply. By 7:06, some customers had received the same answer three times. At 7:10, Arjun pushed a quick fix to the server, which restarted it, and every question in flight vanished. Thirty-seven customers never got an answer at all.

On Monday Meera had the WhatsApp complaints printed out. “Three answers to one question is funny once,” she said. “No answer is not funny. What happened? It worked all week.”

“It worked for one customer at a time,” Priya said. “On Saturday we had six hundred at once. It’s not an AI problem. It’s a kitchen problem.”

How the server actually spent Saturday evening

Arjun pulled up the code. The WhatsApp provider calls a URL on Crumb’s server every time a customer sends a message (this is called a webhook: an HTTP request another service sends to you when something happens). His endpoint looked like this:

@app.post("/whatsapp/webhook")
async def whatsapp_webhook(message: IncomingMessage):
    result = ask(message.text, role="customer")        # 3 to 9 seconds of RAG work
    send_whatsapp_reply(message.sender, result["answer"])
    return {"ok": True}

It does everything in one go: receive the message, run the whole RAG pipeline from Module 09 (scope check, hybrid search, reranking, the Claude call), send the reply, and only then tell the provider “OK, got it”. This style is called synchronous (or sync): the caller waits, connected, until the work is completely finished.

“How many of these can the server run at the same time?” Priya asked.

Arjun checked the deployment. Two server processes. “Two? No, it’s async, so… lots?”

“Let’s test that instead of guessing.”

She had him write a tiny experiment: an endpoint that pretends to be a slow LLM call by sleeping for 3 seconds, and a script that fires 10 requests at it at the same moment.

import time
from fastapi import FastAPI

app = FastAPI()

@app.get("/blocking")
async def blocking():
    time.sleep(3)              # stands in for the slow, blocking RAG call
    return {"ok": True}

He ran it with a single server process. The first request finished after 3.1 seconds. The tenth finished after 30.1 seconds. They had gone through strictly one at a time.

“But it says async,” Arjun said.

“async def tells Python: this function will politely step aside whenever it’s waiting, so others can run. But time.sleep, and the Anthropic client you called inside ask(), are ordinary blocking calls. They don’t step aside. They hold the whole process hostage while they wait. One slow, blocking call inside an async def freezes every other request on that process.”

The quick fix that helped, and the problems it could not fix

Priya showed him a first fix. If you write the endpoint as a plain def instead of async def, FastAPI runs it in a pool of helper threads (separate lines of execution inside one process; by default it allows up to 40 at once), so blocking calls stop freezing everything.

@app.get("/threaded")
def threaded():
    time.sleep(3)
    return {"ok": True}

Same 10 simultaneous requests: all ten finished at 3.0 seconds. With 100 simultaneous requests, the first finished at 3.1 seconds and the last at 9.1, because 40 threads worked through them in three waves.

Arjun deployed it. The following Saturday’s broadcast went better. And then three different problems showed up, none of which more threads could fix.

1. The provider retried, so customers got duplicates. Most webhook providers expect a quick acknowledgement, within a few seconds, and if they do not get one they assume the delivery failed and send the message again. With answers taking 3 to 9 seconds under load, many deliveries looked “failed”. Each retry started a second full RAG run for the same question. More load, more delay, more retries: engineers call this a retry storm.

2. The LLM provider pushed back. With up to 80 threads (two processes of 40), Crumb’s server fired 70 Claude calls in the same second. Every LLM provider enforces a rate limit: a maximum number of requests or tokens your account may use per minute. Crumb’s limit was hit, and the provider answered with HTTP status 429, “too many requests”. Those customers got an error instead of an answer.

3. Restarts still lost work. Any question being processed when the server restarted (to deploy a fix, or because it crashed) was gone. Nothing had written it down anywhere.

“So the problem isn’t speed,” Arjun said slowly. “Each answer takes three seconds no matter what. The problem is what happens when lots of them arrive together.”

“Exactly. Two different numbers. Latency is how long one customer waits. Throughput is how many customers you can finish per minute. Saturday was a throughput problem that showed up as terrible latency.”

The token counter at Vaishali

Priya drew two restaurants on the whiteboard.

Restaurant one: a small café where the person at the counter takes your order, walks into the kitchen, cooks it, comes back and hands it to you, then takes the next order. When it is quiet, it is perfect: no fuss, no tokens. When twenty people walk in at once, the queue goes out of the door, the people at the back give up, and if the cook trips and drops a plate, that order is forgotten. That is synchronous RAG.

Restaurant two: a busy Pune udupi restaurant like Vaishali on a Sunday morning. You order at the counter, pay, and get a token number. The counter clerk never cooks. They pin your order slip on a rail, and the cooks take slips from the rail one by one. When your dosa is ready, they call your number. The counter can take an order every few seconds even when the kitchen is slammed, because taking an order is fast. If the rush is bad, you add a cook, not a counter clerk. And the order slips on the rail survive a cook going on break.

“Map it,” Priya said.

Arjun pointed. “The counter is the web server. It only takes the question and gives back a token. The rail is the queue. The cooks are… workers? Separate programs that take questions off the rail and do the RAG work. And calling your number is sending the reply.”

“And the token itself?”

“A job ID. So the customer, or our code, can ask ‘is number 47 ready?’”

That is the asynchronous design, and every word of it has a technical name:

• A job is one unit of work: “answer this question for this customer”.
• A job ID is the token: a unique identifier you hand back immediately so the work can be tracked.
• A message queue (or job queue) is the rail: a shared, ordered list of jobs waiting to be done, stored somewhere that outlives any single process.
• A worker is a cook: a separate program that repeatedly takes the next job from the queue, does it, and records the result.
• A result store is the pickup counter: somewhere the finished answer waits until it is collected or sent.

Where the analogy breaks: in a restaurant the customer is standing right there to hear their number called. On the web, the server cannot shout. Either the customer’s app has to keep asking “is it ready yet?” (called polling), or your system has to push the answer to them, as the worker can do with a WhatsApp reply. You will build both in this module.

Following one WhatsApp message through both designs

Priya had Arjun trace a single message, from Mrs Kulkarni in Baner, arriving at 7:01:30 p.m. during the broadcast rush: “Is the Diwali Delight hamper eggless?”

Synchronous design (Saturday’s version, threaded):

Time	What happens
0.0 s	Webhook arrives. A thread is free, so work starts immediately
0.6 s	Scope check (a Claude call) says: in scope
0.7 s	Hybrid search and reranking finish (about 100 ms)
0.7 s	The answer call to Claude is sent, but 70 calls are in flight and the provider is near its limit
4.9 s	Provider has not been acknowledged yet and sends the webhook again; a second thread starts the same question
6.8 s	First answer arrives; reply sent; webhook acknowledged
9.5 s	Second answer arrives; Mrs Kulkarni gets the same reply twice

Asynchronous design:

Time	Where	What happens
0.00 s	Web server	Webhook arrives. The server checks it has not seen this message ID before
0.02 s	Web server	Creates job a3f9c1... with the question and sender, puts it on the queue
0.03 s	Web server	Acknowledges the webhook: “got it”. The provider is satisfied; no retry
0.03 to 4.10 s	Queue	The job waits behind 12 others while 4 workers chew through the rush
4.10 s	Worker 3	Takes the job off the queue (dequeues it) and marks it “started”
4.70 s	Worker 3	Scope check done
4.80 s	Worker 3	Hybrid search and reranking done
7.10 s	Worker 3	Claude’s answer arrives. Only 4 calls are in flight across all workers, well under the limit
7.30 s	Worker 3	Sends the WhatsApp reply, stores the result, marks the job “finished”

Look at what changed. Mrs Kulkarni waited about 7 seconds instead of 7 to 10, during the worst minute of the evening. She got one answer, not two. The web server spent 30 milliseconds on her message, so it could acknowledge hundreds of others in the same second. And at no point were more than 4 Claude calls running, because there were only 4 workers: the number of workers is a built-in cap on how hard you hit the LLM provider.

Finally, if the server had restarted at 2 seconds, nothing would have been lost. The job was already written to the queue, which lives outside the web server.

sequenceDiagram
  participant C as Customer (WhatsApp)
  participant API as Web server
  participant Q as Queue
  participant W as Worker
  C->>API: Question (webhook)
  API->>Q: Enqueue job a3f9c1
  API-->>C: 200 OK, received (30 ms)
  W->>Q: Take next job
  W->>W: Scope check, retrieve, rerank, generate
  W-->>C: WhatsApp reply (about 7 s)

Opening the hood: the arithmetic of a rush

How many workers does Crumb need? There is a beautifully simple rule from queueing theory, called Little’s law, which says that on average:

number of jobs being worked on at once = arrival rate × time per job

Saturday’s numbers: 600 messages in 5 minutes is 2 per second. Each job takes about 3.1 seconds of worker time (the median from the Module 09 project). So on average 2 × 3.1 = 6.2 jobs are in progress at any moment. To keep up, you need at least 7 workers, and more for headroom, because arrivals come in clumps, not neatly spaced.

What happens with fewer? With 4 workers, the system finishes 4 ÷ 3.1 ≈ 1.3 jobs per second, while 2 per second arrive. The queue grows by about 0.7 jobs every second. After the 5-minute burst, about 210 jobs are waiting, and the last one waits roughly 210 ÷ 1.3 ≈ 160 seconds, close to three minutes. That is slow, but it is orderly: every question is answered, once, in arrival order, with the LLM provider never overloaded. In the synchronous version, the same overload produced duplicates, errors and lost messages.

That is the deepest difference between the two designs. A sync system under overload fails chaotically. An async system under overload gets slower, predictably, and you can see exactly how far behind it is by counting the jobs in the queue.

What asynchronous costs you

Priya was careful not to oversell it.

• More moving parts. You now run a queue server, a web server and a pool of workers, and each can fail independently. Monitoring three things is harder than monitoring one.
• More code paths. Jobs can be queued, started, finished, failed, or lost when a worker crashes mid-job. Your code and your user interface need an answer for each.
• Harder user experience on the web. A synchronous chat can stream the answer word by word as it is generated (Module 19). With a queue, the browser gets a job ID and must poll or listen for the result, which takes more work to make feel instant.
• Results need a home and an expiry. Finished answers sit in the result store until collected. How long do you keep them? Who may read them?

And so, honestly, sync is often the right choice. An internal tool for six store managers, an answer that takes under a second, a prototype, a demo: a synchronous endpoint (written correctly, with plain def or truly non-blocking calls) is simpler and perfectly adequate. Reach for a queue when at least one of these is true: work takes more than a couple of seconds, traffic comes in bursts, a caller expects a fast acknowledgement (webhooks), you must not lose requests, or you need to control the rate of calls to an expensive downstream service.

For Crumb’s WhatsApp line, all five were true.

Try it yourself

Recreate the experiment. Save the two endpoints above in app.py, run uv run fastapi dev app.py, and fire concurrent requests with this script:

# uv add fastapi httpx
import asyncio, sys, time
import httpx

async def main(path: str, n: int):
    async with httpx.AsyncClient(timeout=120) as client:
        start = time.perf_counter()
        async def one():
            await client.get(f"http://127.0.0.1:8000{path}")
            return time.perf_counter() - start
        times = await asyncio.gather(*[one() for _ in range(n)])
    print(f"{path}: first {min(times):.1f}s, last {max(times):.1f}s")

asyncio.run(main(sys.argv[1], int(sys.argv[2])))

Run it with /blocking 10, /threaded 10 and /threaded 100, then predict, before running, what /threaded 200 will show.

Show the answer

You should see roughly: /blocking with 10 requests, first 3 s and last 30 s (one at a time); /threaded with 10, all done at about 3 s; /threaded with 100, last at about 9 s (40 threads, three waves). With 200, expect the last at about 15 s: five waves of 40. The thread pool raises throughput, but every one of those 200 requests is still holding a connection open for up to 15 seconds, and in real life each would be making a paid LLM call at the same time. That is the gap the queue fills. (The fastapi dev server runs a single process; production deployments often run several, which multiplies the numbers.)

Common mistakes

• Calling blocking code inside async def. It freezes the whole process. Use plain def, an async client, or move the work to a queue.
• Treating async/await as a scaling strategy. It helps one process wait efficiently. It does not survive restarts or cap your LLM calls.
• Acknowledging webhooks only after the work is done. Providers retry slow webhooks. Acknowledge fast, process later.
• Measuring only latency. Throughput under a burst is what breaks. Test with many simultaneous requests, not one.
• Adding a queue to everything. For fast, low-traffic, internal endpoints, sync is simpler and fine.

Recap

Meera asked Arjun to explain the plan to the two pilot store managers on a call. He used the restaurant.

• Our assistant used to work like a café where the cashier also cooks: fine when it’s quiet, chaos in a rush. Orders got duplicated, some got lost.
• We’re switching to a token system like Vaishali: the server takes the question in a fraction of a second, gives it a number, and puts it on a rail.
• Separate “cooks”, called workers, take questions off the rail and do the slow AI work, then send the reply.
• In a rush, customers wait a bit longer, but everyone gets exactly one answer, in order, and nothing is lost if we restart.
• The number of cooks is our dial: more cooks, faster service, up to what our AI provider allows.

The Baner manager asked the only question that mattered to her. “Will it be ready for the next broadcast?”

“The rail needs to live somewhere that never forgets,” Priya said. “That’s tomorrow: a small, very fast server called Redis, running in Docker, and a Python library called RQ that turns it into a job queue.”

Go deeper

• FastAPI documentation (concurrency and async): https://fastapi.tiangolo.com
• HTTPX documentation (async client): https://www.python-httpx.org