Async Python: doing many things while you wait

Monday, 9:20 a.m. Crumb’s support team has arrived, coffee in hand, ready to work through the weekend’s emails. The inbox bot was supposed to have sorted them by now: late deliveries in one pile, refunds in another, allergy questions flagged red at the top.

It’s still running. The progress bar says 287 of 400.

Arjun does the maths. Each email goes to Claude for classification, and each call takes about three seconds. 400 emails × 3 seconds = 1,200 seconds: twenty minutes. Fine on a quiet Monday. But the Diwali weekend brought 2,300 emails, which would take nearly two hours, and the allergy question that arrives at email 2,250 waits all that time.

Meera asks the obvious question. “Can’t you make Claude faster?”

“Claude’s speed isn’t the problem,” Priya says, pointing at the laptop. “Look at what your computer is doing during those three seconds.”

Arjun opens the activity monitor. The processor is at 1%. The script isn’t working hard. It’s waiting.

Arjun’s first attempt: sprinkling async on it

Arjun has heard that “async makes Python fast”. He finds an example online, adds the new keywords to his script, and runs it. To keep things quick to test, he uses a fake classifier that waits a set number of seconds instead of calling a real model:

import asyncio
import time


async def classify(email_id, seconds):
    await asyncio.sleep(seconds)          # pretend to wait for the model
    return f"email {email_id}: done"


async def main():
    start = time.perf_counter()
    for email_id, seconds in [(1, 3), (2, 2), (3, 1)]:
        result = await classify(email_id, seconds)
        print(result, f"at {time.perf_counter() - start:.1f}s")
    print(f"Total: {time.perf_counter() - start:.1f}s")


asyncio.run(main())

email 1: done at 3.0s
email 2: done at 5.0s
email 3: done at 6.0s
Total: 6.0s

Six seconds: exactly what three calls of 3, 2 and 1 seconds take one after another. He’s added async and await everywhere, and nothing is faster.

“Async doesn’t make anything faster,” Priya says. “It lets you stop standing still while you wait. You’ve written the code so that it still stands still.”

One baker, ten ovens

Priya takes Arjun down to the Baner kitchen at 5 a.m., when the morning bake starts. One baker, Sunil, is working ten ovens.

“How long does a tray of croissants take?” Priya asks.

“Eighteen minutes,” Sunil says, sliding a tray into oven 1 and turning to oven 2.

“And does he stand in front of oven 1 for eighteen minutes?” Priya asks Arjun.

“No. He loads all ten, then deals with whichever timer goes off first.”

“How long until all ten trays are done?”

“About eighteen minutes, plus a bit for loading. Not three hours.”

“Is Sunil working faster than a baker who waits by each oven?”

Arjun thinks about it. “No. He’s doing the same work. He’s not wasting the waiting.”

That’s async in one picture. Loading a tray takes a few seconds of real work. Baking takes eighteen minutes of waiting, during which the baker is free. An LLM call is the same: your program spends a millisecond sending the request, then about three seconds waiting while Claude’s servers do the work. If your program waits like the first baker, 400 calls take 400 waits. If it waits like Sunil, 400 calls overlap.

The important limit of the analogy: Sunil only has two hands. If the work itself were slow, kneading dough, say, then loading more ovens wouldn’t help. Async helps with waiting, not with heavy work. We’ll come back to that.

Following three emails through two kitchens

Before writing the fix, trace three emails both ways. Email 1 takes 3 seconds to classify, email 2 takes 2, email 3 takes 1.

One after another (Arjun’s version):

All at once (Sunil’s version):

The concurrent total is the longest single wait, not the sum. With 3 emails, that’s 6 s down to 3 s. With 400 emails of about 3 seconds each, it’s 1,200 s down to roughly 3 s, if nothing else limited you. (Something will. We’ll get there.)

Here is Sunil’s version in code:

import asyncio
import time


async def classify(email_id, seconds):
    await asyncio.sleep(seconds)
    return f"email {email_id}: done"


async def main():
    start = time.perf_counter()
    results = await asyncio.gather(
        classify(1, 3),
        classify(2, 2),
        classify(3, 1),
    )
    print(results)
    print(f"Total: {time.perf_counter() - start:.1f}s")


asyncio.run(main())

['email 1: done', 'email 2: done', 'email 3: done']
Total: 3.0s

Three seconds. One line changed: instead of awaiting each call in turn, asyncio.gather starts all three and waits for them together.

Notice the order of results. Email 3 finished first, but the results come back in the order you passed them in. That’s deliberate and useful: result number i always belongs to input number i, so you can match each classification to its email without any bookkeeping.

Opening the hood: coroutines and the event loop

Now the precise words, one at a time.

async def creates a coroutine function. Calling it does not run it. It gives you back a coroutine: a paused piece of work, like a tray that’s been prepared but not put in an oven.

import asyncio


async def classify(email_id):
    await asyncio.sleep(1)
    return f"email {email_id}: done"


work = classify(1)          # nothing runs yet
print(type(work))
print(asyncio.run(work))    # now it runs

<class 'coroutine'>
email 1: done

The event loop runs coroutines. The event loop is the head baker with a whiteboard. It keeps a list of every piece of work that’s ready to continue, runs one until that work has to wait, notes what it’s waiting for, and switches to another. When a reply arrives, the work waiting for it goes back on the “ready” list. asyncio.run(...) starts an event loop, runs the coroutine you give it until it finishes, and shuts the loop down. You call it once, at the top of your program.

await means “I’m waiting; serve someone else”. Every await is a point where a coroutine can pause and hand control back to the event loop. That’s the whole trick. In Arjun’s first version, the loop said “await email 1” and there was nobody else to serve, because email 2 hadn’t been started yet.

asyncio.gather starts several coroutines and waits for all of them. It wraps each one in a task (a coroutine the event loop is actively running), so they’re all in the ovens at once.

sequenceDiagram
    participant L as Event loop
    participant A as Email 1 (3 s)
    participant B as Email 2 (2 s)
    participant M as Model API
    L->>A: start
    A->>M: send request, await
    L->>B: start (A is waiting)
    B->>M: send request, await
    M-->>B: reply after 2 s
    L->>B: resume, finish
    M-->>A: reply after 3 s
    L->>A: resume, finish

One last precise point, because it causes real confusion: async is concurrency, not parallelism. Concurrency means many tasks are in progress at once, taking turns. Parallelism means many tasks are running at the same instant on separate processor cores. Async Python runs your code on one core, one piece at a time, switching only at await. That’s perfect for waiting on networks and terrible for heavy calculation, as you’ll see in “Where it breaks”.

Real HTTP calls, concurrently

Fake sleeps prove the idea. Now real network calls. httpx is an HTTP library (you’ll learn it properly in the next lesson) with an async client. Here we call a public testing service, httpbin.org, whose /delay/1 address deliberately waits one second before replying:

import asyncio
import time

import httpx


async def fetch(client, n):
    response = await client.get("https://httpbin.org/delay/1", timeout=10)
    return n, response.status_code


async def main():
    async with httpx.AsyncClient() as client:
        start = time.perf_counter()
        for n in range(3):                               # one at a time
            await fetch(client, n)
        print(f"One at a time: {time.perf_counter() - start:.1f}s")

        start = time.perf_counter()
        results = await asyncio.gather(*(fetch(client, n) for n in range(3)))
        print(f"All at once:   {time.perf_counter() - start:.1f}s", results)


asyncio.run(main())

One at a time: 4.1s
All at once:   1.4s [(0, 200), (1, 200), (2, 200)]

Your exact times will vary with your connection and the service’s mood, but the shape won’t: one-at-a-time grows with the number of calls, all-at-once stays close to one call. Two new pieces of syntax:

• async with is the async version of with: it opens the client, and guarantees it’s closed afterwards, even on errors. Reusing one client for many calls is also faster, because it reuses network connections.
• asyncio.gather(*(...)): the * unpacks a sequence into separate arguments, so gather(*[a, b, c]) is the same as gather(a, b, c). It’s how you gather a list of any length.

The same pattern with Claude

The Anthropic SDK has an async client, AsyncAnthropic, with the same methods as the regular one. Each call is awaited:

import asyncio

import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.AsyncAnthropic()      # reads ANTHROPIC_API_KEY

CATEGORIES = "late_delivery, refund, allergy, other"


async def classify(email: str) -> str:
    response = await client.messages.create(
        model="claude-opus-5-5",
        max_tokens=20,
        system=f"Classify the customer email. Reply with exactly one of: {CATEGORIES}.",
        messages=[{"role": "user", "content": email}],
    )
    return next(b.text for b in response.content if b.type == "text").strip()


async def main():
    emails = [
        "My order CR-2210 is 40 minutes late!",
        "Does the plum cake contain walnuts? My son is allergic.",
        "I was charged twice for one hamper.",
    ]
    labels = await asyncio.gather(*(classify(e) for e in emails))
    for email, label in zip(emails, labels):
        print(f"{label:14} <- {email}")


asyncio.run(main())

A typical run prints:

late_delivery  <- My order CR-2210 is 40 minutes late!
allergy        <- Does the plum cake contain walnuts? My son is allergic.
refund         <- I was charged twice for one hamper.

zip(emails, labels) pairs each email with its label, which works because gather returns results in input order. OpenAI’s SDK has the same idea with AsyncOpenAI.

Too many ovens: limiting concurrency

Arjun is thrilled. He points the gather at all 400 emails at once. Within a second, most calls fail with an error whose status code is 429 Too Many Requests.

Every AI provider sets rate limits: caps on how many requests (and how many tokens, the chunks of text models read and write) your account can send per minute. Fire 400 requests in the same instant and you blow straight through them. The provider rejects the excess, and a naive retry makes it worse.

The kitchen has the same rule: Sunil has ten ovens, not four hundred. You need a way to say “at most N at a time”. That’s a semaphore: a counter of free slots. A task takes a slot before its call and gives it back after. When all slots are taken, the next task waits its turn.

import asyncio
import time

limit = asyncio.Semaphore(3)        # at most 3 calls in flight


async def classify(email_id):
    async with limit:               # wait for a free slot, release it when done
        await asyncio.sleep(1)      # pretend each call takes 1 second
        return email_id


async def main():
    start = time.perf_counter()
    results = await asyncio.gather(*(classify(i) for i in range(9)))
    print(results)
    print(f"9 calls, 3 at a time: {time.perf_counter() - start:.1f}s")


asyncio.run(main())

[0, 1, 2, 3, 4, 5, 6, 7, 8]
9 calls, 3 at a time: 3.0s

Nine one-second calls, three at a time: three rounds, three seconds. Now Monday’s numbers: 400 emails of about 3 seconds each, 10 at a time, is 40 rounds × 3 s = about 2 minutes, down from 20. Choose the limit from your provider’s published rate limits, and leave headroom for anything else using the same key.

When some calls fail

In a batch of 400, a few calls will fail: a timeout, a dropped connection, a 500 error on the provider’s side. By default, if one task in gather raises an exception, gather raises it straight away, and you lose easy access to the results that succeeded. For batch work, ask gather to hand back exceptions as results instead:

import asyncio


async def classify(email_id):
    await asyncio.sleep(0.1)
    if email_id == 2:
        raise TimeoutError("model took too long")
    return f"email {email_id}: refund"


async def main():
    results = await asyncio.gather(*(classify(i) for i in range(4)), return_exceptions=True)
    for i, result in enumerate(results):
        if isinstance(result, Exception):
            print(f"email {i}: FAILED ({result}) -> retry later")
        else:
            print(result)


asyncio.run(main())

email 0: refund
email 1: refund
email 2: FAILED (model took too long) -> retry later
email 3: refund

isinstance(result, Exception) asks “is this an exception object?”. Successes and failures come back in their places, and you can retry only the failures.

To stop a slow call from setting the pace for the whole batch, put a deadline on it with asyncio.timeout (Python 3.11 and later):

import asyncio


async def slow_model():
    await asyncio.sleep(5)
    return "finally!"


async def main():
    try:
        async with asyncio.timeout(1):         # give up after 1 second
            print(await slow_model())
    except TimeoutError:
        print("Gave up after 1 second; flag for retry")


asyncio.run(main())

Gave up after 1 second; flag for retry

In Module 19 you’ll combine these pieces, timeouts, retries with growing waits, and fallbacks to a second model, into a proper reliability layer.

Where async matters in AI apps

You’ll meet async throughout this course, wherever a program waits on several things:

• Batch jobs. Classifying an inbox, summarising 1,000 product reviews, or running an evaluation suite of 500 test questions (Module 23). Concurrency turns hours into minutes.
• RAG. Retrieval-augmented generation (looking up relevant documents and giving them to the model with the question, Module 09) often searches several places at once: a vector database, a keyword index, the order database. Three 300 ms lookups take 300 ms together, not 900 ms.
• Agents. When an agent decides it needs the weather, the order status and the delivery partner’s ETA, and they don’t depend on each other, it can fetch all three at once (Module 13).
• Web servers. A chatbot backend serving 200 customers spends almost all its time waiting for model replies. Async web frameworks like FastAPI (Module 05) let one server process handle many conversations at once instead of one per process.
• Streaming. Showing a reply word by word as it’s generated (Module 19) is naturally async: wait for the next piece, show it, wait again.

And where it doesn’t help: a script that makes one model call and exits gains nothing. A data-crunching step that spends its time calculating, not waiting, gains nothing. Async adds complexity, so use it where there’s waiting to overlap.

Where it breaks

Two weeks later, a teammate adds a feature to the async inbox bot, and the Monday batch is slow again. Two minutes has become twenty. Here’s the change, simplified:

import asyncio
import time


async def classify(email_id):
    time.sleep(1)                   # a BLOCKING sleep, not asyncio.sleep
    return email_id


async def main():
    start = time.perf_counter()
    await asyncio.gather(*(classify(i) for i in range(3)))
    print(f"Total: {time.perf_counter() - start:.1f}s")


asyncio.run(main())

Total: 3.0s

time.sleep doesn’t await. It holds the one and only thread hostage, so the event loop can’t switch to anything else. The same happens with any blocking call inside async code: requests.get(...), the regular (non-async) anthropic.Anthropic() client, reading a huge file, or a heavy calculation. In the real case, the teammate had used the synchronous Anthropic client inside an async def. The code looked async. It ran one call at a time.

The other classic failures:

• Forgetting await. result = classify(1) inside an async function gives you a coroutine object, not a result, and Python later prints RuntimeWarning: coroutine 'classify' was never awaited. If you ever see that warning, look for the missing await.
• Heavy calculation in async code. Async takes turns on one core. A 10-second calculation inside a coroutine freezes every other task for 10 seconds. Heavy number crunching belongs in separate processes, or in libraries like NumPy that do the work outside Python (next-but-one lesson).
• asyncio.run inside Jupyter. Notebooks already run an event loop, so asyncio.run(main()) fails with RuntimeError: asyncio.run() cannot be called from a running event loop. In a notebook cell, write await main() directly instead.
• Unlimited concurrency. Covered above, but worth repeating: no semaphore means rate-limit errors and surprising bills.

Arjun explains it to Meera

The next Monday, the inbox bot finishes 412 emails at 8:52 a.m., with 10 calls in flight at a time. Two minutes and ten seconds, start to finish. Three allergy questions sit flagged red at the top of the list when the team walks in.

“Did Claude get faster?” Meera asks.

“No. Each email still takes about three seconds,” Arjun says. “But most of those three seconds, our program sat idle, waiting for Claude to reply. Before, it waited for each reply before sending the next email, like a baker standing at one oven until the croissants are done. Now it sends ten, and handles each reply as it comes back. Same work, without the standing around. I keep it at ten at a time, because Claude’s service only accepts so many requests a minute, and so a bug can’t spend a month’s budget in a second.”

“Two hours at Diwali becomes…?”

“About twelve minutes for 2,300 emails, at ten at a time.”

Try it yourself

Simulate a RAG lookup. Write three async functions that pretend to search for context about a customer’s question:

• search_handbook() waits 0.8 s and returns "handbook: plum cake contains walnuts"
• search_orders() waits 0.5 s and returns "orders: CR-2210 contains 1 plum cake"
• search_menu() waits 0.3 s and returns "menu: plum cake, ₹450"

Then write main() that runs all three concurrently with gather, prints each result on its own line, and prints the total time to one decimal place. Expected:

handbook: plum cake contains walnuts
orders: CR-2210 contains 1 plum cake
menu: plum cake, ₹450
Total: 0.8s

Bonus: what would the total be if you awaited them one by one?

import asyncio
import time


async def search_handbook():
    await asyncio.sleep(0.8)
    return "handbook: plum cake contains walnuts"


async def search_orders():
    await asyncio.sleep(0.5)
    return "orders: CR-2210 contains 1 plum cake"


async def search_menu():
    await asyncio.sleep(0.3)
    return "menu: plum cake, ₹450"


async def main():
    start = time.perf_counter()
    results = await asyncio.gather(search_handbook(), search_orders(), search_menu())
    for result in results:
        print(result)
    print(f"Total: {time.perf_counter() - start:.1f}s")


asyncio.run(main())

handbook: plum cake contains walnuts
orders: CR-2210 contains 1 plum cake
menu: plum cake, ₹450
Total: 0.8s

Common mistakes

• Expecting async alone to speed things up. Awaiting calls one by one in a loop is still one at a time. Use gather (with a semaphore) to overlap them.
• Blocking calls inside async def. time.sleep, requests, and synchronous SDK clients freeze the event loop. Use their async versions.
• Calling asyncio.run more than once, or inside async code. Call it once, at the top level, to start your main coroutine.
• Assuming results come back in completion order. gather returns them in the order you passed them in, which is what you usually want.
• No timeouts. One stuck call holds up the whole batch. Wrap calls in asyncio.timeout, and set timeouts on HTTP clients.

Recap

• AI apps spend most of their time waiting on networks; async lets one program overlap those waits instead of queuing them.
• async def makes a coroutine (paused work); await is where it can pause and let others run; asyncio.run starts the event loop that switches between them.
• asyncio.gather runs many coroutines concurrently and returns results in input order; the total is about the slowest call, not the sum.
• Limit concurrency with asyncio.Semaphore to respect rate limits and budgets; use return_exceptions=True and timeouts so a few failures don’t sink a batch.
• Async is concurrency on one core, not parallelism: it helps with waiting, not with heavy calculation, and one blocking call inside async code stalls everything.

Arjun has been using httpx and API clients for a week without really knowing what goes over the wire. Then the delivery partner gives Crumb access to their rider-tracking API, with a key, a page of documentation and a list of status codes. Arjun’s first script hangs for eleven minutes during a Diwali-night outage. “Every AI API is an HTTP API underneath,” Priya says. “Time to see what’s in the envelope.”

Go deeper

• asyncio documentation, the standard library reference, with a guide to tasks and coroutines.
• HTTPX: its async client, used above.
• Anthropic documentation: rate limits and the async client.