Batching requests for efficiency

The week after Diwali, Meera walked over with her laptop and a spreadsheet. “I want to know what people actually said about us this season. There are 3,014 Google reviews and WhatsApp feedback messages in here. Can the AI sort them into positive, negative and mixed, and pull out the top complaints? I’m presenting to the store managers on Tuesday.”

It was Friday afternoon. “Easy,” said Arjun. He wrote a for loop: for each review, call Claude, save the label. He pressed run and watched the first results print. positive. negative. positive. mixed. About one every four seconds.

Then he did the arithmetic. 3,014 reviews at four seconds each is 12,056 seconds. Three hours and twenty minutes, with his laptop doing nothing but waiting. And if anything crashed at review 2,900, he would start again.

Priya was reading over his shoulder. “Your loop is a bakery with one cashier who takes an order, walks to the oven, watches the cake bake for forty minutes, carries it back, and only then takes the next order.”

Where the four seconds go

“So the model is slow,” Arjun said. “Can we use a faster one?”

“Maybe later. First, tell me what your laptop is doing during those four seconds.”

Arjun opened the activity monitor. His Python process sat at 0.3% CPU.

“Nothing,” he said. “It’s waiting.”

“Right. The request travels to a data centre. The model generates a few dozen tokens on a GPU, a chip built for exactly that, shared with thousands of other customers. The answer travels back. Your machine’s part of the job takes about a millisecond. The other 3,999 milliseconds are waiting.”

Engineers call this kind of work I/O-bound: limited by waiting for input and output (network, disk), not by your own computer’s speed. The opposite is CPU-bound work, like resizing ten thousand photos, where your processor really is the bottleneck. The distinction matters because the fix is completely different. You cannot make the model’s data centre faster. But you can stop waiting for one answer before asking the next question.

“Back to the bakery,” Priya said. “How does a real bakery handle a queue at Diwali?”

“The cashier takes the next order while the first cake bakes. There are several ovens running at once.”

“And is there a limit?”

“The number of ovens. If you put thirty orders in and there are eight ovens, twenty-two wait.”

“That’s the whole design. Many orders in progress at once, and a fixed number of ovens. Let’s build it.”

Following six reviews through two designs

Before writing real code, Priya made Arjun trace a tiny example on the whiteboard: six reviews, each taking exactly 4 seconds.

Design A, one at a time:

Time	What is happening
0 to 4 s	Review 1 waiting
4 to 8 s	Review 2 waiting
…	…
20 to 24 s	Review 6 waiting

Total: 24 seconds, and at every moment exactly one request is in flight.

Design B, up to three at a time:

Time	In flight	Finished
0 s	Reviews 1, 2, 3 start	none
4 s	1, 2, 3 finish; 4, 5, 6 start	3
8 s	4, 5, 6 finish	6

Total: 8 seconds. Same six requests, same model, same cost. Only the waiting overlapped.

flowchart LR
    subgraph Sequential
    A1["Review 1: 4s"] --> A2["Review 2: 4s"] --> A3["Review 3: 4s"]
    end
    subgraph Concurrent
    B1["Review 1: 4s"]
    B2["Review 2: 4s"]
    B3["Review 3: 4s"]
    end

Scale it up. With 8 at a time, 3,014 reviews take about 3,014 × 4 ÷ 8 ≈ 1,507 seconds, roughly 25 minutes. With 20 at a time, about 10 minutes. The bill is identical in all three cases, because you pay per token, not per second.

What is that bill? Each review averages about 60 tokens. With a one-line system prompt and message overhead, each request is roughly 110 input tokens. The reply is one word, but on Claude Opus 5.5 the model may also think briefly, so call it about 40 output tokens. At $4permillioninputtokensand$20 per million output tokens:

• One review: 110 × $0.000004+40\times$0.00002 = $0.00044+$0.0008 = about $0.0012.
• All 3,014: about $3.74, or roughly ₹330 at ₹88 to the dollar.

Hold on to that ₹330. It will drop by half before the end of the lesson.

Show the answer

First, not every call takes exactly 4 seconds. Some take 9, and the whole job cannot finish before its slowest stragglers do (engineers call this tail latency). Second, your account has rate limits, caps on requests and tokens per minute. If 8 concurrent calls push you over them, some calls get a 429 response, wait, and retry, which stretches the run. A third, sneaky one: if a single call hangs and you set no timeout, it can hold one of your 8 slots for minutes.

Opening the hood: what asyncio actually does

asyncio is Python’s built-in toolkit for concurrency. You met the basics in the Python refresher’s async lesson; here is the mental model that matters for AI calls.

Picture one cook in a kitchen with eight pots. The cook puts rice in pot one, covers it, and instead of staring at it, moves to pot two. When pot one bubbles, the cook comes back. One cook, many dishes in progress. That cook is the event loop. Each pot is a coroutine (a function defined with async def, which can pause partway through). The word await is the cook putting a lid on: “this will take a while; go do something else and come back when it is ready”.

This clears up the most common misconception about asyncio: it does not use more CPU cores, and it does not run your Python code in parallel. It is one thread, cleverly switching between tasks whenever one of them is waiting. That is exactly why it is perfect for AI calls, which are almost all waiting, and useless for CPU-bound work like image resizing.

It also explains the second misconception, which caught Arjun ten minutes later. Concurrency only happens at an await. If code inside async def calls something slow without await, the cook stands staring at that one pot, and every other pot waits.

Arjun’s second attempt, and two ways it failed

Arjun, now excited, wrote this:

import asyncio
import anthropic

client = anthropic.Anthropic()   # the normal (synchronous) client

async def tag_review(review: str) -> str:
    response = client.messages.create(...)   # no await: this BLOCKS the event loop
    return next(b.text for b in response.content if b.type == "text")

async def main(reviews):
    return await asyncio.gather(*(tag_review(r) for r in reviews))

He ran it on 30 reviews. It took 2 minutes, exactly as long as the plain loop. The synchronous client does not know about the event loop; while it waits for the network, it holds the whole kitchen hostage. Nothing overlapped.

He fixed that by switching to the async client and adding await. Then he ran all 3,014 reviews. Within two seconds his terminal filled with 429 Too Many Requests. asyncio.gather had started every one of the 3,014 coroutines immediately, and all of them hit the API in the same instant. Most were rejected, the SDK started retrying them all, the retries collided again, and after six minutes he killed it with fewer than 400 results.

“Thirty orders, eight ovens,” Priya said. “You forgot the ovens.”

The real code, built in three steps

Step 1: the async client and one coroutine.

# tag_reviews.py
import asyncio
import time
import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.AsyncAnthropic(timeout=30.0)   # the async twin of anthropic.Anthropic()

SYSTEM = "Label the bakery review as positive, negative or mixed. Reply with exactly one word."

async def tag_review(review: str) -> str:
    response = await client.messages.create(          # await: other reviews run while this waits
        model="claude-opus-5-5",
        max_tokens=1000,
        system=SYSTEM,
        messages=[{"role": "user", "content": review}],
        output_config={"effort": "low"},              # a simple task: ask the model not to overthink
    )
    return next(b.text for b in response.content if b.type == "text").strip().lower()

AsyncAnthropic has the same methods as the client you know, but each one must be awaited. output_config={"effort": "low"} tells Claude Opus 5.5 this is an easy task, which keeps the thinking (and the output-token bill) small. The timeout stops one hung call from holding a slot forever.

Step 2: the ovens. A semaphore is a counter that lets at most N tasks into a section of code at once. The others wait at the door until someone leaves.

limit = asyncio.Semaphore(8)   # at most 8 requests in flight at once

async def tag_review_limited(review: str) -> str:
    async with limit:          # wait here if 8 calls are already running
        return await tag_review(review)

Step 3: start them all, collect them all.

async def main(reviews: list[str]) -> list:
    start = time.perf_counter()
    results = await asyncio.gather(
        *(tag_review_limited(r) for r in reviews),
        return_exceptions=True,   # one failed review must not cancel the other 3,013
    )
    print(f"{len(reviews)} reviews in {time.perf_counter() - start:.1f}s")
    return results

if __name__ == "__main__":
    reviews = [
        "Best plum cake in Pune, ordered three times!",
        "Delivery was two hours late and the cream had melted.",
        "Lovely sourdough, but the staff at Baner were rude.",
    ] * 10   # 30 reviews for a quick test
    results = asyncio.run(main(reviews))
    for review, label in zip(reviews[:3], results[:3]):
        print(f"{label:<10} {review}")

Output from Arjun’s run:

30 reviews in 16.3s
positive   Best plum cake in Pune, ordered three times!
negative   Delivery was two hours late and the cream had melted.
mixed      Lovely sourdough, but the staff at Baner were rude.

Thirty reviews in 16 seconds instead of two minutes. A few details worth understanding:

• asyncio.gather(...) starts every coroutine and waits for all of them. It keeps the order: results[i] is the label for reviews[i], however the calls finish.
• return_exceptions=True means a failed call returns its exception object in that slot instead of crashing the whole run. Afterwards you can find and retry just the failures:

failed = [r for r, res in zip(reviews, results) if isinstance(res, Exception)]
print(len(failed), "to retry")

• asyncio.run(main(...)) starts the event loop. In a Jupyter notebook, which already has a loop running, write await main(reviews) instead.

The full run of 3,014 reviews took 27 minutes at a semaphore of 8, with 6 failures, all timeouts, which a second pass fixed. Meera had her labels before dinner.

How big should the semaphore be? Start small, 5 to 10, and raise it while watching for 429 errors. The right number is the highest one that stays under your account’s rate limits, and the next lesson shows how to back off gracefully when you cross them.

The wholesale order: provider Batch APIs

On Saturday Meera asked for more: “Can it also run every review from the whole year? There are about 40,000. No rush, next week is fine.”

“No rush” changes everything. Providers offer a Batch API for exactly this: you submit thousands of requests in one go, the provider runs them whenever it has spare capacity, and you collect the results later. In exchange for your patience, you get a big discount: about 50% off both input and output tokens on Anthropic and OpenAI at the time of writing.

The bakery analogy is the wholesale order. A café that orders 200 croissants for tomorrow morning pays less than 200 walk-in customers, because the bakery can bake them at 4 a.m. when the ovens are free.

sequenceDiagram
    participant A as Arjun's script
    participant B as Batch API
    A->>B: Create batch (thousands of requests, each with a custom_id)
    B-->>A: batch id, status in_progress
    loop Every few minutes
        A->>B: Retrieve status
        B-->>A: in_progress or ended
    end
    A->>B: Fetch results
    B-->>A: One result per custom_id, in any order

Here is the code, using three reviews so you can try it cheaply:

# batch_reviews.py
import time
import anthropic
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic()

reviews = {
    "rev-0001": "Best plum cake in Pune, ordered three times!",
    "rev-0002": "Delivery was two hours late and the cream had melted.",
    "rev-0003": "Lovely sourdough, but the staff at Baner were rude.",
}

# 1. Submit. Each request gets a custom_id so you can match its result later.
batch = client.messages.batches.create(
    requests=[
        Request(
            custom_id=review_id,
            params=MessageCreateParamsNonStreaming(
                model="claude-opus-5-5",
                max_tokens=1000,
                system="Label the bakery review as positive, negative or mixed. Reply with exactly one word.",
                messages=[{"role": "user", "content": text}],
            ),
        )
        for review_id, text in reviews.items()
    ]
)
print("Submitted batch", batch.id)

# 2. Poll. Most batches finish within an hour; the hard limit is 24 hours.
while client.messages.batches.retrieve(batch.id).processing_status != "ended":
    time.sleep(60)

# 3. Collect. Results arrive in ANY order, so always match on custom_id, never on position.
labels = {}
for item in client.messages.batches.results(batch.id):
    if item.result.type == "succeeded":
        message = item.result.message
        labels[item.custom_id] = next(b.text for b in message.content if b.type == "text").strip()
    else:
        print(item.custom_id, "did not succeed:", item.result.type)   # errored, canceled or expired

print(labels)

Expected output, after a few minutes of polling:

Submitted batch msgbatch_01...
{'rev-0002': 'negative', 'rev-0001': 'positive', 'rev-0003': 'mixed'}

Notice the order of the dictionary: rev-0002 came back first. That is normal. Unlike asyncio.gather, a batch gives no ordering promise. The custom_id is the only thread connecting a result to its input, so make it something meaningful, like your database id for that review.

For the 40,000 reviews, the numbers looked like this: at about $0.0012perreviewatfullprice,$49 (₹4,300) with asyncio; about $25 (₹2,150) with the Batch API. Same labels, half the money, and the work happened while Arjun slept.

In a real system you would not keep a script polling for hours. You would save batch.id in your database and let a scheduled job check it every few minutes.

OpenAI’s Batch API follows the same idea with a different shape. You write requests to a JSONL file (one JSON request per line, each with a custom_id), upload it with client.files.create(..., purpose="batch"), then start it with client.batches.create(...). The discount, the time window and the “match by id” rule are the same.

Show the answer

(a) Neither: one request at a time, as it arrives, with streaming; a human is waiting. (b) Batch API: no one is waiting and it halves the cost. © Batch API again: submit at 9 p.m., results by morning, half price. asyncio would also work fine at this size; the saving is small either way. (d) asyncio with a semaphore: a human is watching a spinner, so minutes, not hours. The rule of thumb: if a person is waiting, go concurrent; if not, go batch.

A third trick: several reviews in one prompt

There is one more way to cut calls: put twenty reviews into a single prompt and ask for twenty labels back. The system prompt is sent once instead of twenty times, and there are 20 times fewer requests to count against your limits.

The trade-offs are real. If the model skips review 13 or merges two, matching labels to reviews gets messy. One bad response loses twenty results. And long prompts with many items tend to get slightly less careful attention per item than short ones. If you do it, number each review in the prompt and use structured outputs from the first lesson of this module, so the reply is a validated list of {id, label} objects, then check that every id came back exactly once.

Where it breaks

A few things that bit Arjun, or nearly did:

• The semaphore was the only brake. It limits requests in flight, not requests per minute. Fast calls at 8-wide can still exceed a per-minute limit. The next lesson adds proper backoff.
• No timeout meant lost slots. In an early run, two calls hung for minutes, quietly turning his 8 ovens into 6.
• Duplicates cost money. The spreadsheet had 214 exact duplicate reviews (people pasting the same text twice). Removing them first saved 7% of the bill. Clean your inputs before you pay to process them.
• Batches can expire. A batch not finished within 24 hours returns expired for the leftovers. Check every result type, resubmit what is missing.
• Batch is the wrong tool for anything urgent. Most batches finish within an hour, but nothing guarantees it. Never put a waiting customer behind one.

Try it yourself

1. Run tag_reviews.py with limit = asyncio.Semaphore(1) and note the time.
2. Change it to Semaphore(8) and run again.
3. Count the labels with collections.Counter(results).

Expected: the second run is several times faster (on Arjun’s machine, 118 s versus 16 s). With 30 reviews built from 3 texts repeated, expect roughly 10 positive, 10 negative and 10 mixed. The third review is sometimes labelled negative; that is a judgement call, not a bug.

Stuck? Hint

If you see RuntimeError: asyncio.run() cannot be called from a running event loop, you are in a Jupyter notebook, which already runs an event loop. Replace asyncio.run(main(reviews)) with await main(reviews). If the second run is not faster, check that you used AsyncAnthropic and wrote await before client.messages.create.

Common mistakes

• Using the sync client inside async def. It blocks the event loop and nothing runs concurrently. Use AsyncAnthropic and await.
• No concurrency cap. gather over thousands of tasks with no semaphore is a self-inflicted rate-limit storm.
• Letting one failure kill the run. Without return_exceptions=True, the first exception aborts gather. Collect failures and retry them.
• Matching batch results by position. Batch results arrive in any order. Key them by custom_id.
• Batching urgent work. A batch can take hours. Live users need live calls.

Explaining it to Meera

On Tuesday morning Meera had her slides, and a question. “Friday you said it would take all weekend. What changed?”

“Before, the computer asked the AI about one review, waited four seconds doing nothing, then asked about the next,” Arjun said. “Now it keeps eight questions going at once, like having eight ovens instead of one. Same work, same cost, an eighth of the time.”

“And the 40,000 from the whole year?”

“Those I sent as a bulk order. The AI company does them overnight when their machines are quieter, and charges half. About ₹2,150 instead of ₹4,300.”

“So why don’t we use the bulk order for everything?”

“Because it can take hours. Fine for a report. Terrible for a customer waiting in the chat.”

Recap

• AI calls are I/O-bound: nearly all the time is waiting, so overlapping the waits gives big speedups at the same cost.
• asyncio overlaps waits on one thread; it only switches tasks at an await, so use AsyncAnthropic.
• A semaphore caps requests in flight; gather(..., return_exceptions=True) keeps order and survives failures.
• Batch APIs trade speed (up to 24 hours) for about half the price; match results by custom_id.
• If a human is waiting, go concurrent; if not, go batch.

Meera’s presentation went well: 71% positive, and “late delivery” was the top complaint by a mile. That evening, to show off, Arjun turned the semaphore up to 50 and reran the season’s reviews. A third of them came back with 429 errors. Priya, packing up, smiled. “Good. You’ve met the rate limiter properly. Tomorrow we learn to live with it, and with its quieter cousin, the token limit.”

Go deeper

• Python asyncio documentation: https://docs.python.org/3/library/asyncio.html
• Anthropic docs (Message Batches): https://docs.anthropic.com
• OpenAI docs (Batch API): https://platform.openai.com/docs