Idempotency keys

The customer was polite about it. Her Christmas plum cake had arrived squashed, the support bot had apologised and issued a refund of ₹850, and then, a minute later, a second refund of ₹850. “I’m honest, so I’m telling you,” she wrote. “But you might want to look at that.”

Arjun searched the payment records for the past week. It was not one customer. It was seven duplicate refunds, ₹5,950 in total (about $70), plus three customers who had received the same WhatsApp order confirmation twice and phoned the store, worried they had been charged twice.

He traced the plum cake first. The support agent from Module 13 had a tool called issue_refund that called Crumb’s payment service. On Saturday the payment service was slow. The refund went through, but the reply took 31 seconds, and the new retry code from the last lesson gave up waiting at 30. It saw a timeout, which is on the “retry” list, waited politely with jitter, and tried again. The payment service had no idea the second request was a repeat. It paid out again.

“Your retry logic is correct,” Priya said when he showed her. “The bug is a question you didn’t ask. A timeout tells you that you didn’t hear back. It doesn’t tell you nothing happened.”

Arjun’s two wrong fixes

His first idea was to turn off retries for refunds. He tried it on staging with a deliberately slow payment service. The refund went through after 31 seconds, but the agent had already given up at 30 and told the customer “Sorry, I couldn’t process your refund.” The customer asked again. The agent issued a second refund. Same duplicate, more steps, and an annoyed customer in the middle.

His second idea was to check before paying:

def issue_refund(order_id: str, amount_inr: int) -> str:
    if payments.has_refund(order_id):          # already refunded? stop
        return "This order was already refunded."
    payments.create_refund(order_id, amount_inr)
    return "Refund issued."

That looked solid until Priya drew a timeline:

14:02:10.000  Request A: has_refund("1042")?  -> no
14:02:10.004  Request B: has_refund("1042")?  -> no     (A hasn't written anything yet)
14:02:10.010  Request A: create_refund(...)    -> paid
14:02:10.013  Request B: create_refund(...)    -> paid again

“Two copies arriving four milliseconds apart both pass the check,” she said. “That’s a race condition: the result depends on which request wins a race. And there’s a second problem. What if a customer legitimately needs two refunds on one order? A squashed cake on Monday, a missing candle box on Tuesday?”

“The check would block the second one,” Arjun admitted.

The coat-check counter

Priya drew a cloakroom counter. “You hand in your coat at a wedding. What do you get?”

“A token with a number. Forty-two.”

“At midnight you come back with token 42. Then your cousin borrows your token and comes back with 42 again. Does the attendant hand over two coats?”

“No. Coat 42 has already gone. They’d say ‘that’s been collected’.”

“Why can they say that with confidence?”

Arjun thought. “Because the number identifies that coat, not the person asking. Any request with 42 is about the same coat.”

“So who invented the number 42, and when?”

“The cloakroom, when I handed the coat in. Before anything could go wrong.”

“In our case the caller makes up the number, before the first attempt, and sends it with every retry. The server keeps a list of numbers it has already handled.” She wrote on the board: Same number, same answer. The work happens once.

Where the analogy breaks: in a cloakroom the attendant hands out the number. With an idempotency key, the client chooses it, which means the client must choose well. Most of the bugs in this lesson come from choosing badly.

What “idempotent” means

An operation is idempotent (roughly “same power” in Latin) if doing it twice has the same effect as doing it once. Some actions are idempotent by nature:

• “Set the delivery address of order 1042 to Koregaon Park.” Twice still means Koregaon Park.
• “Look up the status of order 1042.” Reading never changes anything.
• “Cancel order 1042.” Once it is cancelled, cancelling again changes nothing.

Others are not:

• “Refund ₹850 on order 1042.” Twice is ₹1,700.
• “Send this WhatsApp confirmation.” Twice is a worried phone call.
• “Create a new Diwali hamper order.” Twice is two hampers on the doorstep.

Web developers meet this in HTTP itself. The HTTP standard defines GET, PUT and DELETE requests as idempotent, and POST as not. Creating things is usually a POST, which is exactly why creating refunds, messages and orders needs extra protection.

Show the answer

(a) and (b) are idempotent: reading changes nothing, and setting a slot to the same value twice leaves it the same. © is not: “add 50” twice adds 100. If it were “set points to 1,250”, it would be idempotent, which is a useful design trick in itself. (d) is not: two calls send two messages. Tools © and (d) need idempotency keys.

Why a timeout is ambiguous

When a request times out, one of three things happened, and from the caller’s side they all look the same:

sequenceDiagram
    participant A as Support agent
    participant P as Payment service
    A->>P: Refund 850 on order 1042
    Note over P: Case 1: request never arrived
    Note over P: Case 2: arrived and failed
    Note over P: Case 3: arrived, succeeded, reply was slow
    P--xA: no reply within 30 s
    A->>P: Retry: refund 850 on order 1042

In cases 1 and 2, retrying is right. In case 3, retrying is a duplicate. The caller cannot know which case it is in. So the job of telling them apart has to move to the only party that can know: the server.

Following one refund from start to finish

Priya set up the fixed version on staging, and they traced the plum cake refund through it with the same slow payment service. The support ticket was 5531, the order 1042, the amount ₹850. The agent’s tool built the key refund:5531:1042 and sent it in an HTTP header called Idempotency-Key. The server also computed a fingerprint of the request body (a short hash, a sort of digital thumbprint of the exact JSON) so it could spot a key reused for a different request.

Time	What happens	Key table row for refund:5531:1042
14:02:10.0	Request 1 arrives. Key is new. Server records it, calls the bank	status in_progress, fingerprint 9f2c...
14:02:40.0	Agent’s 30 s timeout fires. It backs off 1.4 s	unchanged
14:02:41.0	Bank confirms payout rf_7Q2X. Server stores the result	status done, result {"refund_id": "rf_7Q2X", "amount_inr": 850}
14:02:41.4	Request 2 (the retry) arrives with the same key and body	key found, status done
14:02:41.4	Server returns the stored result. The bank is not called	unchanged

The agent received rf_7Q2X on its second attempt, told the customer “Your refund of ₹850 has been issued”, and the bank was called exactly once.

They ran it once more with the retry arriving faster, at 14:02:40.6, before the bank had replied. This time the server found the key with status in_progress and answered 409 Conflict, “still processing”. That is on the retry list from the last lesson, so the agent backed off 2.8 seconds, tried again, found done, and got rf_7Q2X. Still one payout.

Opening the hood: what the server keeps

The server’s side of the bargain is a small table:

Column	Purpose
key	The idempotency key, unique
fingerprint	Hash of the request body, to catch a key reused for something else
status	in_progress or done
response	The stored result to replay
created_at	So old keys can be deleted after a day or so

And four rules for each incoming request:

1. New key: insert it as in_progress, do the work, store the result, return it.
2. Known key, done: return the stored result without doing the work.
3. Known key, in_progress: another copy is running right now. Return 409 so the caller retries shortly.
4. Known key, different fingerprint: the caller reused a key by mistake. Return an error rather than guess.

Rule 1 has a hidden requirement: “is it new?” and “insert it” must happen as one indivisible step. Otherwise you are back to Arjun’s race condition, with two copies both seeing “new”. Databases give you this with a unique constraint: if two inserts with the same key race, the database lets exactly one succeed and rejects the other.

Payment companies such as Stripe made this pattern popular, and many payment, messaging and ordering APIs accept an Idempotency-Key header. When a service you call supports it, use it. When you build a service that does things, support it yourself.

Building it, one layer at a time

Layer 1: the refund service with a key table

This version uses SQLite, the small database built into Python, so the “insert if new” step is atomic and you can run it anywhere.

# refunds.py
import hashlib
import json
import sqlite3
import uuid

from fastapi import FastAPI, Header, HTTPException
from pydantic import BaseModel

app = FastAPI()
db = sqlite3.connect("keys.db", check_same_thread=False)
db.execute("""CREATE TABLE IF NOT EXISTS idem (
    key TEXT PRIMARY KEY, fingerprint TEXT, status TEXT, response TEXT,
    created_at TEXT DEFAULT CURRENT_TIMESTAMP)""")


class RefundRequest(BaseModel):
    order_id: str
    amount_inr: int


@app.post("/refunds")
def create_refund(req: RefundRequest, idempotency_key: str = Header()):
    body = json.dumps(req.model_dump(), sort_keys=True)
    fingerprint = hashlib.sha256(body.encode()).hexdigest()

    try:  # the PRIMARY KEY makes this insert atomic: only one copy can win
        with db:
            db.execute("INSERT INTO idem (key, fingerprint, status) VALUES (?, ?, 'in_progress')",
                       (idempotency_key, fingerprint))
    except sqlite3.IntegrityError:  # key already exists
        fp, status, response = db.execute(
            "SELECT fingerprint, status, response FROM idem WHERE key = ?", (idempotency_key,)).fetchone()
        if fp != fingerprint:
            raise HTTPException(422, "This key was already used for a different request")
        if status == "in_progress":
            raise HTTPException(409, "Still processing, try again shortly")
        return json.loads(response)  # replay the original result

    result = {"refund_id": f"rf_{uuid.uuid4().hex[:4].upper()}", "amount_inr": req.amount_inr}  # pretend payout
    with db:
        db.execute("UPDATE idem SET status = 'done', response = ? WHERE key = ?",
                   (json.dumps(result), idempotency_key))
    return result

What is new here:

• FastAPI turns the parameter idempotency_key: str = Header() into the Idempotency-Key HTTP header automatically, and rejects requests that do not send one.
• json.dumps(..., sort_keys=True) makes the fingerprint stable: the same refund always produces the same hash, whatever order the fields arrived in.
• The INSERT either succeeds (this request owns the key) or raises IntegrityError (someone already has it). There is no gap for a race to slip into.

Run it with uv run fastapi dev refunds.py.

Layer 2: the caller makes one key per intention

The key must stay the same across retries of one action and be different for different actions. The classic bug is creating it in the wrong place:

# Wrong: a fresh key on every attempt, so every retry looks like a new refund
for attempt in range(5):
    send_refund(order_id, amount, key=str(uuid.uuid4()))

# Right: make the key once, before the first attempt, and reuse it
key = str(uuid.uuid4())
for attempt in range(5):
    send_refund(order_id, amount, key=key)

A random key made once is fine for an ordinary program. Agents need more care.

Layer 3: keys for agent tools come from business facts

Here is the twist that bit Arjun a second time. Suppose the model call that followed the refund failed, and his app re-ran the whole agent turn. The model reads the conversation again and decides, again, to call issue_refund. Any random key made inside the tool is brand new on this second run. The server sees a new intention and pays out.

The fix is to build the key from facts that identify the intention itself:

import httpx


def issue_refund(ticket_id: str, order_id: str, amount_inr: int) -> str:
    """Refund a Crumb order for a support ticket. Safe to call more than once."""
    key = f"refund:{ticket_id}:{order_id}"  # one ticket + one order = one refund
    resp = httpx.post(
        "http://localhost:8000/refunds",
        json={"order_id": order_id, "amount_inr": amount_inr},
        headers={"Idempotency-Key": key},
        timeout=30.0,
    )
    resp.raise_for_status()
    return f"Refund {resp.json()['refund_id']} issued for ₹{amount_inr}"

Now it does not matter how many times the model asks, or how many times the turn is re-run: one ticket, one refund. And the Tuesday candle-box refund gets its own ticket number, so it gets its own key and goes through.

Show the answer

No. When the turn is re-run, the model produces a brand-new tool call with a brand-new id, so the key changes and the refund is paid again. The tool-call ID does protect against your code executing the same tool call twice, which is useful. But only a key built from business facts, like the ticket and order number, survives the model deciding the same thing twice.

Where it breaks

Idempotency keys solve duplicates cleanly, but they have edges of their own.

• A server crash mid-work. If the service dies after inserting in_progress but before writing done, the key is stuck and every retry gets 409 forever. Production systems treat an in_progress row older than a few minutes as abandoned and check the real outcome (did the bank pay?) before deciding.
• Keys that expire too soon or never. Keep keys long enough to cover every retry and re-run (24 hours is common), then delete them, or the table grows forever.
• A key that is too broad. refund:1042 blocks the legitimate second refund on Tuesday. Include whatever makes the intention unique, usually a ticket or request ID.
• A key that is too narrow. Including a timestamp in the key makes every retry unique, which disables the protection completely.
• Services that do not support keys. If the WhatsApp provider you use has no idempotency support, keep your own record of “message for ticket 5531 sent” before calling it, and accept that a crash at exactly the wrong moment can still send twice. Be honest in your design notes about which actions are fully protected.

What about the model call itself? A plain call (prompt in, text out) changes nothing in the world, so retrying it is safe in that sense. It costs tokens each time, and the second answer will be worded differently, so do not retry once the customer has seen a full answer. The danger is always in the side effects around the call.

Try it yourself

With the refund service running, send the same refund twice, then the same key with a different amount:

curl -X POST localhost:8000/refunds -H "Content-Type: application/json" \
  -H "Idempotency-Key: refund:5531:1042" -d '{"order_id": "1042", "amount_inr": 850}'

Predict all three responses before you run them.

Show the answer

The first call returns a new refund_id, such as rf_7Q2X. The second returns the same refund_id, because the server recognised the key and replayed the stored result without paying again. The third (same key, "amount_inr": 900) returns a 422 error, because the body’s fingerprint does not match. That error protects you from a bug where two different refunds accidentally share a key.

Common mistakes

• Generating the key inside the retry loop. Every attempt looks new. Create it once per intention.
• Random keys inside agent tools. A re-run agent turn produces a new key. Build keys from ticket IDs, order IDs and other business facts.
• Check-then-act without atomicity. Two simultaneous copies both pass “have I seen this?”. Use a unique constraint or an atomic “set if not exists”.
• Assuming a timeout means failure. It means “unknown”. Design every side-effecting call for that.
• Keys that are too broad or too narrow. Too broad blocks legitimate repeats; too narrow (with timestamps) blocks nothing.

Recap

• Retries are only safe for idempotent actions, and refunds, messages and new orders are not idempotent by nature.
• A timeout is ambiguous: the action may already have happened.
• An idempotency key lets the server recognise a repeat and replay the first result instead of acting again.
• The server stores key, fingerprint, status and result, and must claim a new key atomically.
• Make one key per intention and reuse it across retries; for agent tools, build it from business facts, not random IDs.

The next morning Arjun explained it to Meera with the cloakroom. “Every refund now gets a token number made from the ticket and the order. If the same token comes back twice, the payment service says ‘already done, here’s the receipt’ instead of paying again. Even if the bot asks twice. Even if the network hiccups.” Meera asked the question that mattered. “And the seven we paid twice?” Arjun had already listed them. Six customers had not noticed; Meera decided to let them keep it as Diwali goodwill.

Then she put a different printout on the desk, with one row highlighted in yellow. “One customer pasted an entire catering contract into the chat,” she said. “Forty times. Who decides how much one person is allowed to cost us?”

Go deeper

• FastAPI headers and request parameters: https://fastapi.tiangolo.com
• Pydantic models for request bodies: https://docs.pydantic.dev
• HTTPX, for calling services with timeouts: https://www.python-httpx.org
• Anthropic tool use, including designing safe tools: https://docs.anthropic.com