A multi-provider LLM gateway with fallback

Friday, 7:40 p.m., the busiest hour of Crumb’s week. The support bot stopped answering. Arjun’s retries waited politely, backed off with jitter, and failed again. The provider’s status page went orange, then red: “Elevated error rates.” For forty minutes every customer asking about a Diwali hamper got “Sorry, something went wrong.”

On Monday, Priya put the numbers on the whiteboard. 1,130 failed chats. On a normal Friday, about one support chat in eleven ends with an order, and the average Diwali order was ₹1,800. “Roughly a hundred orders that didn’t happen,” she said. “Call it ₹1.8 lakh, or about $2,100, in forty minutes.”

Meera had a simpler way to put it. “When the flour mill in Satara went on strike last year, we didn’t close the bakeries. We called the backup mill in Nashik. Why can’t the computer do that?”

“It can,” Priya said. “We’ll put one front door in front of all the AI providers. The app only talks to the door. Behind it, the door knows who to call first, who to call next, and who to stop calling for a while.”

Arjun’s first attempt

Arjun had API keys for OpenAI and Gemini from a hackathon. His first fix took five minutes:

def ask(system: str, messages: list[dict]) -> str:
    try:
        r = claude.messages.create(model="claude-opus-5-5", max_tokens=800,
                                   system=system, messages=messages)
        return r.content[0].text
    except Exception:
        r = openai_client.chat.completions.create(model=OPENAI_MODEL, max_tokens=800,
                                                  system=system, messages=messages)
        return r.content[0].text

He tested it by switching off his Wi-Fi to Anthropic’s address. The except branch crashed with TypeError: create() got an unexpected keyword argument 'system'. OpenAI’s Chat Completions API has no system argument; the system prompt goes inside the message list. When he fixed that, the next line crashed too: OpenAI’s response has no .content list; the text lives at choices[0].message.content.

“Every supplier has its own order form,” Priya said. “Your fallback code was filling in Satara’s form and posting it to Nashik.”

He fixed both, and it worked. Then Priya asked him to simulate the outage properly, with Anthropic returning errors slowly instead of instantly, the way real outages behave. Each request now tried Claude, waited for the 20-second timeout, let the SDK retry once (another 20 seconds), and only then went to OpenAI. Every single customer waited about 47 seconds for an answer. And every request kept knocking on the door of a provider that was clearly down.

The purchasing clerk and the fuse box

Priya drew Crumb’s purchasing office. “Who orders flour?”

“Ramesh in purchasing.”

“Does each store manager call the mills directly?”

“No. They tell Ramesh ‘we need 200 kilos of maida by Thursday’, and Ramesh deals with the mills.”

“Why is that better?”

Arjun saw it. “Store managers don’t need to know each mill’s forms, prices or phone numbers. Ramesh translates. If we switch mills, nothing changes for the stores.”

“That’s the gateway. Your app is the store manager: it says ‘complete this conversation’. The gateway is Ramesh: it knows that Anthropic wants system as a separate field, OpenAI wants it inside the messages, and Gemini calls the assistant ‘model’.” She paused. “Now, Satara has failed to deliver five mornings in a row. Does Ramesh still phone Satara first every morning and wait on hold for twenty minutes before calling Nashik?”

“No. He’d stop calling Satara for a while. Maybe try them again next week.”

“That’s a circuit breaker.” She drew a fuse box. “When a wire shorts, the breaker trips and cuts the line, so you stop feeding power into a fault. Later, you flip it back and see if the fault is fixed.”

Where the analogy breaks: the backup mill’s flour is the same maida. A backup model is a different model, with its own style and its own mistakes. That difference will come back to bite.

Following one question through the gateway

Priya built the gateway with Arjun (the code is below), and they replayed Friday’s outage on staging. Here is one customer’s question, “Is the kaju katli box available for Saturday delivery?”, arriving at 19:41:03, as the outage starts:

Time	Step	Result
0.00 s	Claude’s breaker is closed (healthy). Call Claude
0.42 s	Anthropic answers 529 “overloaded”	Retryable. SDK retries once
1.95 s	Second 529	Claude failure count goes from 4 to 5: breaker opens
1.95 s	Fall back: OpenAI’s breaker is closed. Call OpenAI
3.10 s	OpenAI answers 200 OK	Reply returned to the app

The app received this, in exactly the same shape it would have received from Claude:

{"text": "Yes! The kaju katli box can be delivered on Saturday if you order by 6 p.m. Friday.",
 "provider": "openai", "model": "gpt-5-mini", "input_tokens": 412, "output_tokens": 38}

The next customer, eight seconds later, had a different experience. Claude’s breaker was open, so the gateway skipped Claude without calling it and went straight to OpenAI. Answer in 1.2 seconds. Sixty seconds after the breaker opened, the gateway let one trial request through to Claude. It failed, so the breaker opened again for another sixty seconds. At 20:19 a trial succeeded, the breaker closed, and traffic flowed back to Claude.

flowchart LR
    App["Crumb support app"] --> GW["LLM gateway"]
    GW --> A["1st: Claude"]
    GW -.->|"if Claude fails or breaker open"| B["2nd: OpenAI"]
    GW -.->|"if both fail"| C["3rd: Gemini"]
    GW -.->|"all down"| D["Canned reply and human handoff"]

Show the answer

Without a breaker, every request tries the broken provider first: 200 × 40 = 8,000 requests, each paying the timeout penalty before falling back. With the breaker, about 5 requests trip it, then roughly one trial per minute for 40 minutes: about 45 requests. The other 7,955 customers skip straight to the backup, and the struggling provider gets breathing room to recover.

Opening the hood: what the gateway hides

An LLM gateway is a layer between your app and the providers. It has three jobs: translate, choose and protect.

Translate. Normalising means agreeing on one shape for requests and responses, then writing a small adapter (a translator function) for each provider. Here is what the adapters absorb:

Detail	Anthropic	OpenAI Chat Completions	Gemini
System prompt	system= argument	First message with role system	system_instruction in the config
Assistant’s role name	assistant	assistant	model
Where the text is	content blocks of type text	choices[0].message.content	.text
Token counts	usage.input_tokens, usage.output_tokens	usage.prompt_tokens, usage.completion_tokens	usage_metadata.prompt_token_count, candidates_token_count
Error classes	anthropic.APIStatusError	openai.APIStatusError	google.genai.errors.APIError

Choose. The gateway walks an ordered list: your best model first, then a capable alternative from a different company (so one outage does not take out both), and finally a polite canned reply that hands the customer to a human. The canned reply is a fallback too: the last, guaranteed-to-work option.

Protect. A circuit breaker per provider has three states:

• Closed (normal): requests flow. Count consecutive failures.
• Open (tripped): after, say, 5 failures in a row, do not call the provider at all for a cooldown (say 60 seconds).
• Half-open (testing): after the cooldown, let one trial request through. Success closes the breaker; failure opens it for another cooldown.

Building the gateway, one layer at a time

Layer 1: one reply shape, three adapters

# gateway.py
import os
from dataclasses import dataclass

import anthropic
from dotenv import load_dotenv
from google import genai
from google.genai import types
from openai import OpenAI

load_dotenv()


@dataclass
class Reply:
    text: str
    provider: str
    model: str
    input_tokens: int
    output_tokens: int


# Short timeouts and one retry each: the gateway, not the SDK, handles longer outages.
claude = anthropic.Anthropic(max_retries=1, timeout=20.0)
openai_client = OpenAI(max_retries=1, timeout=20.0)
gemini = genai.Client()  # reads GEMINI_API_KEY

OPENAI_MODEL = os.getenv("OPENAI_MODEL", "gpt-5-mini")  # model names change: see the provider's models page
GEMINI_MODEL = os.getenv("GEMINI_MODEL", "gemini-2.5-flash")


def call_claude(system: str, messages: list[dict], max_tokens: int) -> Reply:
    r = claude.messages.create(model="claude-opus-5-5", max_tokens=max_tokens,
                               system=system, messages=messages)
    text = next(b.text for b in r.content if b.type == "text")
    return Reply(text, "anthropic", r.model, r.usage.input_tokens, r.usage.output_tokens)


def call_openai(system: str, messages: list[dict], max_tokens: int) -> Reply:
    r = openai_client.chat.completions.create(
        model=OPENAI_MODEL,
        messages=[{"role": "system", "content": system}, *messages],  # system goes inside
    )
    return Reply(r.choices[0].message.content, "openai", r.model,
                 r.usage.prompt_tokens, r.usage.completion_tokens)


def call_gemini(system: str, messages: list[dict], max_tokens: int) -> Reply:
    contents = [{"role": "model" if m["role"] == "assistant" else "user",
                 "parts": [{"text": m["content"]}]} for m in messages]
    r = gemini.models.generate_content(
        model=GEMINI_MODEL, contents=contents,
        config=types.GenerateContentConfig(system_instruction=system, max_output_tokens=max_tokens),
    )
    u = r.usage_metadata
    return Reply(r.text, "gemini", GEMINI_MODEL, u.prompt_token_count or 0, u.candidates_token_count or 0)

Each adapter takes the same three inputs and returns a Reply. The OpenAI adapter leaves out a token cap because the parameter’s name differs between OpenAI’s models and APIs; add the right one from their docs. Library APIs change quickly. If a name has moved, check the docs linked below.

Try one adapter on its own before going further:

print(call_claude("You are Crumb's support assistant.", [{"role": "user", "content": "Hi"}], 200))

Reply(text='Hello! Welcome to Crumb & Co. How can I help you today?', provider='anthropic',
      model='claude-opus-5-5', input_tokens=17, output_tokens=16)

Layer 2: decide what counts as “down”

Not every error should trigger a fallback. Use the sorting from the retries lesson, with one addition:

def status_of(err: Exception) -> int | None:
    # Anthropic and OpenAI errors carry .status_code; Gemini's carry .code.
    return getattr(err, "status_code", None) or getattr(err, "code", None)


def should_fall_back(err: Exception) -> bool:
    status = status_of(err)
    if status is None:
        return True  # timeout or dropped connection
    return status in {401, 403, 408, 409, 429} or status >= 500

The addition is 401 and 403. An authentication failure at one provider (an expired key, a billing problem) says nothing about the others, so the customer should still get an answer. Log it loudly, because a human needs to fix the key.

Show the answer

Usually no. A 400 means your request is wrong, and the same oversized prompt will probably fail at the next provider too, or quietly succeed with a different context window and hide the bug. You want this error to reach your logs and get fixed (the token budgets from the last lesson should have caught it earlier). There are rare designs where falling back to a provider with a larger context window is deliberate, but make that an explicit rule, not an accident.

Layer 3: a circuit breaker

import time


class CircuitBreaker:
    def __init__(self, threshold: int = 5, cooldown: float = 60.0):
        self.threshold, self.cooldown = threshold, cooldown
        self.failures, self.opened_at = 0, None

    def allow(self) -> bool:
        if self.opened_at is None:
            return True                                            # closed
        return time.monotonic() - self.opened_at >= self.cooldown  # half-open after cooldown

    def success(self) -> None:
        self.failures, self.opened_at = 0, None                    # close

    def failure(self) -> None:
        self.failures += 1
        if self.failures >= self.threshold:
            self.opened_at = time.monotonic()                      # open, or re-open after a failed trial

time.monotonic() is a clock that only moves forward, so it is safe for measuring intervals even if the computer’s wall-clock time changes.

Layer 4: put it together

import logging

CHAIN = [("anthropic", call_claude), ("openai", call_openai), ("gemini", call_gemini)]
breakers = {name: CircuitBreaker() for name, _ in CHAIN}
CANNED = Reply("Sorry, our assistant is having trouble. A team member will reply on WhatsApp shortly.",
               "canned", "none", 0, 0)


def complete(system: str, messages: list[dict], max_tokens: int = 800) -> Reply:
    for name, call in CHAIN:
        if not breakers[name].allow():
            continue  # breaker open: skip without waiting
        try:
            reply = call(system, messages, max_tokens)
            breakers[name].success()
            return reply
        except Exception as err:
            if not should_fall_back(err):
                raise  # a bug in our request: surface it
            breakers[name].failure()
            logging.warning("provider=%s failed status=%s, falling back", name, status_of(err))
    return CANNED

The app now calls complete(...) and never imports an AI SDK directly. Log reply.provider with every answer so you can see how often fallbacks happen. The local model you will run in the next module can join this chain as one more adapter.

Layer 5: or use LiteLLM

You do not have to build all of this yourself. LiteLLM is an open-source Python library that calls more than a hundred model providers through one OpenAI-style interface. It can also run as a standalone proxy server (a separate service your apps send requests to) with fallbacks, retries, per-key budgets and spend tracking configured in a file.

# uv add litellm
from litellm import completion

r = completion(model="anthropic/claude-opus-5-5",
               messages=[{"role": "user", "content": "Do you have eggless cakes?"}])
print(r.choices[0].message.content)  # the same response shape for every provider

Change the model string to start with openai/ or gemini/ and the rest of the code stays the same. Its router and proxy add fallback chains through configuration. Library APIs change quickly. If a name has moved, check the docs linked below.

Building your own small gateway once is still worth it. You now know exactly what LiteLLM is doing under the hood, which makes it much easier to debug when it surprises you.

Where it breaks

Two Fridays later, a twelve-minute wobble at Anthropic sent real traffic to OpenAI. No customer saw an error. But three things showed up in Monday’s review.

The backup sounded different. Crumb’s system prompt had been tuned on Claude over months. The fallback model followed it slightly differently: longer answers, Markdown tables that looked broken in WhatsApp, and in one case a cheerful promise of free same-day delivery to Lonavala, 65 kilometres outside Crumb’s delivery area.

Streaming mid-answer. The gateway can switch providers before the first token. Once half an answer is on screen, switching produces a confusing mix of two writers. Fail over only before streaming starts; after that, send the error event from the first lesson.

Shared breakers. Crumb runs four copies of the app server, and each had its own breaker. Each one had to discover the outage separately. For a few servers this is acceptable; at larger scale, keep breaker state somewhere shared, such as Redis.

Experienced engineers watch a few more things: two “different” providers that host on the same cloud region can fail together; tools and structured outputs need translating per provider, which is real work; sending customer messages to three companies means three data agreements to review; and a fallback that works silently can hide a broken primary for days, so alert when the fallback rate rises.

Try it yourself

Test the fallback without waiting for a real outage. In your shell, set ANTHROPIC_API_KEY to a deliberately wrong value. Call complete("You are helpful.", [{"role": "user", "content": "Hi"}]) six times and print reply.provider and the time each call took.

Show the answer

Each of the first five calls gets a 401 from Anthropic, logs a warning and falls back, so every reply comes from openai (or gemini if OpenAI is not configured). After the fifth failure the Anthropic breaker opens, so the sixth call skips Claude entirely: no warning for Anthropic, and a noticeably faster reply. That speed-up is the circuit breaker working.

Common mistakes

• Copy-pasting one provider’s call for another. Request and response shapes differ. Put each provider behind an adapter.
• Falling back on every error. A 400 from a bad prompt gets hidden instead of fixed. Fall back on outage-type errors.
• No circuit breaker. Every customer pays the full timeout penalty during an outage.
• Never testing the backup. The first time the fallback runs should not be on Diwali evening. Evaluate it and trigger it on purpose.
• Not logging which provider answered. You cannot debug quality changes you cannot see.

Recap

• A gateway gives your app one interface to many providers and hides their differences inside adapters.
• Fall back on rate limits, server errors, timeouts and per-provider auth failures; surface errors in the request itself.
• A circuit breaker stops calling a failing provider for a cooldown, then tests it with a single request.
• Always end the chain with a safe canned reply and a human handoff.
• Evaluate every fallback model, and consider LiteLLM when you would rather configure than build.

Meera asked for the explanation in bakery terms, and Arjun was ready. “The app used to phone one mill directly. Now it tells our purchasing clerk what it needs, and the clerk knows three mills and how each one’s forms work. If a mill stops answering five times in a row, the clerk stops ringing them for a minute and only checks occasionally whether they’re back. And if every mill is closed, the customer gets a polite note and a person on WhatsApp.” Meera liked it. Then she frowned. “Three mills. Three American companies. Every customer’s address and phone number and ‘severe peanut allergy, child aged six’ going to all of them?”

Priya smiled. “Then let’s talk about running a model in the back office.”

Go deeper

• Anthropic API docs: https://docs.anthropic.com
• OpenAI API docs: https://platform.openai.com/docs
• Gemini API docs: https://ai.google.dev/gemini-api/docs
• LiteLLM documentation: https://docs.litellm.ai