Deploying your AI app to the cloud

On a wet Tuesday in late November, an unseasonal storm knocked out power across Crumb’s office block. The bakery kitchens had a generator. The office did not. The little server under Arjun’s desk, the one running the WhatsApp bot, lasted 25 minutes on its battery backup and then went dark.

For the next 1 hour 55 minutes, 140 customer messages went unanswered. Nobody noticed for forty minutes, because the alerts were sent by the same server that had lost power.

Meera’s question the next morning was short. “The ovens kept running. Why didn’t the bot?”

Arjun rents a computer

Arjun’s plan was straightforward: if the office computer is the problem, use a computer somewhere else. He rented a small virtual machine (a VM: a slice of a computer in a data centre that you control like your own) in a Mumbai data centre for about ₹1,200 a month. Then he did what he did on his own laptop:

scp -r crumb-bot/ arjun@203.0.113.24:~/        # copy the code over
ssh arjun@203.0.113.24
cd crumb-bot && nano .env                         # paste the API keys
tmux new -s bot                                   # a terminal that survives logout
uv run fastapi dev main.py --host 0.0.0.0

It did not work the first time. The VM came with Python 3.10; the bot used a 3.12 feature. He spent an evening installing the right Python. Then it ran. He was pleased.

Over the next two weeks:

• The VM installed security updates and rebooted itself at 3:12 a.m. Nothing restarted the bot. It was down until Arjun woke up at 8:40.
• fastapi dev is a development server. It watches files and reloads on changes, and it is not meant for real traffic.
• When Arjun changed the code, he copied files over by hand. Once he forgot prompts/, so the server ran new code with an old prompt.
• The API keys sat in a plain text file on a machine that four people could log into.

“You’ve moved the computer under your desk to a desk in Mumbai,” said Priya. “All the problems came with it.”

The shipping container and the food court

Priya sketched two pictures.

The first was a shipping container. “Before containers, every ship was loaded by hand: sacks, barrels, crates of every shape. Slow, and things broke. Then the world agreed on one box size. Now any crane, ship, train or truck can move any container without caring what’s inside.”

A container is that box for software: your code, the exact Python version, every library, and the command to start it, packed together so it runs the same on your laptop, a colleague’s laptop and a server in Mumbai. The “Python 3.10 on the VM” problem disappears, because the right Python is inside the box. Docker is the most common tool for building and running containers. The recipe for building one is a Dockerfile, and the built, frozen result is an image.

The second picture was a food court. “Renting a VM is like renting an empty shop: you sort out the electricity, the locks, the cleaning, and you stand at the door all night. A managed platform is a stall in a food court. The mall provides power, security, cleaning and seating. You bring a kitchen that fits their standard stall, and they handle the rest. If more people arrive, they open more stalls running your kitchen. If nobody comes, they close your stall and stop charging you.”

“So what does the mall need to know about my kitchen?” asked Arjun.

“You tell me.”

He thought about it. “How to switch it on. Which counter window customers come to. Where the keys to the storeroom are, without me leaving them lying around. And… how to tell if it’s working?”

“Exactly four things,” said Priya. “The start command, the port, the secrets, and a health check. Everything else in this lesson is detail around those four.”

Where the analogy breaks: a food stall keeps its stock between customers. A cloud container can be closed or replaced at any moment and comes back empty, so anything stored only inside it, files or memory, can vanish. Keep state in a database or a store like Redis, never inside the container.

Following one deploy and one request

Crumb chose Google Cloud Run, a managed platform that runs containers, scales them automatically (down to zero when idle), and has a region in Mumbai, close to Crumb’s customers. Before writing any configuration, Priya walked Arjun through what would happen, start to finish.

1. Arjun runs one deploy command from the project folder.

2. The platform builds the image from the Dockerfile: start from an official Python 3.12 base, install uv, install the exact library versions from uv.lock, copy in the code and prompts. About 2 minutes 10 seconds; the image is around 250 MB.

3. The image is stored in a registry (a library of images), tagged with a version.

4. A new revision is created: an unchangeable snapshot of “this image, with these settings”. The platform starts one container from it and injects the settings: PORT=8080, and ANTHROPIC_API_KEY fetched from a secrets store.

5. The health check runs. The platform calls GET /health on the new container. It answers {"status": "ok"} after 3.1 seconds of startup. Only now does the platform send it real traffic. If it had failed, the old revision would have kept serving and nothing would have broken.

6. A customer message arrives at 9:00 a.m., after a quiet night with zero containers running. The platform starts one container (a cold start: the delay of starting a fresh container before it can answer), which takes 2.8 seconds, then the model call takes 1.9 seconds. The customer waits 4.7 seconds.

7. The next message, ten seconds later, reaches the already-warm container: 0.04 seconds of app time plus 1.9 seconds of model time. 1.94 seconds.

8. At 1 p.m. lunch traffic jumps. One container is set to handle up to 20 requests at once. When requests exceed that, the platform starts a second, then a third. This is autoscaling: the platform adds and removes copies of your container to match traffic, within limits you set.

9. The cost of a month. Crumb’s 36,000 messages a month each keep a container busy for about 2.5 seconds, mostly waiting for the model. Even if no two requests ever overlapped, that is 90,000 seconds, 25 hours of one container. Cloud Run bills per second of use and includes a monthly free allowance (historically around 180,000 vCPU-seconds; check the pricing page for today’s figures). So hosting costs Crumb somewhere between ₹0 and a few hundred rupees a month, against ₹38,000 for the model API. The hosting bill is a rounding error; the token bill is the one to watch.

Opening the hood: images, secrets, health and scale

Images and containers. The Dockerfile is a recipe. docker build follows it to produce an image, a frozen, versioned package. docker run starts a container from the image. You can start ten identical containers from one image. Each line of a Dockerfile creates a layer, and Docker reuses unchanged layers, which is why you copy the dependency files and install libraries before copying your code: change a prompt, and only the last layer rebuilds.

Environment variables and secrets. An environment variable is a named setting the platform hands to your program when it starts, read in Python with os.environ["NAME"]. Settings that change between environments (port, model name, Redis address) go here, never in code. Secrets such as API keys need more care: a secrets manager stores them encrypted, records who read them, and lets you rotate (replace) a key without rebuilding anything. The platform fetches the secret and hands it to your container as an environment variable at start-up. Your code does not change; it still reads ANTHROPIC_API_KEY, the same as with python-dotenv on your laptop.

Health checks. A health check is a small endpoint the platform calls to decide whether a container is ready for traffic and still alive. Platforms often distinguish a startup check (is it ready yet?) from a liveness check (has it frozen and needs restarting?). Keep both cheap and independent of outside services.

Scaling. Three numbers control it. Concurrency: how many requests one container handles at once; for apps that mostly wait on an LLM, 10 to 40 is reasonable. Minimum instances: 0 means scale to zero and pay nothing when idle, at the price of cold starts; 1 keeps one container always warm for a steady monthly charge. Maximum instances: a ceiling that protects you from runaway scaling and cost.

Region. Put the app near your users and, if possible, near your database. A request from Pune to Mumbai takes a few milliseconds; to a US region, around 250 milliseconds each way. For a chat app making several round trips per message, that adds up.

Building it, step by step

Step 1: an app that is ready for the cloud. Two endpoints: a cheap health check, and the chat endpoint. It reads everything from the environment.

uv add "fastapi[standard]" anthropic

# main.py
import anthropic
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()
client = anthropic.Anthropic()   # reads ANTHROPIC_API_KEY from the environment

class ChatRequest(BaseModel):
    message: str

@app.get("/health")
def health():
    return {"status": "ok"}      # cheap: no model call, no database call

@app.post("/chat")
def chat(req: ChatRequest):
    response = client.messages.create(
        model="claude-opus-5-5", max_tokens=500,
        system="You are Crumb's friendly support assistant.",
        messages=[{"role": "user", "content": req.message}],
    )
    return {"reply": next(b.text for b in response.content if b.type == "text")}

Step 2: the Dockerfile. Each line is commented, because each one solves a problem Arjun hit on the VM.

# Start from an official, slim Python 3.12 image: the right Python is inside the box
FROM python:3.12-slim

# Copy the uv tool in from its official image (pin a version in real projects)
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/

WORKDIR /app

# Install dependencies first, so this layer is reused when only code changes
COPY pyproject.toml uv.lock ./
RUN uv sync --locked --no-dev --no-install-project

# Now copy the code and prompts, all of them, every time
COPY . .
RUN uv sync --locked --no-dev

# Use the virtual environment uv created
ENV PATH="/app/.venv/bin:$PATH"

# The production server (not "dev"), on whatever port the platform gives us
CMD ["sh", "-c", "fastapi run main.py --port ${PORT:-8080}"]

fastapi run starts the production server and listens on all network addresses (0.0.0.0), which a container needs. ${PORT:-8080} means “use the PORT variable if the platform sets it, otherwise 8080”.

Step 3: keep secrets and junk out of the image. A .dockerignore file lists what COPY . . must skip.

# .dockerignore
.env
.venv/
.git/
__pycache__/
*.ipynb
evals/production_samples/

Step 4: run the container on your laptop first. This catches most problems in seconds instead of minutes.

docker build -t crumb-bot .
docker run --rm -p 8080:8080 --env-file .env crumb-bot

-p 8080:8080 connects your laptop’s port 8080 to the container’s. --env-file .env passes your keys in at runtime, without baking them into the image. In a second terminal:

curl localhost:8080/health
curl -X POST localhost:8080/chat -H "Content-Type: application/json" \
     -d '{"message": "Do you have eggless cakes?"}'

{"status":"ok"}
{"reply":"Yes! We have eggless versions of our chocolate truffle, ..."}

You should now see both responses. If /health works on your laptop inside Docker, it will almost certainly work in the cloud.

Step 5: store the key in the secrets manager. These are one-time setup commands with Google Cloud’s gcloud command-line tool, after creating a project and installing it.

gcloud services enable run.googleapis.com cloudbuild.googleapis.com \
    artifactregistry.googleapis.com secretmanager.googleapis.com

# Read the key from your shell, so it never appears in a file
printf "%s" "$ANTHROPIC_API_KEY" | gcloud secrets create anthropic-api-key --data-file=-

# Allow the service's identity to read that one secret (replace PROJECT_NUMBER)
gcloud secrets add-iam-policy-binding anthropic-api-key \
    --member="serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com" \
    --role="roles/secretmanager.secretAccessor"

The last command follows the least-privilege idea from earlier lessons: the bot’s identity can read this one secret and nothing else.

Step 6: deploy.

gcloud run deploy crumb-bot \
    --source . \
    --region asia-south1 \
    --allow-unauthenticated \
    --set-secrets ANTHROPIC_API_KEY=anthropic-api-key:latest \
    --memory 512Mi \
    --concurrency 20 \
    --min-instances 0 \
    --max-instances 5 \
    --timeout 300

Line by line: build from the current folder’s Dockerfile; run in Mumbai; allow public requests (WhatsApp’s servers must reach the webhook, so your code must verify each webhook’s signature instead); inject the secret as an environment variable; give each container 512 MB of memory; up to 20 requests per container; scale to zero when idle; never more than 5 containers; allow requests up to 300 seconds, enough for a slow agent or a long streamed reply.

Building using Dockerfile and deploying container to Cloud Run service [crumb-bot]
✓ Building and deploying... Done.
Service [crumb-bot] revision [crumb-bot-00001-xak] has been deployed and is serving 100 percent of traffic.
Service URL: https://crumb-bot-<your-project-id>.asia-south1.run.app

Your URL will differ. Test it exactly as you did locally, replacing localhost:8080 with the service URL. You should now see the same two responses, from Mumbai.

By default, Cloud Run’s start-up check only confirms the container is listening on its port. To make it call /health instead, add an HTTP startup and liveness probe in the service’s health-check settings in the console or service YAML. Platform settings and flags move; if one has changed, check the Cloud Run docs.

Step 7: read the logs. Anything your app prints goes to the platform’s log viewer, searchable by time and revision:

gcloud run services logs read crumb-bot --region asia-south1 --limit 20

Other platforms

Cloud Run is one good choice, not the only one. The four things (start command, port, secrets, health check) carry over everywhere.

Platform	Good for	Watch for
Render	Connect a GitHub repo, it builds your Dockerfile and deploys on every push; very beginner-friendly	Free tiers sleep when idle, so expect slow first requests
Railway	Very quick setup, usage-based pricing, easy databases alongside	Keep an eye on usage-based bills
Fly.io	Running containers in many regions close to users, from a command-line tool	More networking concepts to learn
AWS (ECS with Fargate) or Azure Container Apps	Companies already on those clouds	More setup: networking, permissions, load balancers

Where it breaks

Listening on the wrong address. A server listening on 127.0.0.1 inside a container only accepts connections from inside that container. The platform sees nothing and the health check fails. Listen on 0.0.0.0; fastapi run does this by default.

Hard-coded ports. Platforms tell you the port through PORT. Ignore it and the container starts but never receives traffic.

State inside the container. Uploaded files, in-memory caches, rate-limit counters and conversation history kept in a Python dictionary all vanish when the platform replaces a container, which it can do at any time. Use a database, object storage or Redis.

Timeouts on long requests. Agents that think for minutes, or long streamed replies, can hit the platform’s request timeout. Set it to match your longest legitimate request, and for very long jobs use a queue, as in the scalable RAG module.

Cold starts on first contact. With minimum instances at 0, the first customer after a quiet spell waits a few extra seconds. For a support bot that is usually fine. If it is not, set minimum instances to 1 and accept the monthly charge.

No ceiling. Without a maximum instance count, a traffic spike or attack can scale you to dozens of containers, each calling the model. Set the ceiling, and keep the spend breaker from the previous lesson.

Try it yourself

Containerise any FastAPI app from earlier in the course:

1. Add a /health endpoint.
2. Write the Dockerfile and .dockerignore.
3. Build and run it locally with --env-file .env, and test both endpoints with curl.
4. Run docker history <image-name> and confirm no layer is suspiciously large or contains your .env.
5. If you have a cloud account, deploy it to one platform and test the public URL.

Show the answer

Steps 1 to 3 should give {"status":"ok"} from /health and a model reply from your main endpoint. If docker run starts but curl hangs, the server is probably listening on 127.0.0.1: check you used fastapi run, not fastapi dev. In step 4, the largest layers should be the base image and the uv sync layer; if you see a layer from COPY . . that is hundreds of MB, your .dockerignore is missing .venv/. For step 5, the most common first-deploy failure is a missing secret permission: the logs show an error reading ANTHROPIC_API_KEY or a permission denied message on the secret.

Common mistakes

• Running the development server in production. fastapi dev reloads on file changes and is not built for real traffic. Use fastapi run.
• Baking secrets into images. Pass them in at runtime from a secrets manager; rotate any key that ever landed in an image or commit.
• Expensive health checks. A health check that calls the LLM costs money all day and turns a provider outage into a restart loop.
• Storing state in the container. Containers are replaced without warning. Keep data in shared services.
• Skipping the local Docker run. Ten seconds on your laptop catches what a five-minute cloud deploy would reveal much more slowly.

Recap

• A container packs code, Python, libraries and the start command so the app runs identically everywhere; a Dockerfile is its recipe.
• A managed platform needs four things from you: a start command, a port, secrets and a health check.
• Secrets live in a secrets manager and reach the app as environment variables at runtime, never inside the image.
• Autoscaling is controlled by concurrency, minimum and maximum instances; scale-to-zero trades cost for cold starts.
• Containers are disposable, so state lives outside them, and hosting usually costs far less than the model API.

By Friday the bot was running in Mumbai. Arjun pulled the plug on the old office server himself, on purpose, while Meera watched the WhatsApp line. Nothing happened. “The bot now lives in Google’s data centre in Mumbai, in a sealed box with everything it needs,” he explained. “If it crashes, the platform restarts it. If lots of people message at once, it starts more copies. If nobody messages at night, it shuts down and we pay nothing. Hosting costs us almost nothing; the AI itself is still the real bill.”

Then Priya asked, “How did the code get there?” Arjun admitted he had typed the deploy command from his laptop, with two files he had not committed yet. Priya smiled. “So the next change goes through a pipeline instead.” That is the next lesson.

Go deeper

• FastAPI, including its guide to deploying with Docker: https://fastapi.tiangolo.com
• uv, including its guide to using uv in Docker: https://docs.astral.sh/uv/
• Docker documentation: https://docs.docker.com
• Google Cloud Run documentation: https://cloud.google.com/run/docs