Dockerised LLM environments

Sneha’s Windows laptop had beaten them both. Her Ollama was four versions old. Her models lived in a different folder. She did not have the crumb-notes preset, her Python was 3.10, and the summariser script crashed on an import Arjun had never seen fail. Twenty minutes in, she said the sentence every engineer has heard: “Well, it works on your machine.”

The same afternoon a delivery van brought the thing Meera had approved after the privacy meeting: a second-hand tower PC for the back office with one NVIDIA graphics card carrying 24 GB of GPU memory, about ₹2.4 lakh (roughly $2,700 at ~₹88 to the dollar). It ran Ubuntu Linux. Nobody at Crumb had ever set up a GPU server.

“So now we have three different machines,” Priya said, “and we want the AI to behave identically on all of them. We are not going to install things by hand three times. We are going to pack it once.”

Tiffin boxes and shipping containers

Priya drew a tiffin box on the whiteboard. “Your mother packs lunch at home: rice, dal, sabzi, pickle, all in their own compartments. You open it at the office, at a friend’s place or on a train, and it is the same lunch. You do not need to find a kitchen with the right ingredients wherever you go.”

Docker is a tool that does that for software. It packs a program together with everything it needs to run (the right Python version, libraries, settings, system files) into a sealed bundle that runs the same way on a Mac, a Windows laptop or a Linux server. Four words carry the whole idea:

• An image is the packed tiffin before anyone opens it: a read-only template containing the program and everything it depends on. ollama/ollama is an image published by the Ollama team.
• A container is a running copy of an image, the tiffin opened and being eaten. You can start many containers from one image, and you throw them away freely.
• A volume is storage that lives outside any container, managed by Docker. Throw the container away, and the volume stays. This will matter a great deal in a minute.
• A port mapping is a door from the host machine into the container. -p 11434:11434 says “traffic arriving at port 11434 on this machine goes to port 11434 inside the container”. A port is a numbered door on a computer that a network program listens behind.

The bigger analogy is the shipping container. Before standard containers, every ship and every port handled cargo differently. Once everything came in the same steel box, any crane and any ship could move it. Docker images are the standard box for software.

Where the analogy breaks: a container is not a separate computer. It shares the kernel (the core of the operating system that talks to the hardware) with the host machine. That makes it light and fast to start, but it also means special hardware, such as a GPU, has to be deliberately handed through from the host. Keep that in mind; it is the hardest part of this lesson.

Install Docker Desktop on macOS or Windows, or Docker Engine on Linux, from the Docker docs. Check it works with docker run hello-world.

Arjun’s first attempt

Arjun started on the GPU server with the command from Ollama’s docs, minus the parts he did not understand yet.

docker run -d -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama pull llama3.1:8b     # about 4.9 GB
docker exec -it ollama ollama pull llama3.2        # about 2.0 GB

-d runs the container in the background (“detached”). --name ollama gives it a name so you can refer to it. docker exec -it ollama ... runs a command inside the running container, here the ollama command-line tool. It worked. The allergy note from the last lesson came back correctly from curl http://localhost:11434/api/chat.

Two days later, Ollama released a new version. Arjun upgraded the obvious way: pull the newer image, delete the old container, start a new one.

docker pull ollama/ollama
docker rm -f ollama
docker run -d -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama list

NAME    ID    SIZE    MODIFIED

Empty. Both models were gone. He pulled them again: 6.9 GB over the office line at about 50 megabits per second, which is 6.9 × 8 = 55 gigabits, or roughly 18 minutes. By the end of the week he had done this three times while experimenting, about 21 GB and nearly an hour of downloads.

And there was a second problem he had not noticed yet. The allergy note took about 3 seconds to summarise on the new ₹2.4 lakh server, slower than on his laptop.

Why the models vanished

“Where did the files go when you pulled the model?” Priya asked.

“Into the container. Into /root/.ollama, I think.”

“And what did docker rm delete?”

He saw it. “The container. And everything written inside it.”

Every container has a thin writable layer on top of its read-only image. Anything a program writes there, including 6.9 GB of models, lives exactly as long as that container. Delete the container and the layer goes with it. That is a feature: it is why containers are disposable and clean. It is a disaster when the thing you wrote is expensive to recreate.

The fix is a Docker volume: a named storage area that Docker keeps on the host’s disk, outside any container, and plugs into a folder inside the container when it starts.

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

-v ollama:/root/.ollama means “plug the volume named ollama into the folder /root/.ollama inside the container”. Docker creates the volume the first time. Ollama writes its models there without knowing anything changed. On Linux the volume’s files sit under Docker’s own data folder on the host; docker volume inspect ollama shows exactly where.

Now the upgrade is safe. Remove the container, start a new one from the newer image with the same -v ollama:/root/.ollama, and ollama list shows both models, because they were never inside the container.

There is a second kind of mount you will see: a bind mount, -v /home/arjun/models:/root/.ollama, which plugs an ordinary host folder into the container. It is handy when you want to see the files directly. Named volumes are simpler for this job because Docker manages permissions and location for you.

Show the answer

After docker rm and a fresh start with the same volume: yes, both models are there. The image version does not matter, because the models live in the volume, not the container or the image. After docker volume rm ollama: they are gone for good, and the next start creates an empty volume. Deleting a volume is the one command here that destroys downloads, so do it on purpose only.

Following one request into a container

The 3-second summary was the GPU problem. To see it, follow the allergy note from the last lesson, “Pls no nuts at ALL, my son (6) is allergic, also deliver after 5pm”, from Arjun’s script on the server into the container and back.

flowchart LR
    S["summarise.py on the host"] -->|"localhost:11434"| P["Port mapping"]
    P --> O["Ollama inside the container"]
    O --> V["Volume: /root/.ollama with the models"]
    O --> G["GPU, only if passed through"]
    O -->|"JSON reply"| S

1. The script sends its JSON to http://localhost:11434/api/chat on the host.
2. Docker’s port mapping forwards it to port 11434 inside the container. The Ollama image is configured to listen on all of the container’s network interfaces (OLLAMA_HOST=0.0.0.0:11434), otherwise nothing outside the container could reach it.
3. Ollama finds llama3.1:8b in /root/.ollama/models, which is really the ollama volume on the host disk.
4. Ollama looks for a GPU. Inside Arjun’s container there was none to find, because he had not handed it through. So all 4.9 GB of weights went into ordinary memory and every token was computed by the CPU.

You can see it from outside:

docker exec -it ollama ollama ps

NAME           ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.1:8b    46e0c10c039e    6.0 GB    100% CPU     4096       4 minutes from now

100% CPU. The numbers from the response told the same story. Illustrative figures for this kind of machine:

	CPU only (container without GPU)	GPU passed through
Writing speed (eval_count ÷ eval_duration)	about 8 tokens/s	about 110 tokens/s
24-token summary	about 3.0 s	about 0.2 s
A 300-token handbook answer	about 38 s	under 3 s

Your numbers will differ by hardware, model and version, but the gap is usually ten times or more. A GPU sitting idle next to a container doing maths on the CPU is the most expensive mistake in this whole module, and Docker gives no warning.

Passing the GPU through

GPU passthrough means giving a container controlled access to the host’s graphics card. Remember where the analogy broke: containers share the host’s kernel, and the GPU’s driver lives on the host. So the host needs the driver, Docker needs a small helper that knows how to connect containers to it, and you have to ask for the GPU when you start the container.

NVIDIA on Linux (Crumb’s server). Three steps, done once:

1. Install the NVIDIA driver on the host. nvidia-smi on the host should list the card.
2. Install the NVIDIA Container Toolkit, the helper that connects containers to the driver. The Ollama Docker docs list the exact apt or yum commands for your distribution.
3. Tell Docker about it and restart Docker:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Then start Ollama with --gpus=all:

docker rm -f ollama
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama run llama3.1:8b "Say hi"
docker exec -it ollama ollama ps        # PROCESSOR should now say 100% GPU

docker logs ollama is worth reading once, too: near the start, Ollama logs which compute device it found. If it mentions no GPU, the passthrough is not working.

AMD on Linux. Ollama publishes a separate image with AMD’s ROCm libraries, and you pass the device files through instead of using --gpus:

docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama ollama/ollama:rocm

Windows. Docker Desktop runs Linux containers inside WSL 2 (Windows’ built-in Linux layer) and can pass NVIDIA GPUs through with --gpus=all on recent drivers. Sneha’s laptop has no NVIDIA card, so for her the container runs on the CPU, which is fine for trying things.

macOS. Here is the catch. Docker Desktop on a Mac runs containers inside a small Linux virtual machine, and that machine cannot reach the Apple GPU (Apple’s Metal graphics system). Containers on a Mac are CPU-only. The same llama3.2 that writes about 45 tokens per second in native Ollama on an M-series MacBook might manage around 12 inside Docker (illustrative numbers again). So on Macs, run Ollama natively, as in the last lesson, and use containers for everything else. Your app in a container can still reach native Ollama on the host at http://host.docker.internal:11434, a special name Docker Desktop provides for “the machine I am running on”.

Show the answer

Sneha’s laptop has no NVIDIA GPU, so there is nothing to pass through; CPU speed is the best it can do (a smaller model would help). Priya’s Mac has a capable GPU that Docker cannot reach; the fix is to run Ollama natively on the Mac and point the containerised app at host.docker.internal:11434. The server is already right. The lesson: “same container everywhere” gives you the same software everywhere, not the same speed.

One file for the whole kitchen: Docker Compose

The docker run commands were getting long, and the summariser app needed to run alongside Ollama. Typing both by hand, in the right order, with the right flags, is exactly the kind of thing Sneha would get subtly wrong.

Docker Compose describes several containers, their volumes, ports and settings, in one YAML file, and starts them together with one command. Think of it as the menu card for a thali: it lists every dish and where it sits on the plate, so anyone in the kitchen can assemble the same meal.

# compose.yaml
services:
  ollama:
    image: ollama/ollama:0.40.14          # pinned: the version we tested, not "latest"
    ports:
      - "127.0.0.1:11434:11434"           # reachable from this machine only
    volumes:
      - ollama:/root/.ollama              # models survive container rebuilds
    environment:
      - OLLAMA_KEEP_ALIVE=30m             # keep models loaded 30 min after last use
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia              # the compose way of saying --gpus=all
              count: all
              capabilities: [gpu]

  summariser:
    build: ./summariser                   # built from the Dockerfile in that folder
    environment:
      - OLLAMA_HOST=http://ollama:11434   # the service name, not localhost
    depends_on:
      - ollama

volumes:
  ollama:

Line by line, the new parts:

• image: ollama/ollama:0.40.14. The version current at the time of writing. latest is a moving label; pinning means Sneha’s laptop and the server run byte-for-byte the same Ollama until you choose to upgrade. Check Docker Hub or the Ollama releases page for current versions.
• 127.0.0.1:11434:11434. Publishes the port only on the host’s private loopback address. More on why in a moment.
• OLLAMA_KEEP_ALIVE=30m. The five-minute unload from the last lesson, stretched so the first order after a quiet spell does not wait for loading. The cost is GPU memory held the whole time.
• deploy.resources.reservations.devices. Compose’s spelling of GPU passthrough. On a Mac or a machine without NVIDIA, delete this block or the container will refuse to start.
• OLLAMA_HOST=http://ollama:11434. The trap in this file. Inside a container, localhost means that container itself, not the host and not the Ollama container. Compose puts all services on a private network where each service is reachable by its name. The Python ollama library reads OLLAMA_HOST, so the app needs no code change.

The summariser is the script from the last lesson with one addition, and a short Dockerfile (the recipe Docker follows to build an image):

# summariser/summarise.py
import ollama

client = ollama.Client()            # reads OLLAMA_HOST from the environment
client.pull("llama3.2")             # one-time download into the volume; quick if already there

reply = client.chat(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "Summarise bakery order notes as: ALLERGY: ... | DELIVERY: ... | OTHER: ..."},
        {"role": "user", "content": "Pls no nuts at ALL, my son (6) is allergic, also deliver after 5pm"},
    ],
    options={"temperature": 0},
)
print(reply["message"]["content"], flush=True)

# summariser/Dockerfile
FROM python:3.12-slim
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
WORKDIR /app
COPY pyproject.toml uv.lock ./
RUN uv sync --frozen
COPY . .
CMD ["uv", "run", "python", "summarise.py"]

Create summariser/pyproject.toml with uv init and uv add ollama inside that folder first. Then, from the folder holding compose.yaml:

docker compose up -d --build
docker compose logs summariser

summariser-1  | ALLERGY: yes, nuts (child, 6) | DELIVERY: after 5 pm | OTHER: none

The model is not baked into any image. It is pulled once into the ollama volume. Images stay small and quick to rebuild; the 2 GB lives where it survives. Sneha ran the same two commands on her laptop (after deleting the GPU block) and got the same line, slower. That was the point.

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks

Other things that bit Crumb in the first month:

• Silent fallback to CPU. One Saturday the server’s NVIDIA driver updated itself. The toolkit and driver disagreed, the container still started, and Ollama quietly ran on the CPU. Arjun noticed only because his log line showing tokens per second dropped from about 110 to 8. Log that number from every response and alert when it falls.
• Disk fills up. Every model is gigabytes. Three experiments, a few tags of each, and old images add up. docker system df shows image and volume usage; ollama rm and docker image prune clean up.
• GPU memory is one shared pool. With OLLAMA_KEEP_ALIVE=30m, an 8B model holds about 6 GB of the card’s 24 GB even when idle. Load a second and third model and the next one may not fit, so it waits or spills to the CPU. Decide which models the server is for.
• “Pinned” only covers what you pin. The Dockerfile above still uses uv:latest and python:3.12-slim, which move. For production, pin those too, and upgrade on purpose after testing.
• Containers do not make data private by themselves. The server is now the place the allergy notes live. Who can log in to it, and who backs it up, is a real question, and Module 24 returns to it.

Try it yourself

Prove to yourself that volumes are what saves the models.

1. Start Ollama with a volume: docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollama.
2. Pull a small model: docker exec -it ollama ollama pull llama3.2:1b (about 1.3 GB). Time it.
3. Delete the container: docker rm -f ollama. Start it again with the exact command from step 1.
4. Run docker exec -it ollama ollama list. Then run docker exec -it ollama ollama ps after one request, and read the PROCESSOR column.

Show the answer

After step 4, llama3.2:1b is still listed, with no second download, because it lives in the ollama volume. If you repeat the experiment without -v, the list is empty after step 3. The PROCESSOR column reads 100% CPU unless you are on Linux or Windows with an NVIDIA GPU, the toolkit installed and --gpus=all added; on a Mac it always reads CPU inside Docker.

Common mistakes

• No volume. Models vanish on every container rebuild and you pay for the download again. Always mount /root/.ollama.
• Forgetting --gpus=all (or the compose deploy block). Everything works, ten times slower. Check ollama ps after every setup change.
• Using localhost between containers. Inside a container it means the container itself. Use the compose service name, http://ollama:11434.
• Running Ollama in Docker on a Mac for speed. Docker cannot reach the Apple GPU. Run Ollama natively and containerise the rest.
• Publishing the port to the world. Ollama has no login. Bind to 127.0.0.1 or a private network.
• Using latest everywhere. The setup that worked on Monday changes under you on Friday. Pin versions and upgrade deliberately.

Recap

• An image is a sealed template, a container is a disposable running copy, a volume is storage that outlives containers, and a port mapping is a door in from the host.
• Mount a volume at /root/.ollama so multi-gigabyte models survive upgrades and rebuilds.
• GPU passthrough needs the host driver, the NVIDIA Container Toolkit and --gpus=all (or ollama/ollama:rocm with device flags for AMD); Docker on macOS cannot use the Apple GPU.
• Docker Compose puts Ollama and your app in one file; services reach each other by name, and the ollama library finds the server through OLLAMA_HOST.
• Bind Ollama’s port to 127.0.0.1, pin image versions, and watch ollama ps and tokens per second for a silent fall back to CPU.

On Friday Arjun showed Meera the server. “The whole AI setup is now one file. Sneha typed two commands and got the same answer as me. The models are kept on the server’s disk separately, so upgrading doesn’t mean downloading 7 GB again. And the graphics card is finally doing the work: the allergy summary takes a fifth of a second instead of three.”

Meera nodded, then asked the question Priya had been waiting for. “Lovely. Now, can I use it? Without typing whatever that was. The store managers keep phoning head office with handbook questions. I’d like them to open a page and ask.”

Go deeper

• Docker documentation, including Get Started and Compose: https://docs.docker.com
• The Ollama repository; docs/docker.mdx has the CPU, NVIDIA and AMD commands and the toolkit setup: https://github.com/ollama/ollama
• uv, including its guide to using uv in Docker: https://docs.astral.sh/uv/
• Ollama’s model library, to check sizes before you pull: https://ollama.com