Ollama as a local runtime

Meera had a list, and she read it out at the Monday meeting. “Customer addresses. Phone numbers. Allergy notes: ‘severe peanut allergy, child aged six’. Every one of those goes to an AI company’s servers when someone chats with our bot. And since last week’s gateway, it can go to three of them. Our lawyer asked me where that data lives. I didn’t know.”

Arjun started to explain data retention policies. Meera held up a hand. “Can we not send it anywhere at all? Can the AI live in our office?”

Priya nodded slowly. “For some jobs, yes. The open models have become good enough for narrow tasks like ‘summarise this allergy note’ or ‘sort this WhatsApp message into a category’. Not for everything. But we can try it on Arjun’s laptop this afternoon.”

By 3 p.m. Arjun had a model answering on his MacBook with the Wi-Fi switched off. Getting there took two wrong turns.

Renting a kitchen versus owning one

So far in this course, every model call has been like ordering from a central kitchen across town. You send the order (your prompt, the text you give the model), they cook it on their expensive ovens (their GPUs, the graphics chips that do the heavy maths of AI fast), and they send back the dish. You pay per dish, and your order slip passes through their building.

Running a model locally is like putting a small oven in your own bakery. You pay once for the equipment, there is no per-dish fee, and the order slips never leave the shop. But your oven is smaller: it cooks simpler dishes, maybe slower.

Where the analogy breaks: an oven does not change, but the “recipe” here (the model) is one big file you download, and you can swap it for a different one in a minute. Your hardware decides which recipes you can cook at all.

This is possible because of open-weights models. A model’s weights (also called parameters) are the billions of numbers it learned during training. Open-weights models publish those numbers for anyone to download and run: Meta’s Llama, Google’s Gemma, Mistral’s models, Alibaba’s Qwen, Microsoft’s Phi and others. Closed models such as Claude and GPT are only available through their companies’ APIs. “Open weights” is not always “open source”: Llama comes with Meta’s community licence, which has conditions. Read a model’s licence before you build on it.

Arjun’s first attempt

Arjun had read that Meta’s Llama 3.3, with 70 billion parameters, scored close to the big cloud models on several benchmarks. If the office was going to run its own AI, it might as well run the best one. He installed Ollama and typed:

ollama run llama3.3:70b

The download bar appeared: 43 GB. On the office line at about 100 megabits per second, that is 43 × 8 = 344 gigabits, or close to an hour. He went for chai. When he came back, the download had finished and the model refused to start, with an error along the lines of:

Error: model requires more system memory (41.5 GiB) than is available (11.8 GiB)

On some machines Ollama does not refuse. It spills over onto the slow disk and produces less than one word per second, which is worse than an error because it looks like it is working.

So he tried a small model, Llama 3.2 with 3 billion parameters. It downloaded in three minutes and answered instantly. He asked it something harder: “What is Crumb’s refund policy for a damaged Diwali hamper?” It replied with a confident, detailed policy. Crumb has never had that policy. The model had never seen Crumb’s policies at all. It filled the gap with something plausible, which is a hallucination: fluent text that is not true.

Two misconceptions in one afternoon: “bigger always works if I wait” and “local means Claude quality, only private”. Priya walked over with a marker.

Why 70 billion numbers do not fit in 16 GB

“How much does one number weigh?” Priya asked.

“Weigh?”

“In memory. Each weight is a number. How many bytes does the computer use to store it?”

Arjun thought. “Depends on the precision. A normal decimal number in Python is 8 bytes.”

“Models are usually stored more cheaply: 16-bit numbers, which is 2 bytes each. So a 70-billion-parameter model is…?”

“140 billion bytes. 140 GB.” He looked at his laptop. “It has 16.”

“And the model has to sit in memory, all of it, every time it produces a single word. It reads every weight to make every token.” (A token is a chunk of text, roughly three-quarters of an English word, that the model reads and writes one at a time.)

“Then how did the download say 43 GB instead of 140?”

“Because Ollama’s version was already squashed. That squashing is called quantization.”

Quantization means storing each weight with fewer bits. Instead of 16 bits per number, you keep about 4. Priya’s analogy was the shop’s ledger. You could record every sale to the exact paisa, or round each one to the nearest ₹10. The rounded ledger is much shorter, and the monthly total barely changes. Most weights in a model are like that: small rounding errors on each one wash out across billions of them.

Where the analogy breaks: squeeze too hard, to 3 or 2 bits, and the errors stop washing out. The model makes noticeably more mistakes, especially on reasoning and less common languages. Four bits is the common sweet spot.

The arithmetic for a few sizes:

Model	At 16 bits (2 bytes each)	At about 4.5 bits (Ollama’s usual download)
Llama 3.2, 3 billion	6 GB	about 2 GB
Llama 3.1, 8 billion	16 GB	about 4.9 GB
Llama 3.3, 70 billion	140 GB	about 43 GB

Why 4.5 bits and not 4? The common formats keep a few sensitive parts at higher precision and store small scaling numbers next to each block of weights, so the average lands a little above 4.

Running the model also needs working memory for the conversation, which grows with the context window (how many tokens the model can look at at once). You will meet that working memory, the KV cache, in Module 21. Rough rule: budget the file size plus 1 to 2 GB, and leave room for your operating system.

Show the answer

Only the 8B model at 4-bit (about 4.9 GB, plus a little working memory). The 8B model at 16 bits needs 16 GB for the weights alone, more than the 11 GB left over. The 70B model at 4-bit needs about 43 GB. Quantization did not make 70B small; it made it possible on a server, not on a laptop.

What Ollama actually does

Running a model yourself used to mean installing drivers, libraries and model files by hand. Ollama wraps all of that into one app:

• It downloads quantized models from its library with one command. Under the hood they are files in a format called GGUF, built for running on ordinary hardware.
• It runs them on your CPU (the computer’s general-purpose processor) or GPU, including the built-in GPU of Apple Silicon Macs.
• It serves them through a local web API on port 11434, so any program on the machine can use them. An API is a set of agreed requests one program can send another.
• It manages memory, loading a model when a request arrives and unloading it after five idle minutes by default.

Install it from ollama.com on macOS or Windows. On Linux, the official script is:

curl -fsSL https://ollama.com/install.sh | sh

Then the daily commands:

ollama pull llama3.2        # download the 3B Llama model, about 2 GB
ollama run llama3.2         # chat with it in the terminal; type /bye to leave
ollama list                 # models on disk
ollama ps                   # models loaded in memory right now
ollama stop llama3.2        # unload it from memory now
ollama rm llama3.2          # delete it from disk

ollama ps is the one beginners skip, and it answers the key question: is the model on the GPU or limping along on the CPU?

NAME               ID              SIZE      PROCESSOR    CONTEXT    UNTIL
llama3.2:latest    a80c4f17acd5    3.1 GB    100% GPU     4096       4 minutes from now

Notice 3.1 GB in memory for a 2 GB file: the extra is working memory for a 4,096-token context. 100% GPU is what you want. Something like 40%/60% CPU/GPU means the model did not fit in GPU memory and part of it runs on the much slower CPU.

Models live in ~/.ollama/models on macOS (other paths on Linux and Windows; the FAQ lists them). Each model name has a tag after the colon, such as llama3.2:1b or qwen2.5:7b, which picks the size and variant. No tag means latest.

Following one allergy note through Ollama

Here is the job Meera cares about. A customer typed this into the order form:

Pls no nuts at ALL, my son (6) is allergic, also deliver after 5pm

The kitchen wants one clean line on the ticket. Follow that note through every step.

Step 1: the download. ollama pull llama3.2 fetched about 2.0 GB, once per machine, not per request.

Step 2: the request. Arjun’s program sends this to http://localhost:11434/api/chat. localhost is the name every computer uses for itself, so the request never touches the network card.

{
  "model": "llama3.2",
  "messages": [
    {"role": "system", "content": "Summarise bakery order notes as: ALLERGY: ... | DELIVERY: ... | OTHER: ..."},
    {"role": "user", "content": "Pls no nuts at ALL, my son (6) is allergic, also deliver after 5pm"}
  ],
  "stream": false,
  "options": {"temperature": 0}
}

"stream": false asks for the whole answer at once instead of piece by piece. Temperature controls randomness when the model picks each next token; 0 means “always pick the most likely one”, which suits extraction jobs.

Step 3: loading. No model is in memory yet, so Ollama copies the 2 GB of weights from disk into memory. On Arjun’s MacBook this took about 1.8 seconds.

Step 4: reading the prompt. Ollama wraps the messages in Llama’s chat template (marker tokens such as “system message starts here”) and turns everything into 61 tokens, more than the word count because of those markers. The model reads all 61 in one pass, in under a tenth of a second.

Step 5: writing the answer. The model then produces tokens one at a time, each one requiring a full read of the weights, until it emits a “stop” token. Here that was 24 tokens.

Step 6: the response. With the timing numbers Ollama attaches. All durations are in nanoseconds (billionths of a second), so divide by 1,000,000,000 to get seconds.

{
  "model": "llama3.2",
  "created_at": "2026-11-02T09:41:07.512Z",
  "message": {
    "role": "assistant",
    "content": "ALLERGY: nuts, severe, child aged 6 | DELIVERY: after 5 pm | OTHER: none"
  },
  "done": true,
  "done_reason": "stop",
  "total_duration": 2412093000,
  "load_duration": 1812004000,
  "prompt_eval_count": 61,
  "prompt_eval_duration": 95120000,
  "eval_count": 24,
  "eval_duration": 498310000
}

Reading it: total 2.41 s, of which 1.81 s was loading, 0.10 s reading the prompt and 0.50 s writing 24 tokens, so 24 ÷ 0.498 ≈ 48 tokens per second on this laptop.

Sent again ten seconds later, load_duration was 9 milliseconds and the total 0.6 seconds: the model was still in memory.

Step 7: the cost. To compare with a cloud model, use made-up round prices: $1permillioninputtokensand$5 per million output tokens (real prices vary a lot by model; check the provider’s page). About 60 tokens in and 25 out costs 60 × $0.000001+25\times$0.000005 ≈ $0.000185,around₹0.016atroughly₹88tothedollar.Tenthousandnotesamonthwouldbeabout$1.85, roughly ₹160.

Locally, the extra cost per note is ₹0 in fees. The laptop drew perhaps 40 extra watts for 2.4 seconds, which is about 0.00003 kilowatt-hours. At ₹8 per unit of electricity, that is ₹0.0002 per note, about ₹2 for ten thousand.

Look at what that says. At Crumb’s volume the cloud bill for this job is tiny, and saving ₹160 a month does not justify a GPU server that costs lakhs. The reason to run this locally is not money. It is Meera’s list: the allergy note, the child’s age and the delivery time never leave the building.

Show the answer

It will be large again, around 1.8 seconds. Ollama keeps a model in memory for five minutes after the last request by default, then unloads it to free memory. After 40 idle minutes the model is back on disk, so the first request pays the loading cost again. You can change this with the keep_alive request field or the OLLAMA_KEEP_ALIVE server setting, at the price of holding the memory the whole time. You will tune this for the server in the next two lessons.

Talking to Ollama from Python

Typing curl commands is fine for a test. For code, the official ollama Python library is shorter.

uv add ollama

# summarise_note.py
import ollama

NOTE = "Pls no nuts at ALL, my son (6) is allergic, also deliver after 5pm"

reply = ollama.chat(
    model="llama3.2",
    messages=[
        {"role": "system", "content": "Summarise bakery order notes as: ALLERGY: ... | DELIVERY: ... | OTHER: ..."},
        {"role": "user", "content": NOTE},
    ],
    options={"temperature": 0},        # same knobs a Modelfile can set
)

print(reply["message"]["content"])
seconds = reply["eval_duration"] / 1e9  # nanoseconds to seconds
print(f"{reply['eval_count']} tokens in {seconds:.2f} s "
      f"= {reply['eval_count'] / seconds:.0f} tokens/s")

ALLERGY: nuts, severe, child aged 6 | DELIVERY: after 5 pm | OTHER: none
24 tokens in 0.50 s = 48 tokens/s

No API key, no .env file. The library talks to http://localhost:11434 unless the OLLAMA_HOST environment variable points somewhere else, which matters once Ollama moves to a server.

The same door the cloud uses

Ollama also speaks a second dialect. At http://localhost:11434/v1 it accepts requests in the same shape as OpenAI’s Chat Completions API. An OpenAI-compatible API is any server that copies that request and response format, so code written for OpenAI works against it by changing the address. Many tools and providers use this shape as a common plug.

# uv add openai
from openai import OpenAI

local = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # key required by the library, ignored by Ollama
resp = local.chat.completions.create(
    model="llama3.2",
    messages=[{"role": "user", "content": "Is a black forest cake usually eggless?"}],
)
print(resp.choices[0].message.content)

Remember gateway.py from Module 19, with its complete(system, messages, max_tokens) -> Reply and one adapter per provider? A local adapter is a few lines, because the request shape is already the OpenAI one:

def call_local(system: str, messages: list[dict], max_tokens: int) -> Reply:
    r = local.chat.completions.create(
        model="llama3.2", max_tokens=max_tokens,
        messages=[{"role": "system", "content": system}, *messages])
    return Reply(r.choices[0].message.content, "ollama", r.model,
                 r.usage.prompt_tokens, r.usage.completion_tokens)

Priya added one rule before Arjun wired it in. “Anything with personal data goes only to the local model. If the local model is down, that request waits or goes to a human. It never falls back to the cloud.” A fallback chain is about staying up; a privacy rule is about where data may go. When they disagree, privacy wins.

A recipe card for the model: the Modelfile

Arjun was pasting the same system prompt and temperature into every script. Priya showed him a Modelfile: a short text file that describes a customised model, naming the base model, the settings and a built-in system prompt. It is like a recipe card that says “start with the standard sponge, bake at 170 degrees, always add cardamom”. Anyone who asks for that card gets the same cake.

# Modelfile
FROM llama3.2

# Lower temperature: more predictable answers
PARAMETER temperature 0.2
# Context window in tokens
PARAMETER num_ctx 8192

SYSTEM """
You are Crumb's internal note summariser. Reply with one line in the form:
ALLERGY: <yes/no and what> | DELIVERY: <any timing request> | OTHER: <anything else>
"""

ollama create crumb-notes -f Modelfile
ollama run crumb-notes "Pls no nuts at ALL, my son (6) is allergic, also deliver after 5pm"

ALLERGY: yes, nuts (child, 6) | DELIVERY: after 5 pm | OTHER: none

crumb-notes now appears in ollama list, and any app can call it by name. It reuses the base model’s 2 GB of weights and stores only the small recipe. ollama show llama3.2 --modelfile prints the Modelfile of an existing model, the best way to learn the format.

The num_ctx line deserves attention. At the time of writing, Ollama defaults to a 4,096-token context on machines with less than 24 GB of GPU memory, and more on bigger machines (the default has changed between versions). If you send a long document and the model seems to ignore the beginning, this is usually why: the extra tokens were cut off without an error. Raising num_ctx fixes it and costs memory, which ollama ps will show you.

Where local models break

Arjun ran forty real notes from last month through crumb-notes. Thirty-six came out right. The four failures were instructive.

• Negation. “Not allergic to nuts, but NO eggs please” came back as ALLERGY: yes, nuts. Small models are noticeably weaker at “not”, “except” and “but”.
• Mixed language. “Bhaiya cake mein anda mat daalna” (Hindi for “don’t put egg in the cake”) came back as OTHER: none. A 3B model is much weaker in Hindi and Marathi, and in the Hinglish customers actually type, than the big cloud models.
• Facts it was never given. Like the refund policy, anything about Crumb’s own rules is invented unless you supply it in the prompt. That problem is what RAG solves, back in Module 09.
• Capacity. By default Ollama processes one request per model at a time and queues the rest. Thirty orders arriving together wait in line. Lesson 4 deals with that.

Then the costs that are not on any invoice. You own the hardware, its updates and its uptime: if the laptop is asleep, the “AI” is down. And Ollama’s API has no password at all. By default it listens only on 127.0.0.1, the machine’s private address for itself, which is safe. Sharing it with the office is the moment to think hard, and the next lesson shows how.

None of this makes local models useless. Choose jobs that fit: narrow, checkable tasks like classify, extract and summarise, with a human or a rule checking anything safety-critical. For allergy notes, the model’s line goes on the kitchen ticket next to the customer’s own words, never instead of them. Module 23 builds systematic tests for these failures.

Try it yourself

1. Pull two sizes of the same model: ollama pull llama3.2:1b and ollama pull llama3.2 (3B).
2. Ask both, through the Python script above, to summarise: “Order 1042, eggless chocolate cake, write Happy 7th Birthday Anaya, deliver Saturday before noon, gate code 4471.”
3. Print the answer and tokens per second for each. Then run ollama ps and compare the memory each one uses.

Show the answer

Typically the 1B model is faster and uses less memory (around 2 GB loaded versus about 3 GB), but is more likely to drop the gate code or “eggless”, or garble the name. The 3B model is slower and keeps more details. Size buys quality and costs memory and speed. Your numbers will vary by machine and version, which is why you test on your own tasks rather than trusting a benchmark table.

Common mistakes

• Pulling the biggest model first. It may not fit in memory and will refuse to start or crawl. Do the bytes arithmetic, start small, and move up only if the small one fails on your real examples.
• Assuming local means as good as the cloud. A 3B or 8B model is a capable assistant for narrow tasks, not a replacement for a frontier model on reasoning or rare languages.
• Forgetting the context window. Long inputs get cut silently at the default context. Set num_ctx in a Modelfile or in request options, and check ollama ps.
• Treating “local” as automatically private. Cloud-tagged models, an exposed port or an unlocked laptop all undo it.

Recap

• Open-weights models can run on your own hardware; Ollama downloads, runs and serves them at localhost:11434.
• Memory is the constraint: parameters times bytes per parameter. Quantization to about 4 bits shrinks an 8B model from 16 GB to about 4.9 GB, with a small loss in quality.
• pull, run, list, ps, stop and rm cover daily use; ollama ps tells you whether the GPU is doing the work.
• Call it through the native /api/chat, the ollama Python library, or the OpenAI-compatible endpoint at /v1, which lets the Module 19 gateway add a local provider in a few lines.
• A Modelfile bakes a base model, parameters and a system prompt into a named preset.
• Local models trade quality and capacity for privacy; pick narrow, checkable tasks and route personal data only to them.

At five o’clock Arjun explained it to Meera without a single acronym. “The AI now lives on my laptop. It’s a smaller brain than the one we rent, about as good as a sharp new trainee at reading notes, and much worse at anything clever. But nothing it reads leaves this machine. For a note like ‘no nuts, my son is six’, it gets it right most of the time, and we print the customer’s own words next to its summary so the kitchen can always check.”

“And the cost?” Meera asked.

“Two rupees of electricity for ten thousand notes. The real cost is that we now look after it ourselves.”

The next morning Arjun tried to show Sneha the setup on her Windows laptop. She had an older Ollama version, models in a different folder, no crumb-notes preset and a different Python. Twenty minutes later it still was not working. “We need this to run the same way everywhere,” Priya said, “including on the GPU server Meera has approved for the back office. Time for containers.”

Go deeper

• Ollama, including the model library with sizes and tags: https://ollama.com
• The Ollama repository; its docs folder has the API reference, the Modelfile reference and the FAQ: https://github.com/ollama/ollama
• Hugging Face, where most open-weights models and their licences are published: https://huggingface.co/docs
• Open versus closed models, and what is inside the weights file: