ChromaDB end to end

Every Crumb store manager has a copy of the store operations handbook: 200 pages covering everything from opening checklists to fire drills to how long a cream cake can sit in the display fridge. Nobody reads it. Instead, they phone head office. Meera counted last month: 128 calls, mostly from the six store managers, mostly with questions the handbook already answers. Each call takes ten minutes of someone’s time, usually hers.

“Make the handbook answer its own questions,” she tells Arjun. “A manager types ‘what’s the refund limit for a damaged delivery?’ and gets the right paragraph. Two things matter. The Baner store has its own closing procedure, so Baner should see Baner’s rules. And we update the handbook every month, so it has to stay current.”

Priya adds one line: “This is a perfect first job for Chroma. It runs on your laptop, it’s free, and it does the index, the metadata and the updates for you.”

Arjun’s first afternoon with Chroma

Arjun installs Chroma, reads the quick-start, and loads the handbook the fastest way he can think of: one record per page.

import chromadb

client = chromadb.PersistentClient(path="./crumb_db")
handbook = client.get_or_create_collection("handbook")

pages = load_handbook_pages("handbook_v7.pdf")          # 200 strings, one per page
handbook.add(ids=[str(i) for i in range(len(pages))], documents=pages)

res = handbook.query(query_texts=["how cold should cream cakes be stored overnight"],
                     n_results=3)
print(res["ids"][0], res["distances"][0])

['57', '42', '12'] [0.92, 0.97, 1.08]

Three problems surface over the next hour.

First, page 57 is about the display fridge layout. The answer he wanted is a single paragraph halfway down page 42, which also covers pastry storage, bread rotation and the freezer defrost schedule. One vector for a whole page points at the average of five topics and matches none of them sharply.

Second, he reads 0.92 as “92% match, excellent”. It is not. These are distances: smaller means closer. And they are measured with a ruler Arjun did not choose.

Third, when he fixes a typo in a page and runs the script again, the change never shows up in search. add with an ID that already exists does not replace the old record.

He takes all three to Priya.

The filing cabinet

Priya draws a filing cabinet. “Think of Chroma as a very well-organised filing clerk. The cabinet is the database. Each drawer is a collection: ‘handbook’, ‘menu’, ‘tickets’. Each folder in a drawer is a record, and every record has three things: a label with a unique ID, the document itself, and some tags on the outside, like ‘store: Baner, section: 4.3, version: 7’. Those tags are metadata.”

“And the clerk does the searching.”

“The clerk does the searching, by meaning. But here’s your first mistake: if you file the whole binder as one folder, the clerk can only hand you the whole binder. File each paragraph as its own folder, and you get the paragraph.”

“So: smaller records.”

“Smaller records. Second mistake: when you tell the clerk ‘here is a new folder with ID 42’, and there already is a 42, what should happen?”

“Replace it?”

“Only if you say ‘replace’. Chroma has two verbs: add means ‘this is new’, and upsert means ‘add it, or replace it if that ID is already here’. For a handbook that changes every month, you want upsert. And the tags let you tell the clerk ‘only folders tagged Baner or All stores’ before she starts looking.”

Following one paragraph from the handbook to an answer

Trace one paragraph through the whole system.

Step 1: the source. Section 4.3 of handbook version 7, on page 42:

4.3 Overnight storage of cream cakes. Cream cakes must be moved from the display counter to the walk-in chiller no later than 30 minutes after closing. The chiller must read between 2°C and 4°C… (about 340 words in total)

Step 2: chunk it. Chunking means cutting a long document into smaller pieces, each embedded separately. Arjun uses pieces of about 200 words with a 30-word overlap (the last 30 words of one chunk repeated at the start of the next), so a sentence cut at the boundary appears whole in at least one chunk. Section 4.3 becomes two chunks.

Step 3: give each chunk an ID and tags.

{"id": "handbook-s4.3-c0",
 "document": "4.3 Overnight storage of cream cakes. Cream cakes must be moved ...",
 "metadata": {"section": "4.3", "title": "Overnight storage of cream cakes",
              "page": 42, "store": "all", "version": 7}}

The ID is built from the section and chunk number, not invented randomly. You will see why that matters when the handbook changes.

Step 4: embed and store. Chroma’s default embedding function (the code it uses to turn text into vectors) runs all-MiniLM-L6-v2 locally: 384 numbers per chunk. The vector is inserted into the collection’s HNSW index, a graph that links each vector to its nearest neighbours so searches can hop instead of scanning everything. The text and metadata go into a small database file next to it.

Step 5: a question arrives. The Baner manager types: “how cold should the fridge be for cream cakes overnight”. Chroma embeds it with the same embedding function.

Step 6: filter, then search. The query is restricted to records whose store is "all" or "baner", then the index finds the nearest vectors among them.

Step 7: the answer comes back with IDs, documents, metadata and distances:

handbook-s4.3-c0   distance 0.31   "4.3 Overnight storage of cream cakes. Cream cakes must be..."
handbook-s4.3-c1   distance 0.48   "...if the chiller reads above 4°C for more than 20 minutes..."
handbook-s4.1-c0   distance 0.62   "4.1 Display counter temperatures. The display fridge..."

The collection was created with the cosine distance, where distance = 1 − cosine similarity. So 0.31 means a cosine similarity of 0.69: a strong match for this model.

Step 8: what it costs. The whole handbook is about 90,000 words. At roughly 170 new words per chunk (200 minus the 30 overlap), that is about 530 chunks.

• Embedding with the local default model: free, and about half a minute on a laptop.
• Embedding with OpenAI text-embedding-3-small instead: about 106,000 words of text including overlap, roughly 140,000 tokens. At about 0.02 US dollars per million tokens, that is 0.0028 dollars, around 25 paise.
• Storage: 530 chunks × 384 numbers × 4 bytes ≈ 0.8 MB of vectors, plus the text. It fits on anything.
• Queries: each query is embedded locally for free, and the search itself takes a few milliseconds.

Opening the hood: what Chroma is made of

Clients. You talk to Chroma through a client. Three kinds matter:

• chromadb.PersistentClient(path=...) runs Chroma inside your Python process and saves to a folder. Perfect for prototypes, notebooks and single-process apps.
• chromadb.HttpClient(host=..., port=...) talks to a Chroma server, which you start with chroma run --path ./crumb_db. Use this when several processes or services need the same data.
• Chroma also offers a hosted cloud service if you would rather not run a server. Its pricing is on Chroma’s website.

Collections. A collection is a named set of records sharing one embedding function and one distance setting. Each collection has its own HNSW index for vectors, and stores documents and metadata alongside.

Records. Every record has an id (a string, unique within the collection), an embedding, and optionally a document (the text) and metadata (a dictionary of simple values: strings, numbers and true/false).

Embedding functions. If you pass text, Chroma embeds it for you using the collection’s embedding function, the local all-MiniLM-L6-v2 by default. You can choose another, such as OpenAI’s, or compute vectors yourself and pass them in directly. Whatever you choose, queries must be embedded the same way as the stored records.

Distance. Chroma supports squared Euclidean (l2, the default), cosine, and inner product (ip). It is set when the collection is created and cannot be changed afterwards. For text embeddings, choose cosine.

Filters. where filters on metadata with operators such as $eq, $ne, $gt, $gte, $lt, $lte, $in and $nin, combined with $and and $or. where_document filters on the text itself, for example {"$contains": "refund"}.

Building the handbook search, step by step

Step 1: install and create a collection

uv add chromadb

# handbook_db.py
import chromadb

client = chromadb.PersistentClient(path="./crumb_db")     # saves to this folder

handbook = client.get_or_create_collection(
    name="handbook",
    metadata={"hnsw:space": "cosine"},       # distance = 1 - cosine similarity
)
print(handbook.count())                       # 0 the first time

Newer Chroma versions also accept a configuration argument for index settings. Library APIs change quickly. If a name has moved, check the Chroma docs linked below.

You should now see a crumb_db folder appear and 0 printed.

Step 2: chunk and add the handbook

def chunk_words(text, size=200, overlap=30):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]

sections = load_sections("handbook_v7.md")    # [{"id": "4.3", "title": ..., "page": 42,
                                              #   "store": "all", "text": ...}, ...]
ids, docs, metas = [], [], []
for s in sections:
    for n, chunk in enumerate(chunk_words(s["text"])):
        ids.append(f"handbook-s{s['id']}-c{n}")
        docs.append(chunk)
        metas.append({"section": s["id"], "title": s["title"], "page": s["page"],
                      "store": s["store"], "version": 7})

handbook.upsert(ids=ids, documents=docs, metadatas=metas)   # upsert: safe to re-run
print(handbook.count())

531

Using upsert from the start means running the script twice does not create duplicates or errors. A dedicated text splitter, such as LangChain’s RecursiveCharacterTextSplitter, splits on paragraphs and sentences rather than word counts; you will use it in the RAG module.

Step 3: query and read the results properly

res = handbook.query(
    query_texts=["how cold should the fridge be for cream cakes overnight"],
    n_results=3,
    include=["documents", "metadatas", "distances"],
)
for id_, doc, meta, dist in zip(res["ids"][0], res["documents"][0],
                                res["metadatas"][0], res["distances"][0]):
    print(f"{1 - dist:.2f}  {id_:20s} p.{meta['page']}  {doc[:60]}...")

0.69  handbook-s4.3-c0     p.42  4.3 Overnight storage of cream cakes. Cream cakes must be...
0.52  handbook-s4.3-c1     p.42  ...if the chiller reads above 4°C for more than 20 minutes...
0.38  handbook-s4.1-c0     p.41  4.1 Display counter temperatures. The display fridge...

Every result is a list inside a list, because you can send several queries at once: res["ids"][0] holds the results for the first query. Converting distance to similarity with 1 - dist makes the numbers readable.

Step 4: filter by store, section and text

# Baner's manager sees rules for all stores plus Baner-specific ones.
res = handbook.query(
    query_texts=["closing procedure for the night shift"],
    n_results=3,
    where={"store": {"$in": ["all", "baner"]}},
)

# Two conditions need an explicit $and.
res = handbook.query(
    query_texts=["refund limit for damaged deliveries"],
    n_results=3,
    where={"$and": [{"store": {"$in": ["all", "baner"]}}, {"version": {"$gte": 7}}]},
)

# Only chunks whose text mentions rupees.
res = handbook.query(
    query_texts=["refund limit for damaged deliveries"],
    n_results=3,
    where_document={"$contains": "₹"},
)

You should now see the Baner closing procedure for the first query, while a Kothrud-only rule never appears.

Step 5: read, update and delete

The other three letters of CRUD (create, read, update, delete):

# Read: fetch by ID, or by metadata, without any similarity search.
rec = handbook.get(ids=["handbook-s4.3-c0"], include=["documents", "metadatas"])
print(rec["metadatas"][0])

baner_only = handbook.get(where={"store": "baner"}, include=["metadatas"])
print(len(baner_only["ids"]), "Baner-specific chunks")

# Update: change metadata (or documents) of existing records.
handbook.update(ids=["handbook-s4.3-c0"], metadatas=[{**rec["metadatas"][0], "reviewed": True}])

# Delete: by ID, or everything matching a filter.
handbook.delete(ids=["handbook-s9.9-c0"])
handbook.delete(where={"version": 6})        # remove all chunks from the old handbook

print(handbook.count())

get is plain lookup, no embeddings involved. It is how you check what is stored, debug a strange result, or build an admin page.

Step 6: choose your own embeddings

The default model is English-first. Crumb’s managers sometimes type in Hinglish, and you may want OpenAI’s stronger model. Two options:

import os
from chromadb.utils import embedding_functions

# Option A: let Chroma call OpenAI for you.
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key=os.environ["OPENAI_API_KEY"], model_name="text-embedding-3-small")
handbook_oa = client.get_or_create_collection(
    "handbook_openai", embedding_function=openai_ef, metadata={"hnsw:space": "cosine"})

# Option B: compute vectors yourself and pass them in.
vecs = my_embed(docs)                                   # your own function, shape (n, d)
handbook_own = client.get_or_create_collection("handbook_own", metadata={"hnsw:space": "cosine"})
handbook_own.upsert(ids=ids, documents=docs, metadatas=metas, embeddings=vecs.tolist())
res = handbook_own.query(query_embeddings=[my_embed([question])[0].tolist()], n_results=3)

Option B gives you full control and works with any model, at the cost of embedding the queries yourself, every time, with the same model.

Step 7: estimate the cost before you scale

Meera will ask. Answer with a small function rather than a guess:

def estimate(n_docs, words_per_doc, dims=384, usd_per_million_tokens=0.02,
             usd_to_inr=88, chunk_words=200, overlap=30):
    chunks = n_docs * max(1, round(words_per_doc / (chunk_words - overlap)))
    tokens = chunks * min(words_per_doc, chunk_words) * 1.33   # about 1.33 tokens per English word
    embed_inr = tokens / 1e6 * usd_per_million_tokens * usd_to_inr
    vector_mb = chunks * dims * 4 / 1e6
    return {"chunks": chunks, "embed_cost_inr": round(embed_inr, 2), "vector_mb": round(vector_mb, 1)}

print(estimate(n_docs=1, words_per_doc=90_000))                      # the handbook
print(estimate(n_docs=3_200_000, words_per_doc=40, dims=1536))      # five years of WhatsApp

{'chunks': 529, 'embed_cost_inr': 0.25, 'vector_mb': 0.8}
{'chunks': 3200000, 'embed_cost_inr': 299.62, 'vector_mb': 19660.8}

The handbook is free in every sense. The WhatsApp archive costs around ₹300 to embed once with OpenAI, but needs about 20 GB for vectors alone, before the HNSW index’s links and the text. Chroma itself is free; the real cost is a server with enough memory to hold that index. Price a machine with your cloud provider, add headroom, and that is your monthly bill. The exchange rate and token price here are assumptions, so update them.

Where it breaks: the two refund limits

The handbook search goes live for the six managers. In its second week, the Kothrud manager approves a ₹1,800 refund for a damaged delivery without calling head office. The new version 8 of the handbook, uploaded the Monday before, had raised the no-questions limit from ₹1,500 to ₹2,000, so that was fine. But the Aundh manager, the same day, refused a ₹1,700 refund, quoting the handbook.

Arjun looks at the Aundh manager’s search. Two chunks came back, one saying ₹1,500 and one saying ₹2,000. The section had been reorganised in version 8, so its chunk IDs changed, and his upload script had upserted the new chunks without deleting the old ones. Both versions were live. The old one happened to rank first.

Arjun’s fix: every upload now runs handbook.delete(where={"version": {"$lt": 8}}) after the new version is in, then asserts that only version 8 remains.

Other ways Chroma projects go wrong:

• Changing the embedding function on an existing collection. If the new model has different dimensions, adds fail. If it has the same dimensions, everything runs and the results are quietly nonsense. New model, new collection.
• Forgetting $and. Two keys in one where dictionary is an error; combine conditions with $and or $or.
• Distance set once. You cannot switch a collection from l2 to cosine later. Create a new collection and re-add the data.
• Several processes writing to one PersistentClient folder. It is built for one process. For a web app with workers, run the Chroma server and use HttpClient.
• No backups. The folder is the database. Stop the app and copy the folder, or back up the server’s data directory on a schedule.
• Metadata that is too rich. Keep metadata to simple values you will filter on. Big text belongs in the document; whole objects belong in your main database, referenced by ID.

Arjun explains it to Meera

“How does it work, in one breath?” Meera asks.

“We cut the handbook into about 530 short passages. Each passage gets a position on our map of meaning and a few labels: section, page, which store, which version. When a manager asks a question, we only look at passages labelled for their store and the current version, find the ones nearest to the question, and show the best three with the page number.”

“And when we change the handbook?”

“The upload replaces the changed passages and deletes the old version completely. After the refund mix-up, it also checks that only one version is left, and stops if it finds two.”

“What does it cost?”

“Nothing to run for the handbook. The software is free and the whole thing is under a megabyte. If we ever put all five years of WhatsApp messages in, the cost is a server with enough memory, around 20 gigabytes, and about ₹300 to embed everything once.”

Calls to head office drop from 128 in a month to 41. Meera notices before Arjun does.

Try it yourself

1. Take any long document you know well (a manual, a policy, your college handbook) and load it into a Chroma collection with section and page metadata, using upsert and stable IDs.
2. Ask five questions and print the top three results with similarity scores.
3. Change one section’s text, re-run the upload, and confirm with get that only the new text is stored.
4. Add a where filter and confirm that filtered-out records never appear.

Expected result: the right paragraph appears in the top three for most questions; after the update, get(where={"section": ...}) shows only the new chunks; filtered searches only return allowed records.

all_meta = handbook.get(include=["metadatas"])["metadatas"]
from collections import Counter
print(Counter(m["version"] for m in all_meta))     # expect a single version
print(Counter(m["section"] for m in all_meta).most_common(5))

Common mistakes

• One record per page or per file. Long records blur into averages. Chunk into passages of a few hundred words.
• Reading distances as percentages. They are distances: smaller is closer, and the scale depends on the distance setting.
• Using add for data that changes. Use upsert with stable IDs, and delete records that no longer exist.
• Mixing embedding models in one collection. One collection, one embedding function, forever.
• Leaving the default distance. For text embeddings, create collections with cosine distance from the start.

Recap

• Chroma stores records (ID, embedding, document, metadata) in collections, each with one embedding function, one distance setting and its own HNSW index.
• PersistentClient runs it inside your script; the Chroma server and HttpClient serve several processes.
• add, upsert, get, update, delete and query cover the full lifecycle; where filters on metadata and where_document on text.
• Chunk long documents, use stable IDs, and delete old versions when content changes.
• Chroma is free; the real costs are embedding (often pennies) and memory for the index (which grows with vectors × dimensions).

A week later, Meera has a bigger idea. Three partner cafés want Crumb’s ordering assistant for their own customers, each with their own menu and policies, and none allowed to see the others’ data. Arjun wonders how many Chroma servers that needs. Priya suggests a different tool: one built to be run for you, with walls between customers.

Go deeper

• Chroma documentation
• Chroma on GitHub
• LangChain documentation (text splitters, used in the RAG module)