Milvus and choosing a vector database

The café deal has made Crumb a little famous. Two weeks later, Meera takes a call from FeastBox, one of Pune’s food-delivery apps. FeastBox has 250 million customer reviews of restaurants and bakeries across India, and they want to pilot “search reviews by meaning” with Crumb’s team building it. If the pilot works, it becomes a real contract.

Meera’s accountant, Mr Deshpande, joins the next meeting. He has one question, and he asks it twice. “Which database will this run on, and what will it cost per month? I need a number, not a technology.”

Arjun now knows two vector databases: Chroma on a laptop and Pinecone as a service. He suspects neither is clearly right for 250 million reviews. He also suspects Mr Deshpande will not accept “it depends”.

Arjun picks the fastest one on the internet

Arjun does what many engineers do first: he searches for “fastest vector database benchmark”. He finds a blog post with a striking bar chart. One database, let’s call it Database X, answers 9,000 queries per second; the others manage between 1,500 and 4,000. The post is confident and well designed. He drafts an email to Meera recommending Database X.

Priya reads the draft and asks four questions.

“How many vectors in the benchmark?”

He checks. One million.

“What recall?”

He checks again. The chart is at 0.90 recall. Each database was tuned to find 90% of the true nearest neighbours.

“Any metadata filters?”

None.

“Who wrote the post?”

The company that makes Database X.

“So the chart tells you which database is fastest at one million unfiltered vectors at 90% recall, according to the people selling it,” Priya says. “FeastBox has 250 million vectors, wants every search filtered by city and restaurant, and Sunita’s experience says we need at least 97% recall. The chart doesn’t describe our problem at all.”

Choosing a vehicle

Priya draws four vehicles on the whiteboard.

"A scooter. Cheap, you can start it in the car park, perfect for getting around town. That’s Chroma: runs on your laptop, nothing to set up.

"A taxi. You don’t own it, don’t maintain it, pay per ride, and someone else worries about servicing. That’s Pinecone, or any managed service.

"A fleet of lorries. Built to move huge loads, very efficient per tonne, but you need a depot, drivers and mechanics, or you hire a company that runs the fleet for you. That’s Milvus.

“And the car already parked in your garage. That’s the database you already run, with a vector add-on. We’ll come to that one tomorrow.”

“So which vehicle do I pick?”

“Which journey are you making? You don’t pick a lorry to buy milk, and you don’t pick a scooter to move house. Crumb has four journeys at the moment. Let’s work out what each one needs.”

Following one workload through the decision

Take the hardest journey, FeastBox’s 250 million reviews, and work out what it needs before naming any database.

Question 1: how many vectors, and how big? Reviews average about 50 tokens, so each is one chunk: 250 million vectors. With text-embedding-3-small at its full 1,536 dimensions, stored as 4-byte floats:

$$250,000,000\times 1,536\times 4\text{ bytes}\approx 1.54\text{ TB}$$

That is the raw vectors alone, before any index.

Question 2: can we shrink it? Two levers from the indexing lesson, applied one at a time:

Choice	Bytes per vector	Total for 250 million
1,536 dimensions, full floats	6,144	1.54 TB
Ask the model for 768 dimensions	3,072	768 GB
768 dimensions, scalar quantised to 1 byte	768	192 GB
768 dimensions, product quantised to 96 bytes	96	24 GB

An HNSW graph also stores links, roughly 128 extra bytes per vector with typical settings, which would add about 32 GB on its own. So an all-in-memory HNSW index over full vectors is out of the question here. A compressed index that keeps full vectors on disk for rescoring is the realistic shape.

Question 3: what does embedding cost? 250 million reviews × 50 tokens = 12.5 billion tokens. At about 0.02 US dollars per million tokens, that is about 250 dollars, around ₹21,000, once. At an assumed throughput of a few million tokens per minute, it takes days, not hours, so it must be a resumable batch job.

Question 4: what does every query need? A filter by city and often by restaurant, at least 97% recall measured against exact search, and under 200 ms at around 100 queries per second during the pilot.

Question 5: isolation? Not between customers this time, but FeastBox requires the data to stay in an Indian data centre. That rules out any service without a region in India.

Question 6: who runs it? Crumb has one engineer, Arjun, and part of Priya. Running a cluster of database machines with on-call duty is not realistic for them alone.

Now, and only now, compare options against those answers:

• Chroma is excellent up to millions of vectors on one machine. 250 million, compressed, with heavy filtering and high availability, is beyond what it is designed for today.
• Pinecone would run it for them, but the data must stay in India: check whether a suitable region is available on their plan, and estimate storage and read costs at this size with their calculator.
• Milvus was designed for this size. Self-hosted on Kubernetes, it is a serious operational job. Its company also offers a managed version, Zilliz Cloud; check its available regions.
• pgvector, inside PostgreSQL, can reach surprisingly large sizes, but 250 million filtered vectors with high recall would need careful tuning and big machines. Worth knowing; probably not the first choice here.

The shortlist for FeastBox is “Milvus, self-hosted or managed” and “a managed service with an Indian region”. The final answer comes from a test, not a chart.

Show the answer

Reducing dimensions (1,536 to 768) halved the size, at a small loss of quality that you should measure on your own search tests. Product quantisation (768 bytes down to 96) did the most, an eight-fold cut, at a larger loss of recall, which you win back by rescoring a shortlist with full vectors stored on disk. Neither is free; both are routinely worth it at this scale. The order matters too: first decide how many dimensions you need, then how aggressively to compress them.

Opening the hood: what Milvus is

Milvus is an open-source vector database, started in 2019 and now hosted by the LF AI & Data Foundation, built to scale from a laptop to billions of vectors. It comes in four shapes that share one API:

• Milvus Lite: a Python library that stores everything in a local file. For notebooks, tests and prototypes.
• Milvus Standalone: one server, usually run with Docker. For small production workloads.
• Milvus Distributed: many cooperating services on Kubernetes (a system for running and scaling containers across many machines). For very large workloads.
• Zilliz Cloud: a managed service run by the company that created Milvus.

How the distributed version is built

Milvus Distributed separates the jobs that a single-machine database does all at once:

flowchart LR
  C["Your app (pymilvus)"] --> P["Proxy"]
  P --> L["Log (message queue)"]
  L --> D["Data nodes"]
  D --> S[("Object storage")]
  S --> I["Index nodes"]
  I --> S
  S --> Q["Query nodes"]
  P --> Q
  M["Metadata store"] --- P

• The proxy receives your requests.
• The log, a message queue, records every insert and delete in order, so nothing is lost.
• Data nodes turn the log into segments (chunks of data) and save them to object storage such as Amazon S3 or MinIO, which is cheap and durable.
• Index nodes build indexes for sealed segments.
• Query nodes load segments and their indexes and answer searches. Need more search capacity? Add query nodes, without touching the rest.
• A metadata store (etcd) keeps track of which segments exist and where.

That separation is why Milvus scales so far: storage, indexing and search grow independently. It is also why running it yourself means running a message queue, object storage, etcd and Kubernetes, and keeping all of them healthy.

The concepts you will use

• Collection and schema. A collection is like a table, with a schema that declares its fields: a primary key, one or more vector fields with a fixed dimension, and ordinary fields such as city or rating.
• Index types. Milvus offers almost every index from the theory lesson: FLAT, IVF_FLAT, IVF_SQ8 (IVF with scalar quantisation), IVF_PQ, HNSW, and DISKANN, a graph index designed to keep most data on fast disks instead of memory. GPU-based indexes are available too.
• Metrics. COSINE, IP (inner product) and L2 (Euclidean). With COSINE and IP, larger scores mean more similar.
• Partitions and partition keys. A collection can be split into partitions. Marking a field such as tenant or city as the partition key makes Milvus route each record to a partition by that field, which speeds up filtered searches and supports multitenancy.
• Filters. Boolean expressions written as strings: 'city == "Pune" and rating <= 2'.
• Hybrid search. Dense vectors plus sparse (keyword-style) vectors in one collection, combined at query time. You will meet the idea properly in the RAG module.

Trying Milvus on a laptop

Milvus Lite lets you learn the API without any servers.

Step 1: install and create a collection

uv add pymilvus sentence-transformers

On some versions, Milvus Lite is a separate optional extra (pymilvus[milvus-lite]). Library APIs change quickly. If a name has moved, check the Milvus docs linked below.

# reviews_milvus.py
from pymilvus import MilvusClient
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
client = MilvusClient("crumb_milvus.db")             # Milvus Lite: one local file

if not client.has_collection("reviews"):
    client.create_collection(
        collection_name="reviews",
        dimension=384,                                # must match the embedding model
        metric_type="COSINE",
    )

This “quick setup” creates a collection with an integer id, a vector field and room for extra fields you add on insert.

Step 2: insert with metadata fields

reviews = [
    {"id": 1, "text": "Cake arrived squashed, box was crushed", "city": "Pune", "rating": 1},
    {"id": 2, "text": "Beautiful packaging, perfect for gifting", "city": "Pune", "rating": 5},
    {"id": 3, "text": "Delivery late by an hour, cake melted", "city": "Mumbai", "rating": 2},
    {"id": 4, "text": "Lovely croissants, fresh and flaky", "city": "Pune", "rating": 5},
]
vectors = model.encode([r["text"] for r in reviews], normalize_embeddings=True)
client.insert(collection_name="reviews",
              data=[{**r, "vector": v.tolist()} for r, v in zip(reviews, vectors)])

Step 3: search with a filter

q = model.encode(["damaged packaging"], normalize_embeddings=True)[0]
res = client.search(
    collection_name="reviews",
    data=[q.tolist()],                       # a list: you can search several queries at once
    limit=2,
    filter='city == "Pune" and rating <= 2',
    output_fields=["text", "city", "rating"],
)
for hit in res[0]:                            # results for the first query
    print(round(hit["distance"], 2), hit["entity"]["text"])

0.58 Cake arrived squashed, box was crushed

Only one review passes the filter: Pune, rated 2 or below. With the COSINE metric, the distance field is a similarity, so higher is better, the opposite of Chroma’s cosine distance. Every database names and scales its scores differently.

Step 4: query, update and delete

# Plain filtered lookup, no vectors involved.
low = client.query(collection_name="reviews", filter="rating <= 2", output_fields=["text"])

# Replace a record by primary key.
client.upsert(collection_name="reviews",
              data=[{"id": 2, "text": "Beautiful packaging", "city": "Pune", "rating": 4,
                     "vector": model.encode(["Beautiful packaging"], normalize_embeddings=True)[0].tolist()}])

# Delete by key or by expression.
client.delete(collection_name="reviews", ids=[4])
client.delete(collection_name="reviews", filter='city == "Mumbai"')

Step 5: what production setup looks like

On Milvus Standalone or Distributed you declare the schema and index explicitly. Milvus Lite supports only a basic subset of index types, so try this on a Docker install:

from pymilvus import DataType, MilvusClient

client = MilvusClient(uri="http://localhost:19530")      # Milvus Standalone in Docker

schema = MilvusClient.create_schema(auto_id=False)
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("vector", DataType.FLOAT_VECTOR, dim=768)
schema.add_field("city", DataType.VARCHAR, max_length=64, is_partition_key=True)
schema.add_field("rating", DataType.INT64)

index_params = client.prepare_index_params()
index_params.add_index(field_name="vector", index_type="IVF_PQ", metric_type="COSINE",
                       params={"nlist": 4096, "m": 96, "nbits": 8})

client.create_collection("feastbox_reviews", schema=schema, index_params=index_params)

These few lines encode every decision from the worked example: 768 dimensions, product quantisation with 96 one-byte pieces, cosine similarity, and the city as partition key so filtered searches touch less data. At search time, nprobe in the search parameters trades speed for recall exactly as you measured with FAISS.

The questions that pick a database

Here is the checklist Arjun now uses for every project, with the reason behind each question.

1. How many vectors now, and in two years? Under a million: almost anything works, including Chroma or pgvector on a small server. Tens of millions: compression and index choice start to matter. Hundreds of millions or more: you need a system designed to scale out.
2. Who will run it? If nobody wants pager duty, choose a managed service. If you have a platform team, self-hosting can be cheaper at scale.
3. Where must the data live? Privacy rules or contracts may require a country or your own servers. That can rule options in or out immediately.
4. What filtering do you need? Heavy filtering (“this restaurant, this month, rating below 3”) is where databases differ most. Test with your real filters.
5. How do you separate customers? Namespaces, partitions, separate collections or row-level rules. Pick a design where a forgotten line of code cannot leak data.
6. Do you need keyword and vector search together? Some databases support hybrid search natively; others need a second system.
7. Do you already run a database that can do vectors? Fewer systems means fewer things to back up, secure and keep in sync.
8. What will it cost, all in? Managed fees, servers, storage, and engineer time. An engineer’s week is a real cost.

The options at a glance

	Chroma	Pinecone	Milvus	pgvector (PostgreSQL)
What it is	Open-source, embedded or server	Fully managed service	Open-source, Lite to distributed; managed via Zilliz Cloud	Extension for PostgreSQL
Easiest start	uv add chromadb	Sign up, API key	Milvus Lite file	CREATE EXTENSION vector
Sweet spot	Prototypes to millions of vectors	Teams that want zero operations	Hundreds of millions to billions	Apps already on Postgres, up to many millions
Who runs it	You	Pinecone	You, or Zilliz	You, or your Postgres host
Multitenancy	Collections, metadata	Namespaces	Partition keys, collections	Rows, schemas, row-level security
Watch out for	Single-machine scale	Region, pricing at scale, lock-in	Operating the distributed version	Tuning at very large scale

Others you will meet: Qdrant and Weaviate (open-source with managed offerings), Elasticsearch and OpenSearch (search engines with vector support, strong at hybrid search), Redis (in-memory, very fast for smaller sets) and LanceDB (embedded, file-based). The same eight questions apply to all of them.

Show the answer

At least these: the recall each system was tuned to (speed at 0.90 recall says little about speed at 0.98); the dataset (size, dimensions, whether it resembles your data); and the workload (whether it included metadata filters, concurrent writes and your query rate). Also check who ran it and on what hardware. Then run your own small test with your real data, filters and recall target. A half-day benchmark of your shortlist beats any chart.

Where it breaks

Arjun runs a two-day test: 5 million FeastBox reviews (a sample), the same embedding model, the same 300 real queries with filters, on Milvus Standalone and on a managed service. He measures recall against an exact search, latency at 100 queries per second, and memory.

The results are close on speed. The surprise is elsewhere. On Milvus, his first attempt at a filtered search for one small restaurant returns only 4 results when he asked for 10. The partition key was set to city, so every Pune restaurant shared one partition, and the filter on a single restaurant left very few candidates for the index to work with. Changing the search parameters and testing again fixes it. Without the test, the pilot would have launched with it.

Other common traps:

• Choosing for the scale you might have in five years. Start with what fits now, with a migration path. Keep source text and IDs so you can re-embed and re-load into anything.
• Lock-in through convenience features. Database-specific embedding services and query languages are pleasant until you want to leave. Keep your embedding step in your own code if portability matters.
• Ignoring backups and disaster recovery. “Managed” usually includes them; “self-hosted” means you build them.
• Two sources of truth. If product data lives in one database and vectors in another, they drift apart. Plan how updates flow, or keep them together.

Arjun explains it to Mr Deshpande

Arjun’s one-page memo does not name a winner at the top. It starts with a table of Crumb’s four workloads:

Workload	Size	Recommendation
Store handbook	530 chunks	Keep on Chroma (or move into our main database)
Café assistants	about 1,200 chunks, 3 tenants	Keep on Pinecone with namespaces
WhatsApp archive	3.2 million	Decide after testing the option in our existing database
FeastBox pilot	250 million	Milvus via a managed provider with an Indian region, pending a 2-week paid trial

Mr Deshpande reads it twice. “You’ve chosen three different databases.”

“Different journeys,” Arjun says. “The handbook is tiny; anything works. The cafés need walls between customers and no servers for us to run. FeastBox is a thousand times bigger than anything else we have, needs to stay in India, and needs a system built for that size. Running that one ourselves would need someone on call at night, which costs more than the managed fee.”

“And the number I asked for?”

“For FeastBox: about ₹21,000 once, to create the vectors, plus the provider’s monthly fee for roughly 200 GB of compressed data, which their calculator puts at a figure I’ve attached for both providers we tested. I’ll confirm it after the trial with real usage.”

Mr Deshpande nods, which from him is high praise.

Priya, reading the memo later, taps the third row. “The WhatsApp archive. Where do our orders already live?”

“PostgreSQL.”

“Then before you add a fourth database to Crumb, try the one we already pay for.”

Try it yourself

1. Install Milvus Lite and recreate the review example with at least 50 reviews of your own, with city and rating fields.
2. Run three searches with different filters and confirm the filters are respected.
3. Fill in the eight-question checklist for a project you care about (a college notes search, a shop’s product search, your company’s help desk) and write a three-sentence recommendation.

Expected result: filtered searches only return matching records; your recommendation names one database and gives at least two reasons tied to the checklist, not to popularity.

Show an example recommendation

“College notes search, 40,000 chunks, two maintainers, must run on the department’s existing PostgreSQL server: use pgvector. It fits easily at this size, adds no new system to maintain, and lets us filter by course and year with ordinary SQL. Revisit if we pass ten million chunks.”

Common mistakes

• Choosing by stars or charts. Popularity and vendor benchmarks are not your workload. Test.
• Ignoring operations. A database nobody can upgrade or restore is a liability, however fast it is.
• Forgetting data residency. Check where the service stores data before you load anything sensitive.
• Over-building. A billion-vector architecture for 50,000 chunks wastes months.
• Not keeping an exit. Keep source texts, IDs and your embedding code, so moving databases is a re-load, not a rewrite.

Recap

• Milvus is an open-source vector database that runs as a local file (Lite), a single server (Standalone) or a distributed cluster, with a managed version from Zilliz.
• Its distributed design separates storage, indexing and search, so each can scale on its own, at the cost of running several supporting systems.
• Collections have schemas, vector fields, index types from FLAT to IVF_PQ, HNSW and DISKANN, filters as expressions, and partition keys for multitenancy.
• Choose a database by workload: size, who runs it, data location, filtering, isolation, hybrid needs, existing systems and full cost including people’s time.
• Never trust a benchmark you did not run; test your shortlist with your own data, filters and recall target.

That evening, Arjun opens Crumb’s order database and looks at the orders table, with its 3.2 million WhatsApp messages linked to customers, stores and dates. Everything a search would need to filter on is already there, in columns. He wonders what it would take to put the vectors right next to them.

Go deeper

• Milvus documentation
• Pinecone documentation
• Chroma documentation
• pgvector on GitHub