Creating embeddings in code

The new search box survived Diwali. “No results” pages fell from one in four searches to fewer than one in ten. Arjun is riding high, and he has a bigger plan: embed everything. Not only the 120 menu items, but all 18,000 customer reviews and all 41,000 support tickets from the last three years. Search the whole history of Crumb by meaning.

Before he starts, Meera leaves a sticky note on his monitor.

1. What does it cost? 2. Do our customers’ messages leave the building?

Arjun opens his mouth to say “basically nothing, and no”, then realises he does not actually know either answer. The support tickets contain names, phone numbers, delivery addresses and, now and then, a sentence about a child’s allergy. Where those go is not a small question.

Arjun’s overnight job

Arjun decides to try the paid route first, because it is the quickest to set up. He writes the most direct code he can think of:

from openai import OpenAI
import json

client = OpenAI(api_key="sk-proj-4f9...")        # pasted straight in, "for now"
tickets = json.load(open("tickets.json"))         # 41,000 tickets

vectors = []
for t in tickets:
    resp = client.embeddings.create(model="text-embedding-3-small", input=t["text"])
    vectors.append(resp.data[0].embedding)

He runs it at 6 p.m. and goes home. In the morning the terminal shows two things. First, it got through 6,212 tickets before dying with an error: 429 Too Many Requests. Second, at roughly a third of a second per request, it had been on course to take nearly four hours for the tickets alone.

Priya looks at the screen, then at the key pasted on line 4. “Three problems. You drove to the shop once for every single page. You got turned away for driving there too often. And that key is going into the GitHub repo the moment you commit.”

The print shop and the home printer

Priya sketches two boxes. “Think about printing. You have two options.”

“A print shop: you send them your file, they print it on a big expensive machine, you pay per page. Great quality, no machine to buy, but they see your documents. Or a home printer: you buy it once, every page after that is nearly free, nobody else sees what you print, but it’s slower and the quality is a step down.”

“So OpenAI is the print shop.”

“Right. And your local model is the home printer. Now, if you had 41,000 pages for the print shop, would you walk in with one page, wait, walk home, and walk back with the next?”

Arjun winces. “I’d bring a big stack each time.”

“That’s called batching: sending many texts in one request. And the shop turning you away is a rate limit: a cap on how many requests a service accepts per minute, so one customer can’t jam the machines for everyone else. Bring stacks, and when they say ‘slow down’, wait a bit and try again.”

“And Meera’s question about where data goes?”

“That is the home printer’s whole selling point. For the menu, nobody cares who sees it. It’s on the website. For support tickets with phone numbers and allergy details, a model on our own machine means the text never leaves Crumb. Let’s work out both properly, with real numbers, before choosing.”

Following one review through both routes

Take one real review and trace it through each road:

“Cake was lovely but the delivery came 2 hours late. Very disappointed, it was for my mum’s birthday.”

Route one: the print shop (OpenAI)

Step 1: count the tokens. A token is a chunk of text the model reads as one unit, roughly three quarters of an English word on average. This review is 20 words, about 25 tokens.

Step 2: send the request. Your code sends a small JSON message over the internet:

{"model": "text-embedding-3-small",
 "input": ["Cake was lovely but the delivery came 2 hours late. Very disappointed, ..."]}

Step 3: receive the vector. About 300 milliseconds later, back comes a list of 1,536 numbers, plus a note of how many tokens you were billed for:

{"data": [{"index": 0, "embedding": [0.0123, -0.0412, 0.0271, "... 1,533 more ..."]}],
 "usage": {"prompt_tokens": 25, "total_tokens": 25}}

Step 4: check its length. Square every number, add them up, take the square root: you get 1.0. OpenAI scales every vector to length exactly 1, which is called normalising. You will see in the next lesson why that is handy.

Step 5: the bill. At the time of writing, text-embedding-3-small costs about 0.02 US dollars per million tokens. 25 tokens costs 0.0000005 dollars, about ₹0.00004.

Route two: the home printer (local model)

Step 1: tokens again. The local model all-MiniLM-L6-v2 has its own tokenizer (the program that splits text into tokens), which cuts the review into about 27 pieces.

Step 2: no request. The text goes into a model file sitting on your laptop’s disk. Nothing crosses the internet.

Step 3: receive the vector. About 10 milliseconds later on a typical laptop processor, you get 384 numbers.

Step 4: normalise. This model does not normalise by default; you ask it to with one setting, and then its length is 1.0 too.

Step 5: the bill. Nothing per review. You paid once, in download time (about 90 MB) and in your laptop’s electricity.

Scaling up to all of Crumb’s history

Now multiply. Reviews average about 60 tokens, tickets about 45.

Time: in batches of 100, the OpenAI route needs 590 requests instead of 59,000, which takes minutes, not hours. The local model processes a few hundred short texts per second on a laptop processor, so it finishes in a few minutes too.

Storage is where the two differ more than you might expect. Each number is stored as a 4-byte decimal:

• OpenAI: 59,000 × 1,536 × 4 bytes ≈ 362 MB.
• Local: 59,000 × 384 × 4 bytes ≈ 91 MB.

So the honest answer to Meera’s first question is: about five rupees to embed three years of history with the paid model, nothing with the local one. The money is not the hard part. The privacy question is.

What happens inside each road

Open the hood one level on each.

The API road. An API (application programming interface, a way for your code to ask another computer to do something) receives your batch, runs each text through OpenAI’s embedding model on their servers, and returns the vectors in the same order you sent them. Facts worth knowing:

• Input limit. Each text can be up to about 8,000 tokens. Longer texts are rejected with an error, so you notice.
• Shorter vectors on request. Pass dimensions=512 and you get 512 numbers instead of 1,536. Quality drops a little; storage drops by two thirds. This works because the model was trained so the first numbers carry the most meaning, a technique nicknamed Matryoshka embeddings, after the Russian nesting dolls.
• Model retirement. Providers retire old models. When that happens, every vector you made with it must be remade with a new model. Keep your source texts.

The local road. The sentence-transformers library downloads a model’s weights (the millions of numbers that make up the trained network) from the Hugging Face Hub, a public library of models, once. After that it runs offline. Facts worth knowing:

• Short reading limit. all-MiniLM-L6-v2 reads at most 256 tokens of each text. Anything after that is silently ignored. No error, no warning.
• Batching still matters. The library processes texts in batches (32 at a time by default), because a processor, and especially a graphics card, is much faster doing many at once.
• Choice of models. There are thousands of open embedding models. A public scoreboard called MTEB (the Massive Text Embedding Benchmark), hosted on Hugging Face, ranks them on many tasks. Multilingual models such as paraphrase-multilingual-MiniLM-L12-v2 handle Hindi and mixed Hindi-English far better than the English-first starter model.

Building the embedder, one step at a time

Set up a project first. uv is a fast tool that manages Python projects and their packages:

uv init crumb-embeddings
cd crumb-embeddings
uv add openai sentence-transformers numpy python-dotenv

(With plain pip, the equivalent is pip install openai sentence-transformers numpy python-dotenv.)

Put your API key (the secret password that lets your code use a paid service) in a file called .env, and add .env to .gitignore before anything else:

OPENAI_API_KEY=sk-proj-...

Step 1: one batch, the right way

# embed_openai.py
import numpy as np
from dotenv import load_dotenv
from openai import OpenAI

load_dotenv()            # copies OPENAI_API_KEY from .env into the environment
client = OpenAI()        # reads the key from the environment; never paste it in code

EMBED_MODEL = "text-embedding-3-small"

texts = [
    "Cake was lovely but the delivery came 2 hours late. Very disappointed.",
    "Best croissant in Pune, flaky and buttery.",
    "Do you have eggless options for a birthday?",
]
resp = client.embeddings.create(model=EMBED_MODEL, input=texts)   # one request, three texts

vectors = np.array([item.embedding for item in resp.data], dtype=np.float32)
print(vectors.shape)                         # (3, 1536)
print(np.linalg.norm(vectors, axis=1))       # lengths: [1. 1. 1.]
print("tokens billed:", resp.usage.total_tokens)

(3, 1536)
[1. 1. 1.]
tokens billed: 38

resp.data comes back in the same order as texts, so the first vector belongs to the first text. np.float32 stores each number in 4 bytes instead of Python’s default 8, halving memory with no practical loss.

Step 2: batches and polite retries

Now handle 59,000 texts without getting turned away:

import time
from openai import RateLimitError

def embed_openai(texts, batch_size=100):
    vectors = []
    for start in range(0, len(texts), batch_size):
        batch = texts[start:start + batch_size]
        for attempt in range(5):
            try:
                resp = client.embeddings.create(model=EMBED_MODEL, input=batch)
                break                                   # success: leave the retry loop
            except RateLimitError:
                wait = 2 ** attempt                     # 1, 2, 4, 8, 16 seconds
                print(f"rate limited, waiting {wait}s")
                time.sleep(wait)
        else:
            raise RuntimeError(f"gave up on batch starting at {start}")
        vectors.extend(item.embedding for item in resp.data)
    return np.array(vectors, dtype=np.float32)

Waiting longer after each failure is called exponential backoff. The OpenAI library already retries a couple of times on its own, so this loop is a second safety net for big jobs. You will build sturdier versions in the Streaming, Retries and Fallbacks module.

Step 3: the local road

# embed_local.py
from sentence_transformers import SentenceTransformer

local_model = SentenceTransformer("all-MiniLM-L6-v2")    # downloads once, then cached

def embed_local(texts):
    return local_model.encode(
        texts,
        batch_size=64,                 # how many texts go through the model at once
        normalize_embeddings=True,     # scale every vector to length 1, like OpenAI's
        show_progress_bar=True,
    )

print(embed_local(["Best croissant in Pune, flaky and buttery."]).shape)   # (1, 384)

Step 4: one door for the rest of your code

The rest of Crumb’s code should not care which road you took. Give it one function, and record which model made the vectors:

import json
import numpy as np

PROVIDER = "local"     # or "openai"

def embed(texts):
    return embed_local(texts) if PROVIDER == "local" else embed_openai(texts)

def save(name, texts, vectors):
    np.save(f"{name}.npy", vectors)
    model = "all-MiniLM-L6-v2" if PROVIDER == "local" else EMBED_MODEL
    json.dump({"model": model, "count": len(texts), "dims": int(vectors.shape[1])},
              open(f"{name}.meta.json", "w"))

reviews = [r["text"] for r in json.load(open("reviews.json"))]
save("reviews", reviews, embed(reviews))

You should now see reviews.npy and reviews.meta.json. The next time you need these vectors, np.load("reviews.npy") takes a fraction of a second, and the metadata file tells you exactly which map they belong to. Embed once; load many times.

The ticket that lost its last sentence

Arjun picks the local road for tickets, for privacy. He builds a quick search and tests it with “does the cake contain nuts”. The results look good, except one ticket Sunita in customer care remembers clearly is missing. He finds it. It is a long, rambling message: three paragraphs about a delayed order and a wrong name iced on the cake, and then, at the very end: “Also my daughter has a nut allergy, please confirm the replacement has no nuts.”

The ticket is about 340 tokens. all-MiniLM-L6-v2 read the first 256 and ignored the rest. The allergy sentence never made it into the vector. No error was raised.

Here is the full list of ways this step goes wrong in real projects:

• Silent truncation. Check the limit (local_model.max_seq_length prints 256) and split long texts into overlapping pieces, called chunks. You will do this properly in the Vector Databases module.
• Empty or junk text. An empty string can be rejected by an API, and a ticket that is only “??” produces a vector that sits near nothing useful. Clean before embedding.
• Query and document prefixes. Some model families expect query: in front of searches and passage: in front of documents. The two models here do not, but read each new model’s card on Hugging Face before using it.
• Tiny number differences. Embedding the same text twice can give numbers that differ in the sixth decimal place, depending on hardware. Harmless for search; surprising if you compare vectors with ==.
• Re-embedding on every run. Slow, and on a paid API it costs money every time. Cache the vectors and only embed new or changed text.

Arjun answers the sticky note

Arjun writes his reply on a fresh sticky note and sticks it on Meera’s laptop.

1. Cost: about ₹5 to embed three years of reviews and tickets with OpenAI’s model. Zero with our local model. Searching is free either way.

2. Data: the menu and public reviews can go to OpenAI. Support tickets stay on our own server, using a local model, because they contain phone numbers and allergy details.

Meera reads it and comes to find him. “That’s clear. One thing: if the local model is free and private, why use OpenAI at all?”

“Quality, mostly,” Arjun says. “The paid model is usually better at subtle matches and at mixed languages. But ‘usually’ isn’t good enough. We should test both on our own data and pick the one that actually finds the right answers.”

Priya nods. “And to test them, you need to understand what you’re looking at. Right now you have 59,000 vectors and no idea what the space they live in looks like. What’s near what, what’s strange, what the model believes. That’s tomorrow.”

Try it yourself

1. Write eight sentences: four about late deliveries in different words (“my cake never arrived”, “order is two hours late”, …) and four about allergies (“does this contain peanuts?”, …).
2. Embed them with both models, plus the query “where is my order?”.
3. Print the similarity between the query and all eight, for each model.

Expected result: with both models, the four delivery sentences should score higher than the four allergy ones. The raw numbers will not match between the models at all, and the OpenAI scores may be lower overall. That is fine: only rankings within one model mean anything.

sentences = [...]                       # your eight sentences
query = "where is my order?"
for name, fn in [("local", embed_local), ("openai", embed_openai)]:
    vecs = fn(sentences + [query])
    scores = vecs[:-1] @ vecs[-1]       # vectors have length 1, so this is cosine similarity
    print(name, scores.round(2))

Common mistakes

• One request per text. Hours instead of minutes, and a rate-limit error halfway. Send batches of around 100.
• Pasting the API key into code. It ends up in Git history, where deleting the line does not remove it. Read it from the environment with python-dotenv.
• Comparing scores across models. 0.45 can be “very close” for one model and “barely related” for another.
• Ignoring the length limit. Local models drop the end of long texts without telling you. Measure lengths and chunk.
• Throwing away the source text. If you only keep vectors, you can never switch models. Always keep the original text next to its vector.
• Not recording the model name. Six months later, nobody remembers which model made vectors.npy. Save it alongside.

Recap

• An embedding call takes a batch of texts and returns one vector per text, in the same order.
• OpenAI’s text-embedding-3-small is paid, hosted, returns 1,536 numbers and reads about 8,000 tokens; all-MiniLM-L6-v2 is free, local, returns 384 numbers and silently stops at 256 tokens.
• At Crumb’s scale, embedding costs a few rupees; privacy, quality and switching costs are the real decisions.
• Batch your requests, back off when rate-limited, keep keys in the environment.
• Embed once, save vectors with the model’s name, and keep the original text so you can re-embed later.

That night, Arjun loads the 18,000 review vectors and tries to plot them. He gets as far as typing plt.scatter( and stops. Scatter plots have two axes. He has 384.

Go deeper

• OpenAI API documentation (the Embeddings guide covers models, dimensions and limits)
• Sentence Transformers documentation
• Hugging Face docs, where you can browse embedding models and their model cards
• uv documentation

Model names and prices change. The current list is on each provider’s models and pricing pages.