What are embeddings?

It is the Monday before Diwali, and Crumb’s online shop is busier than it has ever been. At 10:40 a.m. Meera forwards Arjun a screenshot from a customer. The customer had typed eggless chocolate cake into the search box on the website. The website had replied: No results found.

“We sell an eggless chocolate cake,” Meera writes. “It is our third best seller. Why is our own website telling people we don’t?”

Arjun pulls the search logs. Last week, customers ran 1,140 searches. 262 of them, almost one in four, returned nothing. He scrolls through the failures: eggless chocolate cake, something for my boss diwali, no sugar sweets for dad, anda nahi wala cake. Crumb sells something that fits almost every one. Crumb’s average online order is ₹780. Even if only a third of those 262 people would have bought something, that is around ₹68,000 a week walking out of the door.

“Fix it before the weekend,” Meera says. “Diwali orders close on Thursday.”

The search box that only reads letters

Arjun opens the code behind the search box. It is one line of SQL (the language used to ask a database questions):

SELECT name FROM menu_items WHERE name ILIKE '%' || :query || '%';

ILIKE '%eggless chocolate cake%' means “find items whose name contains these exact letters, in this order, ignoring capitals”. The cake is called Egg-free Belgian Chocolate Gateau. The letters “eggless chocolate cake” appear nowhere in that name. So: no results.

This is keyword search (matching the exact words a person typed against the words stored in your data). It is fast, cheap and predictable. It is the right tool for finding order number CRB-10293 or a customer’s phone number. But it has no idea what words mean. To keyword search, “eggless” and “egg-free” are as different as “eggless” and “bicycle”.

Arjun’s synonym list

Arjun has a plan, and it takes him forty minutes. He splits the query into words, and for every word he looks up a list of synonyms he typed in by hand.

SYNONYMS = {
    "eggless": ["egg-free", "no egg", "without egg", "vegan"],
    "cake":    ["gateau", "torte", "sponge"],
    "gift":    ["hamper", "box", "present"],
}

def expand(query):
    words = query.lower().split()
    expanded = set(words)
    for w in words:
        expanded.update(SYNONYMS.get(w, []))
    return expanded

def keyword_search(query, menu_names):
    terms = expand(query)
    return [name for name in menu_names if any(t in name.lower() for t in terms)]

He tries it. eggless chocolate cake now finds the gateau. He is pleased for about ninety seconds. Then he tries the rest of the failed searches.

• anda nahi wala cake (“the cake without egg”, in Hindi written in English letters): finds every cake on the menu, eggs and all, because only “cake” matched.
• something for my boss diwali: finds nothing. “Boss” is not in the list.
• eggless chocolate cake, looked at more carefully: it also returns Egg Puff, a savoury pastry that is mostly egg. The word “egg” from “egg-free” matched the letters “Egg” in “Egg Puff”.

That last one is worse than “no results”. A customer who avoids eggs is now being shown an egg puff as a match.

Priya has been reading over his shoulder. “How long will that list be by Christmas?” she asks. “Every language your customers mix, every way of saying ‘gift’, every spelling mistake? You are trying to teach a search box what words mean, one rule at a time. That never finishes.”

“So what’s the alternative?”

“Stop matching letters. Start comparing meanings. Come to the whiteboard.”

Priya’s table of cakes

Priya draws a long rectangle. “Imagine this is a huge table in the middle of a store. Your job is to lay out every item on the menu so that similar things sit close together. Where does the butter croissant go?”

Arjun points to one end. “Breakfast end. With the other pastries.”

“And the almond croissant?”

“Right next to it. They’re almost the same thing.”

“The chocolate gateau?”

“Far from the croissants. Near the other cakes. The dessert end.”

“Good. Now a customer walks in and says, ‘I want something sweet for breakfast.’ Where would you stand them?”

Arjun thinks. “Between the croissants and the cakes. Somewhere near the pain au chocolat.”

“And what would you show them?”

“Whatever is nearest to where they’re standing.”

Priya underlines the word nearest. “That is the whole idea. If you can give every item a position, and give every query a position on the same table, then search becomes: ‘what is nearest to where the query landed?’ No shared words needed.”

Arjun frowns. “But a table has two directions. Left to right, front to back. Our menu has more going on than that.”

“Exactly. Say left to right is ‘how sweet’ and front to back is ‘how much of a breakfast thing’. Where do the eggless gateau and the normal chocolate truffle cake go?”

“In the same spot. Both sweet, both dessert.”

“So your customer who avoids eggs can’t tell them apart. You need a third direction: ‘contains egg’. And then a fourth for ‘chocolate’, a fifth for ‘savoury’, a sixth for ‘festive’…” She draws arrows coming out of the board. “Real systems use hundreds of directions. You can’t draw it. But the idea of ‘nearest’ still works, and the maths does not care how many directions there are.”

“Who decides what the directions are?”

“Nobody. That’s the clever part. A model learns them by reading an enormous amount of text. We’ll get to how. First, let’s make ‘nearest’ precise.”

From arrows to a score: cosine similarity

Each item’s position is a list of numbers, one per direction. A list of numbers like that is called a vector, and each number in it is a dimension. Think of a vector as an arrow drawn from the centre of the table out to the item’s spot.

You could measure closeness with a ruler between the two spots. In practice, the most common score for text is cosine similarity, which looks at the angle between two arrows and ignores how long they are.

• Arrows pointing the same way: score 1. Same meaning.
• Arrows at right angles: score 0. Unrelated.
• Arrows pointing opposite ways: score −1. Opposite.

The formula:

$$\cos (\theta )=\frac{a\cdot b}{\Vert a\Vert \Vert b\Vert }$$

It has two parts, and both are arithmetic you could do on paper.

• The top, $a\cdot b$, is the dot product: multiply the two vectors position by position, then add up the results.
• The bottom multiplies the two arrow lengths. The length of $a$ is $\Vert a\Vert =\sqrt{a_1^2+a_2^2+\dots }$, which is Pythagoras’ theorem stretched to any number of directions.

The smallest possible example

Take four items with only two numbers each:

Item	Vector	Length
Chocolate brownie	$a=(3,4)$	$\sqrt{9+16}=5$
Chocolate gateau	$b=(4,3)$	$5$
Masala bun	$c=(4,-3)$	$5$
“The opposite of a brownie”	$d=(-3,-4)$	$5$

Brownie vs gateau. Dot product: $3\times 4+4\times 3=12+12=24$. Lengths multiplied: $5\times 5=25$. Cosine similarity: $24/25=0.96$. Nearly the same direction.

Brownie vs masala bun. Dot product: $3\times 4+4\times (-3)=12-12=0$. Cosine similarity: $0/25=0$. Right angles. Unrelated.

Brownie vs its opposite. Dot product: $3\times (-3)+4\times (-4)=-9-16=-25$. Cosine: $-25/25=-1$. Exactly opposite.

Show the answer

The dot product doubles: $6\times 4+8\times 3=48$. But the brownie’s length also doubles, to $\sqrt{36+64}=10$. So the cosine is $48/(10\times 5)=0.96$, exactly as before. Cosine only cares about which way the arrow points, not how long it is. That is useful for text: a long, rambling review and a short one about the same thing should count as similar, even if something about their length nudges the size of their vectors.

Following one search all the way through

Now trace Meera’s failed search, eggless chocolate cake, through a meaning-based search, one step at a time. To keep the arithmetic on one page, pretend the embedding has only three dimensions, and pretend (for now) that they have names: chocolatey, sweet and contains egg, each from 0 to 1.

Step 1: every menu item already has a vector. This happened once, ahead of time.

Item	chocolatey	sweet	contains egg
Egg-free Belgian Chocolate Gateau	0.9	0.8	0.0
Classic Chocolate Truffle Cake	0.9	0.8	0.9
Butter Croissant	0.0	0.3	0.4
Egg Puff	0.0	0.1	0.9

Step 2: the query gets a vector too. The same model reads “eggless chocolate cake” and places it at $q=(0.9,0.7,0.0)$: very chocolatey, sweet, no egg. Its length is $\sqrt{0.81+0.49+0}=\sqrt{1.30}\approx 1.140$.

Step 3: score the query against every item.

Gateau. Dot product: $0.9\times 0.9+0.7\times 0.8+0\times 0=0.81+0.56+0=1.37$. Gateau length: $\sqrt{0.81+0.64+0}=\sqrt{1.45}\approx 1.204$. Cosine: $1.37/(1.140\times 1.204)=1.37/1.373\approx 0.998$.

Truffle cake. Dot product: $0.81+0.56+0\times 0.9=1.37$, the same as before, because the query’s “egg” number is zero. But the truffle cake is longer: $\sqrt{0.81+0.64+0.81}=\sqrt{2.26}\approx 1.503$. Cosine: $1.37/(1.140\times 1.503)=1.37/1.714\approx 0.80$.

Butter croissant. Dot product: $0+0.7\times 0.3+0=0.21$. Length: $\sqrt{0+0.09+0.16}=0.5$. Cosine: $0.21/(1.140\times 0.5)=0.21/0.570\approx 0.37$.

Egg puff. Dot product: $0+0.7\times 0.1+0=0.07$. Length: $\sqrt{0+0.01+0.81}\approx 0.906$. Cosine: $0.07/(1.140\times 0.906)=0.07/1.033\approx 0.07$.

Step 4: sort by score.

Rank	Item	Cosine similarity
1	Egg-free Belgian Chocolate Gateau	0.998
2	Classic Chocolate Truffle Cake	0.80
3	Butter Croissant	0.37
4	Egg Puff	0.07

The gateau wins, without the word “eggless” appearing anywhere in its name. The egg puff, which Arjun’s synonym list wrongly matched, comes dead last.

Show the answer

Its “contains egg” number (0.9) points its arrow off in a direction the query does not go at all. That extra direction tilts the angle between them, which pulls the cosine down. As for “is 0.80 a match?”: there is no universal answer. Cosine scores are only meaningful compared with each other, for the same model. Here, 0.80 means “clearly second best”. With a different model, a true match might score 0.55. You learn sensible thresholds by looking at real results, which you will do in the next two lessons.

Step 5: what did this cost? With a paid embedding service such as OpenAI’s text-embedding-3-small, you pay per token (a chunk of text, roughly three quarters of an English word). At the time of writing it costs about 0.02 US dollars per million tokens. The query is about five tokens, so embedding it costs 0.0000001 dollars, roughly ₹0.00001. Embedding Crumb’s whole 120-item menu, about 4,200 tokens, costs about 0.00008 dollars, well under one paisa. Then searching is arithmetic on your own computer, which is free. Prices change, so check before quoting them, but at Crumb’s scale the cost of embeddings is a rounding error.

Inside the embedding model

The worked example cheated in one way: Arjun and Priya invented the vectors. In real life, a program called an embedding model produces them. Time to open the hood one level.

An embedding model is a neural network (a program made of layers of simple maths whose settings were tuned on huge numbers of examples). It does one job: text in, a fixed-length vector out. It does not chat or write sentences.

Here is roughly what happens when it reads “eggless chocolate cake”:

1. Tokens. The text is cut into tokens, the same small chunks you met in the LLM mental model module: something like egg, less, chocolate, cake.
2. One vector per token, in context. Each token gets its own vector, and layers of the network let each token’s vector be adjusted by the tokens around it. “Less” next to “egg” ends up meaning something different from “less” in “less sugar”.
3. Pooling. The per-token vectors are combined into one vector for the whole text, often by averaging them. That single vector is the embedding.
4. Fixed length. Whether the input is three words or three hundred, the output has the same number of dimensions: 384 for the popular free model all-MiniLM-L6-v2, 1,536 for OpenAI’s text-embedding-3-small.

How it learned where things go

The model was trained on hundreds of millions of pairs of text that belong together: a question and its accepted answer, a headline and its article, two phrasings of the same thing. Picture one training step:

• It sees the pair “eggless chocolate cake” and “chocolate cake made without eggs”.
• It sees an unrelated text, “how to repair a bicycle chain”.
• It calculates the cosine similarity of the pair, and of the query against the unrelated text.
• If the pair is not clearly closer than the unrelated text, its internal settings get nudged so that next time the pair lands closer and the stranger lands further away.

Repeat this hundreds of millions of times across every topic people write about. The result is a model that places paraphrases side by side, even ones it has never seen. This style of training, pulling matching pairs together and pushing mismatches apart, is called contrastive learning.

Where the table analogy breaks

Priya’s table was a good start. Two things are different in real life.

1. The dimensions have no names. In the worked example, the three numbers meant “chocolatey”, “sweet” and “contains egg”. Real dimensions do not. Nobody told the model to create a sweetness dimension. It invented its own directions while training, and meaning is spread across many numbers at once. Asking “what does dimension 17 mean?” usually has no good answer.
2. There are hundreds of them. You cannot picture 384 directions. But you do not need to: the dot product and the length formula work exactly the same way with 384 numbers as with three. The computer does the same multiply-and-add, only longer.

One more rule follows from all this: only compare vectors made by the same model. Each model invents its own directions. A vector from one model and a vector from another live on different maps, like a street address in Pune and a street address in Lisbon. The maths will happily run and give you a number. The number will be meaningless.

If you have heard that LLMs “use embeddings”, you heard right. The first thing a chat model does is turn each token into a vector. An embedding model is a cousin that is trained to produce one vector for a whole text, specifically for comparing meanings. You will build that first layer yourself in the Transformers from scratch module.

Doing it for real in Python

Time to swap the made-up vectors for real ones. First, check the arithmetic from the worked example with NumPy, the standard Python library for maths on lists of numbers (uv add numpy):

import numpy as np

def cosine(a, b):
    a, b = np.asarray(a, dtype=float), np.asarray(b, dtype=float)
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))  # @ is the dot product

query = [0.9, 0.7, 0.0]
menu = {
    "Egg-free Belgian Chocolate Gateau": [0.9, 0.8, 0.0],
    "Classic Chocolate Truffle Cake":    [0.9, 0.8, 0.9],
    "Butter Croissant":                  [0.0, 0.3, 0.4],
    "Egg Puff":                          [0.0, 0.1, 0.9],
}
for name, vec in menu.items():
    print(f"{cosine(query, vec):.3f}  {name}")

0.998  Egg-free Belgian Chocolate Gateau
0.799  Classic Chocolate Truffle Cake
0.368  Butter Croissant
0.068  Egg Puff

The same numbers you worked out by hand. Now let a real model choose the vectors. Install the free sentence-transformers library (uv add sentence-transformers), which downloads a small model the first time it runs:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")      # about 90 MB, downloaded once

items = [
    "Egg-free Belgian Chocolate Gateau. Rich dark chocolate cake made without eggs.",
    "Classic Chocolate Truffle Cake. Layers of chocolate sponge and ganache.",
    "Butter Croissant. Flaky French pastry, baked every morning.",
    "Egg Puff. Flaky savoury pastry with a spiced boiled egg inside.",
]
vecs = model.encode(items)
q = model.encode("eggless chocolate cake")

print(vecs.shape)                # (4, 384): four items, 384 numbers each
print(q[:6].round(3))            # the first six of the query's 384 numbers
for text, v in zip(items, vecs):
    print(f"{cosine(q, v):.2f}  {text.split('.')[0]}")

Your output will look something like this (exact numbers vary slightly between library versions):

(4, 384)
[-0.061  0.034 -0.012  0.087 -0.045  0.021]
0.71  Egg-free Belgian Chocolate Gateau
0.62  Classic Chocolate Truffle Cake
0.17  Butter Croissant
0.15  Egg Puff

Look at three things. First, the raw numbers in the vector mean nothing on their own; nobody reads them. Second, the real scores are lower and more bunched than in the toy example. That is normal for real models, and it is why you judge by ranking, not by whether a number “looks high”. Third, the ranking is right: the model has never seen Crumb’s menu, yet it put the egg-free gateau first for “eggless”.

Where meaning-search lets you down

Arjun deploys a quick prototype on Wednesday. Meera tests it on her phone and sends back one screenshot with no comment. She had searched nut-free brownie. The top result was Walnut Fudge Brownie.

Arjun checks the scores. “Nut-free brownie” and “walnut brownie” have a cosine similarity of around 0.8. To the model, both texts are strongly about brownies and nuts. The word “free”, which flips the meaning entirely, barely moves the arrow.

This is the best-known weakness of embeddings, and Crumb hit it on day one. Here is the full list of things to watch:

• Negation. “Contains nuts” and “nut-free” are about the same topic, so they land close together, even though for a customer with an allergy they mean opposite things.
• Identifiers and numbers. Order CRB-10293 and order CRB-10239 look nearly identical to an embedding model. “Serves 4” and “serves 40” are close too. Use normal database lookups for exact values.
• Brand-new words. A model trained before Crumb invented the Kesar Cruffin only understands it through its pieces, “kesar” (saffron) and “cruffin”.
• Language coverage. all-MiniLM-L6-v2 learned mostly from English. It handles anda nahi wala cake poorly. Multilingual models exist and do much better, as you will see in the next lesson.
• Long texts get blurry. Pooling averages everything into one vector. A 2,000-word page about fifteen topics becomes an arrow pointing at the average of all fifteen, which matches none of them well. This is why long documents are cut into smaller pieces, called chunks, before embedding, which you will do in the Vector Databases and RAG modules.

Arjun’s fix for the brownie is not a cleverer model. It is a rule: if the query contains “nut-free”, “eggless” or “gluten-free”, filter the results on the allergens column before ranking by meaning. Meaning for the gist, structured data for the facts. You will see that pattern in nearly every system in this course.

Arjun explains it to Meera

On Thursday morning, Meera asks for the short version. “No maths. What did you build?”

Arjun tries. “Every item on the menu now has a position on a kind of map. Not a map of Pune, a map of meaning. Things that mean similar things sit near each other. When someone searches, their words get a position on the same map, and we show them what’s nearest. That’s why ‘eggless chocolate cake’ finds the gateau even though the words don’t match.”

“Who made the map?”

“A model trained on a huge amount of text. It learned that ‘eggless’ and ‘egg-free’ mean the same thing by seeing millions of examples.”

“Can it get things wrong?”

“Yes. It’s good at ‘what is this about’ and bad at small words that flip meaning, like ‘free’ in ‘nut-free’. So for allergies we don’t trust the map. We check the allergen list.”

“What does it cost?”

“Making the map of the whole menu cost less than one paisa. Searching it is free.”

Meera nods slowly. “Then why didn’t we always do it this way?”

Priya answers that one. “Because until a few years ago, models this good were not free to download. Now they are. Which means Arjun’s next job is learning how to make these vectors properly: which model, which service, and what it costs at a larger scale than one menu.”

Try it yourself

1. Without a computer, work out the cosine similarity between $p=(1,0)$ and $q=(1,1)$.
2. Run the real-model code above with your own four items and one query that shares no words with the right answer.
3. Add the query “nut-free brownie” and two items, “Walnut Fudge Brownie” and “Classic Brownie (no nuts)”. See which wins.

Expected result: for step 2, the right item should rank first. For step 3, there is a good chance the walnut brownie scores as high or higher than the nut-free one, which is the negation weakness in action.

Show the answer to step 1

The dot product is $1\times 1+0\times 1=1$. The lengths are $1$ and $\sqrt{2}\approx 1.414$. So the cosine similarity is $1/1.414\approx 0.707$. The arrows are 45 degrees apart, and $\cos (45°)\approx 0.707$.

Common mistakes

• “Each dimension means something.” It is tempting to say “dimension 42 is sweetness”. In real models, almost never. Reason about whole vectors and the distances between them.
• “A score of 0.62 means 62% similar.” Cosine similarity is not a percentage or a probability. What counts as close depends on the model. Compare rankings, and learn thresholds from real examples.
• Comparing vectors from different models. The maths runs and returns nonsense. If you switch models, re-embed everything.
• Using embeddings for exact lookups. Order numbers, phone numbers and SKUs belong in a normal database query.
• Trusting embeddings with negation. “Nut-free” and “contains nuts” can sit side by side. Use structured data for anything safety-related.

Recap

• An embedding is a vector (a list of numbers) that places a piece of text on a map of meaning, made by an embedding model.
• Similar meanings land close together, so search becomes “find what is nearest to the query”.
• Cosine similarity scores closeness by the angle between two vectors, from −1 to 1: dot product divided by both lengths, ignoring length.
• Real embeddings have hundreds of unnamed dimensions, and you can only compare vectors made by the same model.
• Embeddings are great at gist and paraphrase, and weak at negation, exact identifiers, new words and very long texts.

Arjun closes his synonym file and does not open it again. That evening, Priya sends him a message: “Tomorrow: two ways to make real embeddings, one you pay for and one that runs on your laptop. Bring Meera’s question about where our customers’ data goes. It matters.”

Go deeper

• OpenAI API documentation (the Embeddings guide)
• Sentence Transformers documentation
• The Illustrated Word2vec by Jay Alammar, the classic visual introduction to word vectors