Exploring and reasoning about vector spaces

Two things land on Arjun’s desk on the same morning.

The first is from Meera. “The board meets next month. I want to show them this ‘map of meaning’ you keep talking about. An actual picture. Can you make one?”

The second is a bug. Arjun had added a “similar reviews” box to the internal dashboard: pick any review, see the five most similar ones. Sunita from customer care reports that it is broken. “Whatever review I click, a dry cake, a late delivery, a lovely birthday, it shows me the same three giant reviews at the top. The ones that go on for a whole page.”

Arjun has 18,000 review vectors with 384 numbers each. He knows how to make them. He does not, he realises, understand the space they live in at all.

Two quick fixes that did not work

Arjun starts with the picture, because it seems easier. A scatter plot needs two numbers per point. His vectors have 384. So he plots the first two:

import matplotlib.pyplot as plt
import numpy as np

vecs = np.load("reviews.npy")                  # shape (18000, 384)
plt.scatter(vecs[:, 0], vecs[:, 1], s=2)       # dimension 0 against dimension 1
plt.savefig("map.png")

The result is a round, grey smudge. Late deliveries, birthday cakes and allergy questions are mixed together evenly, like salt and pepper shaken in a jar. No islands. No structure.

Then the bug. He opens the “similar reviews” code:

new_vecs = model.encode(new_reviews)            # last week's batch, added on Friday
vecs = np.vstack([old_vecs, new_vecs])
scores = vecs @ vecs[clicked]                   # dot product with the clicked review
top5 = np.argsort(-scores)[1:6]

It looks fine to him. He shows both to Priya.

Shadows on the wall, and the loudest voice in the room

Priya holds her hand between the desk lamp and the wall, making a shadow. “What do you see?”

“Your hand. Well, a shadow of it.”

She turns her hand edge-on to the lamp. The shadow becomes a thin line. “And now?”

“A stick.”

“Same hand. Different angle. A shadow throws away one direction, and which direction you throw away decides whether you can still recognise the thing. Your plot is a shadow too. You threw away 382 of 384 directions, and you picked the two by accident: whichever happened to come first in the list. Of course it looks like a smudge. The trick is to pick the angle on purpose, so the shadow keeps the shape.”

“Fine. And the reviews bug?”

“Look at the Friday batch. Did you pass normalize_embeddings=True?”

Arjun checks. He did not. The old vectors all have length 1. The new ones have lengths between 2 and 7.

“Imagine you ask a room of people which of them agrees with you,” Priya says, “and you measure agreement by how loudly they shout ‘yes’. Who wins?”

“Whoever is loudest. Not whoever agrees most.”

“The dot product is a shout. It grows with how much two arrows point the same way, and with how long they are. Your un-normalised vectors are the loudest people in the room. Long, rambling reviews often end up with longer arrows, so they win every vote.”

Following one query through three rulers

To see it with real arithmetic, use three dimensions. Sunita clicks a short review about a dry cake. Its vector is the query $q$. Two candidates compete:

• $q=(1,1,0)$: the clicked review, “cake was dry”.
• $A=(0.8,0.6,0)$: another short review, “a bit dry, otherwise fine”. Length exactly 1.
• $B=(2,0.5,2.5)$: a page-long review about a late delivery, a wrong name on the cake, and in passing, dryness. Never normalised.

Ruler one: the dot product

Multiply position by position and add:

• $q\cdot A=1\times 0.8+1\times 0.6+0\times 0=1.4$
• $q\cdot B=1\times 2+1\times 0.5+0\times 2.5=2.5$

B wins, 2.5 to 1.4. That is Sunita’s bug, reproduced by hand.

Ruler two: cosine similarity

Cosine similarity divides the dot product by both lengths, so only the direction counts. Lengths first, using $\Vert v\Vert =\sqrt{v_1^2+v_2^2+v_3^2}$:

• $\Vert q\Vert =\sqrt{1+1+0}\approx 1.414$
• $\Vert A\Vert =\sqrt{0.64+0.36+0}=1$
• $\Vert B\Vert =\sqrt{4+0.25+6.25}=\sqrt{10.5}\approx 3.240$

Then:

• $\cos (q,A)=1.4/(1.414\times 1)\approx 0.990$
• $\cos (q,B)=2.5/(1.414\times 3.240)=2.5/4.582\approx 0.546$

Now A wins, and by a mile. The short review about dryness really is the better match.

Making them agree: normalisation

To normalise a vector is to divide every number by its length, so the arrow keeps its direction but has length exactly 1.

• $\hat{q}=(1,1,0)/1.414=(0.707,0.707,0)$
• $\hat{A}=A$, already length 1.
• $\hat{B}=(2,0.5,2.5)/3.240=(0.617,0.154,0.772)$

Now take plain dot products again:

• $\hat{q}\cdot \hat{A}=0.707\times 0.8+0.707\times 0.6=0.990$
• $\hat{q}\cdot \hat{B}=0.707\times 0.617+0.707\times 0.154+0=0.545$

The same numbers as cosine similarity. Once every vector has length 1, the dot product is the cosine similarity. (The 0.545 versus 0.546 is rounding.)

Ruler three: straight-line distance

Euclidean distance is the distance a ruler would measure between the two points: $\Vert a-b\Vert$. Smaller means more similar. For normalised vectors there is a neat shortcut: $\Vert a-b{\Vert }^2=2-2\cos (\theta )$.

• $q$ to $A$: $2-2\times 0.990=0.020$, so the distance is $\sqrt{0.020}\approx 0.14$.
• $q$ to $B$: $2-2\times 0.545=0.910$, so the distance is $\sqrt{0.910}\approx 0.95$.

A is closer. All three rulers now give the same ranking.

Show the answer

No. With length-1 vectors, distance squared is $2-2\cos (\theta )$. A bigger cosine always means a smaller distance, so the two orders are identical. This is why vector databases let you pick “cosine”, “dot product” or “Euclidean” and, on normalised vectors, you get the same top results from all three. The dot product is the cheapest to compute, so normalise once and use it.

Arjun’s fix for Sunita’s bug is one argument: normalize_embeddings=True on the Friday batch, plus a check that fails loudly if any stored vector’s length is not within 0.001 of 1. The giant reviews drop out of the top five.

What “close” means in 384 dimensions

The three-dimensional example is a good picture, but high-dimensional spaces have a few habits that surprise everyone the first time.

Scores bunch together. In 384 dimensions, most pairs of unrelated texts end up at roughly similar angles to each other. Arjun measures it: he takes 10,000 random pairs of reviews and computes their cosine similarity. In his data, the median random pair scores 0.12, 95% of random pairs score below 0.38, and 99% below 0.52. Real matches for a given review mostly land between 0.55 and 0.85.

That table is worth more than any rule of thumb. It tells Arjun that, for this model and this data, a score of 0.40 is barely above background noise, and 0.60 is genuinely close. With a different model, the whole band shifts. OpenAI’s models, for instance, often give unrelated texts scores well above zero, so their “noise floor” sits higher.

Dimensions cost storage. Each number is usually stored as a 4-byte float (a decimal number):

Model	Numbers per vector	Bytes per vector	One million vectors
all-MiniLM-L6-v2	384	about 1.5 KB	about 1.5 GB
text-embedding-3-small	1,536	about 6 KB	about 6 GB

More dimensions give the model room to separate finer shades of meaning. Fewer save memory and speed up search. That trade-off is the heart of the next module, on vector databases.

No single dimension means much. You saw it in Arjun’s smudge: any one dimension, on its own, barely separates topics. Meaning is spread across all of them together.

Choosing the angle on purpose: PCA and UMAP

Back to Meera’s picture. To plot 384 numbers on a flat screen you need dimensionality reduction: squashing many dimensions down to two or three while keeping as much of the structure as you can. Two methods cover most needs, and they make different promises.

PCA: the most informative shadow

PCA (principal component analysis) finds the direction along which your points are most spread out, then the next most spread-out direction at right angles to the first, and so on. It then casts the shadow along those directions. Think of photographing a long baguette: you shoot it from the side, not end-on, because the side view shows the most.

PCA is fast, gives the same picture every time, and keeps large-scale distances roughly honest. It also tells you how much it threw away. When Arjun runs it on the reviews, the first two directions capture only about 7% of the total spread. That is typical for text embeddings: meaning is spread over many directions, so any flat picture loses a lot. PCA’s shadow shows the big contrasts, like “praise versus complaint”, and blurs the rest.

UMAP: keep the neighbours together

UMAP (uniform manifold approximation and projection) makes a different promise. For every point, it finds its nearest neighbours in the full 384-dimensional space. Then it arranges the points on the page so that neighbours stay neighbours. Think of peeling an orange and pressing the peel flat: nearby bits stay nearby, but the overall shape tears and stretches.

UMAP shows clusters far more clearly than PCA. In exchange:

• Distances between clusters mean very little. Two islands far apart on the page are not necessarily very different.
• Cluster sizes mean very little. A big blob is not necessarily a more varied group.
• The picture changes with the random seed and with settings like n_neighbors.

Show the answer

No. UMAP preserves who is next to whom, not how far apart the islands are. The left-to-right layout could flip on the next run with a different seed. To compare groups, compute real numbers in the full space, for example the cosine similarity between the average vector of each group. Use UMAP to spot groupings, then confirm with measurements.

Building the explorer, step by step

Install the tools (uv add scikit-learn umap-learn matplotlib). Each step below runs on its own.

Step 1: check the vectors are healthy

import numpy as np

vecs = np.load("reviews.npy")                       # shape (18000, 384)
texts = open("reviews.txt").read().splitlines()     # one review per line, same order

lengths = np.linalg.norm(vecs, axis=1)
print(lengths.min().round(3), lengths.max().round(3))
# Expect 1.0 and 1.0. Anything else: normalise before doing anything.
vecs = vecs / lengths[:, None]                      # normalise every row, to be safe

Step 2: measure the noise floor

rng = np.random.default_rng(42)
i, j = rng.integers(0, len(vecs), size=(2, 10_000))   # 10,000 random pairs
random_scores = np.sum(vecs[i] * vecs[j], axis=1)     # cosine of each pair
print(np.percentile(random_scores, [50, 95, 99]).round(2))

[0.12 0.38 0.52]

Your numbers will differ. Write them down: they are the background noise your real matches must beat.

Step 3: PCA, and how much it keeps

from sklearn.decomposition import PCA

pca = PCA(n_components=2)
pca_xy = pca.fit_transform(vecs)
print(pca.explained_variance_ratio_.round(3))         # e.g. [0.041 0.029]

explained_variance_ratio_ tells you what share of the total spread each direction keeps. Small numbers are normal for text and a reminder of how much the picture leaves out.

Step 4: UMAP and a coloured plot

import matplotlib.pyplot as plt
import umap

topics = open("review_topics.txt").read().splitlines()   # a hand label per review, e.g. "delivery"

umap_xy = umap.UMAP(n_neighbors=15, min_dist=0.1, metric="cosine",
                    random_state=42).fit_transform(vecs)

fig, axes = plt.subplots(1, 2, figsize=(14, 6))
for ax, xy, title in [(axes[0], pca_xy, "PCA"), (axes[1], umap_xy, "UMAP")]:
    for topic in sorted(set(topics)):
        idx = [k for k, t in enumerate(topics) if t == topic]
        ax.scatter(xy[idx, 0], xy[idx, 1], s=3, label=topic)
    ax.set_title(title)
axes[1].legend(markerscale=4)
plt.savefig("review_map.png", dpi=150)

You should now see two panels. On the left, PCA: overlapping clouds with a rough split between praise and complaints. On the right, UMAP: distinct islands for deliveries, taste, allergies and celebrations, and a few points sitting on the “wrong” island. On 18,000 points UMAP takes a minute or two.

Step 5: the neighbour test

A plot is a starting point. The most reliable check is plainer: pick texts you understand well and read their nearest neighbours.

def neighbours(k_item, k=5):
    scores = vecs @ vecs[k_item]
    for j in np.argsort(-scores)[1:k + 1]:            # position 0 is the item itself
        print(f"  {scores[j]:.2f}  {texts[j][:80]}")

neighbours(1203)

When Arjun runs this on the menu instead of reviews, the Almond Croissant’s neighbours are the butter croissant, the pain au chocolat and Badam Milk. “Badam” means almond. The model groups by ingredient as much as by type of food. Not wrong, but worth knowing before he builds “you might also like” in the next lesson.

Analogies: a lovely trick with real limits

You may have seen the famous result from an older word-embedding model called word2vec:

$$\text{king}-\text{man}+\text{woman}\approx \text{queen}$$

The idea is that the direction from “man” to “woman” is roughly the same everywhere in the space, so adding it to “king” lands you near “queen”. It is a beautiful demonstration that directions in the space can carry meaning.

Arjun tries a Crumb version with his sentence model: croissant − France + India. He hopes for something like masala bun. The nearest item is… croissant. Then butter croissant. Masala bun is eleventh. Here is why the trick disappoints:

• The famous examples quietly exclude the input words. Without that rule, the nearest point to “king − man + woman” is very often “king” itself.
• It works for a few clean relationships, such as capital cities, gender and verb tense, and poorly for most others.
• Sentence embedding models were not trained for it. They were trained to put matching texts close together, not to make arithmetic meaningful.

Treat analogies as a probe of the space, not as a feature to build on.

The island nobody asked for

There is one more thing on Arjun’s UMAP plot. Off to one side sits a tight little island of complaints about late deliveries that is separate from the main late-delivery island. He reads twenty of them. They are all in Hinglish (Hindi and English mixed, written in English letters): “order abhi tak nahi aaya, bahut late hai”.

Embedding models learn from huge amounts of human writing, and they absorb its patterns, including the unfair ones. Researchers have repeatedly found common models placing some job titles nearer to one gender, or associating certain names with negative words. At Crumb, the first bias Arjun finds is about language: the English-first model treats the same complaint differently depending on how it is written. Any system built on top inherits that. A search for “late delivery” misses the Hinglish island. A classifier routes those customers worse.

He measures it directly:

probe = model.encode(["My order is very late",
                      "Delivery delayed again, still waiting",
                      "order abhi tak nahi aaya, bahut late hai"],
                     normalize_embeddings=True)
print((probe @ probe.T).round(2))

[[1.   0.63 0.31]
 [0.63 1.   0.27]
 [0.31 0.27 1.  ]]

The two English complaints score 0.63 with each other. The Hinglish one scores around 0.3 against both: below 0.38, the level that 5% of completely random pairs beat. To this model, it looks barely related, even though it means the same thing. When he repeats the probe with the multilingual model paraphrase-multilingual-MiniLM-L12-v2, the Hinglish row rises to about 0.55.

Arjun shows Meera the map

Arjun brings his laptop to Meera’s office with the UMAP plot open.

“Each dot is one review,” he says. “Dots that are close together mean similar things. This island is late deliveries. This one is people loving our cakes. This small one is allergy questions.”

“And this little one on its own?”

“Late-delivery complaints written in Hinglish. The model we were using learned mostly from English, so it puts them on their own island instead of with the other late deliveries. A search for ‘late delivery’ would have missed them. We tested a multilingual model, and it pulls them back where they belong.”

“Can I show this picture to the board?”

“Yes, with one warning. The picture is a squashed version of something much bigger. Which dots are neighbours is reliable. How far apart the islands look is not. For any claim, I’ll back it with a real number.”

“And the bug Sunita reported?”

“Fixed. Some vectors were the wrong length, so long reviews were shouting over everything else. Now they’re all the same length, and only the direction counts.”

Meera studies the picture for a while. “So this map can tell us which reviews are alike. Can it tell a customer which cake they’d like?”

Try it yourself

1. Embed at least 200 short texts with known categories (reviews, product descriptions, news headlines).
2. Measure your noise floor with 2,000 random pairs and write down the 50th, 95th and 99th percentiles.
3. Make the PCA and UMAP plots coloured by category, and print PCA’s explained_variance_ratio_.
4. Find one point on the “wrong” UMAP island and run the neighbour test on it.

Expected result: UMAP islands are clearer than PCA’s; the first two PCA directions keep well under a third of the spread; and the neighbour list explains the odd point, usually through a shared ingredient, word or style rather than the category you assigned.

Show an example of what you might find

On Crumb’s menu, Chocolate Chip Cookie Dough Jar sits in the cake island rather than with the cookies. Its neighbours are the brownie and two chocolate cakes. The model groups by “chocolate dessert you eat with a spoon”, which is arguably a better grouping than Crumb’s own categories.

Common mistakes

• Mixing normalised and un-normalised vectors. Long vectors win every dot-product contest. Normalise everything once, and assert that lengths are 1.
• Picking a threshold from a blog post. “Above 0.5 is a match” may be noise for one model and a strong match for another. Measure your own noise floor.
• Reading UMAP distances literally. Islands, gaps and sizes on UMAP are not measurements. Confirm with real similarities.
• Shipping analogy tricks. They are a demo, not a dependable feature.
• Never checking groups. Bias stays invisible until you measure each language and customer group separately.

Recap

• Dot product, cosine similarity and Euclidean distance are three rulers; on normalised vectors they always agree on the ranking.
• Normalising means dividing a vector by its length; then the dot product equals cosine similarity, and long texts stop shouting over short ones.
• In high dimensions, random scores bunch together, so measure your own noise floor before choosing thresholds.
• PCA gives an honest but blurry shadow; UMAP keeps neighbours together but distorts distances between groups.
• Analogies are a fun probe with real limits, and embeddings carry the biases of their training text, so test every group.

Priya has been listening from the doorway. “Meera’s question is a good one,” she says to Arjun afterwards. “Search finds items near a query. Recommendations find items near a customer. Same map, different starting point. That’s next.”

Go deeper

• scikit-learn documentation (search for PCA)
• UMAP documentation
• The Illustrated Word2vec by Jay Alammar, for the analogy story