Semantic search and recommendations

Meera has been staring at the website’s analytics dashboard, and she does not like what she sees.

Every product page on Crumb’s site ends with a box labelled You might also like. It has shown the same four bestsellers on every page for two years: the Diwali hamper, the chocolate truffle cake, the plum cake and the croissant box. Last month, 41,000 people viewed a product page. 164 of them clicked something in that box. That is 0.4%. “It’s a shelf of things nobody asked for,” Meera says. “In July it is still showing Christmas plum cake, which we don’t even sell in July.”

She has a second request too. The new meaning-based search box from the last few weeks is still only a prototype on Arjun’s laptop. “Can both go live by the end of the month? Search that understands people, and suggestions that actually fit the person.”

Arjun is about to start two separate projects when Priya stops him. “Look at them again. Search is ‘find items near this query’. Recommendations are ‘find items near this item’, or ‘near this customer’. It’s the same machine. Only the thing you feed into the front changes.”

Arjun’s first “you might also like”

Arjun already has the menu embedded from earlier lessons: 120 items, each a vector of 384 numbers with length 1, so a dot product gives the cosine similarity (a score for how closely two vectors point the same way). The obvious recommender takes him ten minutes:

def similar_items(item_index, k=4):
    scores = menu_vecs @ menu_vecs[item_index]     # similarity to every item
    best = np.argsort(-scores)[1:k + 1]            # skip position 0: the item itself
    return [menu[i]["name"] for i in best]

print(similar_items(BUTTER_CROISSANT))

['Mini Butter Croissant', 'Croissant Box of 6', 'Almond Croissant', 'Chocolate Croissant']

Every answer is correct. The box is useless. Someone looking at a croissant does not need four more croissants.

He tries the Christmas Plum Cake page: the top suggestion is Christmas Fruit Cake Slice, which is out of stock. He tries the search prototype with a query nobody would ever type into a bakery website, “bicycle repair”, and it confidently returns five items, led by the Crumb Cycle Delivery Gift Card. Because something is always nearest.

“It’s finding similar things,” Arjun says. “That’s what I asked for.”

“It is,” Priya says. “Similar isn’t the same as useful. Let’s talk about what a good shop assistant does.”

What a good shop assistant does

Priya draws a counter on the whiteboard. “Think of Ravi at the Koregaon Park store. Best salesperson we have. A regular walks in. What does Ravi do?”

Arjun thinks. “He remembers what they usually buy.”

“Does he show them the exact same thing again?”

“No. Something like it. Or something that goes with it.”

“Would he suggest the plum cake in July?”

“Of course not. It’s not on the shelf.”

“And if they ask for a bicycle repair?”

Arjun laughs. “He’d say ‘sorry, wrong shop’.”

“So Ravi does five things,” Priya says, and writes them down:

1. He listens. He knows what the customer means, not only the words. That is the embedding.
2. He knows the shelves. Everything in the shop has a place in his head. That is the index of vectors.
3. He knows what’s actually available. Out of stock, out of season, not sold at this branch: never suggested. Those are filters.
4. He doesn’t repeat himself. Four croissants is not a suggestion. That is diversity.
5. He admits when nothing fits. That is a minimum score.

“Your code does the first two,” she says. “Let’s add the other three, and then, the part people always skip, check that it works on more than three examples.”

Following one customer through the recommender

Mrs Kulkarni orders from Crumb every week. Her last three orders were a Butter Croissant, an Almond Croissant and a Cold Coffee. Trace her “Picked for you” box from start to finish. To keep the arithmetic on one page, pretend the vectors have three dimensions instead of 384, already normalised to length 1.

Item	Vector	Status
Butter Croissant	(0.943, 0.314, 0.105)	bought
Almond Croissant	(0.843, 0.527, 0.105)	bought
Cold Coffee	(0.105, 0.314, 0.943)	bought
Chocolate Croissant	(0.796, 0.597, 0.100)	in stock
Mini Butter Croissant	(0.943, 0.314, 0.105)	in stock
Pain aux Raisins	(0.704, 0.704, 0.101)	in stock
Masala Bun	(0.970, 0.000, 0.243)	in stock
Masala Chai	(0.099, 0.099, 0.990)	in stock
Christmas Plum Cake	(0.314, 0.943, 0.105)	out of stock

Step 1: build her taste profile. Average the three vectors she bought, position by position. The average point of a group is called its centroid.

• First number: $(0.943+0.843+0.105)/3=0.630$
• Second: $(0.314+0.527+0.314)/3=0.385$
• Third: $(0.105+0.105+0.943)/3=0.384$

Her centroid is $(0.630,0.385,0.384)$. Its length is $\sqrt{0.630^2+0.385^2+0.384^2}\approx 0.833$, not 1. Averages of different directions are always shorter. Divide by 0.833 to normalise it: her profile is $p=(0.757,0.463,0.462)$. It points mostly toward pastries, with a lean toward drinks.

Step 2: score every candidate against the profile. Dot product with $p$. For the Chocolate Croissant: $0.757\times 0.796+0.463\times 0.597+0.462\times 0.100=0.603+0.276+0.046=0.925$. The rest, worked the same way:

Candidate	Score
Chocolate Croissant	0.925
Mini Butter Croissant	0.908
Pain aux Raisins	0.904
Masala Bun	0.846
Christmas Plum Cake	0.723
Masala Chai	0.578

Step 3: filter. Remove anything out of stock (the plum cake) and anything she already bought (they were never candidates). Filters are yes-or-no facts, so they are applied as plain rules, never mixed into the similarity.

Step 4: the naive top three. Chocolate Croissant, Mini Butter Croissant, Pain aux Raisins. Three more croissant-shaped pastries. Ravi would never.

Step 5: pick for variety. Use a method called maximal marginal relevance, or MMR. It picks items one at a time. Each candidate’s MMR score rewards how well it matches the customer and penalises how similar it is to anything already picked:

$$\text{MMR}(j)=\lambda \cdot \text{relevance}(j)-(1-\lambda )\cdot \text{redundancy}(j)$$

Here relevance is the score from step 2, redundancy is the highest similarity between candidate $j$ and anything already picked, and $\lambda$ (the Greek letter lambda) is a dial between 0 and 1: closer to 1 cares more about relevance, closer to 0 cares more about variety. Use $\lambda =0.5$.

Round 1. Nothing is picked yet, so redundancy is 0 for everyone. The top relevance wins: Chocolate Croissant, with $0.5\times 0.925=0.462$.

Round 2. Now each candidate is compared with the Chocolate Croissant.

Candidate	Relevance	Similarity to Chocolate Croissant	MMR
Mini Butter Croissant	0.908	0.949	$0.454-0.475=-0.021$
Pain aux Raisins	0.904	0.990	$0.452-0.495=-0.043$
Masala Bun	0.846	0.796	$0.423-0.398=0.025$
Masala Chai	0.578	0.236	$0.289-0.118=0.171$

Masala Chai wins. It is less relevant than the croissants, but it is a different kind of relevant: it fits the drink side of her taste.

Round 3. Worked the same way, the Masala Bun edges out the croissants (0.025 against −0.021 and −0.043).

Final box: Chocolate Croissant, Masala Chai, Masala Bun. A pastry she will probably love, a drink like the one she already orders, and a savoury option she has not tried. That is what Ravi would say.

Step 6: cost and speed. Every number above is arithmetic on vectors Crumb already has. Scoring 120 menu items with 384 dimensions is about 46,000 multiplications, which a laptop does in well under a millisecond. Recommendations cost nothing per page view.

Show the answer

Their centroid points halfway between “rich dessert” and “savoury snack”, a direction that matches neither very well. The recommender will suggest whatever happens to sit in that empty middle, perhaps a mildly sweet bun nobody wanted. A single average cannot represent someone with two separate tastes. The fix: recommend neighbours of each past item separately, then merge and diversify the lists. Big recommender systems keep several “interest” vectors per customer for the same reason.

Opening the hood: search and recommendation are one machine

Semantic search means searching by meaning rather than by exact words. Every semantic search system, and every content-based recommender, has two halves.

flowchart LR
  subgraph Index["Index time (once)"]
    A[Menu items] --> B["Build text for each item"] --> C["Embed"] --> D[("Vectors and item data")]
  end
  subgraph Query["Query time (every request)"]
    Q["Query, item or customer"] --> E["Get a vector"] --> F["Score against all vectors"] --> G["Filter, diversify, take top k"] --> H[Results]
  end
  D --> F

Index time runs once, or whenever the menu changes: build a text for every item, embed it, store the vector next to the item’s details. Query time runs on every request. The only difference between features is where the starting vector comes from:

• Search: embed the customer’s query with the same model.
• “You might also like”: use the vector of the item on the page.
• “Picked for you”: use the normalised centroid of the customer’s past items.

After that, everything is shared: score, filter, diversify, and take the top-k (the k highest-scoring results).

Two families of recommender

A recommendation system suggests things a person is likely to want. There are two big families, and it is worth knowing both.

• Collaborative filtering learns from behaviour: “people who bought X also bought Y”. It can discover surprising pairings, like plum cake and filter coffee, that no description would suggest. But it needs lots of purchase history, and it knows nothing at all about brand-new items.
• Content-based recommending learns from what items are: “Y is described like X”. That is what you built with embeddings.

The content-based approach has one superpower. On launch day, Crumb’s new Kesar Cruffin has zero purchases, so collaborative filtering cannot suggest it to anyone. But it has a description, so it has a vector, and it appears next to the saffron sweets and the croissants from the first minute. Having no history for new items or new customers is called the cold-start problem, and embeddings are one of the best answers to it. Large shops usually combine both families.

Filters before, or filters after?

There are two places to apply a filter. Pre-filtering removes ineligible items first, then ranks what remains. Post-filtering ranks everything, takes the top few, then removes ineligible ones.

Show the answer

Probably one result, or none. If four of the top five are sold out, post-filtering throws them away and nothing replaces them, even though in-stock items ranked sixth, seventh and eighth would have been fine. Pre-filtering, or asking for many more results than you need before filtering, avoids this. Vector databases, in the next module, have to solve exactly this problem at much larger scale.

Building it for real

Assume menu.json holds Crumb’s items as a list like this:

{"id": "M017", "name": "Egg-free Belgian Chocolate Gateau",
 "description": "Rich dark chocolate cake made without eggs.",
 "category": "cake", "tags": ["eggless", "chocolate", "celebration"],
 "price": 950, "in_stock": true}

Step 1: the index

# menu_search.py
import json
import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
menu = json.load(open("menu.json"))

def item_text(item):
    # Everything a customer might search for, in one string.
    return f"{item['name']}. {item['description']} Tags: {', '.join(item['tags'])}."

menu_vecs = model.encode([item_text(m) for m in menu], normalize_embeddings=True)
print(menu_vecs.shape)          # (120, 384)

The single most important line here is item_text. Embed only the name, and “Egg-free Belgian Chocolate Gateau” loses the word “celebration”; a customer searching “birthday cake” may never find it.

Step 2: search with filters and a floor

def rank(start_vec, k=5, min_score=0.30, exclude=(), **filters):
    scores = menu_vecs @ start_vec
    results = []
    for i in np.argsort(-scores):                         # best first, whole menu
        item = menu[i]
        if item["id"] in exclude:
            continue
        if any(item.get(key) != value for key, value in filters.items()):
            continue                                       # pre-filter: breaks a rule
        if scores[i] < min_score:
            break                                          # everything after is worse
        results.append((round(float(scores[i]), 2), item))
        if len(results) == k:
            break
    return results

def search(query, **kw):
    q = model.encode([query], normalize_embeddings=True)[0]
    return [(s, item["name"]) for s, item in rank(q, **kw)]

print(search("eggless chocolate cake", in_stock=True))
print(search("bicycle repair"))

[(0.71, 'Egg-free Belgian Chocolate Gateau'), (0.58, 'Chocolate Truffle Cake'), ...]
[]

The loop walks the whole ranked menu, so filters never leave you short of results. And “bicycle repair” now returns nothing. Where does 0.30 come from? From the noise floor you measured last lesson: the score that random, unrelated pairs rarely beat. Measure yours; do not copy this one.

Step 3: measure it with a test set

Arjun demos search to Meera. She types three queries; all three look great. Priya is not impressed. “Three queries you chose is not a test. What about the next three thousand?”

The habit that separates engineers from demo-builders is a test set: real queries, each paired with the item a person says is right. Then you measure the hit rate at k: the share of queries where the right item appears somewhere in the top k.

tests = [
    ("eggless chocolate cake", "Egg-free Belgian Chocolate Gateau"),
    ("something with no sugar for my dad", "Sugar-free Oat Cookies"),
    ("diwali gift box for office", "Premium Dry Fruit Hamper"),
    ("kuch teekha snack", "Masala Bun"),                # Hinglish: "something spicy"
    # ... 30 or more, taken from the real search logs
]

def hit_rate(k=3):
    hits = sum(expected in [name for _, name in search(q, k=k, min_score=0)]
               for q, expected in tests)
    return hits / len(tests)

print(f"hit rate at 3: {hit_rate():.0%}")

Arjun’s first score, with name-only item text, is 63%. With descriptions and tags in item_text, it is 80%. Switching to a multilingual model, so the Hinglish queries work, takes it to 87%. Each change now has a number attached, not a feeling.

Step 4: recommendations, with variety

def mmr(start_vec, candidates, k=4, lam=0.5):
    """Pick k items that match start_vec but differ from each other."""
    relevance = {i: float(menu_vecs[i] @ start_vec) for i in candidates}
    chosen = []
    while candidates and len(chosen) < k:
        def score(i):
            redundancy = max((float(menu_vecs[i] @ menu_vecs[c]) for c in chosen), default=0.0)
            return lam * relevance[i] - (1 - lam) * redundancy
        best = max(candidates, key=score)
        chosen.append(best)
        candidates.remove(best)
    return [menu[i]["name"] for i in chosen]

def eligible(exclude_ids):
    return [i for i, m in enumerate(menu) if m["in_stock"] and m["id"] not in exclude_ids]

def similar_items(item_id, k=4):
    i = next(n for n, m in enumerate(menu) if m["id"] == item_id)
    return mmr(menu_vecs[i], eligible({item_id}), k)

def for_customer(bought_ids, k=4):
    idx = [n for n, m in enumerate(menu) if m["id"] in bought_ids]
    profile = menu_vecs[idx].mean(axis=0)
    profile /= np.linalg.norm(profile)               # the average is shorter than 1: fix it
    return mmr(profile, eligible(set(bought_ids)), k)

similar_items(BUTTER_CROISSANT) -> ['Almond Croissant', 'Cold Coffee', 'Masala Bun', 'Orange Marmalade Jar']

Meera calls that list “what Ravi would say”.

Where it breaks

The new box goes live for a week as a test, shown to half of the site’s visitors. Click-through on that half rises from 0.4% to 2.1%. Then Sunita in customer care forwards a complaint. A customer had viewed the Walnut Fudge Brownie, and the box suggested a Peanut Butter Cookie. The customer’s profile clearly says nut allergy.

The recommender was doing its job: those two items really are similar. Nothing told it about the allergy.

Other things experienced engineers watch for:

• Stale vectors. Edit a description, and the old vector keeps answering searches until you re-embed it. Re-embed whenever an item’s text changes.
• Exact codes and rare names. Semantic search is weak at product codes and unusual names; keyword search is strong at them. Real systems usually run both and blend the results, which is called hybrid search. You will build it in the RAG module.
• Popularity blindness. Content-based recommendations ignore whether anyone actually likes an item. A beautifully described cake with a 2-star average will be recommended happily. Blend in ratings or sales once you have them.
• Feedback loops. Whatever you recommend gets clicked more, which makes it look more popular. Watch whether a handful of items start taking over every box.
• Demo-driven confidence. Without a test set and a hit rate, every change is a guess.

Arjun explains it to Meera

At the end of the month, Meera asks for the plain-words version.

“Search and suggestions are now the same machine,” Arjun says. “Every item has a position on our map of meaning. For search, we put the customer’s words on the map and show what’s nearest. For ‘you might also like’, we start from the item on the page. For ‘picked for you’, we start from the middle of everything that customer has bought.”

“And why doesn’t it show four croissants any more?”

“After the first pick, each next pick gets a penalty for being too similar to what’s already in the box. So you get one croissant, then a drink, then something savoury.”

“And the plum cake in July?”

“Gone. Stock, season and allergies are hard rules, checked before anything is ranked. Similarity only decides the order of what’s allowed.”

“How do you know it’s better?”

“Two numbers. Search finds the right item in the top three for 87% of real queries, up from 63%. And people click the suggestions five times as often as the old bestseller box.”

Meera writes both numbers down for the board. Then she pulls up something else. “The support inbox. Four hundred messages a day, and Sunita reads every one to decide who handles it. Could the map sort those?”

Try it yourself

1. Build menu.json with at least 25 items across four categories, or use any product list you know.
2. Write ten test queries with expected answers and measure hit rate at 3.
3. Change item_text to use only the item name. Measure again.
4. Run similar_items on one item with lam=1.0 (no diversity) and lam=0.5, and compare.

Expected result: the name-only version scores noticeably lower, because descriptions and tags carry the words customers search with. With lam=1.0 you see near-clones; with lam=0.5 you see variety.

Show why the item text matters so much

“Egg-free Belgian Chocolate Gateau” contains the idea of “eggless”, but “Sugar-free Oat Cookies” says nothing about “for my dad” or “healthy”. The richer the text you embed, the more ways a query can land near it. This is the cheapest improvement available: it costs nothing and needs no new model.

Common mistakes

• Putting hard rules inside the embedding. Adding “in stock” to the item text does not filter anything reliably. Use real filters on real fields.
• No minimum score. Something is always nearest, so without a floor, off-topic queries get confident nonsense.
• Post-filtering a short list. Taking the top 5 and then removing sold-out items can leave you with nothing. Filter first, or over-fetch.
• Forgetting to normalise a centroid. An average of unit vectors is shorter than 1, so its dot products come out smaller than the true cosine similarities. That quietly breaks your minimum score and tips the MMR balance toward variety.
• Testing with your own favourite queries. Collect real ones from the search logs and measure hit rate.

Recap

• Semantic search and content-based recommendations are one machine: get a starting vector, score everything, filter, diversify, take the top k.
• The starting vector is the query (search), the item (similar items) or the normalised centroid of a customer’s purchases (personal picks).
• Filters are hard yes-or-no rules applied before ranking; similarity only orders what is allowed.
• MMR trades relevance against redundancy, so results are useful rather than four clones.
• Embeddings solve the cold-start problem for new items, and a test set with hit rate at k tells you whether any change actually helped.

That evening, Arjun opens the support inbox export. 3,100 messages, each already sorted into a folder by Sunita over the past year. He realises he is looking at something he did not have for search: thousands of examples with the right answer already written next to them.

Go deeper

• Sentence Transformers documentation (see the semantic search examples)
• OpenAI API documentation (the Embeddings guide has search and recommendation examples)
• Hugging Face docs