Project: Exploring embeddings

Over the past few weeks, Arjun has built a meaning-based search box, a “you might also like” box and an inbox sorter for Crumb. Each one lives in a different notebook. Each embeds the same data again, slightly differently. One normalises its vectors and one, he discovers, does not. Each was tested on whatever examples he happened to think of that afternoon.

Meera wants to show the work to Crumb’s board at the end of the month. “Not slides,” she says. “Something I can type into. And I want to be able to say what it gets wrong, not only what it gets right. The board will ask.”

Priya turns that into a brief on the whiteboard. “One project. One embedding layer that everything shares. Three features on top of it. A real measurement for each. And a written page about what the vector space looks like underneath, including the ugly bits. That’s the difference between a set of demos and a system you understand.”

Arjun counts on his fingers. “So… a week?”

“A week, if you build the foundation first. Three weeks, if you don’t.”

The brief

Crumb has three kinds of text, and each one supports a different feature:

• The menu: 60 to 120 items, each with a name, description, category, tags, price, allergens and a stock flag. Powers search and recommendations.
• Customer reviews: a few hundred short reviews, each tied to a menu item and a star rating. Powers “what are people saying about…” search.
• Support tickets: a few hundred messages, each labelled with a folder (late delivery, allergy, refund, order change, praise, other). Powers the classifier.

Your job is to build one embedding layer and three tools on top of it, then investigate the space they share. Every feature must come with a number that says how well it works, and the project ends with a written report.

You can use Crumb’s data as described, or swap in any domain you know well: a bookshop, a gym, a college canteen, your own company’s help desk. The skills are identical, and a domain you know makes the findings far more interesting to write.

What you will build

Must have

• A cached embedder: every text is embedded once per model, never twice, and all vectors are normalised to length 1.
• Semantic search over the menu with filters (in stock, category, allergen-free) and a minimum score, measured by hit rate at 3.
• Semantic search over reviews (“people saying the cake was dry”).
• Item-to-item recommendations with a diversity step, plus “picked for you” from a customer’s order history.
• A ticket classifier with a held-out test set, a per-label report, an allergy safety net and a “send to a human” rule.
• A UMAP or PCA map of the space and a written FINDINGS.md with evidence.
• A small app Meera can type into.

Nice to have

• A switch between OpenAI and local embeddings, with a side-by-side comparison table.
• Reviews folded into item vectors, so items are described by what customers say as well as by the menu.
• Clustering the “other” tickets to discover folders Sunita might want to add.

Architecture

flowchart LR
  D[("menu, reviews, tickets (JSON)")] --> E["Embedder with cache"]
  E --> V[("Vectors on disk")]
  V --> S["Search"]
  V --> R["Recommend"]
  V --> C["Classify"]
  V --> X["Explore and map"]
  S --> UI["Streamlit app"]
  R --> UI
  C --> UI
  X --> F["FINDINGS.md"]

Data (JSON). Three plain files in a data/ folder. Keeping data separate from code means you can swap in real exports later without touching the logic, and you can re-generate or clean the data without breaking anything.

Embedder with cache. The only place in the whole project that talks to an embedding model. It hashes each text, looks the hash up in a cache file, and embeds only what is missing. Because every feature goes through it, every feature uses the same model, the same normalisation and the same map. This one box prevents most of the bugs Arjun hit over the past weeks.

Vectors on disk. One cache file per model name. Changing model means a new file, never a mix of two models’ vectors in one place.

Search, Recommend, Classify. Three small modules, each a few dozen lines, each with its own evaluation script that prints one headline number.

Explore and map. Plots and probes that help you understand the space. Its output is not code but writing: what you found, with evidence.

Streamlit app. A thin front end. All the real logic lives in the modules, so the interface can change freely without touching anything that has been measured.

Tech stack

uv init crumb-embeddings && cd crumb-embeddings
uv add sentence-transformers numpy scikit-learn umap-learn matplotlib streamlit python-dotenv

A suggested layout:

crumb-embeddings/
  data/        menu.json, reviews.json, tickets.json
  cache/       one .npz file per embedding model
  embedder.py  search.py  recommend.py  classify.py  explore.py  app.py
  eval/        search_tests.json
  FINDINGS.md  README.md

Build it, milestone by milestone

1. Prepare the menu, reviews and tickets

Why. Every number you report later is only as honest as this data. Arjun’s first instinct was to write fifty tidy tickets in an afternoon. Priya’s warning: “If your test data is cleaner than the real inbox, your accuracy is a fairy tale.”

Create three files. Aim for at least 60 menu items, 200 reviews and 300 tickets, with at least 30 tickets per label.

One entry in data/menu.json:

{"id": "M017", "name": "Egg-free Belgian Chocolate Gateau",
 "description": "Rich dark chocolate cake made without eggs.",
 "category": "cake", "tags": ["eggless", "chocolate", "celebration"],
 "allergens": ["milk", "soy"], "price": 950, "in_stock": true}

One entry in data/reviews.json:

{"id": "R0412", "item_id": "M017", "rating": 2,
 "text": "Tasted great but the gateau was a bit dry on day two."}

One entry in data/tickets.json:

{"id": "T1180", "label": "allergy",
 "text": "hi does the badam cookie have peanuts?? my son is allergic"}

Real data is best, anonymised. If you have none, generate a first draft with an LLM, then make it messy by hand: typos, Hinglish, angry capitals, two questions in one message, a ticket that is one word long.

What working looks like. A short script loads all three files and prints counts per category and per label. No label has fewer than 30 tickets; at least a tenth of tickets are in a language other than English or are badly spelled.

Most likely bug. Near-duplicate generated tickets. An LLM asked for “40 late delivery messages” produces many that differ by one word. They will inflate your classifier’s score in milestone 5.

from typing import Literal
import anthropic
from dotenv import load_dotenv
from pydantic import BaseModel

load_dotenv()
client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY

class Ticket(BaseModel):
    text: str
    label: Literal["late_delivery", "allergy", "refund", "order_change", "praise", "other"]

class TicketBatch(BaseModel):
    tickets: list[Ticket]

response = client.messages.parse(
    model="claude-opus-5-5",
    max_tokens=8000,
    messages=[{"role": "user", "content":
        "Write 40 realistic customer support messages for an Indian bakery chain. "
        "Mix English and Hinglish, short and long, polite and angry, with typos. "
        "Spread them across all six labels, and include a few that fit two labels."}],
    output_format=TicketBatch,
)
tickets = response.parsed_output.tickets

2. Embed everything once with a cache

Why. Embedding is the slow and possibly paid step. Doing it once, in one place, with one model, guarantees every feature sees the same map. Arjun’s “similar reviews” bug two lessons ago, where long reviews won every contest, came from one batch that skipped normalisation. A single embedder makes that impossible.

# embedder.py
import hashlib
from pathlib import Path
import numpy as np
from sentence_transformers import SentenceTransformer

MODEL_NAME = "paraphrase-multilingual-MiniLM-L12-v2"
CACHE = Path("cache") / f"{MODEL_NAME.replace('/', '_')}.npz"   # one cache file per model
_model = None

def _key(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

def embed(texts: list[str]) -> np.ndarray:
    """Return one length-1 vector per text, embedding only what is not cached."""
    global _model
    cache = dict(np.load(CACHE)) if CACHE.exists() else {}
    missing = list({t for t in texts if _key(t) not in cache})    # set removes duplicates
    if missing:
        _model = _model or SentenceTransformer(MODEL_NAME)
        new = _model.encode(missing, normalize_embeddings=True, batch_size=64)
        cache.update({_key(t): v for t, v in zip(missing, new)})
        CACHE.parent.mkdir(exist_ok=True)
        np.savez(CACHE, **cache)
    vecs = np.array([cache[_key(t)] for t in texts])
    assert np.allclose(np.linalg.norm(vecs, axis=1), 1, atol=1e-3), "unnormalised vector!"
    return vecs

The cache key is a hash of the text: a short fingerprint that is always the same for the same text and different for different text. If a menu description changes, its hash changes, so it is automatically re-embedded.

What working looks like. The first run over all your data takes a minute or so and creates cache/paraphrase-multilingual-MiniLM-L12-v2.npz. The second run finishes almost instantly. embed(["hello"]).shape prints (1, 384).

Most likely bug. The second run is still slow. That means your texts change between runs, usually because you build item text with tags in a random order (from a Python set) or include something like a timestamp. Build each text deterministically.

3. Build semantic search over menu and reviews

Why. Search is the feature customers feel first, and the easiest to measure. It also forces you to decide what text represents an item, which affects every other feature.

Reuse the rank and search functions from the semantic search lesson, but take vectors from embedder.py. Add an allergen filter that works as a hard rule:

# search.py (excerpt)
def search(query, k=5, min_score=0.30, exclude_allergens=(), **filters):
    q = embed([query])[0]
    scores = menu_vecs @ q
    results = []
    for i in np.argsort(-scores):
        item = menu[i]
        if set(item["allergens"]) & set(exclude_allergens):
            continue                                   # hard rule, checked before ranking
        if any(item.get(key) != value for key, value in filters.items()):
            continue
        if scores[i] < min_score:
            break
        results.append((round(float(scores[i]), 2), item["name"]))
        if len(results) == k:
            break
    return results

Then write search_reviews(query, item_id=None, max_rating=None, k=5) so Meera can ask “complaints about packaging” and get real reviews back.

Finally, measure. Put at least 25 query and expected-item pairs in eval/search_tests.json and print hit rate at 3. Measure your noise floor first (the similarity of random pairs, from the vector spaces lesson) and set min_score slightly above its 95th percentile.

What working looks like. search("nut-free brownie", exclude_allergens=["tree nuts", "peanuts"]) never returns a walnut brownie. search("bicycle repair") returns nothing. Your eval script prints one line such as hit rate at 3: 84% (21/25).

Most likely bug. A high hit rate on queries you wrote. You know the menu wording, so your queries match it too well.

4. Add diverse recommendations

Why. “You might also like” without diversity shows four croissants. With diversity and business rules, it behaves like a good shop assistant. This milestone is where you learn that similar and useful are different things.

In recommend.py, write:

• similar_items(item_id, k=4, lam=0.5): nearest neighbours of an item, re-ranked with MMR (maximal marginal relevance, which trades relevance against similarity to items already picked), skipping out-of-stock items.
• for_customer(order_history, k=4, avoid_allergens=()): start from the normalised average of the customer’s items, apply MMR, and never suggest something containing an allergen the customer has flagged.

def for_customer(bought_ids, k=4, avoid_allergens=()):
    idx = [n for n, m in enumerate(menu) if m["id"] in bought_ids]
    profile = menu_vecs[idx].mean(axis=0)
    profile /= np.linalg.norm(profile)               # an average is shorter than 1
    allowed = [i for i, m in enumerate(menu)
               if m["in_stock"] and m["id"] not in bought_ids
               and not set(m["allergens"]) & set(avoid_allergens)]
    return mmr(profile, allowed, k)

Make three customer histories (“loves chocolate”, “savoury breakfast person”, “buys hampers for Diwali”) and one awkward one (“bought a truffle cake and a masala bun”).

What working looks like. Each normal customer gets four suggestions from at least two categories. Changing lam from 1.0 to 0.5 visibly replaces near-clones with variety. The nut-allergic customer never sees a nut.

Most likely bug. The awkward customer gets odd suggestions: the average of a cake and a bun points somewhere that matches neither.

5. Classify tickets and measure them

Why. This is the feature with the clearest business value (allergy questions answered in seconds) and the clearest way to fool yourself (a big accuracy number hiding a bad allergy recall).

In classify.py:

1. Remove near-duplicate tickets: any pair with similarity above 0.95, keep one.
2. Split 80/20 with stratify, keeping the texts alongside.
3. Train nearest centroid, k-NN and logistic regression; print accuracy for each.
4. For the best one, print classification_report and note the lowest-recall label.
5. Add the routing layer: a low probability bar for allergy, a keyword net of specific allergen words, and "needs_human" when no label reaches 0.6.
6. Print every test ticket the final router gets wrong.

def dedupe(texts, labels, vecs, threshold=0.95):
    keep = []
    for i in range(len(texts)):
        if all(vecs[i] @ vecs[j] < threshold for j in keep):
            keep.append(i)
    return [texts[i] for i in keep], [labels[i] for i in keep], vecs[keep]

What working looks like. Three accuracy numbers, a per-label table, allergy recall of 0.95 or more after routing, a “needs human” rate you can state (for example 6%), and a short list of failures you have actually read.

Most likely bug. Accuracy of 0.99. That is almost never real. Check deduplication first; then check whether some labels can be guessed from a giveaway word your generator overused.

6. Map the space and write up findings

Why. Meera asked to know what the system gets wrong. The map, the probes and the write-up are how you find out, and how you explain it to people who will never read your code.

Write explore.py that:

1. Plots all three datasets together with UMAP, coloured by source (menu, review, ticket), and a second plot of tickets only, coloured by label.
2. Prints PCA’s explained_variance_ratio_ for two components.
3. Runs the neighbour test on five menu items you know well.
4. Probes for language bias: five English tickets and their Hinglish equivalents. Compare average English-to-English similarity with English-to-Hinglish, for your main model and for an English-only model such as all-MiniLM-L6-v2.
5. Tries two analogies and records honestly what happened.

def avg_similarity(a_texts, b_texts):
    a, b = embed(a_texts), embed(b_texts)
    return float((a @ b.T).mean())

english = ["My order is very late", "The cake has not arrived yet", ...]
hinglish = ["order abhi tak nahi aaya", "cake abhi tak nahi pahuncha", ...]
print("en-en", round(avg_similarity(english, english), 2))
print("en-hi", round(avg_similarity(english, hinglish), 2))

Then write FINDINGS.md: at least five observations, each with evidence (a number, a neighbour list, a plot) and what it means for Crumb.

What working looks like. Two saved plots, a printed table of probe results, and a findings page that a non-engineer can read in five minutes. For example: “Allergy tickets overlap with order-change tickets on the map (plot 2). That explains why raw allergy recall was only 0.74, and why we added a keyword safety net.”

Most likely bug. Findings that describe pictures without measurements: “the clusters look nice”. UMAP distances between clusters are not reliable evidence.

7. Put it behind a simple app for Meera

Why. Meera asked for “something I can type into”. A small app also proves your modules are cleanly separated: if the app needs to know anything about embeddings, the design leaked.

# app.py   run with: uv run streamlit run app.py
import streamlit as st
from search import search, search_reviews
from recommend import for_customer
from classify import route

st.title("Crumb: search, suggest, sort")
tab_search, tab_reviews, tab_suggest, tab_triage = st.tabs(
    ["Menu search", "Review search", "Suggestions", "Ticket triage"])

with tab_search:
    q = st.text_input("What are you looking for?")
    no_nuts = st.checkbox("Nut-free only")
    if q:
        allergens = ["tree nuts", "peanuts"] if no_nuts else []
        for score, name in search(q, exclude_allergens=allergens):
            st.write(f"{name}  ({score})")

with tab_triage:
    msg = st.text_area("Paste a customer message")
    if msg:
        st.write("Suggested folder:", route(msg))
# ... the reviews and suggestions tabs follow the same pattern

Streamlit’s core functions (st.title, st.tabs, st.text_input, st.write) have been stable for a long time. Library APIs change quickly, though. If a name has moved, check the Streamlit documentation.

What working looks like. uv run streamlit run app.py opens a browser tab with four tabs. Each query answers in well under a second after the first, because the model and vectors load once.

Most likely bug. Every keystroke reloads the embedding model, taking several seconds. Streamlit re-runs your script on each interaction.

Done means done

• embedder.py `embedder.py` is the only code that calls an embedding model, asserts length-1 vectors, and a second run uses the cache.
• Menu search supports allergen, stock and category filters plus a minimum score, and reports hit rate at 3 on 25 or more test queries written by other people.
• Review search returns real reviews for a vague complaint query.
• Item recommendations include a diversity step and never show out-of-stock items or flagged allergens.
• Customer recommendations work for three taste profiles, and the mixed-taste case is handled or documented.
• The classifier reports accuracy and per-label precision and recall on deduplicated data, has an allergy recall of at least 0.95 after routing, and sends unsure tickets to a human.
• At least two plots are saved as images.
• FINDINGS.md `FINDINGS.md` has five or more observations, each with a number or other evidence.
• The Streamlit app runs and answers in under a second after loading.
• No API key appears anywhere in the repository or its history.

Stretch goals

• Compare models. Make MODEL_NAME configurable. Run every evaluation with an English-only local model, the multilingual one and OpenAI text-embedding-3-small, and put all the numbers in one table with cost and privacy notes.
• Reviews as part of items. Mix each item’s description vector with the average of its reviews’ vectors (for example 70/30) and see whether search hit rate improves.
• Discover new folders. Run k-means clustering on the tickets labelled “other” and read each cluster. Suggest new folders with evidence.
• Search analytics. Log every query and its top score; list queries whose best score was below the floor. Those are menu gaps or vocabulary gaps.

Common pitfalls

• Mixing models in one cache. Name cache files by model and never compare vectors across them.
• Leaking test data. Duplicates across the train and test split make every score look better than reality.
• Only reporting good news. A findings page without weaknesses is not credible. Meera asked for what it gets wrong.
• Over-trusting the plot. Confirm any visual claim with real similarities before writing it down.
• Safety from similarity alone. Allergen filters and the keyword net must stay in charge of anything to do with allergies.
• Logic inside the app. If app.py computes similarities itself, you will end up with two versions of the truth. Keep it a thin shell.

Show it off

• README. One paragraph on the problem; a screenshot of the map; a table with your headline numbers (search hit rate at 3, classifier accuracy, allergy recall after routing, “needs human” rate); and three commands to run it.
• Demo video. Sixty seconds: a vague search that works, a nut-free filter in action, a recommendation with variety, a ticket being routed, and one honest failure with what you did about it.
• LinkedIn post. Lead with a finding, not a list of tools. For example: “I built search, recommendations and ticket triage for a bakery from one set of embeddings. The most useful discovery wasn’t the 92% accuracy. It was that Hinglish messages sat on their own island in the vector space, and that a 92% classifier was quietly missing a quarter of allergy questions. Here’s how I found both, and fixed them.”

Meera’s board meeting goes well. One board member types “something my diabetic father can eat” and gets the sugar-free cookies first. Another asks, a little sceptically, whether “it ever gets things wrong”. Meera opens FINDINGS.md and reads out finding number three, about the Hinglish island, before Arjun can answer.

Afterwards she finds him in the kitchen. “That was the best part,” she says. “Not that it works. That you knew exactly where it doesn’t.” Then she asks the question that leads into the next module. “You did all of this with a few hundred tickets in a file. What happens when we want five years of WhatsApp orders in there? Millions of messages?”

Arjun opens his mouth, then closes it again. Searching every vector one by one will not survive millions. Priya, passing with a coffee, smiles. “That,” she says, “is what a vector database is for.”