Classification with embeddings

Crumb’s support inbox receives around 400 messages a day. Every morning, Sunita from customer care reads each one and drags it into a folder: Late delivery, Allergy question, Refund, Change my order, Praise, or Other. Each folder belongs to a different person. Late deliveries go to the dispatch team, refunds to accounts, allergy questions to the head baker. Sorting takes Sunita until about 1 p.m.

Last Tuesday, a message arrived at 9:10 a.m.: “My son has a severe peanut allergy, does the birthday cake for today’s 4 pm pickup contain any nuts?” It sat in the unsorted pile until 1:20 p.m. By then the mother had cancelled the ₹2,400 order and bought a cake elsewhere. She left a one-star review: “Asked a simple safety question. No answer for four hours.”

Meera brings the review to Arjun. “Allergy questions cannot wait behind refund requests. Can the computer at least put them in the right pile, the moment they arrive?”

Arjun’s rules, and how they failed

Arjun’s first instinct is a set of keyword rules. They take him half an hour:

def route(message):
    m = message.lower()
    if "nut" in m or "allerg" in m or "gluten" in m:
        return "allergy"
    if "refund" in m or "money back" in m:
        return "refund"
    if "late" in m or "not arrived" in m or "where is" in m:
        return "late_delivery"
    return "other"

To test it, he takes 200 messages Sunita has already sorted and checks how often his rules agree with her. They agree on 116. That is 58%, and the mistakes are instructive:

• “Can I add 6 donuts to my order?” went to allergy. The letters “nut” are inside “donut”.
• “Not late at all, delivered early, thank you!” went to late delivery.
• “paisa wapas chahiye, cake kharab tha” (“I want my money back, the cake was bad”) went to other. None of his English keywords appear.
• “My daughter reacts badly to cashews, is the barfi safe?” went to other. No “nut”, no “allerg”.

That last one is the scariest. It is exactly the kind of message Meera asked him to catch.

His second idea is to send every message to an LLM with the prompt “Which folder does this belong in?”. It would work reasonably well. Before he writes it, Priya asks a different question.

“How many messages has Sunita already sorted?”

Arjun checks the helpdesk. Over the past year: 3,100. Every one has a folder.

“Then you’re sitting on a gold mine,” Priya says. “You don’t need a giant model to guess what the folders mean. You have 3,100 examples of what they mean, written by the person who knows best.”

The new postman’s trick

Priya draws a wall of pigeonholes on the whiteboard. “Imagine you start at the post office today. Sorting letters into pigeonholes, but nobody gives you the rules. All you have is the letters already sitting in each hole. A new letter arrives. What do you do?”

“Look at the piles,” Arjun says. “See which pile it looks most like.”

“How exactly? There are three ways you could do it. First way?”

He thinks. “I could form a picture of the ‘typical’ letter in each hole. Then compare the new one to each typical letter.”

“That’s the first method. Second way?”

“Find the few letters already sorted that look most like this new one, and see which hole most of them are in.”

“Second method. And the third is the one experienced sorters end up with: after a while, you stop comparing letters at all. You’ve learned where the boundaries are. Anything with a delivery complaint and a demand for money goes on this side of the line; anything about ingredients goes over there.”

“And ‘looks like’ means…?”

“Close together on the map of meaning. Embed every message, and ‘which pile does it look like’ becomes ‘which examples is it near’. You already know how to measure near.”

Following one message through all three methods

Trace a single new message through each method: “cake still not here, party at 6!!”

To keep the arithmetic readable, pretend the embedding has two dimensions instead of 384, and that Sunita has sorted only nine messages, three per folder. All vectors are already normalised to length 1.

Step 1: embed the new message. The model places it at $t=(0.97,0.23)$.

Method 1: nearest centroid

The centroid of a folder is the average of its examples: its “typical message”.

For late delivery, average each position: $((0.99+0.94+0.99)/3,(0.11+0.35-0.11)/3)=(0.973,0.117)$. Its length is $\sqrt{0.973^2+0.117^2}\approx 0.980$, so normalised it becomes $(0.993,0.119)$.

Doing the same for the other two gives refund $(0.263,0.965)$ and allergy $(-0.880,0.476)$.

Now the dot product of $t$ with each centroid, which on normalised vectors is the cosine similarity:

• Late delivery: $0.97\times 0.993+0.23\times 0.119=0.963+0.027=0.990$
• Refund: $0.97\times 0.263+0.23\times 0.965=0.255+0.222=0.477$
• Allergy: $0.97\times (-0.880)+0.23\times 0.476=-0.854+0.109=-0.744$

Highest score: late delivery.

Method 2: k-nearest neighbours

Skip the averages. Score $t$ against all nine examples and let the closest k vote. With $k=3$, the three nearest are all late-delivery examples, scoring 0.992, 0.986 and 0.935. Three votes to none: late delivery. This method is called k-nearest neighbours, or k-NN.

Method 3: a learned boundary

The third method draws lines across the map, one region per folder, learned from the examples. In two dimensions, picture straight lines between the three clusters. Our message sits deep inside the late-delivery region, far from any line. The method also reports how sure it is, as a probability per folder, perhaps 0.96 for late delivery here.

All three agree, because this is an easy message.

Now a hard one

Try “I want my money back, the cake came 3 hours late”. The model places it at $(0.71,0.71)$, between two clusters, because it really is about both.

• Nearest centroid: refund 0.872, late delivery 0.789. Refund wins, but not by much.
• k-NN with $k=3$: the nearest are a refund (0.951), a late delivery (0.916) and a refund (0.852). Refund wins two votes to one.
• A learned boundary: the message sits near the line, so the probabilities come out split, something like 0.55 refund and 0.41 late delivery.

Every method says refund, and every method is unsure. That is honest. The message belongs to both teams.

Opening the hood

You have now done classification: putting each item into one of a fixed set of buckets, called classes or labels. Here are the pieces, named properly.

Labelled data means examples where a person has already written the right answer, like Sunita’s 3,100 sorted messages.

A training set and a test set. You split the labelled data in two. The method learns from the training set. You measure it on the test set, which it has never seen. Testing on the training examples is like marking your own homework with the answer sheet open: the score looks wonderful and means nothing.

Nearest centroid stores one average vector per label. It is tiny (six vectors for Crumb’s six folders), fast, and easy to explain. Its weakness is labels with several distinct kinds of message inside them.

k-NN stores every training vector and searches them all for each new message. It copes with messy labels and explains itself (“I said refund because these three past messages were refunds”). For thousands of examples it is fast; for millions it needs the indexing tricks of the next module.

Logistic regression is a classic, simple machine-learning method that learns the boundaries. For each label it learns one weight per dimension (so 384 weights per label, plus one extra number), and turns the weighted sum into a probability. It is the “fence-drawing” sorter. With a few hundred labelled examples or more, on top of good embeddings, it is usually the most accurate of the three.

All three rely on the same thing: an embedding model that already places messages by meaning. The embedding model did the hard work, learned from hundreds of millions of texts. Your classifier only has to learn where Crumb’s folders sit on that map, which is why 3,100 examples are plenty. Training from scratch, without embeddings, would need far more.

Building the sorter

Step 1: embed and split

# prepare.py   (uv add sentence-transformers scikit-learn numpy)
import json
from sentence_transformers import SentenceTransformer
from sklearn.model_selection import train_test_split

tickets = json.load(open("tickets.json"))     # [{"text": "...", "label": "late_delivery"}, ...]
texts = [t["text"] for t in tickets]
labels = [t["label"] for t in tickets]

model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")   # handles Hinglish
X = model.encode(texts, normalize_embeddings=True, show_progress_bar=True)

# Keep 20% aside for testing. stratify keeps the same mix of labels in both halves.
X_train, X_test, y_train, y_test, t_train, t_test = train_test_split(
    X, labels, texts, test_size=0.2, stratify=labels, random_state=42)
print(X_train.shape, X_test.shape)

(2480, 384) (620, 384)

Embedding 3,100 short messages takes under a minute on a laptop. You do it once. Everything after this takes seconds. Arjun chose a multilingual model because a fifth of Crumb’s messages are in Hinglish, and an English-first model put those on their own island, as you saw two lessons ago.

Step 2: nearest centroid by hand

import numpy as np

centroids = {}
for label in set(y_train):
    members = X_train[[i for i, y in enumerate(y_train) if y == label]]
    c = members.mean(axis=0)
    centroids[label] = c / np.linalg.norm(c)        # normalise the average

def predict_centroid(vec):
    return max(centroids, key=lambda label: vec @ centroids[label])

msg = model.encode(["cake still not here, party at 6!!"], normalize_embeddings=True)[0]
print(predict_centroid(msg))

late_delivery

A working classifier in about ten lines. Building it yourself once means none of the library versions below will feel like magic.

Step 3: compare all three with scikit-learn

from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier, NearestCentroid

classifiers = {
    "nearest centroid":    NearestCentroid(),
    "k-NN (k=5)":          KNeighborsClassifier(n_neighbors=5, metric="cosine"),
    "logistic regression": LogisticRegression(max_iter=1000),
}
for name, clf in classifiers.items():
    clf.fit(X_train, y_train)                                   # learn from training set
    print(f"{name:22s} accuracy {clf.score(X_test, y_test):.2f}")   # judge on test set

nearest centroid       accuracy 0.84
k-NN (k=5)             accuracy 0.88
logistic regression    accuracy 0.92

Your numbers will depend entirely on your data. The pattern holds surprisingly often, though: logistic regression first, k-NN close behind, nearest centroid a few points back, all three trained in seconds. Compare that with the keyword rules’ 58%.

Where it breaks: the 92% that hid a problem

Meera is thrilled with 92%. Priya is not ready to celebrate. She prints a detailed report:

from sklearn.metrics import classification_report
clf = classifiers["logistic regression"]
print(classification_report(y_test, clf.predict(X_test)))

               precision    recall  f1-score   support
      allergy       0.89      0.71      0.79        24
late_delivery       0.95      0.96      0.95       186
 order_change       0.90      0.91      0.91       124
        other       0.84      0.83      0.83        87
       praise       0.96      0.97      0.96        99
       refund       0.92      0.93      0.92       100

Two numbers per label matter more than overall accuracy:

• Precision: of the messages the classifier put in this folder, what share really belonged there?
• Recall: of the messages that really belonged in this folder, what share did the classifier find?

Look at the allergy row. Recall is 0.71. Of the 24 real allergy questions in the test set, about 7 went somewhere else, mostly Other and Change my order. Allergy questions are only 4% of the inbox, so getting a quarter of them wrong barely dents the overall 92%. A classifier can look excellent overall while failing at the one folder that matters most.

Arjun adds three layers on top:

labels_order = list(clf.classes_)
ALLERGY = labels_order.index("allergy")
ALLERGY_WORDS = ("allerg", "peanut", "cashew", "almond", "gluten", "anaphyla", "reacts")

def route(text, vec):
    p = clf.predict_proba([vec])[0]
    if p[ALLERGY] > 0.20 or any(w in text.lower() for w in ALLERGY_WORDS):
        return "allergy"                           # low bar on purpose: recall first
    if p.max() < 0.60:
        return "needs_human"                       # unsure: let Sunita decide
    return labels_order[p.argmax()]

1. A low bar for allergy. If the allergy probability is above 0.2, even when another folder is higher, route it to allergy. Recall rises from 0.71 to 0.96; a handful of extra messages land in the head baker’s queue, which she says is fine.
2. A keyword safety net. Specific allergen words flag a message whatever the classifier says. Notice the list avoids the bare letters “nut”, so “donut” no longer triggers it. Boring, reliable and exactly right for this job.
3. A “not sure” path. If no folder reaches 0.6 probability, the message goes to Sunita. About 6% of messages take this path, including most of the “money back because it was late” ones.

More failure modes experienced engineers watch for:

• Leaking test data. If near-identical messages appear in both training and test sets, the test is too easy and the score lies.
• Labels that drifted. If Sunita split “Order change” into two folders in June, older labels teach the wrong lesson. Check that labels mean the same thing across your whole dataset.
• New kinds of message. When Crumb launches subscriptions, nothing in the training data looks like “pause my subscription”. Watch the “needs human” pile for new patterns and add labelled examples.
• Switching embedding models. A classifier trained on one model’s vectors cannot read another model’s vectors. Re-embed and retrain together.

No labels yet? Start from descriptions

What if Crumb had no sorted history? You can still begin. Write one sentence describing each folder, embed the descriptions, and use them as the centroids:

descriptions = {
    "late_delivery": "My order is late, has not arrived, or the delivery is delayed.",
    "allergy":       "A question about allergens such as nuts, gluten, eggs or dairy.",
    "refund":        "The customer wants their money back or a refund.",
}
desc_vecs = model.encode(list(descriptions.values()), normalize_embeddings=True)

This is called zero-shot classification: sorting without any labelled examples. It is rougher than a trained classifier, but it starts sorting on day one, and every message a person corrects becomes a labelled example for later.

Embeddings or an LLM?

Arjun’s discarded idea, asking an LLM for each message, deserves a fair comparison.

At Crumb’s volume, 400 messages a day, both are cheap: a small LLM would cost on the order of a couple of hundred rupees a month at the time of writing, and the embedding classifier a few paise. The real differences are speed, consistency and control. At 400,000 messages a day, cost would matter too. Many teams combine them: the embedding classifier handles the confident majority, and the unsure 6% go to an LLM or a person.

Arjun explains it to Meera

The sorter goes live behind a “suggested folder” button, so Sunita can accept or correct each guess with one click. A week later, Meera asks how it is going.

“Every new message gets placed on the map of meaning,” Arjun says. “We already have 3,100 messages that Sunita sorted. The computer looks at which of those the new one sits closest to, and suggests that folder.”

“How often is it right?”

“About 92% of the time overall. But the number I watch is allergies. Out of every 100 allergy questions, it now catches 96 straight away, because we set it to be extra careful there, plus a list of allergen words as a backup. When it’s unsure about anything, it hands the message to Sunita instead of guessing.”

“And the mother with the peanut question?”

“Today, that message would be in the head baker’s queue within a second of arriving.”

Sunita now finishes sorting by 10 a.m. She spends the time she saved answering customers instead of filing them.

Try it yourself

1. Create tickets.json with at least 20 messages for each of four labels. Use real, anonymised messages if you have them; otherwise write them yourself with plenty of variety, including typos and at least a few in another language you know.
2. Train all three classifiers and print their accuracy.
3. Print the classification_report and find the label with the lowest recall.
4. Print every test message the best classifier got wrong.

Expected result: logistic regression or k-NN scores highest; the lowest recall belongs to your smallest or vaguest label; and when you read the mistakes, a few turn out to be labelling errors in your own data.

clf = LogisticRegression(max_iter=1000).fit(X_train, y_train)
for text, true, guess in zip(t_test, y_test, clf.predict(X_test)):
    if true != guess:
        print(f"[{true} -> {guess}] {text}")

Common mistakes

• Testing on training data. The score looks wonderful and means nothing. Always hold out a test set.
• Looking only at accuracy. Rare labels hide inside a good overall number. Read precision and recall per label.
• Forcing every message into a folder. Give the system a “not sure” path to a person.
• Keyword rules on fragments. “nut” matches “donut”. If you use a keyword net, use whole, specific words.
• Mixing embedding models. Retrain the classifier whenever you change the model that makes its vectors.

Recap

• Classification sorts items into fixed labels; learn from a training set, measure on a held-out test set.
• Nearest centroid compares with each label’s average vector: tiny, fast and easy to explain, but weak on scattered labels.
• k-NN lets the closest past examples vote: handles messy labels and explains itself.
• Logistic regression on embeddings learns boundaries and gives probabilities; it is usually the most accurate.
• Read per-label precision and recall, set a low bar for safety-critical labels, add a keyword net, and send unsure cases to a person.

Priya catches Arjun on his way out. “You’ve built search, recommendations and a classifier, each on its own. Next, put them together into one project, with real measurements for each, and a write-up of the map underneath. That’s what turns a set of demos into something you understand.”

Go deeper

• scikit-learn documentation (see nearest neighbours, logistic regression and model evaluation)
• Sentence Transformers documentation
• Hugging Face LLM course