Tokens and tokenisation

Meera wants a number before she approves anything. “If the AI drafts our support replies, what will it cost per month? I need a figure I can put in the budget.”

Arjun is confident about this one. He opens a spreadsheet. Crumb gets about 300 support emails a day, so 9,000 a month. He samples twenty emails: about 120 words each. A good reply is about 80 words. The pricing page for Claude Opus 5.5 says $4permillion\ast inputtokens\ast and$20 per million output tokens. He has seen people say “a token is basically a word”, so he multiplies:

	Arjun’s estimate
Input: 9,000 emails × 120 “tokens”	1.08 million tokens × $4=$4.32
Output: 9,000 replies × 80 “tokens”	0.72 million tokens × $20=$14.40
Per month	$18.72, about ₹1,650

“Less than a team lunch,” he tells Meera, and she approves a one-week trial.

A week later, Priya opens the provider’s usage dashboard. The trial handled 2,100 emails. The bill for the week is $\mathrm{14.20.}Scaledtoamonth,thatisabout$61, more than three times Arjun’s number.

“Same model, same emails,” says Arjun. “Where did the extra money come from?”

Priya scrolls through the trial’s emails. A third are in Marathi or Hindi. Many paste in a full order confirmation. And every request carried a page of instructions about Crumb’s policies. “Three things your spreadsheet did not count. And one wrong assumption underneath all of it: a token is not a word.”

Why a model cannot read letters

“Start here,” says Priya. “What is a model, mathematically?”

“A big calculation. Billions of numbers.”

“Right. Can you multiply the letter c by 0.37?”

“No.”

“So before the model sees anything, your text must become numbers. The question is: numbers for what? Letters? Words? Something else? You design it. Go.”

Arjun thinks out loud. “One number per letter. a is 1, b is 2…”

“Fine. ‘Is your Christmas plum cake eggless?’ is 36 characters, so 36 numbers. A 200-page handbook is half a million. Remember from the last lesson that the model does a full pass of its billions of parameters for every token it writes. One token per letter makes everything long, slow and expensive.”

“Then one number per word.”

“Better. How many words are there?”

“In English? A few hundred thousand?”

“Plus every word in Marathi, Hindi, Tamil, Japanese. Plus every name, every typo, every product code like CR-20931, every emoji, every bit of code. Your list never ends, and when a customer writes ‘eggless-ish’, you have no number for it.”

Arjun grins. “So something in between. Common words get one number. Rare words get built from pieces.”

“You just reinvented the tokenizer.”

Stock codes for text

Priya points through the glass at the stock room. “When the kitchen orders almond flour, nobody writes ‘one 500-gram bag of blanched almond flour’. They write AF-500. Common items get short codes. A rare item, like a special imported vanilla, gets written out piece by piece.”

A tokenizer is the program that does this for text. It splits text into tokens and gives each one a token ID, its number in a fixed list. That fixed list is the vocabulary. The tokenizer used by recent OpenAI models has a vocabulary of about 200,000 tokens. Every model family has its own tokenizer, built before the model is trained and frozen with it forever.

Where the stock-code analogy breaks: stock codes are designed by people for meaning. Tokens are chosen by a statistical process for frequency. Tokens are often fragments that mean nothing on their own, like ievably or aju.

Following one email through the tokenizer

Here is a real customer question, and exactly how OpenAI’s current tokenizer (called o200k_base) splits it. These are the real IDs.

Hi, my order CR-20931 hasn’t arrived. Is the kaju katli nut-free?

#	Token	ID		#	Token	ID
1	Hi	12194		11	.	13
2	,	11		12	Is	2763
3	my	922		13	the	290
4	order	2569		14	kaj	15156
5	CR	14964		15	u	84
6	-	12		16	kat	9015
7	209	23078		17	li	1307
8	31	2911		18	nut	9030
9	hasn't	39519		19	-free	14450
10	arrived	18157		20	?	30

Eleven words became twenty tokens. Look at where the extra ones came from:

• Punctuation is its own token. The comma, the full stop and the question mark each cost one.
• Spaces belong to the token after them. order (with the space) is one token. That is why most tokens in the table start with a space.
• Common English words are one token, even long ones: arrived, and even hasn't.
• Codes and numbers shatter. CR-20931 became four tokens: CR, -, 209, 31. Numbers are chopped into chunks of up to three digits.
• Words the tokenizer saw less often get split. kaju became kaj + u, and katli became kat + li. The tokenizer was built mostly from English-heavy text, where “kaju katli” is rare.

What the model actually receives is not the sentence. It is this list:

[12194, 11, 922, 2569, 14964, 12, 23078, 2911, 39519, 18157, 13, 2763, 290, 15156, 84, 9015, 1307, 9030, 14450, 30]

When the model replies, the same thing happens in reverse. It predicts IDs one at a time, and the tokenizer turns them back into text for you.

How the tokenizer chose its chunks

“But who decided arrived gets one token and kaju gets two?” asks Arjun.

“Nobody, directly. A counting process did.” Priya writes a tiny corpus on the board: the words cake (5 times), cakes (2), bake (3), baked (2).

The most common method is byte-pair encoding, or BPE. It works like this:

1. Start with every word split into single characters: c a k e, c a k e s, b a k e, b a k e d.
2. Count every pair of neighbours across the whole corpus. a k appears 12 times (in every word), k e also 12, c a 7, b a 5, e s 2, e d 2.
3. Merge the most frequent pair into a new token. Merge 1: a+k → ak.
4. Recount and repeat. Merge 2: ak+e → ake (12 times). Merge 3: c+ake → cake (7 times). Merge 4: b+ake → bake (5 times).

After four merges, the vocabulary holds c a k e s b d ak ake cake bake. Now:

• cake → one token: cake
• cakes → two tokens: cake + s
• baked → two tokens: bake + d

Real tokenizers run this on billions of words of text and stop after a couple of hundred thousand merges. Frequent words end up as single tokens. Rare words stay in pieces. And because the starting alphabet is every possible byte (the 256 basic units computers store text in), any text at all, including emoji and scripts the tokenizer barely saw, can always be represented. Nothing is ever “unknown”. Rare things are just long.

The rules of thumb, and where they fail

For ordinary English prose, these estimates hold well:

Measure	Roughly
1 token	about 4 characters
1 token	about three quarters of a word
1 English word	about 1.3 tokens
1 page of prose	about 500 to 700 tokens
200-page handbook	about 100,000 to 140,000 tokens

Arjun’s “1 word = 1 token” was already 30% low for plain English. But Crumb’s inbox is not plain English. Priya runs the same question through the tokenizer three ways:

Text	Words	Tokens	Tokens per word
“Is your Christmas plum cake eggless?”	6	8	1.3
Same question in Hindi	9	17	1.9
Same question in Marathi	7	21	3.0

The Marathi question costs 2.6 times as many tokens as the English one, for the same meaning. Here is how it was split:

['त', 'ुम', 'चा', ' ख', '्र', 'िस', 'म', 'स', ' प्ल', 'म', ' के', 'क', ' अ', 'ंड', '्य', 'ाश', 'िव', 'ाय', ' आहे', ' का', '?']

Many tokens are single letters or vowel signs. And this is the improved tokenizer. OpenAI’s previous tokenizer, cl100k_base, needed 40 tokens for the same Marathi sentence, five times the English cost. Tokenizers have become much fairer to Indian languages, but the gap has not closed.

Structured data is also expensive. Priya compares two ways to give the model the same order information:

Format	Example	Tokens
JSON from the order database	{"order_id": "CR-20931", "status": "out_for_delivery", "items": [{"sku": "DIW-HAMP-L", "qty": 1}]}	38
A plain sentence	Order CR-20931 is out for delivery. Items: 1 large Diwali hamper.	19

Every bracket, quote and underscore is a token. The plain sentence carries the same facts for half the price. The model reads both equally well.

Why the model cannot count the r’s in “strawberry”

Tokens also explain one of the most famous LLM failures. For a long time, if you asked a chat model “How many r’s are in strawberry?”, it often said two.

Look at what the model receives. In the middle of a sentence, strawberry (with its leading space) is one token: ID 101830. Written alone without a space, it becomes three: st, raw, berry. Either way, the model never sees the individual letters. Asking it to count the r’s is like handing someone the stock code AF-500 and asking how many l’s are in “almond flour”. The information is not in the code. The model has to have memorised the spelling of each token, which it partly has, imperfectly.

Newer models usually get this right, often by spelling the word out letter by letter first. But the lesson stands: anything that depends on individual characters (exact spelling, counting letters, reversing a string, checking that an order code has exactly eight characters) is a weak spot. When you need character-level precision, do it in ordinary Python code, not in the model.

Back to the bill: recounting in tokens

Now Priya and Arjun redo the estimate with real token counts from the trial week. Per email, the request contained:

Part of each request	Average tokens
Instructions about Crumb’s policies (sent every time)	450
Order record from the database, as JSON	240
The customer’s email (two thirds English at ~160, one third Marathi or Hindi at ~420)	250
Input total	940
Output: the drafted reply (some written in Marathi)	150

Monthly, at 9,000 emails:

	Tokens	Price per million	Cost
Input	9,000 × 940 = 8.46 million	$4	$33.84
Output	9,000 × 150 = 1.35 million	$20	$27.00
Total			$60.84, about ₹5,350

That matches the trial bill. Look at where Arjun’s estimate went wrong: he counted only the email (250 of the 940 input tokens), he counted words instead of tokens, and he forgot the Marathi multiplier.

It also points straight at cheap fixes. Converting the order JSON into a plain sentence saves about 120 tokens per email. Trimming the instructions from 450 to 300 tokens saves another 150. Together that cuts the input cost by about 29%. Small changes in tokens, multiplied by 9,000 emails, become real money.

The real code: counting tokens yourself

Now do this properly in code. Use the crumb-ai project from the setup lab later in this module, or any folder with uv.

Step 1: count locally with tiktoken. OpenAI publishes its tokenizer as a small library that runs on your laptop, with no API key and no cost.

uv add tiktoken

# count_tokens.py
import tiktoken

enc = tiktoken.get_encoding("o200k_base")  # the tokenizer of recent OpenAI models

text = "Hi, my order CR-20931 hasn't arrived. Is the kaju katli nut-free?"
ids = enc.encode(text)                     # text -> list of token IDs
pieces = [enc.decode([i]) for i in ids]    # each ID -> its chunk of text

print(len(text.split()), "words ->", len(ids), "tokens")
print(ids)
print(pieces)

11 words -> 20 tokens
[12194, 11, 922, 2569, 14964, 12, 23078, 2911, 39519, 18157, 13, 2763, 290, 15156, 84, 9015, 1307, 9030, 14450, 30]
['Hi', ',', ' my', ' order', ' CR', '-', '209', '31', " hasn't", ' arrived', '.', ' Is', ' the', ' kaj', 'u', ' kat', 'li', ' nut', '-free', '?']

encode turns text into IDs. decode turns IDs back into text; decoding one ID at a time lets you see each chunk.

Step 2: a cost estimator. Wrap the counting in a function that turns tokens into money. Keep prices as variables, because they change.

PRICE_IN = 4.00     # dollars per million input tokens (Claude Opus 5.5 at time of writing)
PRICE_OUT = 20.00   # dollars per million output tokens
INR_PER_USD = 88    # check today's exchange rate

def monthly_cost(input_tokens: int, output_tokens: int, requests: int) -> None:
    dollars = (input_tokens * PRICE_IN + output_tokens * PRICE_OUT) * requests / 1_000_000
    print(f"${dollars:,.2f} per month (about ₹{dollars * INR_PER_USD:,.0f})")

monthly_cost(input_tokens=940, output_tokens=150, requests=9_000)

$60.84 per month (about ₹5,354)

Step 3: ask the providers to count. tiktoken is exact for OpenAI models and only an estimate for others, because Claude and Gemini use their own tokenizers that do not run on your laptop. Both offer a free counting call. Once your keys are set up:

import os
import anthropic
from google import genai
from dotenv import load_dotenv

load_dotenv()
text = "Hi, my order CR-20931 hasn't arrived. Is the kaju katli nut-free?"

claude = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
count = claude.messages.count_tokens(
    model="claude-opus-5-5",
    messages=[{"role": "user", "content": text}],
)
print("Claude:", count.input_tokens)

gemini = genai.Client()  # reads GEMINI_API_KEY
result = gemini.models.count_tokens(
    model=os.getenv("GEMINI_MODEL", "gemini-2.5-flash"),
    contents=text,
)
print("Gemini:", result.total_tokens)

Library APIs change quickly. If a name has moved, check the docs linked below.

You will see different numbers from each provider, and both will be a little higher than you expect for one short sentence. Each provider wraps your message in a few formatting tokens that mark where a message starts and who is speaking. That overhead is small, but it is real and billed.

The most accurate count of all comes free with every real request: each response reports exactly how many input and output tokens it used. You will read those numbers in the lesson on the anatomy of an API call.

Where tokens bite in production

• Budgets in words. Always convert to tokens, and measure on a sample of your real data, not on a tidy English example.
• Hidden repeated text. Instructions sent with every request are paid for every time. 450 tokens × 9,000 requests is 4 million tokens a month.
• Structured data. JSON, HTML and long IDs are token-hungry. Send the model only the fields it needs, in the plainest form that works.
• Other languages. Indian-language text can cost two to three times more than English. Budget for it, and test reply quality in each language you support.
• Character-level tasks. Counting, spelling and exact-length checks belong in Python, not in the model.
• Different tokenizers. Switching providers changes your token counts, and therefore your costs, even for identical text.

Try it yourself

1. Run count_tokens.py on these three strings and predict the token count before you run each one: CAKE, Cake, Koregaon Park.
2. Take one real-looking email of your own (write it in English, then translate it into Hindi or Marathi using any translator) and compute tokens per word for both.
3. Use monthly_cost to answer Meera’s follow-up: “What if we get 15,000 emails a month in Diwali season, and replies get longer, about 250 tokens?”

Show the answers

1. Cake is 1 token, but CAKE is 2 (CA + KE): capitals are rarer, so they split. Koregaon Park is 4: K, ore, gaon, Park. Place names outside the English-speaking world often split.
2. Expect English at around 1.3 tokens per word and Hindi or Marathi at around 2 to 3. Your exact figure depends on the text.
3. monthly_cost(940, 250, 15_000) gives $131.40 (about ₹11,563). Output was 150 tokens and is now 250, and output is the expensive side, so longer replies hurt more than more emails.

Common mistakes

• “A token is a word.” In English it is about three quarters of a word, and in many other languages much less. Budget in tokens.
• Counting only the user’s message. Instructions, data and conversation history all count, on every request.
• Using tiktoken as the exact count for Claude or Gemini. It is a decent estimate. Use each provider’s counting endpoint or, better, the usage numbers in real responses.
• Forgetting the output side. Output tokens cost several times more than input. Shorter replies are the cheapest optimisation.
• Asking the model for character-level precision. It sees tokens, not letters. Check lengths, formats and spellings in code.

Recap

• A tokenizer turns text into tokens with ID numbers, from a fixed vocabulary of around 100,000 to 200,000 entries.
• BPE builds that vocabulary by repeatedly merging the most frequent pairs, so common words are one token and rare ones split into pieces.
• In English, one word is about 1.3 tokens; Indian languages, codes, numbers and JSON cost noticeably more.
• Prices and limits are counted in tokens, with input and output priced separately and output costing more.
• Each model family has its own tokenizer, so measure with the one you actually use.

Arjun sends Meera the corrected sheet with a one-line summary: “The model reads in chunks called tokens, not words. Marathi and order codes take more chunks, and we were sending a page of instructions every time. Real cost is about ₹5,400 a month, and I can cut a third of that by sending less.”

Meera replies a minute later: “₹5,400 is fine. But I read that these models cost hundreds of crores to make. If the model already exists, why are we paying every single time we use it?”

Go deeper

• OpenAI docs, including tokens and pricing: https://platform.openai.com/docs
• Claude docs, including the token counting endpoint: https://docs.anthropic.com
• Gemini API docs, including count_tokens: https://ai.google.dev/gemini-api/docs
• Hugging Face LLM course, the tokenizers chapter: https://huggingface.co/learn/llm-course