Context windows and why LLMs forget

Meera holds out her phone. On the screen is a test chat with Arjun’s prototype support assistant, which Crumb’s WhatsApp team has been trying out:

Customer: Hi, I’m Rohan. My order CR-20931 hasn’t arrived. Bot: Sorry to hear that, Rohan! Let me help with order CR-20931. Could you tell me what was in the order? Customer: A large Diwali hamper. Bot: Thanks! Could you share your order number so I can look into it?

“It asked for the order number,” says Meera, “fifteen seconds after he gave it. If a person on our team did that, I would have a word with them.”

Arjun winces. He had tested every message on its own, and every reply was good. He had never tested a conversation.

“The model is fine,” says Priya. “It has never met Rohan. Each time Arjun’s code calls it, it is meeting Rohan for the first time.”

Arjun’s two attempts to make it remember

Arjun opens his code. The core of it is this:

def reply_to(customer_message: str) -> str:
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=1000,
        system=CRUMB_INSTRUCTIONS,
        messages=[{"role": "user", "content": customer_message}],
    )
    return next(b.text for b in response.content if b.type == "text")

Every time a WhatsApp message arrives, the code sends Crumb’s instructions plus that one message. When Rohan wrote “A large Diwali hamper.”, that was all the model saw. No name, no order number, no earlier reply. Given only “A large Diwali hamper.”, asking for the order number was the sensible thing to do.

His first fix is to add a line to the instructions: “Always remember what the customer has told you earlier in the conversation.” He runs the same chat. The bot still asks for the order number.

His second fix works: he keeps a Python list of every message in the chat and sends the whole list each time. The bot remembers Rohan, the order and the hamper. Arjun is pleased, until the WhatsApp team runs a long stress-test chat that afternoon, pasting in the entire operations handbook “so the bot can check the rules”. After a while, every request fails with an error saying the prompt is too long.

“Two different problems,” says Priya. “First, the model forgets because it has no memory at all. Second, when you give it memory by resending everything, you eventually run out of desk.”

The counter clerk with no memory

Priya sketches the counter at the FC Road store.

“Imagine the most brilliant clerk you have ever hired. She knows everything about baking. But she has a strange condition: the moment a customer steps away from the counter, she forgets the entire conversation.”

“How would you work with her?”

“What would you do?”

Arjun thinks. “Write everything down. Every time the customer says something, hand her the full transcript. She reads it, answers, and forgets again.”

“That is exactly how every chatbot you have ever used works. The model is the clerk. In technical terms, it is stateless: it keeps no state, no memory, between one request and the next. ChatGPT, Gemini and Claude’s apps all seem to remember you because the app keeps the transcript and quietly sends the whole conversation, every time you press Enter.”

“And the desk?”

“The clerk can only read what fits on her desk. That desk has a fixed size.”

The context window is that desk: the maximum number of tokens (chunks of text, about three quarters of an English word each) that a model can handle in a single request. And everything shares it:

• the standing instructions (the system prompt),
• the conversation history,
• any documents or data you paste in,
• the new message,
• and the reply the model is about to write.

The last item surprises people. The model writes its answer onto the same desk. If the input fills almost all of it, there is no room for the reply.

Where the clerk analogy breaks: a human reading a long transcript skims. The model processes every token on the desk, every time, and you pay for every one.

Following one conversation, request by request

Let us trace Rohan’s chat after Arjun’s second fix, with real-looking token counts. Crumb’s instructions are 400 tokens. Each customer message is about 30 tokens, and each bot reply about 90.

Turn	What is sent	Input tokens this request	Output tokens
1	Instructions + message 1	400 + 30 = 430	90
2	Instructions + message 1 + reply 1 + message 2	430 + 90 + 30 = 550	90
3	Everything so far + message 3	550 + 120 = 670	90
4	Everything so far + message 4	790	90
…		grows by 120 each turn
10	Everything so far + message 10	1,510	90

Two things jump out.

First, the request grows every turn, because every earlier message rides along again. Turn 10 sends Rohan’s first message for the tenth time.

Second, the total adds up faster than the conversation does. Over 10 turns the model reads 9,700 input tokens. Over 20 turns, not 19,400, but 31,400. Double the length, more than triple the input. At $4permillioninputtokensthat20-turnchatcostsabout$0.13 in input alone, plus about $0.04 for output. Pennies for one chat, but the shape matters: long conversations get expensive faster than they get long.

sequenceDiagram
  participant App as Arjun's app
  participant Model as LLM
  App->>Model: Instructions + "Hi, I'm Rohan, order CR-20931"
  Model-->>App: "Sorry to hear that, Rohan..."
  Note over Model: Forgets everything
  App->>Model: Instructions + turn 1 (both sides) + "A large Diwali hamper"
  Model-->>App: "Thanks Rohan, let me check CR-20931..."
  Note over Model: Forgets everything again

How big is the desk?

Context windows have grown enormously. Early chat models handled about 4,000 tokens, roughly six pages. Today’s large models from OpenAI, Google and Anthropic accept hundreds of thousands of tokens, and several, including Claude Opus 5.5 and Gemini’s recent models, accept around a million. A million tokens is several thick novels, or Crumb’s 200-page handbook about eight times over.

There is usually a second, smaller limit: the most the model may write in one reply. For Claude Opus 5.5 it is 128,000 tokens. You choose your own cap below that for each request, using a setting called max_tokens that you will meet in the lesson on API calls.

Limits change with every model release. Each provider lists the current numbers on its models page.

“So we have a million tokens of desk,” says Arjun. “Why did the stress test break?”

Priya checks the logs. The WhatsApp team had pasted the handbook, about 120,000 tokens, into the chat, as a customer message. Then they kept chatting, and pasted it again twice more “to be sure”. Every pasted copy rode along in the history on every later request. After a few hours of testing, the history had grown past the limit, and the API refused the request.

Opening the hood: what the window really is

Why is there a limit at all? Two reasons.

Training. A model learns to use positions in text up to a certain length. If it was trained and tuned on sequences up to some length, it has no reliable experience beyond that, so providers set the window at what the model handles well.

Cost of looking back. When the model predicts each token, it looks back at every token in the window to decide what matters. That mechanism is called attention, and you will build it yourself in Module 30. Looking back over a million tokens takes far more memory and compute than looking back over a thousand. Providers have made this much cheaper, but it is still why long requests are slower and why the window cannot grow forever.

Watch the messages pile up on the desk. In the clip, when the window is full, the oldest messages fall off the edge. Real APIs do not do that for you. If you send too much, you typically get an error, and deciding what to drop is your code’s job. Some chat apps trim silently, which is why a long ChatGPT conversation can “forget” its beginning.

And a big window is not the same as a perfect memory. Researchers found that models use information near the start and the end of a long input better than information buried in the middle, an effect named lost in the middle. Newer models are much better at it, but the principle holds: one relevant paragraph is easier to use than the same paragraph hidden among 200 pages.

Statelessness is a feature, not a bug

It feels like a flaw at first. But a stateless model is a gift to engineers:

• You control exactly what the model sees. Want it to forget a mistake? Remove it from the history. Want it to know today’s date or the customer’s order status? Put it in.
• It scales easily. Any server can handle any request, because nothing is stored on a particular machine.
• It is debuggable. If a reply is strange, you can read the exact text that went in. There is no hidden memory to blame.

The price is that memory is your job. Some providers offer conveniences that look like memory: OpenAI’s Responses API can store a conversation on its side and continue it from an ID, and Google’s Gemini SDK has a chat helper that keeps the history in your program for you. Underneath, the model is still stateless: the earlier messages are fed back in, and you are still billed for them as input tokens.

The real code: forgetting and remembering

You will set up all three API keys in the lab two lessons from now. Come back and run this then. It uses Claude, but every provider behaves the same.

Step 1: see the forgetting.

# forgetful.py
import anthropic
from dotenv import load_dotenv

load_dotenv()
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY

def ask(messages):
    response = client.messages.create(
        model="claude-opus-5-5",
        max_tokens=1000,
        output_config={"effort": "low"},  # a simple task: think briefly
        messages=messages,
    )
    return next(b.text for b in response.content if b.type == "text"), response.usage

# Two separate requests. The second never sees the first.
print(ask([{"role": "user", "content": "Hi, I'm Rohan. My order is CR-20931."}])[0])
print(ask([{"role": "user", "content": "What is my order number?"}])[0])

The second reply says it does not know your order number. Each messages list contains exactly one message.

Step 2: carry the history. Keep a list, append each message and each reply, and send the whole list every time.

history = []
for text in [
    "Hi, I'm Rohan. My order is CR-20931.",
    "It was a large Diwali hamper.",
    "Can it go to my office instead?",
    "What was my order number again?",
]:
    history.append({"role": "user", "content": text})
    reply, usage = ask(history)
    history.append({"role": "assistant", "content": reply})
    print(f"turn {len(history) // 2}: sent {usage.input_tokens} input tokens")
print("Last reply:", reply)

Each message is a small dictionary with a role (who is speaking: user for the customer, assistant for the model) and its content. You will explore roles properly in Module 02.

You should see output shaped like this:

turn 1: sent 26 input tokens
turn 2: sent 97 input tokens
turn 3: sent 171 input tokens
turn 4: sent 248 input tokens
Last reply: Your order number is CR-20931.

The model now “remembers”, and the input grows every turn, exactly as in the table.

Step 3: check the desk before you send. The provider can count tokens for you before you spend money on a request. Use it to check that history plus reply space fits.

CONTEXT_LIMIT = 1_000_000   # Claude Opus 5.5's window; check the models page
REPLY_SPACE = 1_000         # the max_tokens we will ask for

needed = client.messages.count_tokens(
    model="claude-opus-5-5",
    messages=history,
).input_tokens

if needed + REPLY_SPACE > CONTEXT_LIMIT:
    print("Too big: trim the history before sending.")
else:
    print(f"OK: {needed:,} tokens of history, {CONTEXT_LIMIT - needed:,} left on the desk.")

Library APIs change quickly. If a name has moved, check the docs linked below.

Where it breaks in production

• The request is rejected. Exceed the window and the API returns an error instead of a reply. A chatbot with no trimming will eventually break on its longest, most engaged customers.
• The reply gets squeezed. If the input nearly fills the window, the reply runs out of room and stops mid-sentence.
• Costs climb with chat length. Every turn resends everything. A support chat that wanders for 40 turns costs many times what four short chats would.
• Quality can drop. A focused context beats a bloated one. Burying the one fact that matters in a huge history makes it easier to miss.
• Private data travels every time. Whatever is in the history, including phone numbers and addresses, is sent to the provider on every request. Send only what is needed.

The WhatsApp team’s fix was straightforward once the cause was clear: never paste reference documents into the chat history, keep only the most recent part of each conversation, and put the order number into the instructions once it is known. You will build all of that in Module 02, and learn to fetch only the relevant handbook paragraphs in Module 09.

Try it yourself

1. Run Step 2 and record the input tokens per turn. Add four more questions and predict the token count for turn 8 before you run it.
2. Change Step 2 to keep only the last two messages (history[-2:]) when calling ask. Ask “What was my order number again?” at turn 4. What happens, and why?
3. Meera asks: “If we paste our 120,000-token handbook into every request, what does a 10-turn chat cost in input tokens?” Use $4 per million.

Show the answers

1. The increase per turn is roughly constant (one question plus one reply), so extend the pattern: each turn adds about the size of one message and one reply. Your exact numbers depend on reply lengths.
2. The model no longer knows the order number, because the message containing it was trimmed away. Trimming saves tokens but loses information. Choosing what to keep is a real design decision; you will learn smarter strategies, like summarising old turns or saving key facts separately, in Module 02.
3. Each request carries the handbook, so 10 requests × 120,000 = 1.2 million tokens, plus a little conversation. 1.2 million × $4permillion=about$4.80 for one chat, roughly ₹420. Across thousands of chats a month that is lakhs of rupees, spent re-reading the same handbook.

Common mistakes

• “The model remembers our earlier chat.” It never does between requests. Memory is something your app builds by resending history.
• “Telling it to remember will help.” Instructions cannot conjure information that is not in the request.
• Forgetting the reply needs room. The output shares the window. Leave space.
• Treating the window as free storage. Every token is paid for, on every request, and slows the first word.
• “A bigger window means better answers.” More context can mean more noise. Send what is relevant.

Recap

• LLMs are stateless: each request starts from nothing.
• Chat apps fake memory by resending the conversation history on every turn.
• The context window is the maximum tokens per request, shared by instructions, history, documents, the new message and the reply.
• Resending history makes total cost grow faster than the conversation does.
• Exceeding the window causes errors, so trimming and choosing what to send is the engineer’s job.

That evening, Arjun explains it to Meera. “The model is like a brilliant clerk with no memory. Our app hands her the whole chat every time, on a desk of fixed size. If we hand her too much, she cannot work, and we pay for every page she reads. So we decide carefully what goes on the desk.”

Meera nods, then scrolls to another test chat. The same customer asked about a squashed cake twice, in two separate chats. Once the bot offered a full refund. Once it offered store credit. “And this? Is this the desk as well?”

“No,” says Priya. “That is the dice.”

Go deeper

• Claude docs, context windows and token counting: https://docs.anthropic.com
• Gemini API docs, long context: https://ai.google.dev/gemini-api/docs
• OpenAI docs, conversation state: https://platform.openai.com/docs