Context engineering and context window management

Mrs. Kulkarni had been chatting with Crumb’s website chatbot for twenty-five minutes, planning her granddaughter Anaya’s sixth birthday. In her third message she wrote: “Important: Anaya is allergic to sesame.” Then came thirty-seven more messages: sizes, flavours, whether the unicorn topper was edible, delivery slots, whether the Baner store could do a 4 p.m. pickup instead.

Message forty-one: “Okay, so of the cakes we discussed, which is safest for her?”

The bot replied, cheerfully: “The til-gud celebration cake would be a lovely choice!” Til is sesame.

Mrs. Kulkarni, who knew exactly what til was, phoned the store instead. No harm done. But Meera put the printed transcript on Arjun’s desk without a word, and that was worse than any word.

The same week, the finance team flagged something else: the chatbot’s API bill had gone from ₹18,000 in August to ₹36,500 in October, while the number of conversations had grown by only 20%.

Priya read the transcript, then the bill, then Arjun’s code. “These are the same bug,” she said.

Arjun’s chatbot, as it was

Here is the history-handling part of the chatbot Arjun had built in Module 02. Every turn, it appended the new message and sent everything. To keep costs under control, he had added a cap:

MAX_HISTORY_TOKENS = 4000

def estimate_tokens(text: str) -> int:
    return len(text) // 4          # rough rule of thumb for English text

def build_messages(system: str, history: list[dict]) -> list[dict]:
    # drop the oldest messages until the history fits under the cap
    while sum(estimate_tokens(m["content"]) for m in history) > MAX_HISTORY_TOKENS:
        history = history[1:]
    return [{"role": "system", "content": system}] + history

By message forty-one, Mrs. Kulkarni’s conversation held about 9,800 tokens. To fit under 4,000, the loop dropped the oldest messages, starting with message one. Message three, the allergy, went in the first batch. On the call that recommended til-gud cake, the model had never seen the word “sesame”. From its point of view, nobody had mentioned an allergy at all.

“Okay,” Arjun said. “So the cap is the problem. I’ll remove it. The model’s context window is 200,000 tokens; one conversation will never fill that.”

“Try it,” Priya said, “and then look at two things: the bill, and message forty-one.”

He removed the cap and replayed the conversation. This time the bot did not recommend the til-gud cake. But it recommended the “nutty caramel crunch”, which contains sesame brittle, with no allergy caveat. It had the allergy in its context this time, buried at message three of forty-one, between a long debate about unicorn toppers and a discussion of the delivery slots. It did not connect the two.

And the cost of that single reply had tripled.

What a good support agent keeps on the desk

Priya sketched a desk on the whiteboard. “Picture Kavita handling a long phone call. Does she keep a word-for-word transcript of the whole call in front of her?”

“No. She has the customer’s screen open. Name, order history. And she scribbles notes.”

“What goes in the notes?”

“The important stuff. Allergy, order number, the date of the party.”

“Where on the screen?”

“At the top. There’s a red allergy flag on the customer profile, actually. It’s always visible.”

“So she has: the standing rules she knows by heart, a few critical facts flagged at the top, a rough summary of the call so far in her notes, the last couple of things the customer said fresh in her mind, and the question in front of her. And she does not have a forty-minute transcript she has to re-read before every sentence.”

“That’s not what my bot has.”

“Your bot has everything in a pile, and when the pile gets too tall, the oldest papers fall off the edge. Including the one with the red flag on it.”

The model’s desk has a name: the context window, the maximum amount of text, measured in tokens (chunks of text, roughly three-quarters of a word each), that a model can read in one call. Everything the model knows about this situation has to be on it: instructions, history, documents, the question. And unlike Kavita, the model has no memory between calls. Every API call starts with an empty desk, and your code lays out the papers again from scratch.

From prompt engineering to context engineering

In the early lessons of this module, most of what the model saw was a prompt you wrote by hand. In a real chatbot, the hand-written part is a fraction of each call. Most of it is assembled by code: conversation history, the customer’s profile, product details, policies, results of earlier steps.

So the hard question has moved from “how do I phrase this?” to “what should the model see, right now, for this step?”. The discipline of answering that question is called context engineering. The term took off in 2025 and it is now how most teams describe the core work of building LLM applications. Writing good prompts is still part of it; it is no longer the whole job.

Following message forty-one through a designed context

Priya and Arjun redesigned what goes on the desk, in layers. Then they traced Mrs. Kulkarni’s message forty-one through it, with real numbers.

flowchart TD
    A["1. System instructions (fixed)"] --> B["2. Policies and allergen table (fixed)"]
    B --> C["3. Pinned facts for this customer"]
    C --> D["4. Product details relevant to this turn"]
    D --> E["5. Summary of the older conversation"]
    E --> F["6. Last six turns, word for word"]
    F --> G["7. The new message"]

Here is what each layer contained for message forty-one, and how many tokens it took:

Layer	Content	Tokens
1. System instructions	Crumb’s role, tone and rules	450
2. Policies	Allergen table for all cakes, delivery and pickup rules	600
3. Pinned facts	allergy: sesame (Anaya, granddaughter), event: 6th birthday, 14 Dec, fulfilment: pickup, Baner, 4 p.m.	40
4. Product details	The four cakes discussed, with ingredients	300
5. Summary	“Customer is choosing between four cakes for 14 Dec; prefers chocolate; asked whether the unicorn topper is edible (yes); switched from delivery to Baner pickup.”	150
6. Recent turns	Messages 35 to 40, verbatim	700
7. New message	“Okay, so of the cakes we discussed, which is safest for her?”	30
Total		2,270

The reply: “Since Anaya is allergic to sesame, I’d avoid the til-gud cake and the nutty caramel crunch, which both contain sesame. The chocolate truffle and the vanilla unicorn cake are made without sesame, though our kitchen does handle sesame, so please mention the allergy when you collect it at Baner on the 14th.”

That is the answer Kavita would have given. The allergy is now in a pinned block near the top of every call, not buried in turn three. The product ingredients are on the desk, so the model does not have to remember what was in which cake.

The cost, turn by turn. Use illustrative mid-size model prices: $3permillioninputtokens,$15 per million output tokens, and cached input at a tenth of the input price (prices change; check your provider’s pricing page).

• Old design, no cap, message forty-one: 9,800 input tokens × $3/M=$0.029, about ₹2.60 for one reply’s input.
• New design, message forty-one: layers 1 and 2 (1,050 tokens) are identical on every call, so they are served from the cache at $0.30/M:$0.0003. The other 1,220 tokens at full price: $\mathrm{0.0037.}Total:about$0.004, or ₹0.35.

Now the part that explained the doubled bill. In the old design, every turn resent the whole history. Turn one sent 1 message, turn two sent 2, turn forty-one sent 41. A forty-one-turn conversation resent roughly 200,000 tokens in total. Costs grew with the square of the conversation length: twice as long a chat cost about four times as much. October’s chats were longer than August’s, mostly because of Diwali planning, and the bill followed. In the new design, each turn costs roughly the same however long the chat runs.

Opening the hood: order, caching and why less is more

Two mechanisms explain why the order of those layers matters as much as their content.

Attention favours the edges. Models weigh every token, but in practice they use the start and the end of a long context most reliably. So the layout puts standing rules at the top, the latest question at the bottom, and promotes vital facts into a short pinned block near the top instead of leaving them wherever they happened to come up in conversation. If something is critical, it should never be “somewhere in the middle”.

Caching rewards a stable prefix. Prompt caching means the provider saves the work it did processing the beginning of your prompt, and when your next call starts with exactly the same text, it reuses that work: cheaper (often around a tenth of the normal input price) and faster. The catch is in the word exactly. The cache matches the longest identical prefix, the start of the prompt up to the first character that differs. One changed character near the top and everything after it is processed at full price.

So the rule is: stable first, volatile last. Instructions and policies (same for everyone, every call) at the top. Customer-specific facts next (same for the whole conversation). Recent turns and the new message last (different every call). You will see how providers implement caching, and how to measure hit rates, in Module 21.

Less is often more. Even within the window, more tokens means more for the model to sift through, and quality tends to slide as contexts grow long and noisy. This gradual decline is sometimes called context rot. A focused 2,000-token context often beats a sprawling 50,000-token one, even when both technically fit.

Building it, one layer at a time

Step 1: pin the critical facts. After each user message, run a small, cheap extraction call with structured output (from earlier in this module) to update a dictionary of facts that must never be lost:

from pydantic import BaseModel

class PinnedFacts(BaseModel):
    allergies: list[str] = []        # e.g. ["sesame (Anaya)"]
    event_date: str | None = None
    order_id: str | None = None
    fulfilment: str | None = None    # e.g. "pickup, Baner, 4 p.m."

def update_pins(pins: PinnedFacts, message: str) -> PinnedFacts:
    """Ask a small model to return the updated facts, given the old ones and the new message."""
    prompt = (f"Current facts: {pins.model_dump_json()}\n"
              f"New customer message: <message>{message}</message>\n"
              "Return the updated facts. Keep existing facts unless the customer changed them.")
    return extract(prompt, PinnedFacts)   # your structured-output helper from the earlier lesson

With a small model this costs a fraction of a paisa per message, and after message three, pins.allergies is ["sesame (Anaya)"] for the rest of the conversation.

Step 2: fold old turns into a rolling summary. When the history passes a threshold, summarise everything except the recent turns and keep only the summary:

KEEP_RECENT = 6

def compact(summary: str, history: list[dict]) -> tuple[str, list[dict]]:
    if len(history) <= KEEP_RECENT * 2:
        return summary, history                     # nothing to fold yet
    old, recent = history[:-KEEP_RECENT], history[-KEEP_RECENT:]
    transcript = "\n".join(f"{m['role']}: {m['content']}" for m in old)
    new_summary = ask(                              # any plain chat call
        f"Previous summary: {summary or 'none'}\n\nNew messages:\n{transcript}\n\n"
        "Update the summary in under 120 words. Keep decisions, preferences and open questions.")
    return new_summary, recent

Step 3: assemble the layers in cache-friendly order.

def build_context(system: str, policies: str, pins: PinnedFacts, products: str,
                  summary: str, recent: list[dict], new_message: str) -> list[dict]:
    stable = f"{system}\n\n<policies>\n{policies}\n</policies>"          # identical every call
    per_customer = (f"<pinned_facts>\n{pins.model_dump_json(indent=1)}\n</pinned_facts>\n"
                    f"<products_discussed>\n{products}\n</products_discussed>\n"
                    f"<earlier_conversation>\n{summary or 'none'}\n</earlier_conversation>")
    return ([{"role": "system", "content": stable + "\n\n" + per_customer}]
            + recent
            + [{"role": "user", "content": new_message}])

Step 4: check what you built. Print the token count of each layer for a real conversation:

msgs = build_context(SYSTEM, POLICIES, pins, products, summary, recent, "Which is safest for her?")
for m in msgs:
    print(m["role"], estimate_tokens(m["content"]))

For Mrs. Kulkarni’s conversation you should see a system message of about 1,500 tokens (layers 1 to 5), six recent turns of around 100 to 150 tokens each, and a short final user message, about 2,300 in total. The len(text) // 4 estimate is fine for budgeting; for exact counts, use the provider’s token-counting tool or a tokenizer.

Where it breaks

The redesign fixed the sesame bug and halved the bill. Running it in production for a month surfaced the next layer of problems.

The summary that changed a fact. In one conversation, the rolling summary turned “my son can’t have eggs” into “customer prefers eggless”. A preference and a medical restriction are very different things. Summaries are lossy and occasionally wrong. That is exactly why safety-critical facts live in the pinned block, extracted with a specific schema, and not only in the summary.

The pin that went stale. A customer changed from pickup to delivery halfway through, but the extraction prompt said “keep existing facts” a little too firmly and the pin still said pickup. Pinned facts need an explicit way to be changed, and a test conversation where the customer changes their mind.

The timestamp that broke the cache.

Too many product pages. An early version put all forty cakes’ ingredients on the desk “just in case”. It cost 4,000 extra tokens per turn and the bot started mixing up which cake had which nuts. Only the products actually being discussed belong there. Choosing the few relevant pieces from a large library is exactly the problem retrieval solves, the heart of RAG, which you will build in Module 09.

Try it yourself

Simulate a long conversation and watch the costs. Write fake history with history = [{"role": "user" if i % 2 == 0 else "assistant", "content": f"Turn {i}: " + "blah " * 60} for i in range(40)] (each turn is about 75 tokens). Then compute:

1. The total input tokens sent across all 40 turns with the old “send everything” design (turn n sends n messages).
2. The same total with the new design, assuming a fixed 1,500-token system block, the last 6 turns, and the new message on every call.
3. With the illustrative prices above ($3/Minput,$0.30/M cached), the rupee cost of each design for one conversation, if the 1,050-token policy-and-instructions part is cached in the new design.

Show the answer

1. Old design: 75 × (1 + 2 + … + 40) = 75 × 820 = 61,500 tokens of history, plus a 450-token system prompt 40 times (18,000): about 79,500 tokens.
2. New design: each call is roughly 1,500 + 6 × 75 + 75 = 2,025 tokens, times 40 calls: about 81,000 tokens. Surprise: about the same total for this short-message conversation!
3. But price them. Old: 79,500 × $3/M=$0.24, about ₹21. New: 40 × 1,050 cached tokens at $0.30/M($0.0126) plus 40 × 975 tokens at $3/M($0.117), about $0.13, or ₹11.4. And the old design’s cost keeps climbing with the square of the length: at 80 turns it is roughly three and a half times as much, while the new one only doubles.

The lesson: caching and a fixed-size context change the shape of the cost curve. And the real win was never only money. The pinned allergy is the part that would have saved Mrs. Kulkarni the phone call.

Common mistakes

• Sending everything, every time. Costs grow with the square of conversation length, and accuracy drops as context gets noisy.
• Blind truncation. Cutting the oldest messages loses early, critical details. Pin them first.
• Volatile content at the top. One changing line at the start breaks prompt caching for the whole prompt.
• Dumping whole documents. Put only the relevant pieces on the desk; retrieval does this properly.
• Trusting the summary with safety facts. Summaries are lossy. Critical facts get their own pinned, structured block.
• Assuming a bigger window fixes it. A larger window raises the ceiling; it does not make the model use buried facts reliably.

Recap

• The context window is the model’s whole world for one call, and your code rebuilds it from scratch every time.
• Context engineering is choosing what goes in, in what order and in what form, for each step.
• Layer it: fixed instructions and policies, pinned facts, relevant documents, a rolling summary, recent turns, the new message.
• Order it stable-to-volatile so prompt caching works, and keep vital facts near the edges, never lost in the middle.
• Sending full history makes cost grow with the square of conversation length; summaries, pins and caching keep each turn roughly constant.

The following Monday, Arjun walked Meera through the new design using Kavita’s desk as the example. “Before, the bot had a giant pile of every message, and when the pile got too tall, the oldest pages fell off, including the allergy. Now it works like Kavita does: our rules are always at the top, the important facts get a red flag that never falls off, older chat gets squashed into short notes, and only the last few messages are kept word for word. And because the top part never changes, the provider charges us a tenth for re-reading it.”

“And the bill?”

“October was ₹36,500. This month is tracking at about ₹14,000.”

Meera picked up the transcript she had left on his desk a week ago and put it in the bin. “Right,” she said. “Then let’s put all of this to work. I have a list.” It was a long list: summaries, translations, ticket routing, code, a training bot. That list is Module 04.

Go deeper

• Building effective agents (context design is central): https://www.anthropic.com/research/building-effective-agents
• Anthropic documentation, context windows and prompt caching: https://docs.anthropic.com
• OpenAI documentation, prompt caching: https://platform.openai.com/docs