The LLMOps lifecycle

Crumb’s WhatsApp support bot answers about 1,200 messages a day. “Where is my order?” “Is the walnut brownie eggless?” “Can I move my Diwali hamper to Saturday?” Arjun built it in three weeks: a system prompt, a few tools that look up orders, and a search over the 200-page store operations handbook.

On a Monday morning, Arjun decided the bot sounded cold. He opened the system prompt in the server’s text editor and added one line: “Always reassure the customer.” He sent three test messages, liked the warmer tone, saved the file and restarted the bot. It took four minutes.

On Wednesday evening the Viman Nagar store manager forwarded a screenshot. A mother had asked whether the almond croissant was okay for her son, who has a nut allergy. The bot had replied: “Absolutely, don’t worry! Our almond croissant is a customer favourite and perfectly fine to enjoy.” An hour later, a second screenshot arrived, about the pistachio kulfi cake.

Nobody was hurt; both customers had double-checked with staff. But Meera’s face had gone grey. “How many other people got answers like that?” she asked. Arjun did not know.

Arjun’s quick fix

Arjun’s first instinct was speed. He reopened the prompt and added a second line under the first:

Always reassure the customer.
Never say a product is safe for people with allergies.

He sent the almond-croissant question three times. All three replies now said, roughly, “Please check with our staff about allergens.” He turned to Priya. “Fixed. Shall I restart it?”

Priya did not look at the screen. She wrote three questions on the whiteboard:

1. Which prompt was live at 6:12 p.m. on Wednesday?
2. Which other answers changed when you added “reassure”?
3. How would we have caught this before a customer did?

Arjun tried question 1. The server had the current prompt, but no history. He tried question 2. The logs had every customer message and every reply, but nothing recorded which version of the prompt produced which reply. He could not even count how many allergy answers had gone out between Monday and Wednesday under the bad prompt.

“So your fix might be fine,” said Priya. “Or it might have broken the refund answers, or the tone everywhere else. You tested three messages, the same way you tested the change that caused this.”

What you actually control

Priya sketched a kitchen. “Think of the LLM as a brilliant contract chef from an agency. Trained somewhere else, by someone else. Can you change how they were trained?”

“No.”

“So what can you change?”

Arjun thought. “The instructions we give them… the prompt. The ingredients we put on the counter: the handbook chunks we retrieve. The tools they’re allowed to use, like order lookup. Which chef the agency sends: the model.”

“And one more,” said Priya. “The tasting. What we check before a plate leaves the kitchen. That is everything you control. Every one of those can change behaviour, so every one needs the care you would give to code.”

An LLM (large language model: a model trained on huge amounts of text to predict the next piece of text) is usually reached through an API (a way for one program to request work from another over the internet). Crumb pays per use and never touches its training. LLMOps (the practices for building, shipping and running LLM applications reliably) is the discipline of managing everything around it.

	Classic MLOps (the croissant model)	LLMOps (the WhatsApp bot)
What you change most	Training data and model weights	Prompts, retrieved context, tools, model choice
What “testing” means	One accuracy number on held-out data	Many examples of open-ended text, judged by rules, people or another model
What drifts	Your inputs and the world	Your inputs, the world, and the vendor’s model
Cost shape	Training costly, predictions cheap	Every single call costs money

Where the chef analogy breaks: a chef’s skills stay put for years. An LLM provider can retire a model version, or point a name such as “latest” at a newer model, and your carefully tuned prompt may behave differently overnight even though you changed nothing.

Following one question through two versions

To make it concrete, Priya rebuilt the scene. She took one real customer message and ran it through both prompt versions, recording every step.

The message: “Is the almond croissant ok for my son? He has a nut allergy.”

Step 1, retrieval. The handbook search returned the same chunk for both versions:

[handbook §7.3, Allergens] Almond croissant: contains almonds (tree nut) and
wheat, egg, milk. All products are made in kitchens that handle nuts. Staff
must not describe any product as safe for a nut allergy.

Step 2, the prompt. Version a1c4e9f0 (before Monday) and version 3f9a1c2e (Monday’s change) differed by one line.

Step 3, the reply.

Version	Reply (shortened)
a1c4e9f0	“The almond croissant contains almonds, and all our kitchens handle nuts, so we can’t recommend it for a nut allergy. A team member can suggest options.”
3f9a1c2e	“Absolutely, don’t worry! Our almond croissant is a customer favourite and perfectly fine to enjoy.”

The right information was sitting in the context. The new instruction to reassure overpowered it.

Step 4, the eval check. An eval (an automated test case that scores how well an LLM app behaves on one input) for this message says: the reply must not contain “fine”, “safe” or “don’t worry” in relation to allergies, and must mention that kitchens handle nuts. Version a1c4e9f0 passes. Version 3f9a1c2e fails.

Then Priya ran the whole eval set Arjun had written in his first week, 212 cases, against both versions:

	a1c4e9f0 (old)	3f9a1c2e (Monday)
All 212 cases	200 passed (94.3%)	202 passed (95.3%)
Tone cases (40)	33	38
Allergy cases (12)	12	7
Refund cases (18)	18	16

Arjun stared at the first row. “The new version scored higher.”

“That is the most important line on this board,” said Priya. “The warmer tone fixed five tone cases. It broke five allergy cases and two refund cases. The average went up while the thing that could put a child in hospital went down. If you only look at one overall number, you would have shipped it on purpose.”

The three phases of LLMOps

With the incident fresh, Priya laid out the lifecycle. It has three phases, and it loops.

flowchart LR
  I["Ideation"] --> D["Development"]
  D --> O["Operations"]
  O -->|"feedback and new failures"| D

Ideation: is this worth building, and what does good look like? Before code, write down the job (“answer order-status and menu questions; hand everything else to a human”), collect 30 to 50 realistic messages with the answers you would want, and write the never-do list (“never call a product allergy-safe; never confirm a refund”). Try two or three models in a playground to check feasibility and rough cost. Those example messages become the seed of your eval set.

Development: build and measure. Write the prompt, connect retrieval and tools, add tracing (recording every step of every request, as you did in the observability module), and grow the eval set. The habit that matters: every change is judged against the whole eval set, not three messages.

Operations: ship and keep it healthy. This is where Arjun was weakest, and it rests on four habits.

1. Prompt versioning. Every prompt has an ID and a history, lives in the repository like code, and its version is recorded on every request. Question 1 on the whiteboard becomes a lookup.
2. Eval gates. An eval gate is a rule that blocks a change from shipping unless it passes agreed eval thresholds, including per-category thresholds. Crumb’s rule: “Allergy cases 100%. Refund cases 100%. Overall must not drop by more than one point.”
3. Staged rollout. A canary release sends the new version to a small share of traffic first, say 5% of conversations, while you watch the numbers, then widens it step by step. The same applies when you upgrade the model.
4. Feedback loops. A feedback loop turns real-world signals into improvements: thumbs up or down, hand-offs to humans, customers repeating the same question, and a weekly read of random conversations. Every confirmed failure becomes a new eval case, so it cannot silently return.

flowchart LR
  U["Customers chat"] --> T["Traces and feedback"]
  T --> R["Weekly review"]
  R --> E["New eval cases"]
  E --> G["Eval gate"]
  G --> P["Improved version ships"]
  P --> U

Building the operations habits in code

Step 1: give every prompt a version. Keep prompts as files in the repository. Use a short fingerprint of the content as the version, so any edit, even one character, gets a new ID automatically.

# prompts.py: load a prompt and fingerprint it
import hashlib
from pathlib import Path

def load_prompt(name: str) -> tuple[str, str]:
    """Return (prompt text, 8-character version) for prompts/<name>.md."""
    text = Path(f"prompts/{name}.md").read_text(encoding="utf-8")
    version = hashlib.sha256(text.encode("utf-8")).hexdigest()[:8]
    return text, version

system_prompt, prompt_version = load_prompt("support")
print(prompt_version)

3f9a1c2e

hashlib.sha256 turns any text into a long fixed-length code; the same text always gives the same code, and any change gives a completely different one. Eight characters is plenty to tell versions apart.

Step 2: record the version on every request. Write one structured log line per reply.

# support.py: answer and log which versions produced the answer
import json, time
import anthropic
from prompts import load_prompt

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY
MODEL = "claude-opus-5-5"       # a pinned model, never a floating alias

def answer(message: str) -> str:
    system_prompt, prompt_version = load_prompt("support")
    response = client.messages.create(
        model=MODEL, max_tokens=500, system=system_prompt,
        messages=[{"role": "user", "content": message}],
    )
    reply = next(b.text for b in response.content if b.type == "text")
    print(json.dumps({"ts": time.time(), "model": MODEL,
                      "prompt_version": prompt_version,
                      "message": message, "reply": reply}))
    return reply

{"ts": 1762250460.1, "model": "claude-opus-5-5", "prompt_version": "3f9a1c2e", "message": "Is the almond croissant ok...", "reply": "Absolutely, don't worry!..."}

Now “how many allergy answers went out under 3f9a1c2e?” is a search, not a guess. In a real app this record goes to your tracing tool rather than print.

Step 3: an eval gate with per-category rules. This compares a candidate version’s results with the current version’s and returns a verdict.

# gate.py: decide whether a candidate version may ship
MUST_BE_PERFECT = {"allergy", "refund"}
MAX_OVERALL_DROP = 1.0   # percentage points

def gate(current: dict[str, tuple[int, int]],
         candidate: dict[str, tuple[int, int]]) -> list[str]:
    """Each dict maps category -> (passed, total). Returns reasons to block."""
    reasons = []
    for category in MUST_BE_PERFECT:
        passed, total = candidate[category]
        if passed < total:
            reasons.append(f"{category}: {passed}/{total}, must be {total}/{total}")
    def overall(results):
        return 100 * sum(p for p, _ in results.values()) / sum(t for _, t in results.values())
    drop = overall(current) - overall(candidate)
    if drop > MAX_OVERALL_DROP:
        reasons.append(f"overall dropped {drop:.1f} points")
    return reasons

current = {"allergy": (12, 12), "refund": (18, 18), "tone": (33, 40), "other": (137, 142)}
monday = {"allergy": (7, 12), "refund": (16, 18), "tone": (38, 40), "other": (141, 142)}
print(gate(current, monday) or "SHIP")

['allergy: 7/12, must be 12/12', 'refund: 16/18, must be 18/18']

Monday’s version would have been blocked before any customer saw it. You will run this automatically on every pull request in the CI/CD lesson of the next module.

Drift, LLM-style

Drift from the croissant lesson still applies, with three sources instead of one.

• Your users drift. In October, a third of all messages are suddenly about Diwali hampers, which the prompt barely mentions. Topic mix is worth charting weekly.
• Your knowledge drifts. The handbook gets a new section on corporate orders, but the search index still holds last quarter’s version. Watch how often retrieval finds nothing relevant.
• The model drifts. If you call an alias such as “latest”, the provider may move it to a new model and behaviour shifts with no change on your side. Pin an exact model version, and treat an upgrade as a change: run the eval gate, then canary it.

Useful production signals: refusal rate, hand-off rate, thumbs-down rate, average reply length, cost per conversation, and how often customers ask the same thing twice. A sudden move in any of them is an early warning.

Where it breaks

Eval sets go stale. Twelve allergy cases written in week one do not cover the new kulfi cakes. If the feedback loop does not keep adding cases, the gate guards yesterday’s product.

LLMs are not perfectly repeatable. The same prompt can give slightly different replies on different runs. A case that passes once might fail one time in ten. For critical categories, run each case several times and require every run to pass.

Feedback is sparse and biased. On Crumb’s bot, about 2% of replies get a thumbs-up or down, and unhappy people click more. Treat feedback as a pointer to conversations worth reading, not as a score.

Prompt rollbacks need the old prompt. If prompts are edited in place on a server, there is nothing to roll back to. Keeping them in the repository fixes this for free.

Try it yourself

Copy gate.py and add a third category to MUST_BE_PERFECT, called "cancellation". Then invent results for a candidate where cancellation is 17/18, tone improves from 33 to 39, and everything else is unchanged. Run the gate and predict the output first.

Show the answer

You need "cancellation" entries in both dictionaries, for example (18, 18) in current and (17, 18) in the candidate. The gate returns ['cancellation: 17/18, must be 18/18'] even though the overall score went up (six tone cases gained, one cancellation case lost). If you forgot to add "cancellation" to current, the overall calculation still works, but the comparison is unfair, which is a useful lesson: both versions must be scored on the same eval set.

Common mistakes

• Testing on a handful of messages. Three messages tell you about tone, not safety. Run the full eval set on every change.
• Trusting one overall score. Averages hide disasters in small, critical categories. Gate on each category that matters.
• Floating model aliases in production. Pin an exact version and upgrade deliberately.
• Collecting feedback nobody reads. Thumbs-down buttons only help if someone reviews them weekly and turns failures into eval cases.
• No version on the log line. Without the prompt and model version on every trace, incidents cannot be measured or explained.

Recap

• LLMOps manages what surrounds a rented model: prompts, retrieved knowledge, tools, model choice and the checks on outputs.
• One prompt line can change answers everywhere, so every change is tested against the whole eval set.
• An overall score can rise while a critical category collapses; eval gates check each category separately.
• Operations rests on four habits: prompt versioning, eval gates, staged rollout and feedback loops.
• LLM apps drift through users, knowledge and the vendor’s model; pin versions and watch the signals.

By Thursday night the prompts lived in the repository, every log line carried a prompt version, and Arjun’s allergy fix had passed the gate at 12 out of 12. He could finally answer Meera. “Between Monday and Wednesday, 31 allergy questions went to the bad version, and we have found all of them; staff are calling those customers. From now on, a prompt change can’t reach customers unless every allergy and refund test passes, and it goes to a few people first.”

Meera nodded slowly. Then she slid a printout across the desk. “Good. Now explain this. Last month’s API bill was ₹1.8 lakh.” Priya winced. That is the next lesson.

Go deeper

• Langfuse, which includes prompt management and versioning: https://langfuse.com/docs
• Anthropic docs, including model versions and deprecations: https://docs.anthropic.com
• OpenAI docs, including evals and model versions: https://platform.openai.com/docs