The MLOps lifecycle: deploy, monitor, drift
You can walk through the MLOps loop, tell data drift from concept drift, and build a simple weekly monitor that checks data quality, input drift and prediction error.
Words in this lessonMLOps · Model · Inference · Model registry · Model monitoring · Data drift · Concept drift
- MLOps
- The practices and tools for running machine learning models reliably after they are built. Like the maintenance crew for a factory machine: someone has to check it, service it and recalibrate it, or it slowly starts making bad parts.
- Model
- A program with adjustable settings that turns an input into a prediction or an output. Like a recipe card plus exact quantities: the same card with different quantities makes a different cake.
- Inference
- Using a trained model to get an answer, with its parameters fixed. Like an experienced baker making today's order: no new learning, just doing.
- Model registry
- A catalogue of every version of a model, with what it was trained on, how it scored and who approved it. Like a recipe book where every version of a recipe is dated, tasted, signed off and never thrown away.
- Model monitoring
- Watching a deployed model’s inputs, outputs and results so you notice when it starts going wrong. Like a shop manager who checks not only that the lights are on, but that the shelves are stocked and customers are leaving happy.
- Data drift
- When the inputs a model sees in production start to look different from the data it was trained on. Like a chef trained on summer produce suddenly cooking with winter vegetables: the recipes stay the same, but the ingredients have changed.
- Concept drift
- When the right answer for the same inputs changes, so a model’s learned rules stop being true. Like a map that is still drawn perfectly, but the city has rerouted the roads.
On this page
Every evening at 9 p.m., a small program at Crumb & Co. decides how many croissants each of the six stores should bake tomorrow. It looks at the day of the week, the weather forecast, the festival calendar and the last four weeks of sales, and it prints one number per store. A data scientist built it two years ago, before Arjun joined. It worked so well that everyone forgot it existed.
Then came July. In one week the Koregaon Park store threw away 140 croissants. At ₹45 each, that is ₹6,300 of butter and flour in the bin. Across town, the Baner store sold out by 10 a.m. five days running, and the manager kept forwarding photos of an empty display case with the caption “again??”.
Meera called Arjun into her office. “The croissant computer is broken. Can you fix it before the weekend?”
Important
The idea in one sentence: A trained model is not finished software; MLOps is the discipline of deploying it, watching it, and retraining it carefully as the world it learned from drifts away.
Arjun goes looking for the bug
Arjun did what any backend developer would do. He looked for the error.
He opened the job logs. Every night for two years: forecast job finished in 4.2s, 6 stores, exit 0. No exceptions. No warnings. He checked the code history. The last change was eight months ago, a dependency update. He reran last night’s job by hand and got exactly the same numbers.
“Nothing is broken,” he told Priya, a little annoyed. “The program does exactly what it did last year.”
“I believe you,” said Priya. “So what did you try next?”
“I retrained it. If the data changed, retrain on the data, right?” He showed her his notebook. He had taken all two years of sales, retrained the model, and saved it as version 15. For Baner on a Tuesday, the old version 14 predicted 182 croissants. The shiny new version 15 predicted 186.
“Four more croissants,” said Priya. “And Baner sold out by ten. How many would it have sold if you had baked enough?”
Arjun opened his mouth, and closed it again. He did not know. Nobody did.
Why the bug is not in the code
Priya pulled a chair over. “Let me ask it differently. Who taught this program how many croissants Baner needs?”
“Nobody. It learned from the sales history.”
“Right. It is a model: a program whose behaviour was learned from examples instead of written as rules. Imagine we hired a new shift manager and, instead of training them, we gave them two years of old sales ledgers. They study hard. They learn that a rainy Tuesday at Baner means about 180 croissants. Now, in June, a huge office complex opens next to Baner. What does our manager do?”
“Keeps ordering 180,” said Arjun slowly. “Because that is what the ledgers say.”
“Do they crash? Do they phone you with an error?”
“No. They keep being confidently wrong.”
“That is the core difference,” said Priya. “Ordinary software is like a vending machine. If B4 stops giving crisps, something inside is broken, and usually there is an error to find. A model fails silently. It keeps running, keeps returning healthy-looking numbers, and gets quietly worse as the world moves away from the examples it learned from.”
MLOps (machine learning operations: the practices and tools for running models reliably in production) exists because of that silence. It is the noticing, and the careful fixing, that the model cannot do for itself.
Where the analogy breaks: a human shift manager would eventually notice the queue of office workers and adjust. A model never notices anything. Someone has to build the noticing for it.
Following one Tuesday through the model
Priya picked one day to trace in full: Tuesday, 15 July, for two stores.
Baner. The model received these inputs:
| Input | Value |
|---|---|
| Day of week | Tuesday |
| Rain forecast | Yes |
| Temperature | 27 °C |
| Festival | None |
| Average Tuesday sales, last 4 weeks | 198 |
It predicted 182. The store bakes the prediction plus a 10% buffer, so it baked 200. Sales recorded at closing: 200. The display case was empty at 10:05 a.m.
“There’s the trap in your retraining,” said Priya. “The database says Baner sold 200. That is not what customers wanted. That is how many we had. Every sold-out day records a number that is too low. When you retrained on two years of data, the model learned from 100 normal weeks and a handful of capped ones. Of course it barely moved.”
This is called censored demand: when you run out, you only see how much you sold, not how much people wanted. It is one of the classic traps in forecasting, and it hides inside perfectly clean data.
Arjun checked the till system for the morning after 10:05. Staff had logged 47 “croissant requested, sold out” notes that day. True demand was probably near 247, not 200.
Koregaon Park. Here Priya did something clever. She found a day last year with nearly identical inputs: Tuesday, 16 July 2025, rain, 26 °C, no festival.
| Last year | This year | |
|---|---|---|
| Inputs | Tue, rain, 26 °C, no festival | Tue, rain, 27 °C, no festival |
| Prediction | 175 | 176 |
| Baked | 193 | 194 |
| Sold | 178 | 112 |
| Wasted | 15 | 82 |
The inputs barely changed. The prediction barely changed. But the right answer dropped by a third. Eighty-two croissants in the bin, ₹3,690 lost in one day. The reason was across the road: a new patisserie had opened in late June.
“Two stores, two completely different problems,” said Priya, “and neither is a bug in the code.”
Note
Pause and think. Baner’s inputs changed (many more office workers) and Koregaon Park’s did not. Why does that difference matter for how quickly you could have noticed each problem?
Show the answer
At Baner, you can see the change in the inputs on day one: weekday morning footfall and last-four-weeks sales were climbing, with no need to wait for the real answer. At Koregaon Park, every input looked normal. The only signal was the outcome (waste), which you see only after the day ends, and only if someone compares predictions with what actually sold. Problems in the inputs can be caught early; problems in the relationship between inputs and answers need you to track outcomes.
Opening the hood: two kinds of drift
The general name for “the world moved, the model did not” is drift. There are two kinds, and Tuesday showed one of each.
Data drift means the inputs themselves have changed shape. The model now sees kinds of days, customers or values it rarely saw in training. Baner’s new office crowd is data drift. So is a quieter, nastier cause: a bug upstream. If someone switches the weather feed from Celsius to Fahrenheit, every day suddenly looks like 80-degree heat, and the model happily forecasts for a heatwave.
Concept drift means the relationship between the inputs and the right answer has changed. The same inputs now deserve a different output. Koregaon Park’s competitor is concept drift. The model’s learned rule (“rainy Tuesday means about 175”) was true and is now false.
| Data drift | Concept drift | |
|---|---|---|
| What changed | The inputs | The link between inputs and the right answer |
| Crumb example | Office crowd at Baner | Rival bakery at Koregaon Park |
| Visible before real answers arrive? | Yes, by comparing input distributions | Usually not; you need actual outcomes |
| Typical fix | Retrain on recent data; fix broken feeds | Retrain on recent data; often add a new input, such as “competitor nearby” |
There is a timing problem hiding here. You only learn the true answer, called the label, after some delay. For croissants it is the same evening. For a model that predicts whether a loan will be repaid, it could be a year. This label delay is why teams watch input drift so closely: it is the early warning you can see today, while the truth is still on its way.
The lifecycle is a loop, not a road
Arjun’s instinct had been “find the bug, fix it, done”. Priya drew the alternative on the whiteboard: a circle.
flowchart LR
A["Frame the problem"] --> B["Collect and prepare data"]
B --> C["Train and evaluate"]
C --> D["Register the version"]
D --> E["Deploy"]
E --> F["Monitor"]
F -->|"drift or decay"| B
- Frame the problem. What decision does the model support, and what does “good” mean in business terms? For the croissant model: waste under 5% and sell-outs before 4 p.m. on fewer than one day a week per store. Notice the target is a business outcome, not a maths score.
- Collect and prepare data. Sales, weather, calendar, and now sell-out times. Remove the day the till broke and the week a store was closed for repairs.
- Train and evaluate. Training means letting the model adjust its internal numbers to fit the examples. You test it on the most recent weeks, which it has never seen, because the future is what you care about.
- Register. Store the trained model in a model registry: a catalogue of every model version with the data it was trained on, its test scores, and who approved it. “Which model is running?” now has a precise answer:
croissant-forecast v16, trained 21 July, approved by Priya. - Deploy. Put the model where it can make predictions. Using a trained model to make predictions on new inputs is called inference.
- Monitor. Watch inputs, outputs and, once they arrive, the real answers. When something drifts, go round again.
There are two common shapes for step 5. Batch deployment predicts on a schedule for many items at once, like the 9 p.m. job; minutes are fine and the main risk is a job silently not running. Online deployment predicts the instant a request arrives, like a fraud check at checkout, where you need answers in milliseconds and the risk is slowness under load.
Teams also switch models carefully. A shadow deployment runs the new version beside the old one, logs its predictions, and uses none of them, so you can compare at zero risk. A canary gives the new version a small slice of real traffic first, say one store, before the rest. You will lean on both ideas again when we roll out LLM features in the next lesson.
Note
Pause and think. Arjun’s version 15 was “retrained on all the data”. Using what you now know, give two reasons it barely helped.
Show the answer
First, censored demand: on sold-out days the data recorded what Crumb had, not what customers wanted, so the training data itself understated Baner’s demand. Second, dilution: a few weeks of the new pattern were drowned out by about 100 weeks of the old one, so the model’s average barely shifted. The fix was to estimate lost demand from sell-out notes, weight recent weeks more heavily, and add a “competitor nearby” input for Koregaon Park.
Who does what
In a large company, the loop is shared by several roles. At Crumb, Arjun wears most of these hats, which is normal for a small company. What matters is that each job has a name next to it.
- Data scientist: explores data, chooses inputs, trains and evaluates models.
- Data engineer: builds the pipelines that deliver clean, on-time data, like the sell-out notes from the tills.
- ML engineer: turns a notebook into a reliable service and deploys it.
- MLOps or platform engineer: builds the shared plumbing: registry, monitoring, scheduled retraining.
- Domain expert: the store managers, who know Diwali week is chaos and the IT park empties on Fridays.
- Owner: the person who decides whether a new version ships and who is accountable if it hurts. For croissants, that is Meera.
Building the noticing, step by step
Priya asked Arjun to build a weekly health check, starting simple and adding one layer at a time.
Step 1: measure the error per store. Compare what the model predicted with what actually sold.
# monitor.py, step 1: how wrong were we last week?
last_week = [
# store, predicted, sold, sold_out_before_4pm
("Baner", 182, 200, True),
("Koregaon Park", 176, 112, False),
("Aundh", 150, 147, False),
]
for store, predicted, sold, sold_out in last_week:
error_pct = (predicted - sold) / sold * 100
flag = "SOLD OUT (true demand unknown)" if sold_out else ""
print(f"{store:14} predicted {predicted:4} sold {sold:4} error {error_pct:+6.1f}% {flag}")
Run it with uv run monitor.py:
Baner predicted 182 sold 200 error -9.0% SOLD OUT (true demand unknown)
Koregaon Park predicted 176 sold 112 error +57.1%
Aundh predicted 150 sold 147 error +2.0%
Baner’s error looks small, only 9%, which is exactly the trap: the true error is hidden by the sell-out. That is why the line is flagged.
Step 2: check the inputs for drift. A common statistical tool is the Kolmogorov–Smirnov test (KS test), which asks: could these two sets of numbers plausibly come from the same distribution? It returns a statistic between 0 (identical shape) and 1 (completely different).
uv add numpy scipy
# monitor.py, step 2: does this month's Baner weekday traffic look like training data?
import numpy as np
from scipy.stats import ks_2samp
rng = np.random.default_rng(42)
training_footfall = rng.normal(loc=180, scale=20, size=500) # weekday mornings, last year
recent_footfall = rng.normal(loc=235, scale=25, size=30) # weekday mornings, since June
result = ks_2samp(training_footfall, recent_footfall)
print(f"KS statistic: {result.statistic:.2f} p-value: {result.pvalue:.2g}")
if result.statistic > 0.3:
print("Drift alert: Baner inputs look different from training data.")
KS statistic: 0.84 p-value: 1.9e-22
Drift alert: Baner inputs look different from training data.
Your exact numbers will differ slightly. Notice the alert is based on the statistic (how different), not only the p-value (how sure). That choice matters, and you will see why below.
Step 3: add data-quality checks before trusting anything. Drift tests on broken data give nonsense.
# monitor.py, step 3: catch broken plumbing first
def quality_problems(rows: list[dict]) -> list[str]:
problems = []
for row in rows:
if row["sold"] < 0:
problems.append(f"{row['store']}: negative sales")
if row["temperature_c"] is None:
problems.append(f"{row['store']}: missing temperature")
elif row["temperature_c"] > 50:
problems.append(f"{row['store']}: {row['temperature_c']} °C, wrong units?")
return problems
print(quality_problems([{"store": "Aundh", "sold": 147, "temperature_c": 81}]))
['Aundh: 81 °C, wrong units?']
Put the three steps together and you have the five layers Priya insisted on, ordered from cheapest to most valuable:
- System health: did the job run, and how long did it take?
- Data quality: missing, impossible or wrongly-scaled values.
- Input drift: do today’s inputs look like training data?
- Model performance: how far off were predictions, once real sales arrived?
- Business outcome: waste and sell-outs per store, the numbers Meera actually cares about.
Watching all five, with alerts at thresholds you agreed in advance, is called model monitoring. Tools such as Evidently and the monitoring built into cloud ML platforms automate this across hundreds of inputs.
Where it breaks
Monitoring has its own failure modes, and experienced engineers watch for these.
Big samples make every difference “significant”. With 50,000 rows, the KS test will report a tiny p-value for a shift too small to matter. If you alert on p-values alone, alerts fire daily and people mute them. Alert on the size of the change (the statistic) and on business impact.
Seasons look like drift. Diwali week looks wildly different from a normal October week. If last year’s Diwali is in the training data, that is expected, not drift. Compare like with like: this Diwali against last Diwali.
Retraining on poisoned data. An automatic “retrain every Sunday” job that runs after a week with a broken till teaches the model the bug. Always run the data-quality layer before retraining, and never let a new version replace the old one without evaluation and a named approver.
Your model changes the data it learns from. Baner can never sell more than it bakes, so a model that under-forecasts produces data that justifies under-forecasting. Breaking this loop needs extra signals, such as the sold-out notes, or occasionally baking a little extra to measure true demand.
Warning
A model with no monitoring does not stay “the same”. It gets steadily worse, and the cost shows up as waste, lost sales or bad decisions long before anyone suspects the model. Budget time for monitoring before launch, not after the first complaint.
Try it yourself
Use the step 2 code. Change loc=235 in recent_footfall to loc=184 and run again. Then change size=30 to size=5000 for both samples, keeping loc=184, and run a third time.
Show the answer
With loc=184 and 30 recent samples, the statistic drops to about 0.2 (with a p-value near 0.08): no alert, which is right, because a four-person shift is noise. With 5,000 samples each, the p-value becomes very small (the test is now sure the distributions differ slightly) but the statistic stays small (around 0.12), so the size-based alert still stays quiet. This is exactly why the code alerts on the statistic rather than on the p-value alone.
Common mistakes
- Treating deployment as the finish line. The day you ship is the day the model starts ageing. Plan monitoring and retraining before launch.
- Monitoring only uptime and errors. A drifting model returns perfectly healthy responses. Watch inputs and outcomes, not only servers.
- Trusting recorded outcomes blindly. Sold-out days, cancelled orders and missing feedback all distort the “truth”. Ask what your labels really measure.
- Retraining on everything, forever. Recent weeks matter more when the world has changed. Weight them, or train on a recent window.
- No record of what is running. Without a registry, “which model made this prediction?” becomes guesswork. Version the code, data snapshot, model and settings together.
Recap
- Models fail silently: they keep running and quietly get worse as the world changes.
- The MLOps lifecycle is a loop: frame, data, train, register, deploy, monitor, and around again.
- Data drift means the inputs changed; concept drift means the right answer for the same inputs changed.
- Outcomes can lie: censored demand and label delay hide the truth, so check what your data really measures.
- Monitor five layers, from system health to business outcome, and alert on the size of a change, not only its statistical significance.
On Friday Arjun shipped version 16. It was trained mostly on the last eight weeks, it estimated lost demand from the sell-out notes, and it had a new “competitor nearby” input. He also set up a weekly report with the five layers. He explained it to Meera in one breath: “The program wasn’t broken. Baner got busier and Koregaon Park got a rival, and the model was still living in last year. Now it learns from recent weeks, and every Monday it tells us if the world has moved again.” By mid-August, waste was back under 5%.
Meera nodded. “Good. And the WhatsApp bot? Does it drift too?” Priya smiled. “Worse. And we can’t even retrain it, because we never trained it in the first place.” That is the next lesson.
Go deeper
- MLflow, an open-source model registry and experiment tracker: https://mlflow.org/docs/latest/
- Hugging Face docs, for hosting and versioning models: https://huggingface.co/docs
- OpenTelemetry, the common standard for collecting system health signals: https://opentelemetry.io/docs/