Why this matters: Test Case Strategy tells you whether a feature is correct. This note is about whether the product is working for real users right now, which is a different question with a different answer. The ideas come from Google’s Site Reliability Engineering and the SRE Workbook.
Testing and reliability are not the same question
A feature can pass every test case and still be failing customers, because tests run in a controlled environment on clean data with one user at a time.
Acceptance criteria say: valid card payment succeeds
Test case proves: it succeeds in the test environment
Production reality: it succeeds 96.2% of the time, and the other
3.8% are timeouts at 9pm when the batch job runs
No test case catches that. Only measurement in production does. This is the thing a growing product needs and a small one can survive without, which is why teams usually discover it too late.
The three terms
SLI, Service Level Indicator. What you measure. A number, from real traffic.
The percentage of payment requests that complete successfully.
The percentage of invoice list page loads that render within 2 seconds.
SLO, Service Level Objective. The target you promise yourselves.
99.9% of valid payment requests succeed, measured over 30 days.
95% of invoice list loads complete within 2 seconds, measured over 30 days.
Error budget. The amount of failure the objective allows. This is the part that changes behaviour.
99.9% over 30 days = about 43 minutes of allowed failure.
Used 10 minutes so far: you have room to ship and take risks.
Used 43 minutes: you are out of budget, and reliability work now takes priority.
The first two are measurement. The third is a decision rule, and that is what makes the whole idea useful rather than just another dashboard.
Why the error budget is the actual idea
Every team has the same unresolvable argument: product wants features, engineering wants stability, and both sides are arguing from opinion. Nobody can win, so it gets decided by whoever is more senior or more persistent that week.
The error budget converts that argument into arithmetic. You agreed the target together, so:
- Budget remaining: ship freely. Take risks. Move fast. The numbers say you have earned it.
- Budget exhausted: feature work pauses, reliability work takes priority, until the number recovers.
The SRE Workbook presents SLOs exactly this way, as the mechanism for resolving the tension between new functionality and reliability. The important consequence is that 100% is never the goal. A perfectly reliable service means you are shipping too slowly and spending too much on reliability nobody asked for. The budget exists to be spent.
This also transforms how leadership sees the pause. “We need two weeks for stability work” is a request that gets refused. “We have exceeded the reliability target we agreed, so the next two weeks are stability work per the policy” is a rule that was accepted in advance. Agreeing the policy in calm times, before the outage, is the entire trick.
How to choose what to measure
Start from the user’s experience, not from your infrastructure. CPU usage is not an SLI, because no customer has ever cared about your CPU. What they care about is whether the thing they clicked worked, and how long it took.
Four categories cover almost everything:
- Availability: did the request succeed? Percentage of non error responses.
- Latency: was it fast enough? Use a percentile, typically the 95th or 99th, never the average. Averages hide exactly the users who are suffering.
- Correctness: did it produce the right result? For example the percentage of bank feed transactions that process without manual intervention.
- Freshness: is the data recent enough? For example the percentage of accounts whose feed synced within the last 24 hours.
Then pick the critical user journeys, not every endpoint. For an accounting product that might be: log in, create an invoice, sync the bank feed, run a report. Three to five journeys is enough to start, and starting with three good ones beats designing forty and implementing none.
Setting a first target
Do not invent a number. Measure what you currently achieve for a few weeks, then set the objective slightly above it and improve gradually.
Measured over 4 weeks: 99.2% of payment requests succeed
First SLO: 99.3%, review in a quarter
Not: 99.99%, because it sounds impressive
Every extra nine costs real money and real engineering time, and often none of it is noticed by users. Ask instead: at what point do customers actually complain or leave? Set the objective just above that, because the objective’s job is to protect the customer experience, not to look good on a slide.
The error budget policy
Write this down before you need it. Four questions, answered in advance:
- What are the SLOs, exactly, and over what window?
- What happens when the budget is 50% consumed? Usually a warning and a review.
- What happens when it is fully consumed? Usually: feature releases pause, reliability work is prioritised, until the trailing window recovers.
- Who can override it, and how is the override recorded?
Question four matters. There will be a launch that must go out during a budget breach. The policy is not meant to prevent that, it is meant to make the override a conscious, recorded decision rather than a quiet habit. The same principle as the explicit trade in Feedback and Scope Control: the decision is fine, the invisibility is not.
Alerting: page on symptoms, not causes
A rule that removes a large amount of pain: alert on things that affect users, not on every internal anomaly.
High CPU at 3am, when every user request is succeeding within its latency target, is not worth waking anybody. A drop in payment success rate at 3am is. Alerting on causes produces dozens of alerts a day, which produces alert fatigue, which produces an ignored alert on the night it actually mattered.
Tie alerts to the SLO through burn rate: page when the budget is being consumed fast enough to exhaust it well before the window ends. Slow burns become tickets, fast burns become pages.
Where this connects to everything else
Some acceptance criteria are really SLIs in disguise. The example in Acceptance Criteria of “invoice save stays under 800ms at the 95th percentile under load” is a criterion for the Story and a candidate indicator for production. Write those criteria as numbers and you get both for free.
Reliability work belongs in the backlog like everything else. Not as invisible work done at night by whoever cares. Real Stories, estimated, prioritised, tracked, as covered in Work Breakdown Hierarchy.
Toil is a measurable cost. The SRE Workbook defines toil as manual, repetitive, automatable operational work that scales with usage: restarting things, fixing stuck jobs, running the same manual correction every month end. Measure how much of the team’s time it eats. When it passes roughly half of someone’s time, automating it is a better investment than a new feature, and you will have the numbers to argue that.
Blameless postmortems. After an incident, write what happened, the timeline, the impact, the contributing causes, and the actions, without naming anyone as the cause. The reason is not politeness. It is that blame makes people hide information, and hidden information means the same failure repeats. The action items from a postmortem are backlog items like any other, and if they are never prioritised then the postmortem was theatre.
When to start this
Not on day one of a product, and not after your first serious outage either.
The sensible trigger is: when customers depend on it and you would not know they were suffering unless they told you. If your current early warning system is a customer emailing support, you are already late.
Start absurdly small. Two indicators, on your two most important journeys, on one dashboard that somebody actually looks at. Then add one per quarter. A comprehensive SLO framework that nobody maintains is worth less than two numbers that the team checks every Monday.