SLI, SLO and error budget: measuring reliability from the user side Intermediate

The problem

The CPU and memory dashboards are green, Pods are not restarting, yet users still report checkout failures. A machine-level metric only tells you the machine is running; it does not answer “how many user transactions completed correctly and fast enough?”.

Without an agreed way to measure the service from the user’s point of view, operations and product teams easily argue by feel: one side says the system is fine, the other says the experience is bad.

SLIs, SLOs and error budgets turn that conversation into a measurable contract. They do not promise zero failures. They help choose the reliability level you need, detect when quality is degrading, and decide when to prioritize stability over new features.

Core concepts

An SLI (service level indicator) is a ratio or measurement of what users receive.

• An availability SLI for the checkout endpoint over 28 days: the number of eligible requests that returned a successful result divided by the total eligible requests.
• A latency SLI can be the proportion of requests completed under 500 ms, not necessarily just a p95 number.

A metric name is not enough; you must define eligible requests, good requests, where they are measured, the unit and the time window.

An SLO (service level objective) is the target set on an SLI: at least 99.9% of eligible checkout requests must be “good” over a rolling 28-day window. “Good” can require both a valid status and latency below a threshold, but it must be stated explicitly.

An SLO is an internal reliability target; an SLA is usually a contractual commitment with commercial consequences. Do not use the three terms interchangeably.

The error budget is the portion allowed to miss the SLO within a window. With 10,000 eligible requests and a 99.9% SLO, at most 10 requests may be “bad” in this illustrative calculation:

• 7 bad so far: 3 left.
• 5 more bad in the same window (assuming total eligible stays the same): the budget is exceeded by 2.

In a real system the denominator grows with traffic, the rolling window slides continuously, and integer/rounding arithmetic has to be defined consistently.

10,000 eligible requests
   ├─ 9,993 good
   └─ 7 bad

Target 99.9% → budget 10 bad → 3 bad left

The error budget here is a count/ratio of bad events, not automatically a number of downtime minutes. You can convert it to minutes only if the SLI is defined in time, or if you add assumptions about traffic.

An hour with 10 requests and an hour with 100,000 requests affect a request-based SLI very differently.

How it works

Start with one important user journey, such as “order placed successfully”, rather than measuring every endpoint the same way. Determine where the user sees the result: the load balancer, the API gateway or a synthetic probe.

• Measuring too deep inside the service: may miss DNS, TLS and gateway failures and truncated responses.
• Measuring at the client: closer to the user, but you have to handle sampling, user networks and missing data.

You can use several measurement sources, but each SLO must state its primary source.

Loading diagram…

The diagram shows why two teams both claiming “99.9%” can be talking about different things: for the same system, measuring at the Service handler will never see a request that died at TLS or the gateway. That is why every SLO must state its measurement point, not just a number.

Next, define eligible events. Do you exclude health checks, internal traffic or requests cancelled by the client? If so, why, and does that hide real failures?

A 4xx caused by bad input usually does not count as a server error, but a 4xx caused by a broken auth service can be a failed experience.

Do not choose the denominator to make the dashboard look good; check traces and user tickets to verify that the definition of “bad” is right.

Then choose a target that is both valuable and achievable. 100% sounds attractive but leaves no room to change the system; a target that is too low does not protect the experience. Use historical data, user expectations and business risk to pick the SLO.

Some features may need their own SLO, for example checkout versus reading articles. Combining everything into one ratio can hide failures in an important flow beneath the heavy traffic of a less important endpoint.

Finally, track the burn rate: how fast the budget is being consumed compared to the average allowed pace.

burn_rate = error_ratio / (1 - SLO)
time to exhaust budget = SLO window / burn_rate

• Burn rate 1: the budget runs out exactly at the end of the window.
• Burn rate 14.4 on a 30-day SLO: the budget runs out after about 2 days.
• A high burn rate over a short window signals an acute incident; a moderate but sustained burn rate shows gradually degrading quality.

A common alert configuration (SRE Workbook, 30-day SLO) combines a long window to avoid false alarms with a short window so the alert clears quickly once errors stop:

Burn rate	Long / short window	Budget consumed	Action
14.4	1 hour / 5 minutes	2%	Page immediately
6	6 hours / 30 minutes	5%	Page
1	3 days / 6 hours	10%	Open a ticket

The alert condition is both windows over the threshold: the long window prevents false alarms, the short window lets the alert clear quickly once errors stop. The SRE Workbook formula: budget consumed = burn rate × alert window / SLO window. For example 14.4 × 1 hour / 720 hours = 2%.

PromQL alert for a 99.9% SLO (budget 0.001), following the Workbook’s sample expression:

(
  sum(rate(http_requests_total{job="checkout", status=~"5.."}[1h]))
    / sum(rate(http_requests_total{job="checkout"}[1h])) > (14.4 * 0.001)
and
  sum(rate(http_requests_total{job="checkout", status=~"5.."}[5m]))
    / sum(rate(http_requests_total{job="checkout"}[5m])) > (14.4 * 0.001)
)

In practice, create a recording rule for each window (1h, 5m, 6h, 30m…) to keep alerts concise and cheap. An alert should include the SLI, the SLO, the remaining budget and a runbook.

The “nines” table and its limits

Converting to time is only for intuition, because real failures are usually a fraction of requests failing, not a complete outage:

SLO	“Full outage” allowed in 30 days	In 28 days
99%	7.2 hours	6.72 hours
99.9%	43.2 minutes	40.3 minutes
99.95%	21.6 minutes	20.2 minutes
99.99%	4.32 minutes	4.03 minutes

Dependencies multiply: the SLO ceiling

A service calls three dependencies sequentially, each at 99.9%, and needs all three. The availability ceiling is 0.999³ ≈ 0.997, or 99.7%, lower than the SLO of each dependency.

So you cannot set a 99.9% SLO for this service without retries within the deadline, fallbacks, or removing dependencies from the critical path. See timeout budget and retry, backoff, jitter. (The multiplication assumes independent failures; correlated failures change the result.)

Error budget policy: agree before the incident happens

Budget consumed (SLO window)	Action
Below 50%	Release as normal
50-100%	More risk review; prioritize fixing recurring failure causes
Budget exhausted (100%)	Freeze feature releases; accept only reliability fixes until the budget recovers
One incident consumes more than 20% of the budget	Mandatory postmortem with action items

The thresholds above are a template to tune for your organization; what matters is that product and engineering sign off in advance, and that exceptions are deliberate.

When to use it

Use SLOs and error budgets	Not needed yet
A service with real users and stable traffic	A prototype with no users yet
You need to balance release speed against reliability	Traffic too low for ratios to mean anything (1 error is 10%)
Many teams depend on each other	No data yet to define good/eligible

Real-world example

Checkout at 99.9% over 28 days

Eligible means POST /orders from real users at the gateway, excluding health checks and test traffic. Good means a valid business result within 800 ms; a 200 where the order was not saved does not count as good.

An SLO of 99.9% means at most 0.1% bad requests, equivalent to about 40.3 minutes of “full outage” if converted.

An incident makes 23.5% of requests fail for 2 hours: burn rate 0.235 / 0.001 = 235, consuming 235 × 2 / 672 = 70% of the 28-day window’s budget (672 hours). Per the policy, the 50% mark has been passed, so the team prioritizes fixing the cause; if 100% is consumed, releases freeze.

Two weeks later the overall SLI still passes, but mobile users in one region are slow. The team splits reporting by region to detect it, and does not quietly change the denominator mid-period to make the numbers look better.

Run the simple calculation:

npx --prefix examples ts-node examples/src/slo-budget/slo-demo.ts

The demo uses 10,000 requests, a 99.9% target, 7 bad initially and then 5 more bad. It holds the denominator fixed so the subtraction is easy to see; with a real rolling window, each new request both increases total eligible and may push old requests out of the window.

Try changing the target to 99.95%: the budget drops from 10 to 5, so the same 7 bad already exceed the target. The target number is therefore a product/operations decision, not just a way to color a dashboard.

To verify real metrics, take a sample of requests from the access log, classify good/bad yourself according to the contract, and compare with the dashboard query. Also test these cases:

• a 200 response with the wrong business outcome;
• a timeout at the gateway;
• a client cancel;
• a flow with very low traffic.

A syntactically correct Prometheus query does not prove the SLI definition matches the user experience.

🎮 Try it

• 3 a.m. incident: the error budget burning during incident response.
• Canary & feature flag: SLO-based stop criteria before ramping up.

Production checklist

1. Each SLO states the journey, eligible, good, measurement point and window (28 or 30 days).
2. The SLI is computed with recording rules from metrics at the gateway or load balancer.
3. Two-window burn-rate alerts (14.4 and 6 for pages, 1 for tickets), with a runbook.
4. The error budget policy is signed off in advance by product and engineering.
5. The service’s SLO is less than or equal to the ceiling computed from critical-path dependencies.
6. Review the definition of good against real tickets and traces every quarter.

Trade-offs

Benefits

• Moves reliability debates onto a measurable definition tied to the user journey.
• The error budget helps balance the pace of change with operational risk.
• Burn rate helps prioritize incidents by how fast they consume the target, not by a single metric point.

Limits

• Choosing the wrong eligible/good can keep the dashboard green while users still fail.
• Rolling windows, low traffic and sampled data make interpreting ratios tricky.
• An SLO does not replace root-cause investigation or business prioritization.

Common mistakes

Common pitfall

Measuring infrastructure health and calling it user availability. A live Pod does not mean checkout works. Place the SLI at a boundary that reflects user outcomes and cross-check it against real incidents.

Common pitfall

Combining every endpoint into one SLO. Content-reading traffic can hide rare but serious checkout failures. Set SLOs per important journey and explain the denominator clearly.

Common pitfall

Interpreting 99.9% as a fixed number of downtime minutes for every SLI. For a request-based SLI, varying traffic makes that conversion wrong. Keep the calculation in event units or state your assumptions explicitly.

Summary

Key takeaways

• The SLI is how you measure; the SLO is the target; the error budget is the acceptable failure under that target.
• Defining eligible and good matters more than picking a nice percentage.
• A request-based budget is a count/ratio of events, not downtime minutes by default.
• Use burn rate and an agreed policy to decide when to prioritize stability.

Self-check: (1) With 10,000 requests and a 99.9% SLO, what is the maximum number of bad requests? (2) Why is a 200 not always good? (3) Why does a green CPU dashboard not prove the checkout SLO is met?

References

• Implementing SLOs — Google SRE Workbook
• SLO monitoring — Google Cloud