SLIs, SLOs, and Error Budgets

Topics Covered

What an Indicator Actually Measures

Good events over valid events

The validity filter is where the arguments happen

Three families, and what each one is for

Why a percentile is not an indicator

Request based and window based, and when each is honest

A worked indicator, written out in full

Most teams that say they have service level objectives do not have service level indicators. They have dashboards with thresholds drawn on them, and somebody at some point agreed that the threshold was three hundred milliseconds. That is not an indicator, and the difference is not pedantic. It is the difference between a number that can be budgeted, alerted on and negotiated, and a number that can only be argued about.

A service level indicator is a ratio. It is the count of good events divided by the count of valid events, expressed as a proportion, over some window. That shape is the whole idea, and almost every mistake people make with reliability targets comes from departing from it.

Good events over valid events

Writing the indicator as a ratio forces three decisions into the open, and each of them is a decision somebody has to make deliberately.

The first is what counts as an event. For a synchronous API the natural event is a request. For an asynchronous pipeline it might be a record processed, or a file delivered. For a batch job it might be a run. For a mobile application it might be a screen load. The event is the unit the user experiences, which is not always the unit your system is built around, and choosing the system's unit instead of the user's is the most common first error.

The second is which events are valid. Not everything that arrives at your service is something you promised to serve. Health checks from a load balancer are not user traffic. Requests from a synthetic prober might be, or might not be, depending on whether you want the indicator to keep reporting when real traffic goes to zero at four in the morning. Requests that are malformed, that fail authentication, or that ask for a resource that genuinely does not exist, are usually not your failure. The validity filter is the denominator, and it is where almost every disagreement about an indicator actually lives.

The third is what counts as good. For availability this is typically a response that is not a server error. For latency it is a response that arrived faster than some threshold. For a data pipeline it is a record that arrived within some freshness bound and passed some correctness check. Notice that goodness is a predicate on a single event, not an aggregate. That is what makes the ratio meaningful.

Every indicator is this shape. Traffic narrows to valid events through a filter somebody chose, and valid events narrow to good events through a predicate somebody chose. The ratio is what you can budget.

The validity filter is where the arguments happen

Consider a checkout API receiving nine thousand requests a minute. Of those, six hundred are health checks, four hundred are requests carrying an expired session token, and two hundred are requests for order identifiers that do not exist because somebody is scraping. If you count all nine thousand as valid, a background rate of rejections sits permanently in your error budget and you have less room than you think. If you exclude all of them, you have excluded a category of failure: a bug that causes your service to reject valid sessions will look like a normal rate of authentication failures and will not move the indicator at all.

There is no universally correct answer, but there is a correct way to decide. Ask whether a user who sees this outcome would say the service was broken. A user whose session genuinely expired would not. A user whose valid session was rejected by your bug would. Since you cannot distinguish them at the status code level, the usual resolution is to exclude the class but add a separate indicator or alert on the rate of rejections, so a change in that background rate is still visible somewhere.

The same reasoning applies to client errors generally. A four hundred level response usually means the client asked for something it should not have, and counting it against your reliability means your indicator degrades when somebody else ships a bad client. That is a defensible choice for a public API where the client is a browser you also wrote, and an indefensible one for a platform API where the clients are other companies.

Three families, and what each one is for

Almost every useful indicator falls into one of three families, and naming them helps because each has its own failure mode.

Availability indicators ask whether the service responded successfully. They are the easiest to compute, the easiest to agree on, and the least informative, because a service that responds in nine seconds is available and useless. Nobody should ship availability as their only indicator.

Latency indicators ask whether the service responded fast enough. They are computed as the proportion of valid requests completing under a threshold, which is not the same thing as a percentile, and the distinction matters enough that it has its own subsection below.

Quality indicators ask whether the response was correct, complete or fresh. A search service that returns results from a stale index is available and fast and wrong. A recommendations service that falls back to a generic list under load is degrading in a way no availability or latency indicator will ever see. These are the hardest to build and the ones that most closely match what a user would call broken.

Which family is load bearing depends on the service. A write path lives on availability, a search path lives on latency and freshness, and a recommendations path lives on quality nobody is measuring.

Why a percentile is not an indicator

A great many teams write their latency objective as "p99 under three hundred milliseconds" and then discover that it cannot be budgeted. The reason is arithmetic. A percentile is a statistic computed over a window, so it has a value per window, and it does not decompose. The p99 of an hour is not derivable from the p99 of its twelve five minute buckets. You cannot add percentiles, average them, or subtract one from a target to get a remaining allowance.

The proportion formulation does all of those things. "The proportion of valid requests completing in under three hundred milliseconds, which should be at least ninety nine percent" is countable. Every request is good or bad. Counts add across windows, across regions and across services. A month is the sum of its days. That single property is what makes error budgets possible, and it is why every serious treatment of this subject writes latency objectives as proportions rather than percentiles.

The two formulations are closely related, and at the exact threshold they agree, but only one of them is a quantity you can spend.

What this looks like at different levels

Request based and window based, and when each is honest

There is a second formulation worth knowing. A request based indicator counts individual events, as described above. A window based indicator divides time into fixed windows, labels each window good or bad according to some rule, and then computes the proportion of good windows.

Window based indicators are useful when the unit of user pain is time rather than requests. A batch pipeline that must produce a file every hour is naturally measured in windows. A service whose traffic is extremely bursty can be measured in windows so that a five minute outage during a quiet period counts the same as a five minute outage during a peak, which is sometimes exactly what you want and sometimes exactly what you do not.

The hazard with window based indicators is that they hide magnitude. A one minute window in which one percent of requests failed and a one minute window in which a hundred percent failed are both bad windows, counted identically. For most request serving systems the request based form is the honest one, because the user count scales with the request count.

A worked indicator, written out in full

Here is one, written out the way it should appear in a document somebody can disagree with.

The event is an HTTP request to any route under the checkout service. A request is valid if it is not a health check path, if it carries a well formed and unexpired session, and if it is not a request this service deliberately rejects for rate limiting. A valid request is good if the response status is below five hundred and the server side duration is under eight hundred milliseconds. The indicator is the count of good requests divided by the count of valid requests, measured over a rolling twenty eight day window, computed from the server side request log rather than from the load balancer, because the load balancer cannot see the session validity filter.

Every clause in that paragraph is a decision, and every one of them will be questioned by somebody within a month of publishing it. That is the point. An indicator nobody can question is an indicator nobody understands.