Almost every alerting system starts the same way. Something breaks, somebody investigates, they find the proximate cause, and they write an alert for that cause so it never surprises anyone again. A pod ran out of memory, so there is now an alert on container memory above ninety percent. A connection pool was exhausted, so there is now an alert on pool utilisation. A disk filled, so there is now an alert on disk usage. Each of these decisions is locally reasonable and the accumulation of them is the single most common reason that on call is miserable.

The alternative is not to stop measuring those things. It is to stop paging on them. An alert that wakes somebody up should describe something a user is experiencing, and everything else should be a dashboard, a ticket, or a line in the runbook that the person reads after they have been paged.

What a cause alert actually promises

When you write an alert on connection pool utilisation above ninety percent, you are making a claim that you probably would not defend if it were stated plainly. The claim is that whenever pool utilisation exceeds ninety percent, a user is being harmed, and that whenever a user is being harmed by pool exhaustion, utilisation will exceed ninety percent.

Neither half of that claim is true. A pool can sit at ninety five percent utilisation all afternoon under a healthy traffic peak with every request served inside its latency objective, because high utilisation is what a correctly sized pool looks like when it is busy. And a pool can cause user visible harm at sixty percent utilisation if the checkouts are all waiting on one slow connection while the rest sit idle. The threshold is a proxy, and like all proxies it is wrong in both directions.

The cost of the first kind of error is paid every week in pages that resolve themselves. The cost of the second is paid once, during the incident where the thing you were sure you were monitoring failed in a way your monitor did not notice.

The sprawl, and why it never shrinks

Cause alerts accumulate monotonically. Every incident adds one, and nothing removes any, because removing an alert requires somebody to argue that a specific failure mode no longer needs watching, which nobody wants to be wrong about. A four year old service with an active on call rotation will typically have between forty and two hundred alert rules, of which fewer than ten have ever fired during a real user affecting incident.

The sprawl has a second cost beyond noise. Each rule has a threshold, and each threshold was correct for the system as it existed on the day it was written. Thresholds do not follow the system as it changes. The memory limit doubles, the pool is resized, the instance type changes, and the alert either fires constantly or stops being able to fire at all. Nobody notices the second case until the incident.

What a symptom alert covers

A symptom alert makes a much weaker and much more defensible claim: users are not getting what we promised them right now. There are only a handful of ways to say that, which is exactly why the approach scales.

Consider the same connection pool exhaustion. If it matters, it shows up as requests failing or requests becoming slow, and both of those are already covered by the availability and latency indicators from the previous lesson. You do not need a pool alert to be paged for pool exhaustion. You need a pool dashboard so that the person who was paged can see, in the first minute, that the pool is where the problem is.

This is the division of labour that makes the approach work. Symptoms decide whether to wake somebody. Causes decide what they look at once they are awake. Collapsing the two is what produces both the noise and the blind spots.

Where a cause alert is still correct

The rule is not absolute, and treating it as absolute is how teams end up unable to see a disk fill. There is a specific and identifiable class of condition that symptom alerting cannot cover, and it is worth naming precisely rather than keeping as a vague exception.

A cause alert is right when the condition will cause harm in the future, is not causing harm yet, and will be expensive or impossible to fix once it does. A TLS certificate expiring in two weeks is harming nobody. A disk that will be full on Friday is harming nobody. A cloud quota that will be hit at the end of the month is harming nobody. In each case, by the time the symptom appears the remediation takes hours and the outage is total.

What these share is a deadline rather than a population. Burn rate alerting, which the rest of this lesson builds on, requires a stream of events that can be classified good or bad. A certificate has no such stream. The correct mechanism is a plain threshold on time remaining, with enough lead time to do the work calmly, and it belongs in a different and much smaller category than the pages that mean something is wrong now.

Deciding which of your alerts is which

There is a quick test that resolves most cases. For each existing alert, ask what you would do if it fired and you had no further information. If the honest answer is "look at the dashboards to find out whether anything is actually wrong", it is a cause alert and it should not page. If the answer is "start an incident, because users are affected", it is a symptom alert. If the answer is "schedule work before a deadline", it is the third category and it should create a ticket with a due date rather than a page.

Running this test across a mature alerting set is uncomfortable, because the first category is usually most of it. The useful framing for the conversation is that nothing is being deleted. The signals stay, the dashboards stay, the queries stay. What changes is which of them is allowed to wake a person.

What the audit usually finds

The first time a team runs this classification across its whole rule set, three patterns appear reliably enough to be worth predicting.

The first is duplication. The same underlying failure is covered by four or five rules written at different times by different people, usually one on the resource, one on a derived ratio, one on a log pattern and one on a synthetic probe. During an incident all of them fire within a few minutes of each other, producing a burst of pages that looks like a cascading failure and is in fact one problem described four ways. Deduplicating this alone often removes a third of the volume.

The second is rules that cannot fire. A threshold written against a metric that was renamed in a library upgrade, a query against a label that no longer exists, a rule scoped to a cluster that was decommissioned. These are invisible because a rule that never fires looks identical to a rule guarding a condition that never happens. The only way to find them is to check, for every rule, when it last evaluated to true, and to treat anything that has never fired in two years as suspect rather than as proof of stability.

The third is rules nobody will claim. The author has left, the team has been reorganised, and the rule survives because deleting something unowned feels riskier than keeping it. These are worth attacking first precisely because there is nobody to argue with, and because an unowned rule is one that will page somebody who has no context for it.

None of these three is an argument about symptoms and causes. They are just what accumulates when nothing ever removes anything, and clearing them is the cheapest part of the work.