0%
Observability and Delivery: Running Production Systems
The Signals
Metrics Systems
Logs and Traces
Reliability Targets
Delivery
Incident Command and Communication
An incident is a declaration, not a fact about the world. Something is wrong with production whether or not anybody says so, and declaring an incident is the act of changing how the organisation behaves in response: it assembles people, it suspends normal work, it starts a clock and it creates a record. The question is never whether something is broken. It is whether this particular broken thing is worth that machinery.
Most organisations get this wrong in the same direction. They declare too rarely, too late, and only once somebody senior has become uncomfortable, which means the first twenty minutes of every serious outage are spent by one engineer working alone and deciding whether to bother anybody.
The cost of declaring, and the cost of not
Declaring an incident costs something real. It interrupts people, it generates noise in channels, it may notify customers, and in some organisations it triggers reporting obligations. Teams that have been burned by a noisy incident process learn to hesitate, and the hesitation is rational given what they experienced.
Not declaring costs more, but the cost is invisible. It is paid as the lone responder who spends twenty five minutes on a problem that two people would have scoped in four, as the customer facing team who learned about the outage from a customer, and as the missing record that makes the next occurrence just as slow.
The asymmetry is what matters. A declared incident that turns out to be minor costs perhaps fifteen minutes of three people's attention and produces a short record. An undeclared incident that turns out to be major costs the difference between a coordinated response and an improvised one, which in practice is most of the duration.
Designing for this asymmetry means making declaration cheap, reversible and uncontroversial. Anybody can declare. Nobody is criticised for declaring. Closing an incident five minutes later with the note "not an incident, single pod restarted" is a normal and expected outcome rather than an embarrassment.
Who is allowed to declare
The answer that works is everyone, with no approval step. The responder who is paged, the engineer who notices something in a dashboard, the support agent who sees three identical tickets arrive in a minute, the person on another team whose integration started failing.
This is frequently resisted on the grounds that it will produce noise. In practice it does not, because the people in a position to declare are the people who understand the systems, and because a declaration that is cheap to close does not accumulate. What produces noise is not broad permission to declare, it is an alerting set that fires constantly, which is a separate problem with a separate fix.
The role that does need a named holder is the opposite one: deciding when an incident is over. That decision has consequences, including the status page and the postmortem, and it belongs to whoever is holding command rather than to whoever is tired of being on the call.
Severity, and keeping the scale small
A severity level is a routing decision compressed into a number. It answers who is woken, how fast updates go out, and whether anybody outside engineering needs to know. It is not a measure of how bad somebody feels about the problem.
Three levels is usually enough and five is usually too many. The distinctions that earn their place are: is this affecting users right now, is it affecting all of them or some of them, and is there a path to mitigation that does not need anybody else. Anything finer than that produces arguments during the incident about whether something is a two or a three, and those arguments consume exactly the attention the scale was supposed to protect.
The definitions should be written in terms of observable user impact rather than internal state. "Checkout is failing for more than five percent of attempts" is a definition a responder can evaluate in thirty seconds. "Significant degradation of a critical service" is a definition that requires a meeting.
It is also worth writing down that severity can change in both directions. An incident that starts as minor and turns out to be broad should be raised immediately and loudly, and one that was declared at the top of the scale and turns out to be a single region should be lowered without anybody treating it as an error.
Detection is a separate problem
It is worth separating two numbers that are often summed into one. The time from the fault beginning to anybody knowing about it is detection. The time from somebody knowing to an incident existing is declaration. They have different causes and different fixes, and a team that reports only total time to response will work on the wrong one.
Detection is an alerting and instrumentation problem, covered by the signals earlier in this course. If the first notification came from a customer, no declaration policy will help; the fix is a symptom alert that would have fired.
Declaration is a process and culture problem. If a responder knew at 14:02 and the incident channel appeared at 14:26, those twenty four minutes were spent deciding, and no amount of extra monitoring will recover them.
Measuring the two separately is cheap, because both timestamps usually already exist: the alert fired at a recorded time and the incident record has a creation time. Most teams that look at this for the first time are surprised by how much of their total response time sits in the second gap.
The incidents that are declared by somebody else
Not every incident starts with a page. A meaningful share start with a support ticket, a complaint from another team, or a customer email, and these arrive through channels that have no connection to the paging system.
This matters because the declaration path for those routes is usually undefined. A support agent who sees four identical reports in ten minutes has better evidence of a user facing problem than most alerts produce, and if their only option is to open a ticket that will be triaged tomorrow, the organisation has thrown away its best signal.
The fix is to give the non engineering routes a real path in: a documented way for support to declare, a channel they are welcome in, and an explicit statement that they will not be asked to prove it first. The cost is occasional declarations that turn out to be a single confused user, which is the same cheap, reversible cost as any other minor declaration.
The thirty second declaration
The act of declaring should take under a minute and produce four things: a channel, a name, a severity and an initial statement of the symptom. Everything else can follow.
The symptom statement is the part most often skipped and most valuable. It should describe what a user experiences, not what a monitor says. "Checkout returns an error for about one in five attempts since 14:02" is a symptom. "Elevated 5xx on the payments service" is a monitor reading that may or may not be the same thing, and starting from it anchors everybody on a hypothesis before anybody has checked.
Naming the incident matters more than it sounds. A name that describes the symptom lets somebody joining forty minutes later know what they are joining without reading the scrollback, and it becomes the thing the postmortem is filed under.
What declaring actually changes
The moment an incident exists, three behaviours should change automatically rather than by request.
Communication moves to one place. The incident channel becomes the only record, and anybody who learns something posts it there rather than in a direct message. This is the single most valuable mechanical effect of declaring, because it converts a dozen private conversations into one readable timeline.
Normal work pauses for the people involved. The responder is not expected to attend their meetings, and the organisation accepts that whatever they were doing is now late. Making this explicit prevents the common failure where somebody tries to handle an incident between calls.
And the clock starts. Not as a stick to measure people with, but because the duration of an incident is the one number that makes the next conversation about investment concrete.