An incident is a search problem under a deadline. You have four kinds of evidence, a system nobody fully understands, and a page that went off ninety seconds ago. What separates a twelve minute incident from a ninety minute one is almost never the depth of anybody's knowledge. It is the order in which they look at things.

The order that works is fixed, and it is worth learning as a sequence rather than as a principle, because under pressure people fall back on sequences and argue about principles. Establish the blast radius, establish what changed, then and only then establish the mechanism. Most incidents are resolved before the third step, because the second step produced something to revert.

Blast radius first, because it decides everything else

The first question is not what is broken. It is how much is broken and for whom. A checkout error rate of four percent and a checkout error rate of a hundred percent are different incidents with different responses, and the difference between them is visible in one panel. So is the difference between one region, one customer tier, one deployment version, and everybody.

This matters practically. If the failure is confined to one availability zone, the mitigation is to shift traffic and the investigation can happen afterwards at leisure. If it is confined to one version, the mitigation is a rollback. If it is confined to one customer, the incident is probably not an incident. The blast radius does not tell you the cause, but it very often tells you the mitigation, and mitigating first is almost always correct.

Then what changed, because systems do not spontaneously degrade

Software that worked at ten in the morning and failed at eleven was changed by somebody or something in between. The candidates are a code deploy, a configuration change, a feature flag flip, an infrastructure action, a dependency's own deploy, a data change such as a large customer arriving, or a scheduled job starting. That list is short enough to check in sixty seconds if the information is in one place, and it is the single highest yield minute in incident response.

The reason this works is uncomfortable but true: most incidents are self inflicted and recent. The distribution of causes is heavily weighted toward things your organisation did in the last hour. Starting from the mechanism means reasoning forward through a system with a thousand components. Starting from the change list means reasoning backward from a list of six.

Only then the mechanism

The third question is what is actually happening inside the system, and it is the one engineers enjoy, which is exactly why it has to be third. Understanding the mechanism is necessary for the fix and almost never necessary for the mitigation. An incident where the team understood the mechanism beautifully and restored service forty minutes later than they could have is a failure, however good the write up reads.

What each signal is actually for

Four kinds of evidence exist, and each one answers a different question well and the others badly. Metrics tell you how much and when, across everything, cheaply. They are the only signal you can afford to have for every component at all times, which is why they go first. They cannot tell you about an individual request.

Traces tell you where time went and what called what, for one request. They are the only signal that crosses service boundaries in a single view. They cannot tell you how often, because you are looking at a sample.

Logs tell you what a specific piece of code decided, in that code's own words. They carry detail nothing else does, including the exception, the parameter values, and the branch taken. They are expensive, they are per service, and the correlation across services has to be built deliberately.

Events tell you what people and automation did to the system. Deploys, flag flips, scaling actions, configuration pushes, schema migrations. This is the signal most organisations have the least of, and it is the one that answers the second question above.

The transitions matter more than the signals

A practitioner's real skill is not reading any one of these. It is moving between them quickly: from a metric spike to an example trace, from a slow span to the logs that service emitted during that span, from a log line to the deploy that introduced it. Each of those transitions is either one click or ten minutes of manual filtering, and which one it is was decided months earlier by whoever set the platform up.

That is the central argument of this lesson. Correlation is not something you do during an incident. It is something you build before one, and what you are buying is the ability to make those transitions without thinking. The rest of this lesson is about how each link is built, which ones are worth the effort, and what the investigation looks like when they are missing.

Who is doing what while this happens

The three questions are faster in parallel than in sequence, and parallelism needs roles rather than volunteers. The useful split is small. One person owns the incident, meaning they decide what happens next and they talk to everybody outside the channel. One person owns the investigation, meaning they are the one actually running queries. If the incident is large enough, one person owns communication, because writing updates for the people who are not in the channel is a full task and it is the first thing a busy investigator drops.

The reason to name these explicitly is that the failure mode without them is remarkably consistent. Four engineers all open the same dashboard, all reach the same conclusion a minute apart, and nobody is reading the change stream. Naming an owner per question costs ten seconds and removes that duplication. It also makes the handover legible when somebody with more context joins twenty minutes in, because there is a single person to ask what has been ruled out.

There is a related discipline that matters more than it sounds. Say out loud, in the channel, what you have ruled out and how. Not what you suspect, which is cheap, but what is now excluded and on what evidence. An incident channel full of theories grows without converging. An incident channel that records exclusions converges, because the search space visibly shrinks and arriving engineers can see where the remaining space is.

The trap of the plausible first theory

The most expensive minutes in an incident are usually spent confirming a theory formed in the first thirty seconds. Somebody recognises a symptom shape from a previous incident, proposes the cause that applied then, and the next ten minutes are spent gathering evidence for it rather than against it. Sometimes that recognition is genuine expertise and it saves the incident. Often the resemblance is superficial and the pattern match is costing time.

The cheap defence is a habit rather than a process: before investigating a theory, name the single observation that would falsify it, and look for that observation first. If the theory is a bad deploy, the falsifying observation is the symptom beginning before the rollout started. That is one glance at a marked panel, and it either kills the theory in five seconds or promotes it to the thing worth spending ten minutes on.

A worked example to carry through the lesson

Keep one scenario in mind. At 11:04 an alert fires: checkout latency above two seconds, sustained. Error rate is normal. Traffic is normal. The service was deployed at 10:58. By the end of this lesson you should be able to say precisely which panel you open first, which link takes you to a trace, which span you expect to be slow, which log line confirms it, and how long the whole path should take when the platform is built properly. The answer is about four minutes, and roughly forty when the links are missing.