0%
Observability and Delivery: Running Production Systems
The Signals
Metrics Systems
Logs and Traces
Reliability Targets
Delivery
Postmortems and Follow Through
A postmortem is not a report. It is a mechanism for converting one organisation's expensive hour into a change that makes the next hour cheaper, and every part of the practice should be judged against whether it does that. Most postmortem processes fail this test while producing documents that look entirely correct.
The failure is recognisable. A template is filled in, a root cause is named, a list of action items is appended, the document is filed in a folder that nothing links to, and six weeks later a strikingly similar incident happens. Nobody did anything wrong. The process ran, and it produced nothing.
Learning is the only output that counts
The things a postmortem can produce are a shared understanding of what happened, a changed system, a changed process, and a record somebody will read later. Only the middle two alter the future, and they are the two most often skipped in favour of the first and the last.
This is worth stating plainly because it determines what to optimise. A beautifully written narrative that produces no change is worth less than three sentences that produce one deleted alert. A ninety minute meeting where everybody now understands the failure, and which produces nothing owned, has spent six person hours buying a feeling.
The test to apply to any postmortem practice is simple: six months from now, will anybody be able to point at something in the system that is different because of this document. If the honest answer is usually no, the practice is ceremonial and should be changed or stopped.
Why the incident is the cheapest teacher available
An incident has already been paid for. The outage happened, the people were woken, the customers were affected, and that cost is sunk regardless of what happens next. The postmortem is the only opportunity to get anything back.
This framing matters because it changes how much effort the postmortem deserves. Teams routinely spend eleven hours of engineering time on an incident and then forty minutes on understanding it, which is a strange allocation given that the forty minutes is the only part with a return.
It also explains why near misses deserve postmortems. An event where the system nearly failed but did not has delivered the same information at a fraction of the cost, and skipping it because nobody was affected throws away the cheapest version of the lesson available. The teams that run postmortems on near misses are usually the ones that have fewer real incidents to run them on.
What belongs in the document, and what does not
A useful postmortem answers a small number of questions. What was the user visible impact, and for how long. What sequence of events produced it. Why did it take as long as it did to detect, to understand and to mitigate. What was surprising. And what will be different.
Several things that commonly appear do not belong. A blow by blow of every command run is noise unless it illustrates something. A long description of the system's normal architecture belongs in the architecture documentation and linked. An apology is for the customer communication, not the internal document. And a section asserting that the team responded admirably is a social gesture that makes the document harder to read honestly.
The strongest signal of a healthy document is that it contains something uncomfortable. If no postmortem in a team's history records that a decision was wrong, a system was known to be fragile, or a warning was ignored, the documents are being written for an audience rather than for the next engineer.
The audience is a future engineer
Postmortems are written as though the audience is management, and read, if at all, by an engineer debugging something similar eighteen months later. Writing for the actual audience changes several things.
It means naming systems precisely rather than generically, because the future reader is searching for a component name. It means including the specific error text, which is what somebody will paste into a search box. It means stating what was tried and did not work, because that is the part they cannot derive. And it means saying what the team believed at the time and why that belief was reasonable, since the reader's instinct will be to wonder how anybody could have missed it.
Writing for a management audience produces the opposite document: vague about systems, confident about causes, silent about dead ends, and defensive about judgement. It is also, not coincidentally, the document nobody reads.
The meeting is optional, the document is not
Many teams treat the review meeting as the postmortem and the document as its minutes. This is backwards, and it is why postmortems feel expensive.
The document is the artefact. It persists, it is searchable, and it is what the future engineer finds. The meeting is a tool for improving the document, and it is worth holding when the sequence of events is genuinely contested, when several teams hold different pieces of the picture, or when the conclusions are uncomfortable enough that they need to be said in a room rather than written by one person. For a straightforward incident with one team and an agreed timeline, a document circulated for comment is better: it costs one person an hour instead of eight people an hour, and the written comments are usually more considered than what gets said out loud.
The failure mode worth avoiding is the meeting that exists to read the document aloud to people who have not read it. If attendance is required and nobody prepares, the meeting is a reading group and should be replaced with a deadline for comments.
When not to write one
Not every incident earns a full postmortem, and pretending otherwise is how the practice becomes resented.
A sensible bar is that a postmortem is required when the incident had meaningful user impact, when it was surprising, when the response was difficult regardless of impact, or when it was the second occurrence of something. A brief outage with a known cause, a known fix and no surprises needs a note and a link, not a meeting.
What should never happen is the reverse, where an incident is skipped because it is embarrassing, because it was somebody's first week, or because a similar one was written up recently and nothing came of that either. Those three are the strongest signals that a postmortem is needed.
The relationship with the incident itself
Everything in the previous lesson about running an incident determines how good the postmortem can be. A response with a contemporaneous timeline, announced state changes and a hypothesis queue produces a postmortem in an hour. A response with none of those produces a day of archaeology and a document that is partly fiction.
This is worth making explicit to a team, because the incentive to keep a timeline during an incident is otherwise abstract. The concrete version is that the scribe is not doing it for the record, they are doing it so that nobody has to spend Thursday reading scrollback and guessing.
The two questions that produce most of the value
If a team can only answer two questions about an incident, the highest value pair is not what broke and why. It is why did it take this long to detect, and why did it take this long to understand.
The cause is usually specific and often unrepeatable. A particular config field, a particular version interaction, a particular queue depth. Fixing it prevents exactly that failure and very little else. Detection and comprehension time, by contrast, generalise completely: whatever made this failure invisible for eleven minutes will make the next unrelated failure invisible too, and whatever made it take thirty five minutes to understand will do the same again.
This is why the timeline matters more than the narrative. The gaps between the event, the alert, the declaration, the first correct hypothesis and the mitigation are the only numbers in the document that describe the organisation rather than the bug.