0%
Observability and Delivery: Running Production Systems
The Signals
Metrics Systems
Logs and Traces
Incidents
Delivery
Dashboards That Get Used
Every organisation with a monitoring system has far more dashboards than it has questions. The count grows the same way the alert set grows, by one board per incident and one per new service, and it is subject to the same quiet decay, because a board that has stopped being correct looks exactly like a board that is still correct.
The useful starting point is to stop thinking of a dashboard as a view of a system and start thinking of it as an answer to a question somebody asks repeatedly. A board with no question behind it is a collection of panels that happened to be available, and nobody will open it, because there is nothing they were trying to find out.
What the usage data actually says
Most monitoring systems record which dashboards were viewed and by whom, and almost nobody looks at that data. When a team looks for the first time, the shape is consistent across organisations of very different sizes.
A small number of boards, usually between three and eight, account for the overwhelming majority of views. A second group is opened a handful of times a month, almost always from a link in an alert or a document. The long tail, which is normally more than half the total, has not been opened by a human in six months.
The tail is not harmless. It costs query load on the metrics backend when boards refresh on a schedule, it costs confusion when somebody finds an abandoned board during an incident and reads stale panels as current, and it costs credibility, because a directory with four hundred entries and no organisation is indistinguishable from no directory at all.
How the tail gets created
Three mechanisms produce almost all of it. The first is automatic generation: a service template that creates a board per service, which guarantees coverage and guarantees that none of the boards were designed. The second is the incident board, created at two in the morning to hold the four panels somebody needed, saved with a name nobody will recognise, and never revisited. The third is the copy, where somebody duplicates an existing board to change one query and leaves both.
None of these is unreasonable at the moment it happens. The problem is purely that nothing ever removes them, so the set accumulates at a steady rate and the fraction that is worth keeping falls every quarter.
The three audiences, and why one board cannot serve them
Dashboards fail most often because they try to answer three different questions at once, for three groups of people who want very different things.
The first audience wants to know whether the service is meeting its promises right now. They want four numbers, large, with the targets beside them, and nothing else. This audience includes executives, support staff and anybody deciding whether to escalate, and the board they need can be read from across a room in two seconds.
The second audience is a responder during an incident. They already know something is wrong and they want to narrow it: which journey, which dependency, which instance, which deploy. They need ordering and drill-down rather than breadth, and they need the time range to be wrong in a specific way, which is to say pinned to the incident rather than to the last six hours.
The third audience is an engineer doing capacity or performance work on a Tuesday afternoon. They want detail, they want a long time range, they want to compare this week against last month, and they are perfectly happy with a dense board because they are reading it deliberately with time to think.
A single board that attempts all three is too dense for the first audience, too broad for the second and too shallow for the third. The fix is not cleverness in layout. It is three boards with three names, and links between them in the direction people actually travel, which is from the summary into the drill-down and almost never the reverse.
The question test
Before a board is created, it is worth writing the question it answers in one sentence, and putting that sentence in the board's description. Is checkout meeting its target today. Which dependency is causing checkout to miss its target. How much headroom does the payment service have before the next peak.
A board whose question cannot be written in one sentence is usually several boards, and a board whose question is already answered by an existing board is a duplicate that will divide attention. This single discipline removes most of the sprawl before it happens, costs about thirty seconds, and is almost never done.
It also changes the panels. A board that exists to answer whether checkout is meeting its target does not need a panel for node memory, however available that metric is. Panels that do not contribute to the question are not neutral, because every one of them adds to the time it takes to read the board and to the chance that somebody reads the wrong one under pressure.
What people do instead of using a dashboard
It is worth noticing what happens when the board set is bad, because the behaviour is recognisable. People keep private boards, or a saved query in a text file, or they ask a specific colleague who they know can find the answer. During incidents they share screenshots in chat rather than links, because a link would require the reader to reconstruct the time range.
Every one of those is a workaround for the same gap, which is that the board that should have answered the question either does not exist or cannot be found among the four hundred that do. Treating the workarounds as evidence rather than as individual habits is what turns dashboard work from a tidying exercise into something with a measurable result.
The cost of a board that is merely available
There is a quieter cost that rarely enters the argument, which is what a large directory does to the people searching it. A responder who knows that four hundred boards exist and that most of them are untrustworthy does not search; they ask a colleague, or they open the two boards they have memorised and work from those, whether or not those boards answer the question.
This is the mechanism by which a monitoring investment stops paying. The data is present, the panels exist, and the behaviour of the organisation is indistinguishable from an organisation that never built them, because nothing in the directory signals which entries are worth reading. Search ranking does not solve it, because recency and title matching are poor proxies for correctness.
What does solve it is a set small enough that people can hold its shape in their heads, with names that say what question each board answers. Teams that reach that state describe the change in terms of behaviour rather than tooling: people start navigating rather than asking, and the two memorised boards stop being a coping strategy.
Generated boards, and when they are worth having
It is worth being fair to the automatic generator, because the argument against it is often overstated. A per service board generated from a template is genuinely useful for the long tail of services that nobody has thought about, where the alternative is no board at all, and it is the only way to get coverage across a hundred services without a hundred afternoons of work.
The problem is not generation. It is that generated boards and hand designed boards sit in the same directory with the same visual weight, so a reader cannot tell which one somebody thought about. Separating them, by folder, by naming convention or by tag, costs almost nothing and restores the signal: the hand built boards are the ones that answer a named question, and the generated boards are the fallback when no such board exists.
The second problem is that a generated board is sometimes treated as evidence that the journey is covered, which it is not. A template cannot know which of a service's calls matter to a user, so it shows every metric equally, and equality is the opposite of what a reader under pressure needs.