0%
Observability and Delivery: Running Production Systems
The Signals
Metrics Systems
Logs and Traces
Reliability Targets
Incidents
Runbooks and Automating Toil
Every operations team has a category of work that nobody argues about and nobody enjoys. Rotating a certificate. Clearing a stuck queue. Re-running the batch job that fails on the first attempt roughly twice a week. Adding a user to a group because the self service path does not cover that group. None of it is hard, none of it is interesting, and all of it has to happen.
The useful name for this is toil, and the reason it deserves a name is that it behaves differently from other work. Project work is bounded: it ends, and when it ends the team has something it did not have before. Toil is unbounded and renewable. It scales with the size of the system, it arrives on somebody else's schedule, and when it is finished the system is exactly where it was before it started. A team that spends sixty percent of its time on toil is not a team that is sixty percent busy. It is a team with forty percent of a capacity to improve anything, including its own toil.
Toil is not simply work you dislike
The term gets used loosely, and the looseness causes real arguments, so it is worth being precise. Toil has a specific shape. It is manual, in that a human performs the steps. It is repetitive, in that it has been done before and will be done again. It is automatable, in that a machine could in principle do it. It is tactical rather than strategic, meaning it is reactive to an event rather than chosen as part of a plan. It has no enduring value, so the system is no better afterwards than before. And it grows at least linearly with the size of the service.
Several kinds of unpleasant work are not toil by this definition, and conflating them leads teams to automate the wrong things. Reading an unfamiliar codebase to understand a bug is not toil, because it produces understanding that persists. A long migration is not toil, because it ends and leaves the system changed. Writing documentation is not toil, however tedious it feels. The property that matters most in this list is automatable, because it is the one that makes the work a choice rather than a fact of life.
Measuring it without turning it into a timesheet
Teams resist measuring toil, usually for good reason: nobody wants an hourly accounting of their day, and the data would be unreliable anyway. The practical approach is coarser and works better. Take the ticket queue and the paging history for a month, classify each item as toil or not, and attribute a rough duration to each class rather than to each item. A class that fires forty times a month at twenty minutes each is thirteen hours, and the estimate does not need to be better than that to drive a decision.
The second thing worth counting is what fraction of the work arrives as an interrupt. An hour of planned toil on a Tuesday afternoon costs an hour. An hour of toil that arrives as a page during focused work costs the hour plus the twenty minutes of re-entry on either side of it, and it costs it to whoever is unlucky rather than to whoever is free. Interrupt driven toil is roughly twice as expensive as its clock time suggests, and that premium is the reason it is usually the right thing to attack first.
Where the hours actually go
When a team measures for the first time, the result is almost always surprising in the same direction. The work everybody complains about turns out to be a small fraction of the total, because it is memorable rather than frequent. The real volume is in two or three dull, fast tasks that happen several times a day and that nobody has ever mentioned in a retrospective, precisely because each instance is too small to be worth mentioning.
This is why the measurement has to come before the automation. Engineering intuition about which task consumes the most time is reliably wrong, because memory weights unpleasantness rather than duration. The queue data weights duration, which is what the budget cares about.
Toil is a reliability problem, not only a morale problem
It is tempting to treat toil reduction as a quality of life initiative, something a humane manager does for the team. That framing is true but it undersells the argument, and it loses funding competitions against features. The stronger argument is that toil is a direct reliability risk.
A team saturated with toil stops doing the work that prevents incidents. Postmortem actions slip. Monitoring gaps stay open. The upgrade that would have removed a known failure mode waits another quarter. Beyond that, toil is performed by tired humans under time pressure, and manual steps executed at 3am by somebody who has done them a hundred times are exactly where destructive mistakes come from. Every manual step is a place where the wrong cluster name can be typed. The incident caused by a mistyped command during routine maintenance is not an outlier; it is a predictable consequence of having routine maintenance that requires typing.
The fifty percent rule, and what it is for
A common guideline holds that an operations team should spend no more than half its time on toil. The number itself is arbitrary and arguing about whether it should be forty or sixty misses the point. The rule exists to force a conversation at a threshold rather than letting the fraction drift upward unnoticed, because the drift is self reinforcing: the more toil a team carries, the less capacity it has to remove any of it, and a team that crosses into permanent saturation does not climb back out without an explicit intervention.
What makes the rule work in practice is agreeing in advance what happens when the threshold is crossed. The options are genuinely limited. Hire, so there are more hours. Automate, which costs hours now to save hours later. Shed the work by handing it back to the teams that generate it or by declaring that a class of request is no longer served. Or accept a lower reliability bar explicitly, which is a legitimate choice as long as somebody says it out loud. A team that crosses the threshold and does none of these four things has chosen the fourth one without telling anybody.
Automation is not free, and sometimes not worth it
The arithmetic for whether to automate is simple and is frequently skipped. A task that costs twenty minutes and happens twice a week is thirty five hours a year. An automation that takes two weeks to build and two days a year to maintain pays for itself in roughly three years, which is longer than the system may live and longer than most of the team will be on it. The same task at twice the frequency pays back in eighteen months, which is a straightforward yes.
What this arithmetic consistently undervalues is risk and fatigue, both of which argue for automating sooner than the hours alone suggest. What it consistently overvalues is the first version of the automation, because the version that handles every edge case is five times the work of the one that handles the common case and escalates the rest. The honest version of the calculation includes the interrupt premium, the cost of the mistakes that manual execution will eventually cause, and a scope that deliberately leaves the hard ten percent to a human.
The cheapest automation is a written procedure
Before any of this becomes a scripting project, there is a step that costs an afternoon and captures a surprising share of the benefit. Writing the procedure down converts a task that only two people can perform into one that anybody on the rotation can perform, which immediately spreads the load and removes the escalations that happen because the person who knows is asleep.
It also does something less obvious. A task cannot be automated until somebody has articulated the decisions inside it, and the act of writing the procedure forces that articulation. Teams regularly discover during this step that the task they were about to automate has three branches nobody had named, or that two of the five steps are no longer necessary, or that the whole thing exists to work around a configuration default that could simply be changed. A written procedure is both the first increment of automation and the specification for the rest of it.
Loading problem...