0%
Observability and Delivery: Running Production Systems
The Signals
Metrics Systems
Logs and Traces
Reliability Targets
Incidents
Efficiency and Cost as a Reliability Concern
Infrastructure cost is usually treated as a finance problem that occasionally interrupts engineering, which is why the conversation arrives once a quarter, in the form of a number that nobody recognises and a request to reduce it. Treated that way it produces the worst possible outcome: a round of cuts chosen by whoever is least able to defend their line item, followed six months later by an incident whose root cause was the headroom somebody removed.
Cost belongs in the same conversation as reliability because it is the same conversation. Every reliability mechanism you have read about in this course is paid for. Replication is paid for. Headroom is paid for. Multi zone redundancy, retained logs, retries, warm standby capacity and the second region are all bought with money, and the question is never whether to spend but what each unit of spend buys. A team that cannot say what its spend buys is going to cut the wrong things, because in the absence of a stated return every line looks equally optional.
Start with the shape, not the total
The total is not actionable. A quarter of a million dollars a month means nothing until it is decomposed, and the first decomposition should be by component rather than by team, by region or by account, because components are what engineers can actually change. Compute, storage, data transfer, managed services, observability and licences behave differently and respond to different interventions, so a single total hides the fact that four of them are fine and one of them has doubled.
The decomposition almost always contains a surprise. Observability spend is the most common one: teams routinely discover that logs and metrics cost more than the application tier that produces them, which is a perfectly defensible state of affairs if somebody chose it and an embarrassing one if nobody did. Data transfer is the second: cross zone traffic generated by a service mesh that was configured for safety rather than locality can quietly become a larger line than the compute it connects.
The units that make a number mean something
An absolute monthly figure is almost useless for engineering decisions because it moves with traffic, and traffic moves for reasons unrelated to efficiency. The number that carries information is a ratio: cost per request, per active user, per gigabyte processed, per order, per tenant. A doubling of total spend accompanied by a tripling of traffic is an efficiency improvement reported as a crisis, and without a unit metric nobody in the room can tell which of the two things happened.
Pick the unit that matches how the business grows. A platform that charges per seat should track cost per seat. A pipeline should track cost per terabyte processed. The unit does not need to be perfect, and arguing about the denominator is a common way to avoid starting. What it needs to be is consistent, so the trend is honest even when the level is approximate.
A second useful habit is to carry the unit metric alongside the reliability metrics in the same review. Cost per request belongs on the same page as latency and error rate, because all three are properties of the same design and the trade between them is the actual subject of most architecture decisions.
Cost is a lagging, noisy signal
Infrastructure billing arrives late, aggregated and in a schema built for invoicing rather than for engineering. A change made on Tuesday may not be visible until the following month, and the attribution may be wrong in ways that take a day to untangle. This matters because it shapes how you can use the signal: cost is a review metric, not an alerting metric, with one exception, which is a sudden step change.
The step change is worth an alert because it is almost always a defect: a retry storm, a runaway job, a log level left at debug in production, a misconfigured autoscaler, or a test fixture that forgot to clean up. These are reliability events that happen to show up on a bill, and they are frequently the first visible symptom of a problem that will become an outage later. A team that alerts on a daily spend anomaly often catches a bug a week before its users do.
Not all spend is equal, and some of it is insurance
A great deal of infrastructure spend buys capacity that is deliberately unused. Standby replicas, spare headroom, the second availability zone, retained backups, the extra broker in the cluster: in steady state these do nothing, and in a spreadsheet they look like waste. They are insurance, and insurance is evaluated by the probability and severity of the event it covers, not by whether the event happened last month.
This is why cost conversations go badly when the people in them have different models of the same line. The engineer sees a replica that will carry the system through a zone loss. The analyst sees an instance at four percent utilisation. Both descriptions are accurate. The resolution is not to argue about utilisation but to name what the line insures against and what the organisation thinks that event is worth, which converts an aesthetic disagreement into a decision somebody can actually make.
The cheapest resource is the one you did not provision
Most durable efficiency work is not tuning. It is deletion and avoidance: the environment that nobody has used since the migration, the dataset retained for a compliance requirement that expired two years ago, the three services that could be one, the replicas of a cache that nothing reads. These are the cheapest wins available and they are unglamorous, which is why they persist.
The standard reason they persist is that nobody is sure they are unused, and nobody wants to be the person who deleted the thing that turned out to matter. The solution is procedural rather than technical: make it safe to find out. Turn the thing off rather than deleting it, announce it, wait a defined period, then delete. A documented decommissioning path converts a frightening irreversible action into a routine reversible one, and teams that have one delete an enormous amount of inherited spend.
Who owns the number
Cost has the same ownership pathology as any cross cutting concern. It is produced by every team and owned by none, it appears in a finance review attended by nobody who can change it, and the engineers who could change it see the number quarterly if at all. The organisations that handle this well do one unremarkable thing: they put the number in front of the people who create it, at a cadence short enough that the connection between a decision and its cost is still legible.
That does not require a chargeback model, which is a heavier instrument than most organisations need and which produces its own distortions. It requires visibility with attribution, and attribution requires tagging discipline that somebody has to enforce before it is needed rather than during the first cost crisis. A tagging standard applied from the beginning costs almost nothing. Applied retroactively across three years of infrastructure it is a project, and it is the project nobody funds.
The practical minimum is that every resource can be traced to a team, every team can see its own trend without asking anybody, and one person is accountable for the total. The last part matters more than it sounds: a total that is everybody's responsibility moves in exactly one direction.
Efficiency work competes with everything else
It is worth saying plainly that efficiency work has the same problem as toil reduction: it is never the most urgent thing, it does not ship a feature, and its benefits accrue to a budget rather than to a user. Teams that get it done do so by making it visible and bounded. A named target with a deadline, a single owner, and a number that somebody reports on produces results. A general exhortation to be mindful of cost produces nothing at all, in every organisation, every time.