An incident starts with a question, and the question decides which signal can answer it. The four signals are not four vendors competing to sell the same thing. They are four different measurements, each of which answers a class of question cheaply and answers every other class badly or not at all.

The question is the thing to classify, not the tool

There are only a handful of questions anyone asks of a production system during an incident. Is it broken, and for whom. How broken, and since when. Where in the path did the time or the errors appear. What is the process actually doing with the cycles it is burning. And what happened to this one specific request that the customer is complaining about on the phone.

Each of those maps cleanly onto one signal. Is it broken and how broken is a metric. Where in the path is a trace. What is the process doing is a profile. What happened to this one request is a log line, or a trace if the request crossed services. The mistake that wastes the most money in this field is answering a metric question with logs, because logs can technically answer it and the bill arrives a month later.

A metric is a number sampled on a schedule

A metric is a named number with a set of labels, recorded at a regular interval. The checkout service exposes a counter of requests handled, a Prometheus server reads it every 15 seconds, and the pair of a timestamp and a value is appended to a time series.

The defining property is that the cost of a metric does not depend on traffic. A counter that has been incremented four million times in the last fifteen seconds produces exactly the same single sample as a counter that was incremented twice. This is the only signal in the set with that property, and it is why metrics are the backbone of alerting and dashboards.

Prometheus stores a sample in a little over one byte on average after compression, using delta of delta timestamps and XOR encoded values. A series scraped every 15 seconds produces 5,760 samples a day, which is about 7 KB a day per series, or 2.5 MB a year. Ten thousand series costs 70 MB a day. That number is small enough that teams stop thinking about it, which is exactly the habit that the cardinality lesson later in this course exists to break.

A log is a record of one event

A log line is a statement that one specific thing happened, with whatever context the author chose to attach. Its cost is strictly proportional to how often that thing happens.

Take a service handling 20,000 requests per second that writes one 400 byte structured line per request. That is 8 MB per second, 691 GB per day, 20 TB per month before indexing overhead, which in most hosted log products multiplies the stored volume by somewhere between 1.2 and 3 depending on how many fields are indexed. At a typical ingest price of around 50 cents per GB, that single service's access log costs roughly ten thousand dollars a month to keep for thirty days.

The value you get for that is total specificity. A log line can carry the order id, the coupon code, the exact exception and the stack. No other signal can tell you that request 4f2a9c failed because the coupon had expired four minutes earlier. Metrics cannot, because the label would have to contain the order id, and then it would not be a metric any more. That tension between specificity and cost is the whole economics of observability.

A trace is one request's path across services

A trace is a tree of spans. Each span records an operation with a start time, a duration, a parent, and some attributes. A span for the checkout handler contains a span for the cart lookup, which contains a span for the database query. The spans are stitched together by a trace id that travels in the request headers across every network hop.

A trace answers exactly one question, and answers it better than anything else: where did the time go. Given a request that took 900 milliseconds, the trace says that 63 milliseconds were in auth, 189 in the cart service of which 162 were one database query, and 207 in the payment call. That decomposition is not derivable from metrics, because each service's metrics are aggregates over all requests and cannot be joined back to this one. It is not derivable from logs unless every service logs its own timings and something reassembles them, which is tracing done badly.

A profile is what one process was doing with its cycles

A profile is a statistical sample of a running process. A CPU profiler interrupts the process on a timer, typically 100 times per second, records the stack at each interrupt, and aggregates. After 30 seconds you have 3,000 stacks, and the proportion of samples containing a given function is an unbiased estimate of the proportion of CPU time spent in it.

A profile is the only signal that can see inside a single process. When the trace says that the cart service spent 189 milliseconds and the database span inside it accounts for 162, the trace has told you where to look but cannot explain the remaining 27 milliseconds. A profile can, and the answer is routinely something like JSON serialisation at 41 percent of CPU, or a regular expression compiled inside the request path rather than once at startup.

The four cost shapes, side by side

Per event handled, a metric costs roughly nothing, since it amortises over the scrape interval. A trace span costs around 500 bytes and there are typically 10 to 50 of them per request, so a sampled trace costs real money and an unsampled one costs a great deal. A log line costs its own size, every time. A profile costs a fixed amount per process per interval, independent of traffic entirely, which makes continuous profiling surprisingly cheap at scale: a few hundred kilobytes per process per minute, regardless of whether that process served ten requests or ten thousand.

Those cost shapes, rather than any feature comparison, are what should decide the division of labour. Put everything you need to alert on in metrics, because the volume is flat. Put everything you need to explain a slow request in traces, and sample them. Put the facts you will need to reconstruct a specific business event in logs, and resist the urge to log anything you could count instead. Run profiles continuously, because they are nearly free and they answer the question nothing else can.

Why the boundaries blur, and why that is fine

Modern systems convert between signals constantly. A log pipeline can count lines matching a pattern and emit a metric. A tracing backend can compute request rate and error ratio from spans and expose them as metrics, which is what span metrics means. A profile can be attached to a trace span so you can see the stacks sampled during that specific slow request.

None of that collapses the four into one. A metric derived from logs still costs what the logs cost, because you had to ship them to derive it. Conversion is a convenience at the query layer, not an escape from the economics underneath.

Two of the four signals exist to answer the question that metrics structurally cannot: inside this one slow request, what was the system doing. Traces answer it across process boundaries and profiles answer it inside a process, and they are most useful when read one after the other.

Context propagation is the entire mechanism

A trace works because an identifier survives every hop. The incoming request either carries a traceparent header or does not, in which case the first service to see it generates one. That header carries the trace id, the id of the current span, and a flag saying whether this trace was sampled. Every outbound call the service makes copies the trace id and sets its own span id as the parent.

Everything else about tracing is bookkeeping around that one rule. If a single service drops the header, the trace splits into two unrelated trees and the waterfall shows a mysteriously short parent with no children. In practice the commonest cause of broken traces is not a missing library but a hop that reconstructs the request: a message queue where the producer does not put the context into the message attributes, a thread pool that does not carry the context across the handoff, or a proxy configured to strip unknown headers.

What a span actually contains

A span is a name, a start timestamp, a duration, a parent span id, a status, and a bag of attributes. In OpenTelemetry the wire representation of a typical HTTP client span is around 400 to 600 bytes once the standard attributes are present: method, route, status code, peer address, and the resource attributes identifying the service, version and host.

Two of those fields do most of the work. The parent id builds the tree, and the attributes are what makes a trace searchable later. A span with no attributes beyond its name is almost useless in aggregate, because the only queries you can run are by name and duration.

Reading a waterfall from the top down

A waterfall is the tree flattened into rows, each row a bar positioned by start time and sized by duration. The discipline for reading one is to go top down and subtract.

Start with the root: 900 milliseconds. Look at its direct children and add their durations. If the children account for 520 milliseconds and the root is 900, then 380 milliseconds happened inside the root span itself, in code nobody instrumented. That subtraction is the single most valuable operation in trace analysis, because the time that is not in any child span is invisible to everyone who only reads the span list and sorts by duration.

The three things a waterfall shows at a glance

Serial work appears as bars starting one after another, each beginning roughly where the previous ended. That shape is an invitation to ask whether the calls are independent, because two independent 150 millisecond calls run in sequence cost 300 milliseconds and run in parallel cost 150.

A repeated small bar, forty times in a row, is the N plus one query pattern rendered visually. The total may be only 90 milliseconds, but the shape is diagnostic: one query to fetch a list and then one query per item.

A single bar wider than everything else is the easy case, and it is the only case most people look for. The other two shapes are where the cheap wins live.

A profile is a statistical estimate

A CPU profile does not measure time spent in a function. It samples the stack at a fixed frequency and counts. At 100 hertz over 30 seconds there are 3,000 samples, and a function that appears in 1,230 of them is estimated to have used 41 percent of the CPU time in that window.

Because it is an estimate, the error bars matter and most people never think about them. The standard error on a proportion from n samples is roughly the square root of p times one minus p over n. For p of 0.41 and n of 3,000 that is about 0.9 percentage points, which is fine. For a function appearing in 15 of 3,000 samples, p is 0.005 and the estimate is 0.5 percent plus or minus 0.13 percentage points in absolute terms, but that is a relative error of 26 percent, which means the bottom of a profile is noise and should not be optimised against.

Reading a flame graph

A flame graph puts the stack on the vertical axis and the proportion of samples on the horizontal. Width is what matters and the horizontal order carries no meaning at all, which is the thing almost everyone misreads on first contact: the bars are sorted alphabetically so that the same code produces the same picture twice, not by time.

Read it by scanning the widest plateau at each level downward. A wide block near the top with a narrow parent is a leaf doing real work. A wide block whose children are all narrow and varied is a function whose own body is expensive rather than its callees. In a Go service the classic finding is a wide encoding/json.Marshal under the response handler, which is not a bug in anything, just an expensive default choice.

When the trace and the profile disagree

A trace says the cart service spent 189 milliseconds on this request. A profile of the cart service over the same minute says the process was 70 percent idle. Both are true and the combination is informative: the request was not CPU bound, so the time went to waiting. Waiting on what is then a question for the span attributes, the connection pool metrics, or an off CPU profile, which samples goroutines or threads that are blocked rather than running.

This is the single most useful cross signal inference in the set. CPU profiles only see work that is on CPU. A service that is slow because it is waiting for a lock, a connection from an exhausted pool, or a downstream response will look perfectly healthy in a CPU profile, and that very healthiness is the finding.

Span kinds, and why they change the reading

OpenTelemetry labels each span with a kind: server, client, producer, consumer or internal. The pairing of a client span in one service with a server span in the next is what lets a backend show network time, because the client span includes the round trip and the server span does not. The difference between them is connection setup, queueing in the proxy, and the wire.

When that difference is large and the server span is small, the downstream service is innocent and the path between them is not. That is a distinction metrics on either side cannot make, because each service measures only its own handler.

Clock skew is real and the backend hides some of it

Spans carry timestamps from different machines, and those clocks disagree. With a well run NTP setup the disagreement is a few milliseconds, which does not matter for a 900 millisecond request and matters a great deal for a 3 millisecond one. Without one it can be hundreds of milliseconds, which produces waterfalls where a child appears to start before its parent or to finish after it.

Most backends correct the obvious violations by clamping children inside their parents, which makes the picture readable and quietly destroys the evidence that the clocks are wrong. If a waterfall looks physically impossible, suspect the clocks before suspecting the code, and check the host's time synchronisation metrics, which is a measurement worth having on every node for exactly this reason.

Metrics are cheap because they are aggregated before they are stored. Everything useful and everything misleading about them follows from that one fact, so it is worth being precise about which aggregations preserve what.

The four instrument types, and what each one can answer

A counter only goes up. It records how many times something happened, and its value at any instant is meaningless on its own. What you read from a counter is its rate of change, which is why the first thing anyone writes in PromQL is rate().

A gauge goes up and down. Queue depth, memory in use, number of connections. A gauge is a snapshot, so the value between two scrapes is unobserved, and a gauge that spiked to 40,000 and came back inside a single 15 second interval leaves no trace at all.

A histogram counts observations into predefined buckets. A request duration histogram with buckets at 5, 10, 25, 50, 100, 250, 500, 1000 and 2500 milliseconds stores ten cumulative counters plus a sum and a count, so twelve series per label combination. From those counters you can estimate any quantile after the fact.

A summary computes quantiles in the client process and exports them directly. That sounds better and is usually worse, for one specific reason covered below.

Why you cannot average a percentile

This is the most common statistical error in production monitoring, and it is worth stating bluntly: the average of the p99 of ten instances is not the p99 of the fleet.

Take two instances. Instance A serves 10,000 requests with a p99 of 100 milliseconds. Instance B serves 100 requests with a p99 of 2,000 milliseconds. The mean of the two p99 values is 1,050 milliseconds. The true fleet p99 is close to 100 milliseconds, because 99 percent of the 10,100 total requests come from the fast instance. Averaging produced a number that is wrong by a factor of ten and wrong in the direction that generates false alarms.

The reverse error is equally common. If instance B is the slow one and it serves most of the traffic, averaging the percentiles understates the fleet tail and hides a real problem.

Why histograms win and summaries lose

A histogram stores bucket counts, and bucket counts add. To get the fleet p99 you sum the per instance bucket counters, then interpolate within the bucket where the 99th percentile falls. The arithmetic is exact up to the bucket width.

A summary stores precomputed quantiles, and quantiles do not add. There is no operation on ten p99 numbers that recovers the fleet p99, because the information needed to combine them was thrown away in the client. This is why a service exporting summaries can be monitored per instance and can never be monitored as a fleet, and why the Prometheus ecosystem defaults to histograms despite their higher series count.

The cost of the choice is bucket resolution. A p99 estimated from a histogram whose buckets jump from 500 to 1000 milliseconds can only say that the p99 is somewhere in that range, and the linear interpolation inside the bucket is an assumption rather than a measurement. Choosing bucket boundaries is therefore a real design decision: put the boundaries where your objective is, so that the number you report against your target is measured precisely at the threshold that matters.

Averages hide the shape, and the shape is the problem

A mean response time is the single least useful latency statistic, because the distributions in question are not symmetric. A service whose p50 is 40 milliseconds and whose p99 is 900 milliseconds has a mean around 70 milliseconds, and the mean will barely move when the p99 doubles.

The practical consequence is that a dashboard showing mean latency can look completely flat through an incident in which one percent of users are timing out. One percent of a million daily requests is ten thousand people.

Aggregating across instances hides the one that is broken

The second thing aggregation throws away is which instance. A service level error ratio of 0.9 percent across fifteen pods can mean every pod is failing slightly, which is a dependency problem, or that one pod is failing on 13 percent of its requests while fourteen are perfect, which is a bad host, a corrupted cache or a pod that missed a configuration update.

Those two situations call for completely different actions, and the aggregate cannot distinguish them. This is why the instance label is worth its cardinality in a way that the customer id is not: the instance count is bounded by the fleet size, and the ability to split by it turns a vague number into a specific pod name.

What to pre aggregate and what to leave alone

Recording rules let you compute an expression once per evaluation interval and store the result, which makes expensive dashboard queries cheap. The rule for choosing what to pre aggregate is to look at what the dashboard asks for repeatedly and whether the aggregation is associative.

Sums and counts are safe to pre aggregate because they combine. Quantiles are not, and a recording rule that stores a p99 per service has created exactly the summary problem described above one layer higher up. The correct pattern is to pre aggregate the bucket counters, which still combine, and compute the quantile at query time.

Cumulative and delta temporality

A Prometheus counter is cumulative: it reports a total since process start, and the reader computes the difference. A delta counter reports what happened since the last report, and the writer computes the difference. OpenTelemetry supports both and the choice propagates through the whole pipeline.

Cumulative is resilient to lost data points, because a missed scrape costs resolution rather than correctness: the next scrape still carries the total, so the rate over a window spanning the gap is right. Delta is the opposite, since a dropped report is a permanently lost increment, but it suits push based pipelines and short lived processes that would otherwise disappear before anyone scraped them. A serverless function that lives for 200 milliseconds cannot be scraped at all, which is the case delta temporality exists for.

The failure to watch for is mixing them in one pipeline. A collector configured to convert cumulative to delta keeps state per series, and when that state is lost on a collector restart, the first delta after the restart is either dropped or enormous. Knowing which temporality each hop produces is part of knowing whether a spike on a chart is real.

Counter resets, and why rate is not a derivative

A counter in a process resets to zero when the process restarts, and in a fleet with rolling deployments that happens constantly. A naive difference between two samples would show a large negative number at every restart.

The rate function handles this by detecting that a sample is lower than its predecessor and treating the gap as a reset, adding the pre reset value. The consequence worth knowing is that rate is an estimate over a window rather than an instantaneous derivative, and the window must be at least four times the scrape interval to tolerate a missed scrape. A 1 minute rate over a 30 second scrape interval has only two samples and goes blank the moment one scrape fails, which is precisely when you are looking at it.

Native histograms change the arithmetic

Classic histograms fix the bucket boundaries at instrumentation time, so twelve series per label combination and a resolution you chose months ago. Native histograms, supported in recent Prometheus versions, store exponential buckets with a configurable factor and carry the whole distribution in a single series.

The practical effects are that series count falls by an order of magnitude for the same information, that quantile estimates stop depending on whether someone guessed the right boundaries, and that merging across instances remains exact because the bucket schema is shared. Where they are available, they remove most of the tension between histogram accuracy and cardinality, and the bucket boundary advice above becomes advice about the resolution factor instead.

Observability spend is one of the few infrastructure costs that grows without anyone deciding to grow it. Nobody approves a tripling of the log bill. It happens because a hundred engineers each add a debug line to a hot path, and no single addition is unreasonable.

Work the bill as a formula, not as an invoice

For each signal the cost is a product of a few numbers you can estimate in advance, and writing the product down converts an argument about principles into an arithmetic disagreement, which is much easier to settle.

Logs cost events per second, times bytes per event, times seconds of retention, times a storage and indexing multiplier. Metrics cost active series, times samples per second, times bytes per sample, plus a memory term for the active set. Traces cost requests per second, times spans per request, times bytes per span, times the sampling rate. Profiles cost processes, times profiles per hour, times bytes per profile.

A worked example for one service

Consider a service at 20,000 requests per second, running on 60 pods, with 40 spans per request.

Logs: one 400 byte line per request is 8 MB per second, 691 GB per day. At 30 day retention that is 20.7 TB stored, and a hosted product charging 50 cents per GB ingested bills on the 691 GB per day, so about 10,400 dollars per month for one access log.

Metrics: suppose 400 distinct series per pod, which is a realistic number for a service exporting a handful of histograms plus runtime metrics. That is 24,000 active series, scraped every 15 seconds, so 1,600 samples per second at roughly 1.3 bytes compressed, which is 180 MB per day and about 5.4 GB for a month, costing single digit dollars in storage. The real constraint is memory in the ingesting server, at roughly 3 KB per active series, so 24,000 series is about 72 MB of head block, which is nothing. Metrics are approximately free until cardinality breaks them, and then they are not free at all.

Traces: 20,000 requests per second times 40 spans is 800,000 spans per second. At 500 bytes a span that is 400 MB per second unsampled, which is 34 TB per day, which no one pays. At 1 percent head sampling it is 4 MB per second and 345 GB per day, which is comparable to the log bill and buys a far more useful signal.

Profiles: 60 pods producing a CPU and a heap profile every minute at roughly 120 KB each is 14.4 MB per minute, 20 GB per day, and compresses well. For the only signal that can see inside the process, that is the cheapest line on the invoice.

The three levers, in order of effect

The first lever is volume at the source. A debug line on a path taken 20,000 times a second is a recurring monthly expense, and deleting it is free. Sampling logs is the second best version of this, and dropping fields is the third, because a structured line with forty attributes where four are read costs ten times what it needs to.

The second lever is retention, and it is the one teams reach for first because it is a single number in a configuration file. It is also the one most likely to be regretted, because the question you cannot answer in six months is usually a question about something that happened five months ago. A better shape is tiered: full fidelity for 7 to 14 days where interactive debugging happens, then either aggregate or move to cold object storage where the price per GB falls by an order of magnitude and the query latency rises to minutes.

The third lever is the shape of the data. A field that is indexed costs more than a field that is merely stored, so indexing only what you filter on routinely is often a 30 to 50 percent saving with no loss of information at all.

The question to ask of every line item

For each large item, ask what question it answers, when that question was last asked, and whether a cheaper signal answers it. The access log at ten thousand dollars a month is usually answering two questions: what is the request rate and error ratio, which is a metric, and what happened to this one request, which is a sampled trace with exemplars pointing at it.

That is not an argument for deleting logs. It is an argument for logging the events that cannot be reconstructed any other way, which are typically business events rather than requests: the coupon was applied, the payment was captured, the refund was issued.

Queries cost money too, and they are invisible

Ingest is the line item everyone looks at, but on many platforms query is billed separately and can be the faster growing half. A dashboard with thirty panels, auto refreshing every ten seconds, open on a wall display all day, issues 259,200 queries a day by itself. If each one scans a few hundred megabytes, the scanned volume dwarfs the data ingested.

Three habits control it. Set dashboard refresh intervals to something a human can actually use, which is rarely faster than 30 seconds. Use recording rules so the expensive aggregation happens once per evaluation interval instead of once per viewer. And put a time range limit on exploratory log queries by default, because the pattern where someone searches 30 days to find something that happened an hour ago is both common and expensive.

Alert evaluation is a constant background cost

Every alert rule is a query running on a schedule forever. Two hundred rules evaluating every 30 seconds is 576,000 query executions a day, and a rule written as a nested subquery over a high cardinality metric can take seconds to evaluate. When a metrics server starts falling behind on rule evaluation the symptom is delayed alerts, which is the worst possible failure mode because the system looks healthy.

The fix is the same recording rule discipline as dashboards: an alert should read a pre aggregated series, not perform the aggregation itself. It also pays to measure rule evaluation duration, which every mature metrics system exposes, and to alert on it.

Self hosted and hosted move the cost, they do not remove it

Running Prometheus, Loki and Tempo yourself converts a per gigabyte invoice into machines, object storage and the engineering time to operate them. That trade is usually favourable at large volume and usually unfavourable at small volume, and the crossover is further out than enthusiasts expect, because the operational burden is not proportional to volume. Someone has to be on call for the monitoring system, and the monitoring system going down during an incident is a special kind of bad.

The honest comparison includes three terms people leave out. The storage amplification of replication, which is typically a factor of two or three for the components that need it. The cost of the object storage requests rather than the bytes, which for a query pattern that lists many small objects can exceed the storage line itself. And the engineering time, which at a loaded cost of a few hundred thousand dollars a year for a part time owner is often larger than the hosted bill it was meant to replace.

The decision that usually survives contact with reality is a split: self host the high volume, low query value signal, which is typically logs, and buy the signal whose value depends on a good query experience, which is typically traces.

What a reasonable budget looks like

As a rough benchmark from mature teams running at scale, observability tends to land between 5 and 15 percent of infrastructure spend. Below 5 percent usually means something is missing rather than that the team is efficient, and above 20 percent almost always means a log index that nobody has pruned. The ratio is not a target to optimise, it is a tripwire: a sharp move in it is a signal that someone changed what is emitted.

The last thing to understand about the four signals is that what you can ask at 03:40 is not what the system emitted. It is what survived instrumentation, sampling, cardinality limits, retention and the query interface. Each of those stages discards data, and the question is whether what is left answers the questions you will actually have.

The coverage funnel

Start with the events the system handles, say 1.7 billion requests a day. Instrumentation covers some fraction of the code paths, perhaps 80 percent of services fully and the rest partially. Sampling keeps 1 percent of traces. Cardinality limits mean that some labels were dropped at ingest, so certain breakdowns are simply not available. Retention means anything older than 14 days exists only in aggregate. And the query interface imposes the final filter: data that is stored but cannot be grouped the way you need is data you do not have.

Writing that chain down for a real system is a sobering exercise, and it is the right one to do before an incident rather than during it. The useful output is a short list of questions you cannot currently answer, which is a far more actionable artefact than a list of tools you have installed.

A decision procedure for the signal to reach for

If the question is whether something is broken and for how many users, use a metric, and if the metric does not exist, that is the gap to fix after the incident.

If the question is where in the path the time or the errors appeared, use a trace, and if tracing coverage does not reach the suspect service, fall back to comparing the latency metrics of adjacent services, which approximates the decomposition badly but cheaply.

If the question is why one process is slow while nothing it calls is slow, use a profile, and prefer a differential profile against a healthy instance of the same service.

If the question is what happened to one specific entity, use logs, and search by the identifier rather than by time, because searching a time range in a large index is both slow and expensive.

The order to use them in during an incident

The order that works is metric, trace, log, profile, and the reason is funnel width. A metric tells you which service and which minute, narrowing from everything to one component. A trace for a request in that minute tells you which call inside that service, narrowing again. A log line for that specific request tells you what the code said while it was failing. A profile tells you what the process was doing if the answer was none of the above.

Going in the other direction, which is what people do under pressure, means searching logs across the whole estate for a minute you have not yet identified. That is the most expensive query you can run and the slowest path to the answer.

Correlation is a design requirement, not a feature

The handoff between signals only works if the identifiers line up. A trace id present in every log line makes the trace to log step a single click. An exemplar on a histogram bucket makes the metric to trace step a single click. A service and instance label shared by metrics, logs and profiles makes the comparison possible at all.

None of that happens by itself. It requires that the logging library reads the active trace context, that the metric library supports exemplars and that the collector is configured to keep them, and that resource attributes are set consistently across all four pipelines. A team that invests one week in making the identifiers consistent gets more operational value than one that spends a quarter adding a fifth tool.

What to do when a signal is missing

In a real incident you will often find the signal you want does not exist. The useful move is to name the substitute explicitly and record its weakness. No traces for the payment path, so use the gateway's upstream latency histogram as a proxy, remembering it includes queueing in the proxy. No profile, so use the runtime metrics for garbage collection pause time and allocation rate, remembering they describe the process as a whole rather than the slow request.

Then write the gap down. The output of an incident should include the list of questions that took longer than they should have because the signal was not there, and that list is how an observability roadmap gets written by evidence rather than by vendor demonstration.

What to instrument first in a new service

The order that gives the most value per hour of work is fixed, and it is worth having as a checklist. First, the three request metrics: rate, error ratio and a duration histogram with buckets placed around the objective. That alone supports an alert, a dashboard and a service level objective. Second, context propagation, before any custom spans, because propagation is what makes every later span useful and is the part that is painful to retrofit. Third, structured logging with the trace id included, which costs almost nothing once propagation works. Fourth, the dependency metrics: for each downstream call, its rate, errors and duration, which is what lets you attribute your own latency. Fifth, continuous profiling, which is usually a one line agent addition. Custom business metrics come after all of those, however interesting they are, because they answer product questions rather than operational ones.

Run a coverage review before you need one

A signal coverage review takes an afternoon. Take the five questions most likely to arise during an outage of this service, and for each one, try to answer it right now against yesterday's data. Not in theory, actually run the query. The questions that cannot be answered, or that take more than a few minutes, are the gaps.

This exercise consistently finds the same three things. A service on the critical path with no tracing, so the decomposition stops there. A metric that exists but cannot be split the way the question requires, because the label was dropped at ingest. And a dashboard whose panels compute something subtly meaningless, usually an averaged percentile, which nobody noticed because it has never been checked against a known incident.

Change events are the signal people forget

Most incidents are caused by a change, and the record of changes is rarely treated as observability data. Deployments, feature flag flips, configuration pushes, infrastructure applies, certificate rotations and schema migrations all belong on the same timeline as the metrics, because the single highest value correlation in practice is an inflection that lines up with a change.

Making this work is unglamorous plumbing: every system that changes production emits an event with a timestamp, an actor, a target and a short description, into one store that dashboards can overlay as annotations. Teams that have this answer what changed in seconds. Teams that do not spend the first twenty minutes of every incident asking in chat.

The same record pays for itself outside incidents. Change frequency and change failure rate, two of the four delivery metrics this course returns to in the delivery section, are both computed directly from it.

The signals are a system, not a shopping list

The four signals interlock. Metrics tell you to look, traces tell you where, profiles tell you what the code was doing, and logs tell you the specific fact. A team with excellent metrics and no traces spends its incidents guessing which service to blame. A team with traces and no metrics does not know an incident has started. A team with everything and inconsistent identifiers has four tools and no workflow.

The goal is not coverage of every signal for every service. It is that for each question you expect to ask, some signal can answer it, and the path from one signal to the next is one click rather than one hour.