Tracing is the only signal whose volume is proportional to both your traffic and your architecture. Metrics scale with the number of series you chose to create. Logs scale with how talkative your code is. Traces scale with requests multiplied by the number of services each request touches, and neither of those factors is under the control of the person deciding how much to store.

That is why sampling is not an optimisation bolted onto tracing. It is part of the design, and the only real question is whether you make the decision deliberately or inherit whatever default your library shipped with.

The arithmetic that forces the decision

Take a fleet serving twenty thousand requests per second at peak. A request that enters at the gateway touches the gateway itself, an authentication service, the product service, two databases, a cache and a message producer. That is eight spans, and eight is a modest number: a service mesh adds a span per hop, an ORM adds one per statement, and a retry adds a whole subtree.

Twenty thousand requests per second times eight spans is one hundred and sixty thousand spans per second. At five hundred bytes per span on the wire, which is what a span with conventional attributes and no stack traces actually costs, that is eighty megabytes per second, or about six point nine terabytes per day.

Keep that for seven days and you are storing forty eight terabytes. The bill for storing it is not the painful part. The painful part is the ingest path: eighty megabytes per second sustained, with peaks of three times that during an incident, through a collector tier, a queue and an indexing layer that has to make every one of those spans findable by trace identifier, service, duration and a dozen attributes.

Nobody does this. Everybody samples. The disagreement is only about how.

What you are actually buying

It helps to be precise about what a stored trace is for, because the answer changes the policy completely.

A trace is read by a human during an investigation. That is the dominant use, and it has a striking property: the reader does not need many of them. Given a well chosen example of the failing behaviour, one trace answers the question. Ten identical traces of the same slow path add nothing. The value of the hundredth copy of a healthy checkout trace is approximately zero.

A trace is also read by a machine, to compute aggregates: service dependency maps, per route latency distributions, error rates by downstream. These uses do need volume, because they are statistical, but they are also the uses that metrics serve better and more cheaply.

So the honest framing is this. You are not buying coverage of your traffic. You are buying the probability that, when something interesting happens, at least one example of it is in the store. That is a very different objective from keeping a representative sample, and it is why uniform random sampling is such a poor fit for the thing people actually do with traces.

Four ways to spend less, only one of which is sampling

Before choosing a sampling policy, it is worth knowing which lever you are pulling, because teams routinely reach for sampling when a different lever was cheaper and lossless.

The first lever is span count per request. A service that emits a span for every ORM statement produces forty spans for a request that made one logical database call. Turning off statement level spans and keeping one span per logical operation can halve your volume without discarding a single trace.

The second lever is bytes per span. A span carrying the full SQL text, a stack trace and a copy of the request body is three to six kilobytes. The same span with a statement digest and conventional attributes is five hundred bytes. This is pure compression of the same information.

The third lever is retention. Traces are overwhelmingly read within hours of being produced. Dropping trace retention from thirty days to seven cuts storage by three quarters and costs almost nothing in practice, provided your metrics retention is long.

The fourth lever is sampling, which is the only one of the four that actually loses information. Pull it last, and pull it knowing what you are giving up.

Sampling is a question about questions

The useful way to choose a policy is to write down the questions you expect to ask of traces, and then ask what sampling would have to preserve for each one to be answerable.

"A customer reports that order 88213 failed, show me what happened" requires that the specific trace for that request exists. No probabilistic policy can promise this. Only a policy that keeps all errors, or all traces for a flagged customer, can.

"Why did p99 latency for checkout rise at 14:20" requires at least one slow trace from that window, and ideally several. A policy that keeps a fixed fraction of slow requests answers it; a one percent uniform policy answers it only if the slow population was large.

"Which downstream service is responsible for most of our 500s" requires enough errors to compare, which a keep all errors policy gives you trivially and a uniform policy gives you only at high volume.

"What does a normal request look like" requires a handful of healthy traces, which any policy provides.

Four questions, four very different requirements, and the policy that serves all of them is not a single percentage. It is a set of rules with a baseline underneath.

Where the decision is made, and why it matters

There are exactly two places a sampling decision can be made, and the difference between them is the entire subject of this lesson.

Head sampling decides at the start of the trace, in the first service, before anything has happened. The decision is then propagated to every other service so that the whole trace is kept or the whole trace is discarded. It is cheap, it is stateless, and it is blind: at the moment of deciding, nobody knows whether this request will fail or take four seconds.

Tail sampling decides at the end, after the trace is complete, in a component that has buffered all the spans. It can see the outcome, so it can keep exactly the interesting traces. It costs memory, it costs a component that has to be scaled and operated, and it requires all spans of a trace to reach the same decision maker.

Everything else in trace sampling is a variation on, or a hybrid of, these two. The rest of this lesson works through what each one costs, what each one silently breaks, and how to combine them without producing numbers that lie.

Sampling is not the first lever you should reach for

Before touching a sampling rate, it is worth asking whether the volume is real. In most systems a large fraction of span volume is waste that nobody would defend if they looked at it, and removing waste is lossless in a way that sampling never is.

Four patterns account for most of it. A database library instrumented per statement rather than per logical operation, producing forty spans where one would do. Retry loops that emit a span per attempt inside a parent that already describes the operation. Internal function calls wrapped in spans by an enthusiastic developer, crossing no process boundary and telling you nothing a profile would not. And attributes carrying full SQL text, serialised request bodies or stack traces on every span, which multiply bytes per span by five or ten without changing what questions you can answer.

Fixing those typically halves span volume, and it halves it without losing a single trace. Sampling, by contrast, always loses traces: the question is only which ones. So the order of operations is to remove the spans nobody wants, shrink the ones that are too large, and only then decide how many of the remaining traces to keep.