A trace is a record of one request's journey through a system, assembled after the fact from pieces that were emitted independently by every service it touched. That assembly is the whole trick, and understanding how it works explains almost every surprising thing tracing does, including why a trace is sometimes incomplete, why timings sometimes look impossible, and why the thing people most want from tracing is the thing it is worst at.

The unit is a span. A span records one operation: a name, a start time, a duration, a status, a set of key value attributes, and two identifiers. The first identifier belongs to the span itself. The second is the identifier of its parent, the operation that caused it. A trace is simply every span sharing a trace identifier, and the tree structure is reconstructed by following parent references.

Nothing holds the trace while it is happening

This is the part people find counterintuitive. There is no object anywhere representing the trace in progress. Each service holds only the current span and enough context to name a parent, and every span is shipped independently as it finishes. The backend receives thousands of unrelated spans a second and groups them by trace identifier.

Several consequences follow immediately. A trace is complete only when every span has arrived, so opening a trace a second after a request finished often shows a partial tree. A service that crashes mid request emits no span for the operation that killed it, so the gap in the waterfall is the evidence rather than an error entry. And because each service timestamps its own spans with its own clock, a child span can appear to start before its parent on a machine whose clock has drifted, which is a measurement artefact and not a causality violation.

The three identifiers worth knowing by name

The trace identifier is sixteen bytes, generated at the first service to handle the request, and carried unchanged everywhere afterwards. The span identifier is eight bytes, new for every operation. The parent span identifier is whatever span identifier arrived in the incoming context, which is how a child in one process links to a parent in another.

There is a fourth field that matters more than its size suggests: the sampling flag. It travels with the context, it is decided once near the start, and it tells every downstream service whether this trace is being recorded. Without it, services would make independent sampling decisions and you would collect fragments of traces rather than traces, which is the single most common way a tracing deployment produces data nobody can use.

Span kind, and why it changes the picture

Spans carry a kind: server, client, producer, consumer, or internal. The distinction matters because a single network call produces two spans, one on each side, and their durations differ by exactly the thing you usually want to measure. The client span includes connection setup, serialisation, the network, and the time the request waited in the server's accept queue. The server span includes only what the server did after picking the request up.

When a client span shows two hundred milliseconds and the matching server span shows eleven, the two hundred is not a lie and the eleven is not a lie. The difference is queueing, network, or connection pool exhaustion, and reading those two numbers together is one of the fastest diagnoses available in any observability tool. A tracing setup that instruments only the server side throws that away.

Events and links, for the cases the tree cannot express

Two more parts of the model exist because the parent and child tree is not quite enough. A span event is a timestamped annotation inside a span, used for moments that have no duration of their own, such as the instant a retry began or a cache was found empty. Events cost far less than spans and they keep the tree readable.

A link is a reference from one span to another trace, which exists because some relationships are not parent and child. The clearest case is a batch: a consumer processing forty messages cannot have forty parents, so it links to the forty producing traces instead. The same applies to a fan in, where several requests are coalesced into one downstream call. Tools display links less prominently than parents, which is a reason to prefer a real parent relationship when one honestly exists, and to use links only where the tree genuinely cannot express the truth.

The other half of the division is that a trace is a sample of one. Any conclusion of the form this operation is slow, drawn from a single trace, is a conclusion drawn from one observation, and the request you are looking at was very likely retained precisely because it was unusual. The honest reading is that a trace tells you what happened in this request, and that metrics tell you how often that is representative. Reading traces without checking where they sit in the distribution is how teams spend a sprint optimising the tail of an endpoint whose median was always fine.

What it costs the request to be traced

The overhead of tracing lands in three places, and it is worth knowing the rough size of each because the fear of it is usually larger than the reality.

The first is span creation inside the process. Allocating a span, generating an identifier, recording two timestamps and attaching a handful of attributes is on the order of a microsecond, which against a request taking tens of milliseconds is invisible. This is why instrumenting process boundaries and external calls costs nothing anybody can measure, and why instrumenting a loop body costs something everybody can.

The second is export. Spans are batched, serialised and sent on a background thread, so the request itself does not wait for them. The cost shows up as CPU and as memory held by the batch queue, and the failure mode worth knowing is that a queue which fills because the backend is slow will start dropping spans, which produces incomplete traces for reasons that have nothing to do with propagation. A metric counting dropped spans per process is one of the few pieces of meta telemetry genuinely worth having.

The third is the bytes on the wire for context propagation, which for two headers of under a hundred bytes is negligible until somebody starts putting large payloads into baggage.

What all of this means practically is that the argument against tracing is almost never a latency argument. It is a storage and index argument, which is a decision about sampling and attributes rather than about whether to instrument. Teams that refuse instrumentation on performance grounds are usually defending against a cost that does not exist, while the cost that does exist sits in the backend bill.

What this model is good at, and what it is not

Tracing answers where time went in one request, with precision no other signal offers. It also answers which services participate in a request path, which is the only reliable way to keep an architecture diagram honest in a system that changes weekly.

It is much weaker at questions about populations. Asking what the ninety ninth percentile latency of a service is, from traces, means asking a question of a sampled subset, and the answer is only as good as the sampling. That question belongs to metrics, where every request is counted. The healthy division of labour is that metrics tell you something is wrong and how often, and traces tell you where in one example it went wrong. A team that tries to compute reliable rates from traces will get numbers that drift with the sampling configuration and will not understand why.