Instrumentation is the part of observability that lives in your code, and it is the part that decides what every tool downstream can possibly show. A dashboard cannot display a dimension the process never emitted, and no amount of query sophistication recovers a span that was never created. This is why instrumentation is worth understanding as a design problem rather than as a library you install.

Why there is a standard at all

Before OpenTelemetry, instrumenting a service meant choosing a vendor's agent and writing against that vendor's client library. Changing backend meant touching every service, and running two backends during a migration meant double instrumentation. Worse, a library author who wanted to emit spans had to pick one vendor on behalf of every user of the library, which is why most libraries emitted nothing at all.

OpenTelemetry exists to break that coupling. It separates an API, which your code and your libraries call, from an SDK, which decides what actually happens to the data. The API is deliberately almost inert: if no SDK is registered, every call is a no operation with negligible cost. That property is what makes it safe for a library to depend on the API, and library instrumentation is the thing that was missing from the ecosystem for a decade.

The split that matters: API, SDK, collector

There are three layers and they fail in different ways, so it pays to keep them straight.

The API is the set of calls in your code: start a span, add an attribute, record a measurement, emit a log record. It defines types and semantics and nothing else. The SDK is the implementation registered at process start: it holds the sampler, the span processors, the metric readers and views, and the exporters. The collector is a separate process, not part of your service, which receives telemetry, transforms it and forwards it to one or more backends.

When a trace does not appear, the question is which of the three stages dropped it. The API call may not have run because the code path was not taken. The SDK may have sampled it away, or buffered it and lost it at shutdown. The collector may have rejected it, queued it, or exported it to a backend that returned an error nobody is watching. Each has a different diagnostic, and all three publish their own self telemetry, which is the first thing to read when data goes missing.

Three pipelines, one configuration

Traces, metrics and logs are separate pipelines inside the SDK, and treating them as one thing causes confusion. Each has its own exporter, its own batching behaviour and its own failure mode.

The trace pipeline is push based and batched. Spans are finished, handed to a span processor, buffered, and exported in batches of up to 512 by default every five seconds. If the process exits without flushing, the last batch is lost, which is why a crash loop produces traces that stop short of the interesting part.

The metric pipeline is periodic. A reader collects from the instruments on a fixed interval, typically 60 seconds, applies the configured aggregation and exports. Metrics are therefore coarse in time by design, and a spike shorter than the interval is averaged into the window rather than visible as a spike.

The log pipeline is the youngest of the three and in most deployments is still the existing logging library writing structured lines to standard output, with the OpenTelemetry part limited to injecting the trace id. That is a perfectly reasonable configuration, and this course treats it as the default rather than as a compromise.

What the SDK actually decides

It is worth being concrete about what lives in the SDK, because these are the knobs that get changed during an incident and the ones that cause the strangest behaviour when they are wrong.

The sampler decides, at span creation, whether the span will be recorded. The default in most distributions is parent based with a probabilistic root, which means a service respects the decision in the incoming context and makes its own decision only when it is the first service in the path.

The span processor decides what happens to a finished span. The simple processor exports one span at a time synchronously, which is correct for a short lived function and ruinous for a server. The batch processor queues spans and exports in batches, with a queue size, a batch size and a timeout, and it is the one you want everywhere else.

Views, on the metric side, decide what an instrument becomes: which aggregation it uses, which attributes are kept, and what the histogram boundaries are. This is the mechanism that lets you drop a label without changing the code that records it, which is the fastest cure for a cardinality accident.

Exporters decide the protocol and the destination. The protocol in practice is OTLP, over gRPC or HTTP, and the destination is almost always a local collector rather than a vendor endpoint, for reasons the collector section covers.

Resource attributes are the join key

Every piece of telemetry a process emits carries a resource: a set of attributes describing the producer rather than the event. Service name, service version, deployment environment, host name, Kubernetes pod and namespace, cloud region.

These are the attributes that let you compare a metric, a trace and a profile from the same process, so consistency matters more than completeness. A service that calls itself checkout in its metrics and checkout-service in its traces has broken every cross signal query, and the failure is silent: both pipelines work, the join just returns nothing.

The practical rule is to set the resource once, from the environment, in one shared initialisation function, and to treat the service name as a value from the deployment rather than a string in the source. The semantic conventions name these attributes precisely, and following the convention is what makes backend dashboards work without configuration.

Instrumentation is a contract with your future self

The reason to take instrumentation seriously as design is that it is the only part of observability you cannot change retroactively. A dashboard can be rebuilt in an afternoon and a query can be rewritten during the incident itself, but a label that was never emitted is a question that cannot be asked about last Tuesday, no matter what you are willing to spend.

That asymmetry suggests a specific posture. Be generous about what you record where the cost is flat in traffic, which means metrics and their bounded labels, because the cost of a series is small and the cost of not having it is a blind spot. Be careful where the cost is linear in traffic, which means logs and spans, because every decision there multiplies by request volume. And be explicit about the attributes you deliberately do not record, writing the reason down, so the next person does not rediscover the cardinality incident that led to the decision.

What to instrument, in order

Instrumentation work expands without limit if you let it, so the order matters more than the ambition. Request rate, error ratio and a duration histogram come first, because they support an alert, a dashboard and an objective by themselves. Context propagation comes second, before any custom spans, because propagation is what makes every later span useful and it is the piece that is painful to retrofit. Trace ids in log lines come third, which is nearly free once propagation works. Dependency metrics for each outbound call come fourth. Everything else, including the business metrics that are genuinely interesting, comes after those four.