Almost every surprising thing Prometheus does follows from one decision made early and never reversed: the server fetches metrics from targets rather than receiving them. Everything else, the service discovery machinery, the staleness rules, the awkwardness around short lived jobs, is downstream of that choice.

What pulling actually means

A Prometheus server holds a list of targets, each one an HTTP endpoint. On a timer it issues a GET to each endpoint, parses the text it receives, attaches labels, and appends the resulting samples to its local database. The target is a passive participant: it renders its current counter and gauge values into a response body and forgets about it.

This has three consequences worth stating plainly, because they shape operational reality.

The server controls the load. Scrape volume is a function of the server's configuration, not of how enthusiastic any application is about emitting. A service under heavy load does not generate more metric traffic, because the response is the same size whatever the request rate. This is the single largest practical difference from a push based system, where a traffic spike produces a telemetry spike at exactly the wrong moment.

The target must be reachable. A pull based server needs network access to everything it monitors, which is straightforward inside a cluster and genuinely painful across a network boundary, behind a NAT, or for a job that exits after forty seconds.

Target health is free. If the scrape fails, the server knows immediately and records up as zero for that target. There is no ambiguity between a silent service and a broken pipeline, which in a push system is a real and recurring problem.

Service discovery is the interesting half

In a cluster where pods come and go every few minutes, a static list of targets is useless. Service discovery turns a dynamic environment into that list, and it runs continuously rather than at startup.

A discovery mechanism, whether Kubernetes, EC2, Consul or a file on disk, returns a set of targets each carrying a bag of metadata labels, conventionally prefixed with two underscores. Those labels are not yet part of your metrics. They are raw material for the relabelling stage, which decides three things: whether to scrape this target at all, what address to scrape, and which labels end up attached to every sample it produces.

This is where most Prometheus configuration mistakes live, and the reason is that relabelling is a small stack machine written in YAML, evaluated in order, and almost impossible to read. A rule that drops targets is spelled action: drop with a regex against a source label, and a rule that renames metadata into a real label is action: replace with a target label name. Get the order wrong and you either scrape everything in the cluster or nothing at all.

The identity of a target, and why it matters

Prometheus identifies a target by the full set of its labels after relabelling. Change any one of them and, as far as the server is concerned, the old target disappeared and a new one arrived.

This is the mechanism behind a class of confusing incidents. A deployment that adds a version label to every pod does not update existing series: it ends the old ones and starts new ones. Every rate calculation that spans the deployment sees a counter that vanished and another that appeared at zero, which is exactly the pattern that rate is designed to interpret as a reset. The graphs show a dip, the dip is not real, and the cause is three characters in a label.

The same applies to the instance label, which defaults to the host and port that was scraped. In an environment where pods get new addresses on every restart, instance churn is continuous and is the main source of the series churn discussed in the cardinality lesson.

The scrape budget nobody writes down

A scrape is not free, and the arithmetic is worth doing once.

Take a server with 900 targets at a 15 second interval. Each target exposes around 2,700 series, which gives 2.4 million active series and 160,000 samples a second. Each scrape is an HTTP request, a text parse of perhaps 300 kilobytes, label interning, and an append per sample.

The parse is the expensive part, and it is why the exposition format matters: the text format is human readable and costs real CPU, while the protobuf and OpenMetrics variants trade readability for speed. At this scale a single server spends a meaningful fraction of a core purely on turning text into samples.

The interval multiplies everything. Halving it to 7.5 seconds doubles samples per second, doubles disk write rate, doubles the chunk cut frequency, and does not double the information, because almost nothing in a production system changes meaningfully in seven seconds that did not also change in fifteen.

Timeouts, intervals and the rule that follows

The scrape timeout must be shorter than the scrape interval. If it is not, a slow target can have two scrapes in flight at once, and the samples arrive out of order, which the database will reject.

The default is a ten second timeout inside a fifteen second interval, and the margin exists because a target under load renders its page slowly. An endpoint that takes longer than the timeout produces a failed scrape, up goes to zero, and every series from that target goes stale. Importantly, this is all or nothing: there is no partial scrape. A target exposing fifty thousand series that times out at ninety nine percent complete contributes nothing at all.

That failure mode is the argument for keeping the number of series per target modest. A large page is slow to render, slow to transfer and slow to parse, and when it fails it fails completely, taking every metric from that process with it.

The metrics the scrape itself produces

Every scrape generates a handful of synthetic series that describe the scrape rather than the application, and they are the first place to look when something is wrong.

up is one or zero. scrape_duration_seconds is how long the request took, and watching its trend against the timeout is how you see a target getting slower before it starts failing. scrape_samples_scraped is how many samples the target exposed, which is the per target cardinality number you need when hunting a growth problem, and scrape_samples_post_metric_relabeling is the same count after your drop rules ran, so the difference between the two tells you exactly how much work your relabelling is doing.

There is also scrape_series_added, which counts series seen for the first time in this scrape. A target with a steady workload should report close to zero once it has warmed up. A target reporting hundreds on every scrape is creating new series continuously, which means a label somewhere contains something that changes, and this one series will find that faster than any dashboard.

These five are worth putting on a permanent panel. They cost nothing, they exist whether or not anybody configured them, and between them they diagnose slow targets, failing targets, cardinality growth and misconfigured relabelling without touching the application at all.