A single Prometheus server is remarkably capable. On a machine with sixty four gigabytes of memory it will comfortably hold several million active series, scrape tens of thousands of targets and answer dashboards for a few hundred engineers. Teams reach for a distributed system long before they reach that ceiling, usually for reasons that have nothing to do with throughput, and the first thing worth doing is naming which reason applies, because the four reasons have four different answers.

The four reasons people outgrow one server

The first reason is retention. Local storage is sized by the product of active series, sample interval and retention period, and the relationship is linear in all three. Two million series at a fifteen second interval produce about one hundred and fifteen thousand samples a second, which at the compacted average of 1.3 bytes a sample is roughly thirteen gigabytes a day. Fifteen days of that is two hundred gigabytes, comfortable on a local volume. Thirteen months of it is five terabytes, which is a different conversation about a different kind of disk.

The second reason is the global view. A server scrapes what it can reach, and in a multi region or multi cluster estate that means one server per cluster and therefore one dashboard per cluster. Nobody wants nine tabs to answer a question about the whole fleet, and no alert can express "error rate across all regions" if no single server can see all regions.

The third reason is availability. A single server is a single point of failure for its own monitoring, and the standard answer is to run two identical servers scraping the same targets. That gives redundancy and immediately creates a new problem: two servers that scraped at slightly different moments hold slightly different numbers, so a query that reads both sees two overlapping series and a graph that zigzags between them.

The fourth reason is tenancy. Once more than one team shares a server, one team's cardinality mistake is everybody's outage, one team's expensive query is everybody's latency, and there is no mechanism in a single Prometheus to put a limit on either per team.

Vertical first, and how far it goes

Before any of this, the cheap move is to make the one server smaller rather than the cluster bigger. Cardinality work is the highest leverage activity available to a monitoring team: dropping a single high cardinality label at scrape time can halve a server's memory, and the series it removes were nearly always never queried. A server at two million series after a careful relabelling pass is a healthier system than one at eight million behind a distributed query layer, because every component downstream is sized from that number.

The practical ceiling of a single server is set by memory rather than by CPU or disk. Each active series in the head costs memory for its label set, its index entries and its open chunk, and the commonly used planning figure is around eight kilobytes per series when everything is counted. At that rate five million series is forty gigabytes before any query runs, and queries need headroom because a range query materialises its intermediate results. Churn makes this worse: a cluster that replaces its pods every few hours holds several generations of series at once, so the head can be two or three times the count of series currently being scraped.

What an HA pair really gives you

Two servers with identical configuration scraping identical targets give you survival of one machine failure and nothing else. They do not give a global view, they do not extend retention, and they actively make querying harder. The two servers are not replicas in any database sense. There is no log shipping and no consensus. Each one independently scraped the same endpoint a few hundred milliseconds apart and independently stored what it got, so their series are genuinely different data that happens to be nearly the same.

This is why every long term storage system has an opinion about deduplication, and why every one of them requires that you label the two servers differently so that it can tell them apart and then collapse them. The usual convention is an external label such as replica: a and replica: b, which is attached to every series as it leaves the server. Getting this wrong in either direction is a classic mistake: forget the label and the two servers look like one server with duplicate data, which corrupts aggregations; keep the label in the output of the query layer and every panel shows two of every line.

Where the data physically lives

It is worth being concrete about the layers, because the cost conversation later depends on it. The head is memory, measured in gigabytes and holding two hours. The local blocks are a fast local volume, measured in hundreds of gigabytes and holding days to weeks. Object storage is measured in terabytes and holds months to years, at roughly a fiftieth of the cost per gigabyte of block storage and with latency measured in tens of milliseconds rather than microseconds.

That last ratio is the whole economic argument for long term storage on object storage, and the latency ratio is the whole engineering argument for why querying it needs caches, index headers and a separate component to do it. A query that would read two hundred megabytes from a local SSD in forty milliseconds reads the same two hundred megabytes from object storage in several seconds across dozens of requests, which is survivable for a monthly report and unacceptable for an on call dashboard.

The decision in one paragraph

If the problem is retention only, and there is one cluster, remote write to a hosted or self run long term store and keep the local server small. If the problem is the global view and retention is fine, the lightest answer is a query layer in front of the existing servers rather than a new storage system. If the problem is availability, run a pair and solve deduplication. If the problem is tenancy, the answer is a system with per tenant limits built in, which is the strongest argument for the Cortex and Mimir line of designs over the Thanos line. Most estates eventually want three of the four, which is why the mature architectures all look similar in the end.

A worked sizing, before any component is chosen

Take an estate of four clusters, each running an HA pair, each pair holding 1.8 million active series at a fifteen second interval. The scrape rate is 120,000 samples a second per server, so 960,000 a second across the eight servers. Deduplicated down to one copy per cluster that is 480,000 a second reaching storage.

At the compacted figure of 1.3 bytes a sample, 480,000 samples a second is 54 gigabytes a day, or about 20 terabytes a year of raw resolution. Downsampled to five minute resolution the same year is 1.1 terabytes, and at one hour resolution it is 90 gigabytes. Those three numbers are the entire retention policy conversation, and they differ by more than two orders of magnitude.

The query side has its own number. A global dashboard panel that sums across four clusters fans out to four store components, and each of those must find the relevant blocks, read their index headers and then read chunks. The latency of the panel is the latency of the slowest of the four, not the average, which is why a single slow region degrades every global panel and why partial response handling is a real feature rather than a nicety.

Write these four numbers down before choosing a system: active series after cardinality work, samples a second after deduplication, retention at each resolution, and the fan out width of the widest query. Every component in the rest of this lesson is sized from one of them.