Cardinality is the single concept that separates engineers who can run a metrics system from engineers who keep being surprised by it. It is not complicated. It is a multiplication, and the trouble comes entirely from how fast multiplication grows when one of the factors is a user supplied string.

A metric is not one thing, it is a family

When you write a line of code that increments a counter named http_requests_total, you have not created one thing to store. You have created a family of time series, one for every distinct combination of label values that line of code will ever produce.

A time series is identified by its metric name plus the full set of its labels. Change one label value and it is a different series, with its own identifier in the index, its own chunk of samples on disk and its own slot in memory. The metric name is almost irrelevant to the storage system. The labels are what it is actually storing.

So the question to ask of any instrument is never "how much does this metric cost" but "how many series will this metric produce", and that number is the product of the number of distinct values each label can take.

The multiplication, done out loud

Take a counter on an HTTP server with four labels. The method label takes five values in practice, because almost nobody serves more than GET, POST, PUT, PATCH and DELETE. The route label takes forty values, one per registered route pattern. The status label takes six values if you group by class and about fifteen if you do not. The region label takes three values.

Five times forty times six times three is three thousand six hundred series. That is a comfortable number. A Prometheus server holding a few million series will not notice it, queries over it return instantly, and the storage cost is in the low megabytes a day.

Now add one more label: customer id. The service has twelve thousand customers. Three thousand six hundred times twelve thousand is forty three million two hundred thousand series, from one line of code, in one service. Nothing else about the code changed. The developer added one word to a map literal.

Histograms multiply again

A counter produces one series per label combination. A histogram produces many more, and this is the detail that turns a merely expensive metric into an outage.

A histogram with twelve explicit bucket boundaries stores thirteen bucket series, because the implicit +Inf bucket is always there, plus a _sum series and a _count series. That is fifteen series per label combination. The same four label set that gave a counter three thousand six hundred series gives a histogram fifty four thousand.

Apply the customer id label to the histogram instead of the counter and you have six hundred and forty eight million series. At that point the question is no longer how to query it, it is whether the process that tried to create it survived.

The rule that follows is simple and worth internalising: be generous with labels on counters, careful with labels on histograms, and treat every bucket boundary you add as a decision that is multiplied by your entire label space.

Active series, not total series

There is a second number that matters and it is easy to confuse with the first. Active series are the series that received a sample recently, typically within the last five minutes. Total series are every series the storage system has ever seen within its retention window.

A time series database holds its recent data in memory. In Prometheus this is the head block, and it holds roughly the last two hours of samples for every active series along with the index that maps labels to series. Memory is sized by active series. Disk and query cost are sized by total series across the retention window.

These two numbers diverge when series stop being written and new ones take their place, which is the subject of the churn section later in this lesson. For now it is enough to know that a system can have a perfectly reasonable two million active series and still be unqueryable, because the index contains two hundred million series that existed at some point in the last fifteen days.

Why the system does not just refuse

Most metrics systems will accept the forty three million series without complaint until something breaks. There is usually no error at the point of the mistake, which is what makes cardinality a recurring problem rather than a lesson learned once.

The application side is cheap. Recording a sample against a new label combination is a map lookup and an atomic increment, measured in tens of nanoseconds. The application allocates a small object per series and that is the whole cost it pays. A service can happily emit two million series while using a hundred megabytes of extra heap, and the developer sees no symptom at all.

The cost lands on the storage system, which is shared, which belongs to another team, and which fails in the middle of the night rather than in code review. This structural separation between the person who adds the label and the person who absorbs the cost is the real reason cardinality incidents keep happening, and it is why the controls described at the end of this lesson have to be mechanical rather than cultural.

Counting series without guessing

Everything above is arithmetic you can do on paper, and you should, but the system will also tell you the truth directly, which is better. Three queries cover almost every situation.

To find which metrics dominate, count series grouped by metric name. In Prometheus that is topk(20, count by (__name__)({__name__=~".+"})), which is expensive enough that it should be run deliberately rather than put on a dashboard, and which almost always reveals that two or three metrics account for more than half the total.

To find which service is responsible, count grouped by job instead. To find which label inside a suspect metric is the problem, count the distinct values of each label on that metric, which count(count by (label_name)(metric_name)) gives one label at a time.

There is also a cheaper and often overlooked source: the target's own /metrics endpoint. Fetching it and counting lines tells you exactly what one instance is producing, with no load on the storage system at all, which matters when the storage system is the thing that is sick. A service exporting forty thousand lines from one process is a finding you can act on before anything is ingested.

The number to put in the design document

When a design is being written rather than debugged, the useful habit is to write the expected series count next to each proposed instrument, the way you would write an expected queries per second next to an endpoint. It takes a minute, it is usually wrong by less than a factor of two, and it moves the conversation from opinions about whether a label is reasonable to a number that can be compared against a budget.

The one sentence version

A label is not a field on a record. It is a dimension of a cross product, and every value it can take multiplies the storage, the memory and the query cost of every other label on that metric. Ask what the cross product is before you add the label, not after the alert fires.