Lag is the most watched number in a Kafka deployment and the most frequently misread. There are three distinct quantities that all get called lag, they disagree, and the one most dashboards show is the least useful of them.

The three quantities

Committed lag is the log end offset minus the group's committed offset, which is what kafka-consumer-groups.sh --describe prints and what most exporters scrape. It includes the records that are consumed but not yet committed, so it is always an overestimate.

Position lag is the log end offset minus the consumer's position, exposed client-side as records-lag per partition in consumer-fetch-manager-metrics, with records-lag-max as the rollup. This is the one that measures actual progress, because the position advances with every poll rather than every commit.

Time lag is the difference between now and the timestamp of the record at the consumer's position, and it is the only one that answers the question anybody actually has, which is how stale the output is.

The three answer different questions. Committed lag answers "where would a restart resume from". Position lag answers "how many records behind is this consumer". Time lag answers "how old is the data this consumer is producing", and that is usually what the business cares about.

The lag floor, which makes committed lag unusable at the small end

Committed lag has a hard floor equal to the commit interval times the consumption rate, and that floor is often larger than the threshold people set alerts at.

A consumer handling 50,000 records per second and committing every five seconds sits permanently at a committed lag of around 250,000 records, perfectly healthy, nothing wrong. An alert at 100,000 fires continuously, gets muted, and then does not fire for the real incident.

So committed lag is a coarse instrument: useful at the scale of millions when a consumer is genuinely falling behind, meaningless below the floor. The floor should be computed once per group and written next to the alert threshold, and if the threshold is below the floor one of the two is wrong.

Position lag has no such floor, because the position advances on every poll rather than on every commit. A consumer keeping up reports a position lag in the hundreds, bounded by the fetch size rather than by the commit interval, and that is a number an alert can be built on.

Why time lag is worth the effort

Record counts are not comparable across topics, and that is the practical problem with every record-based lag number.

A lag of 100,000 on a topic receiving 1,000 records per second is a hundred seconds behind. The same 100,000 on a topic receiving 100,000 per second is one second behind. One is an incident and one is noise, and a single alert threshold across both topics cannot distinguish them.

Time lag normalises that. The computation is to read the record at the consumer's position and subtract its timestamp from now, and because a consumer already has that record in hand it is nearly free client-side: emit the age of the newest record processed as a gauge.

The caveat is which timestamp. message.timestamp.type is CreateTime by default, meaning the producer's clock, so time lag inherits every producer's clock skew; with LogAppendTime the broker stamps it and the measurement is clean but the producer's own event time is lost. For lag purposes LogAppendTime is better and for event-time processing it is worse, which is a real trade rather than a default to flip casually.

The idle-partition artefact

A specific confusion that costs an afternoon the first time it appears.

Time lag on a partition that has received nothing for an hour reads as an hour, because the newest record processed is an hour old. The consumer is perfectly caught up and the metric says it is an hour behind.

So time lag has to be reported as the age of the newest record available, not processed, or it has to be suppressed when the position equals the log end offset. The simplest correct form is to report time lag only when position lag is above zero, which is exactly when the question is meaningful.

The same artefact appears in reverse with record lag on a partition whose producer has stopped: lag stays at whatever it was and never drains, because there is nothing to drain. Both cases are the same underlying fact, which is that a lag metric describes a relationship between two moving things and says nothing when one of them has stopped.

Per-partition, not aggregate

Summing lag across a group's partitions discards the information that matters, and the shape of the failure is specific.

A group of 24 partitions where 23 are at zero and one is at 2 million has an aggregate lag of 2 million, which looks like a group-wide problem and is actually one partition. The investigation for the two is completely different: a group-wide problem is capacity, and one partition is a hot key, a poison record, or a single stuck member.

So the metric to alert on is the maximum per-partition lag, not the sum, and the dashboard should show the per-partition distribution rather than a single line. records-lag-max gives this directly per member, and the group-level view needs the per-partition breakdown from the admin API or an exporter that preserves it.

A useful second number is the count of partitions above a threshold, because one partition above it is a partition problem and all of them is a capacity problem, and that one distinction resolves most lag investigations in the first minute.

What the exporters actually measure

Three common ones, and knowing which quantity each reports prevents a long argument about which dashboard is right.

kafka_exporter and most Prometheus exporters poll ListOffsets and OffsetFetch and publish committed lag, inheriting its floor and its scrape-interval granularity. They cost one admin round trip per scrape per group, which for hundreds of groups at a short interval is real broker load.

Burrow takes a different approach deliberately: it watches the __consumer_offsets topic, keeps a sliding window of each group's committed offsets, and evaluates whether the group is making progress rather than comparing against a threshold. Its output is a status, not a number, and its value is that it needs no per-group threshold, which is the thing nobody maintains.

The consumer's own metrics, exported by whatever the application already uses, give position lag and are the only source that does. Every serious lag alert should be built on these, with an exporter's committed lag kept as the coarse cross-check that catches a consumer which has stopped emitting metrics entirely.

What to alert on, concretely

Four alerts cover it, and the thresholds come from the arithmetic in the next section rather than from a round number.

Time lag above the staleness the downstream can tolerate, which is the primary alert because it is expressed in the same units as the requirement. If the requirement is that dashboards are under a minute stale, the threshold is a minute.

Maximum per-partition position lag growing monotonically for longer than the drain time of a normal burst, which catches a consumer that is behind and not recovering, as distinct from one absorbing a spike.

A group with committed offsets and zero members, from the previous lesson, which no lag alert catches because nothing is advancing.

A discontinuous drop in lag, which is the offset-reset signature and means records were skipped rather than processed. That one is the least obvious and the most damaging, and it is the alert that nobody has until it has happened once.

The cost of measuring it

Lag measurement is not free, and at a few hundred groups it becomes a visible load on the cluster, which is a surprise the first time it appears in a broker's request metrics.

Every exporter scrape is an OffsetFetch to each group's coordinator plus a ListOffsets to the leader of every partition the group consumes. For 300 groups across 4,000 partitions at a 15-second scrape interval, that is 20 OffsetFetch and several hundred ListOffsets per second, all of which land on the brokers alongside the real traffic.

The pathological case is a single exporter scraping every group from one process, which concentrates that load and makes the exporter itself a single point of failure for all lag visibility. Sharding by group across several exporters fixes both.

The cheaper alternative is to let each consumer export its own position lag, which costs nothing because the consumer already has the numbers, and to keep one slow broker-side scrape, every minute or two rather than every fifteen seconds, purely as the cross-check for a consumer that has stopped reporting.

What lag does not tell you

Three things it is routinely used to decide and cannot, which is worth stating because each has caused an incident.

Whether records are being processed correctly. A consumer that reads every record, throws away the ones it cannot parse, and commits has a lag of zero and is losing data. Lag measures position, not effect, and a healthy lag graph is entirely compatible with a broken pipeline.

Whether the data is complete. A producer that stopped publishing shows as zero lag, because there is nothing to be behind. The alert for a stopped producer is on the topic's messages-in-rate, not on any consumer's lag, and that gap is the single most common blind spot in a streaming deployment's monitoring.

Whether a consumer skipped anything. A reset to latest drops lag to zero instantly, which looks exactly like a successful recovery. Only the discontinuity in the lag series distinguishes the two, which is why that alert is worth having before it is needed rather than after.