A committed offset is not a piece of consumer state that the broker happens to remember. It is a record in a Kafka topic, written and read through the ordinary log machinery, and almost every surprising behaviour in this lesson follows from that.

The committed offset is a record in a topic

__consumer_offsets is an internal topic with 50 partitions by default, set by offsets.topic.num.partitions, and cleanup.policy=compact. An OffsetCommit request appends a record to it; an OffsetFetch request reads the group's current value from the coordinator's in-memory cache of it.

The key is the triple of group id, topic and partition. The value carries the offset, the leader epoch of the partition at commit time, an optional metadata string the application may attach, and the commit timestamp.

Keying on that triple is what makes compaction do the right thing: every commit for the same partition by the same group has the same key, so compaction keeps the latest and discards the rest. A group committing every five seconds for a year leaves one record per partition, not six million.

The leader epoch in the value was added for a specific correctness reason: on fetching the committed offset, a consumer can detect that the offset belongs to a partition lineage that was truncated away by an unclean leader election, rather than silently seeking into a log that no longer contains what it was promised.

What "committed" actually means numerically

The committed offset is the offset of the next record to consume, not the offset of the last record processed. If a consumer has finished record 99, it commits 100.

Off-by-one here is the single most common bug in hand-written commit code, and its symptom is one duplicate record per partition per rebalance, which is small enough to go unnoticed for years and large enough to matter for anything billing or counting.

The client's own APIs mostly hide it. commitSync() with no arguments commits the position, which is already defined as the next offset poll would return, so it is correct by construction. The explicit form takes an OffsetAndMetadata and the application has to do the arithmetic itself.

There is also a semantic distinction worth keeping straight. The position is where the consumer will read next, maintained locally and advanced by poll. The committed offset is what is durable in __consumer_offsets. They differ by exactly the records consumed since the last commit, and that difference is the duplication window.

Which broker holds a group's offsets

The same derivation as the group coordinator, because it is the same thing. The partition of __consumer_offsets is abs(group.id.hashCode) % offsets.topic.num.partitions, and the broker leading that partition is the group's coordinator.

So offsets and coordination are co-located by construction, which is why an OffsetCommit goes to the coordinator rather than to the leader of the partition being committed. A consumer reading from twelve brokers commits to exactly one.

Two consequences. The commit path has a single broker on it per group, so commit latency is a property of that one broker rather than of the cluster, and a group whose coordinator is on a struggling host has slow commits even though consumption is fine.

And offsets.topic.replication.factor, default 3, governs the durability of every group's position. A cluster where that topic was created at replication factor 1, which happens when the first broker came up alone, loses every group's position when that broker loses its disk. The topic is created once and does not pick up the setting later, so this has to be checked rather than assumed.

The broker-side settings that matter

Four, and the defaults are right for almost everyone, which is why knowing them matters mainly for diagnosis.

offsets.topic.num.partitions, 50, fixed at creation. Changing it later redistributes every group to a different coordinator, which is why it is effectively immutable.

offsets.commit.timeout.ms, 5,000, which bounds how long the coordinator waits for the commit record to be replicated before failing the request. A commit is acknowledged only when it is durable on the required replicas, so a commit is a real write with real latency, typically a few milliseconds and occasionally much worse.

offsets.load.buffer.size, about 5 MB, which governs reading the offsets partition into memory when a coordinator takes over. A coordinator failover has to replay its partition before it can answer, and for a cluster with thousands of groups that load is measurable, which is part of why rolling broker restarts produce a visible pause for consumers.

offsets.retention.minutes, 10,080, which is seven days and is the one that causes incidents. It gets its own treatment below.

The retention trap

Committed offsets expire. Specifically, when a group becomes empty, meaning it has no members, a timer starts, and after offsets.retention.minutes the group's offset records are tombstoned and compacted away.

The behaviour on restart after that is the damaging part. The consumer finds no committed offset, falls back to auto.offset.reset, and with the default of latest it begins at the end of the log, silently skipping everything produced while it was down. No error, no warning, and lag reports as zero because the group is at the end.

This is the mechanism behind a specific and recurring incident: a service is turned off for a quarter, turned back on, and quietly loses a quarter of data with every dashboard green.

The clock behaviour changed for the better in 2.1. Before that, expiry was measured from the last commit, so a group that was running but idle, committing nothing because it had nothing to consume, could expire while alive. Since then it is measured from the group becoming empty, which fixed that case but not the long-shutdown case.

The defences are to set auto.offset.reset=none for anything that must not silently skip, which turns the condition into an exception at startup, and to treat a group with recent offsets and zero members as an alert rather than as nothing.

Reading the topic directly

Occasionally necessary, and the only way to see what is actually stored rather than what the tooling reports.

1# The offsets topic is readable like any other, with a formatter that
2# decodes the binary key and value into something legible.
3#
4#   kafka-console-consumer.sh \
5#     --bootstrap-server broker:9092 \
6#     --topic __consumer_offsets \
7#     --from-beginning \
8#     --formatter 'kafka.coordinator.group.GroupMetadataManager$OffsetsMessageFormatter'
9#
10# Output interleaves two record kinds sharing the topic:
11#   [group,topic,partition]::OffsetAndMetadata(offset=4821, leaderEpoch=7, ...)
12#   [group]::GroupMetadata(generation=19, protocol=cooperative-sticky, members=...)

Two record kinds share the topic, which is worth expecting: offset commits keyed on the triple, and group metadata keyed on the group id alone, holding the generation, the protocol in use and the member list.

That second kind is how a coordinator reconstructs a group after failover, and reading it is the most direct way to confirm which assignor protocol a group is actually running, as opposed to which one its configuration files name.

The practical uses are narrow but real: confirming a group's committed offset when the tooling disagrees, confirming the protocol after a cooperative migration, and confirming that a group's offsets have genuinely been tombstoned rather than merely not reported.

What the consumer group tool shows, and from where

kafka-consumer-groups.sh --describe --group <g> prints, per partition, the current offset, the log end offset and the lag between them.

The current offset comes from OffsetFetch against the coordinator, so it is the committed offset, not the consumer's position. A consumer that is processing happily but committing rarely looks like it is behind, because the tool can only see what was committed.

The log end offset comes from a ListOffsets request to each partition leader, so lag is computed across two sources at two instants and is approximate by construction. For a fast topic the two can differ by thousands of records with nothing wrong.

And for a group whose members have their own offset store, the tool reports against a committed offset nobody maintains, which is either stale or absent. Monitoring built on it stops meaning anything in that case, which is a cost of the external-offset pattern worth paying deliberately rather than discovering.

Deleting a group, and what that writes

kafka-consumer-groups.sh --delete --group <g> removes a group, and like a reset it requires the group to have no active members, for the same reason: a live member would simply recreate it on its next heartbeat.

What the deletion does is write tombstones, a null-valued record for each of the group's keys, which compaction then removes. So deletion is a write to a log rather than a mutation of a table, which is the recurring theme of this section and is why it is not instantaneous: the records vanish from the coordinator's view immediately and from the log at the next compaction.

There is also --delete-offsets --group <g> --topic <t>, which removes a group's committed offsets for one topic while leaving the group and its other subscriptions intact. The use for it is narrow and real: a group that stopped consuming a topic still holds offsets for it, which show up in --describe as partitions with growing lag that nobody is responsible for, and that noise is what makes a genuine lag alert get ignored.

The reason both operations matter here is that a deleted group is indistinguishable from a group that never existed, so a restarted consumer applies auto.offset.reset. Deleting a group to clean up a dashboard is therefore the same action as the retention incident above, performed deliberately.