A consumer group is a set of consumers that divide a subscription's partitions among themselves, and the division is negotiated rather than configured. Understanding where that negotiation happens, and what holds its state, explains most of what a group does under stress.

What a group actually is

A group is identified by a single string, group.id, and that string is the whole of its identity. There is no create operation, no registration and no schema: a consumer that sets group.id to a value nobody has used before creates a group by joining it, and a group whose last member leaves ceases to exist.

What the group owns is two things. A division of partitions among its current members, so that each partition of each subscribed topic is consumed by exactly one member. And a committed position per partition, which is where a new member starts when it takes that partition over.

Both are held by a broker rather than by the consumers, because both must survive any individual consumer's death. That broker is the group coordinator, and which broker it is follows from the group id by a rule worth knowing.

Finding the coordinator

A starting consumer sends a FindCoordinator request to any broker, naming its group id, and receives the address of the broker that coordinates it. From then on every group operation goes to that one broker.

The rule the broker applies is a hash. The group's state lives in one partition of the internal topic __consumer_offsets, chosen as the absolute value of the group id's Java hashCode modulo the topic's partition count, and the leader of that partition is the coordinator.

offsets.topic.num.partitions defaults to 50 and is immutable after the topic is created, which is a detail with consequences: changing it on a running cluster would re-map every group to a different partition, so Kafka does not permit it, and a cluster created with the wrong value keeps it forever.

offsets.topic.replication.factor defaults to 3, with the important caveat that on a cluster with fewer than three brokers the topic is created at whatever is available, so a single-broker development cluster has a group state with no redundancy and a three-broker production cluster grown from it may still have a replication factor of 1 on that topic. That is worth checking once, because it is a single point of failure for every group in the cluster.

Why the coordinator is a partition leader

The choice to derive the coordinator from a partition leader rather than elect one is a deliberate reuse, and it buys three things at no cost.

Failover is already solved. If the coordinator broker dies, the partition's leadership moves to another in-sync replica by the ordinary mechanism from the replication lesson, and that replica becomes the coordinator. No separate election, no separate failure detector.

State durability is already solved. The group's assignments and committed offsets are records in a replicated log, so they are as durable as any other topic's data, with the same acks and in-sync-replica semantics.

And load spreading is already solved. Fifty partitions distributed across the brokers means fifty coordinator roles distributed across the brokers, so a cluster with many groups spreads their coordination work without anyone configuring it.

The cost is that group load is distributed by hash rather than by weight, so two very large groups can hash to the same partition and land on the same broker. There is no remedy other than renaming a group, which is why a group id is worth choosing as deliberately as a topic name.

What the coordinator does

Five responsibilities, and separating them matters because they fail independently.

It tracks membership: which consumers are currently in the group, identified by a member id it assigns.

It detects failure, by requiring each member to heartbeat within session.timeout.ms and removing those that do not.

It drives rebalances, by collecting every member's subscription, handing the set to one designated member, and distributing the result.

It stores committed offsets, as records in its __consumer_offsets partition, and serves them on request.

And it fences stale members, using a generation number, so that a consumer which missed a rebalance cannot commit offsets for partitions it no longer owns.

Notably it does not compute the assignment, which is the next section's subject and one of the more unusual design decisions in Kafka.

The generation, and the fencing it provides

Each time the group's membership changes, the coordinator increments a generation id, a monotonically increasing integer carried in every subsequent request from every member.

The coordinator rejects any request carrying a generation lower than its current one, with ILLEGAL_GENERATION, and the member's response is to rejoin. This is the same fencing-token pattern as the controller epoch, the leader epoch and the producer epoch, which is now the fourth instance in this course and should be read as a house style rather than a coincidence.

The problem it solves is specific and would otherwise be severe. A consumer that was paused long enough to be removed from the group, by a GC pause or a blocked thread, wakes up believing it still owns its partitions. Those partitions have been reassigned and are being consumed by someone else. Without fencing, the woken consumer's offset commit would overwrite the new owner's position, and the group would reprocess or skip records with nothing to indicate why.

With fencing, the commit is rejected, the consumer discovers it has been removed, and rejoins. The work it did between being removed and discovering it was removed is duplicated, which is at-least-once behaving as designed, and the difference from the unfenced case is that the duplication is bounded and the offsets stay correct.

The group state machine

The coordinator holds each group in one of five states, and they appear in kafka-consumer-groups.sh --describe --state, so reading them is a diagnostic skill.

Empty: the group exists, with committed offsets, and has no members. A group whose consumers are all stopped sits here, and its offsets remain until offsets.retention.minutes expires them, 10,080 minutes or seven days by default.

PreparingRebalance: a membership change has been detected and the coordinator is waiting for every member to send JoinGroup. A group stuck here is waiting for a member that will never arrive, which is the single most common group pathology and is covered in the timeouts section.

CompletingRebalance: every member has joined, the designated member is computing the assignment, and the coordinator is waiting for it. Brief in normal operation.

Stable: an assignment is in force and members are heartbeating. This is where a healthy group spends essentially all of its time.

Dead: the group's metadata has been removed, either because it was deleted or because it was empty past the offset retention.

The transition worth internalising is that no member consumes during PreparingRebalance or CompletingRebalance under the eager protocol, so the duration of those states is consumption downtime for the whole group. That is the fact the rebalance lesson is built around.

Where the group's state is written

The __consumer_offsets topic holds two kinds of record, both keyed, and the distinction explains an operational surprise.

Offset commit records are keyed on group, topic and partition, with the committed offset and optional metadata as the value. Since the topic is compacted, the latest value per key survives indefinitely, which is why a group's position is not lost to retention.

Group metadata records are keyed on the group id alone, holding the membership and the assignment, written on each rebalance.

The topic's cleanup.policy is compact, so the log holds the current state of every group that has committed within the retention of its tombstones. That makes the topic small relative to its write rate, and it makes a group's offsets durable without any time limit while the group has members.

The surprise is that an empty group's offsets are deleted after offsets.retention.minutes, seven days, by writing tombstones. A consumer group stopped for a two-week holiday restarts with no committed offsets and falls back to auto.offset.reset, which defaults to latest, meaning it silently skips everything that arrived while it was stopped. That is one of the few data-skipping behaviours Kafka has by default, and it has surprised a great many people.