A rebalance is the mechanism by which a group adapts to change, and under the original protocol it is also the mechanism by which a group stops. Understanding where the time goes is what makes the rest of this lesson actionable.

The stop-the-world property

Under the eager protocol, which is what RangeAssignor, RoundRobinAssignor and StickyAssignor all use, a rebalance begins by every member revoking every partition it holds and ends when every member has received its new assignment.

Between those two points no member of the group consumes anything. Not the members whose assignment is about to be identical, not the members that were not involved in whatever triggered it. The whole group stops.

That is a surprising amount of collateral damage from, say, one instance being redeployed, and it is the single fact that makes rebalancing an operational concern rather than an implementation detail. A 40-member group that rebalances for 20 seconds has lost 800 member-seconds of consumption to one instance restarting.

The reason is the barrier in the protocol: the coordinator cannot respond to any JoinGroup until every member has sent one, and no member can be told its new partitions until the leader has computed the whole assignment. A protocol built on a global barrier makes every participant wait for the slowest.

Where the time actually goes

Four components, and they differ by orders of magnitude, so knowing which dominates in a given group is the whole diagnosis.

Detection. For a member that left without closing, this is session.timeout.ms, 45,000 by default, and it dwarfs everything else. For a member that called close() and sent LeaveGroup, it is milliseconds.

Waiting for the slowest member to join. A member mid-batch does not send JoinGroup until it returns to poll(), so this is bounded by how long a batch takes, up to max.poll.interval.ms.

Computing and distributing the assignment. Microseconds for the built-in assignors at any realistic scale; seconds for a Streams assignor with many tasks and state stores.

Recovering after the assignment. Each member seeks to the committed offset of each newly assigned partition, re-establishes fetch sessions and rebuilds any local state. For a stateless consumer, milliseconds. For a Streams application with a large state store and no standby, minutes to hours.

The practical consequence is that the first and the last dominate, and they have completely different fixes: the first is fixed by how the process shuts down, the last by not moving partitions that do not need to move.

The four things that trigger one

Worth enumerating, because two of them surprise people and are the cause of rebalances nobody can account for.

A member joins. A new instance, or an existing one rejoining after being removed.

A member leaves. Either explicitly, by close() sending LeaveGroup, or implicitly by missing heartbeats past session.timeout.ms, or by removing itself for exceeding max.poll.interval.ms.

A subscription changes. A member calling subscribe() with a different topic set triggers a rebalance for the whole group, because the assignment covers the union of subscriptions.

The subscribed metadata changes. This is the surprising one. A topic's partition count increasing triggers a rebalance for every group subscribed to it, since there are new partitions to assign. And a group that subscribed by pattern rebalances when any topic matching that pattern is created or deleted, which in an environment with automatic topic creation means an unrelated service's deployment can rebalance your group.

How quickly the last is noticed is metadata.max.age.ms, default 300,000, which is why a pattern-subscribed group can rebalance up to five minutes after the topic that caused it appeared, making the correlation genuinely hard to see.

The fifth trigger, which is not a membership change

Coordinator failover. If the broker leading the group's __consumer_offsets partition fails or its leadership moves, members discover this through a NOT_COORDINATOR error, re-issue FindCoordinator and rejoin.

The new coordinator has the group's metadata, because it replicated the log, so the group's state is intact. But the members must re-establish themselves with it, and the usual result is one rebalance.

This matters during ordinary cluster maintenance. A rolling broker restart moves partition leadership, including __consumer_offsets leadership, so every group whose coordinator lands on a restarting broker rebalances. On a cluster with many groups, a rolling restart produces a wave of rebalances unrelated to any consumer deployment.

The mitigation is the ordinary one for rolling restarts: a controlled shutdown so leadership moves cleanly rather than by failure detection, and enough time between brokers for groups to settle. It is also an argument for offsets.topic.replication.factor being at least 3, since a coordinator whose partition has no other in-sync replica cannot fail over at all.

Measuring it, rather than guessing

Six metrics in consumer-coordinator-metrics, and the first three are the ones to put on a dashboard.

rebalance-latency-avg and rebalance-latency-max, in milliseconds, which say how long a rebalance takes. A group whose max is in the tens of seconds has a detection problem, since the computation is never slow.

rebalance-rate-per-hour, which says how often. A group deploying once a day should show a handful per day; anything higher without deployments is one of the timeout problems from the previous lesson.

failed-rebalance-rate-per-hour, which counts rebalances that did not complete, usually because a member timed out mid-rebalance. Nonzero here means the group is thrashing rather than merely rebalancing.

last-rebalance-seconds-ago, which is the single most useful one during an incident: a value that keeps resetting to zero is a group in a loop, and that is visible in a single sample over a few seconds.

And assigned-partitions per member, which under the cooperative protocol is how you see that a member kept its partitions through a rebalance rather than having them taken and given back.

Converting the measurement into a cost

The number worth computing, because it is the number that justifies the work: group-seconds of lost consumption per deployment.

For a 40-member group with a 20-second eager rebalance and a deployment that restarts all 40 instances one at a time, each restart is two membership changes, so roughly 80 rebalances at 20 seconds each. Even assuming the group is stopped for only one rebalance's duration per restart, that is 40 members times 20 seconds times 80, or around 64,000 member-seconds. Note that those 80 rebalances are themselves 1,600 seconds, so the deployment's own wall clock cannot be under about 27 minutes no matter how fast the instances restart.

Expressed as lag, a group consuming 200,000 records per second that is stopped for 1,600 seconds in total accumulates 320 million records of backlog over the deployment, and then has to drain it at whatever headroom it has. Headroom is the point here and it is easy to get wrong: a group running at 70 percent of capacity can consume about 286,000 records per second, so while 200,000 keeps arriving the backlog falls at only 86,000 per second. That drains 320 million in roughly 3,700 seconds, a little over an hour. Add the 27 minutes of deployment and the group has elevated lag for about an hour and a half, against a change that looked like a twenty-minute rollout.

That calculation is what turns "rebalances are annoying" into a number a team can act on, and the two fixes in this lesson, static membership and the cooperative protocol, both reduce it by more than an order of magnitude.

What this lesson changes and what it cannot

Three things are addressable, and being clear about the fourth prevents wasted effort.

Detection time is addressable, by making shutdown explicit or by making the departure not count at all with static membership.

The group-wide stop is addressable, by the cooperative protocol, which confines the pause to the partitions actually moving.

Movement itself is addressable, by a sticky assignor, which keeps a partition with its existing owner whenever balance permits.

What is not addressable within this lesson is recovery cost for a stateful consumer whose partitions genuinely do move. A Streams task that must rebuild a 50 GB state store from a changelog takes as long as it takes, and the answer there is standby replicas and warmup, which belongs to the Streams lesson rather than to the protocol.

Reading the coordinator's log, which records the reason

The most underused diagnostic in the whole subject. The broker acting as group coordinator logs a line for every rebalance it begins, and that line names the cause.

The shape is a preparing-to-rebalance message carrying the group id, the old generation, the member count, and a reason string. The reasons are specific: adding a new member, removing a member on heartbeat expiration, removing a member on an explicit LeaveGroup, and updating metadata for a member whose subscription changed.

That distinction is exactly the one the consumer's own logs cannot make. A consumer sees REBALANCE_IN_PROGRESS and knows a rebalance happened; only the coordinator knows whether the cause was a new arrival, a session expiry or a subscription change, and those have completely different fixes.

The practical workflow is to find which broker leads the group's __consumer_offsets partition, then grep its log for the group id over the window in question. A run of heartbeat-expiration reasons for the same client id points at GC or CPU throttling on one host; a run of add-and-remove pairs points at a crash loop; metadata-update reasons point at the topic-creation trigger.