A Kafka cluster has to agree on things that are not records: which brokers are alive, which topics exist, how many partitions each has, which broker leads each partition, and which replicas are in sync. None of that fits in a partition log, because a partition log needs a leader and choosing a leader is exactly the problem. So the cluster needs a separate agreement mechanism, and the component that owns it is the controller.

The state the cluster must agree on

Being specific about the contents helps, because the volume is what drives every design decision that follows.

Per broker: its identifier, its advertised listeners, its rack, and whether it is currently alive. Per topic: its name, its configuration overrides, and its partition count. Per partition: the full replica assignment, the current leader, the current in-sync replica set, and the leader epoch. Plus cluster-wide items: ACLs, client quotas, dynamic broker configuration, delegation tokens, and in a tiered cluster the pointers into remote storage are tracked separately but their topic configuration is here.

The per-partition entries dominate. A cluster with 10,000 partitions at replication factor 3 has 10,000 replica assignments, 10,000 leader entries, 10,000 ISR sets and 10,000 epochs, all of which change during failures and none of which any broker can decide unilaterally.

Why one component decides

The alternative to a controller is for brokers to negotiate among themselves, and the reason Kafka does not do that is worth understanding because it shapes the whole architecture.

Leader election has to produce exactly one answer. Two brokers believing they lead the same partition means two brokers accepting writes to the same log, which is not a performance problem but a correctness one: records are lost, offsets collide, and no downstream consumer can tell. Avoiding that by consensus among all brokers would require a round of agreement per partition per failure, which at 10,000 partitions is not feasible on a failure path that needs to complete in seconds.

The controller pattern instead elects one leader for the metadata, once, and that single leader then makes every per-partition decision locally without further coordination. The hard problem is solved once per controller election rather than once per partition, and partition decisions become cheap local writes.

This is a common and important shape: rather than reaching consensus about the data, reach consensus about who decides, and let that one decider be fast. Its cost is that controller failover is the serialising event, which is why how long failover takes is the number that matters in both of Kafka's implementations.

What a broker does with metadata

Brokers are not passive. Each one needs an accurate local view, because that view answers questions on the hot path.

When a client asks which broker leads a partition, any broker answers from its local metadata cache. When a producer sends to a partition, the receiving broker checks whether it is the leader and returns NOT_LEADER_OR_FOLLOWER if it is not. When a follower fetches, the leader checks the follower's epoch against its own. All of that is local and must be fast, so every broker holds the full metadata in memory.

Which means metadata propagation is a broadcast problem: the controller makes a decision and every broker in the cluster must learn it promptly. The window between a decision and its propagation is a window of stale answers, during which clients are told to talk to a broker that no longer leads, get an error, refresh, and retry. That window is why clients have retry logic at all, and shortening it is most of what the KRaft redesign achieved.

Metadata is also a client-facing interface

The last thing to note is that metadata is not purely internal. Clients fetch it, cache it, and act on it, which makes its propagation part of the client-visible behaviour.

A producer fetches metadata on startup and refreshes it every metadata.max.age.ms, 300,000 by default, and immediately on any error implying its view is stale. A consumer does the same, plus it needs metadata to discover its group coordinator. So a client's view can be five minutes old in the absence of errors, and the system's correctness does not depend on it being fresh, only its efficiency: a stale client is told it is wrong and corrects itself.

That design choice, that stale metadata produces a retryable error rather than a wrong result, is what allows propagation to be asynchronous at all. It is worth holding onto, because it is the reason the cluster can have thousands of brokers and clients with inconsistent views and still never lose a record.

The leader epoch, which is the mechanism underneath all of it

One concept recurs everywhere in this lesson and is worth establishing now rather than meeting four times.

Every partition has a leader epoch, a monotonically increasing integer the controller advances each time it assigns a new leader. The epoch is not bookkeeping; it is carried in fetch requests, stamped into the leader epoch checkpoint file beside the log, and included in the record batch headers written while that leader held the role.

What it buys is the ability to detect a stale actor without any shared clock or coordination. A follower fetching with epoch 7 from a leader on epoch 9 is told its epoch is stale and must truncate. A producer whose request carries knowledge of an old leader is rejected. A broker receiving a controller request with a lower controller epoch than it has already seen discards it.

This is the fencing token pattern and it appears at three levels in Kafka: the controller epoch fences deposed controllers, the leader epoch fences stale replicas and clients, and the producer epoch fences zombie producers in the transactional protocol. Recognising it as one idea makes each appearance easier, and the recurring shape is the same: a monotonic number attached to every authoritative action, with receivers rejecting anything older than the highest they have seen.

What happens with no controller at all

A useful thought experiment, because it tells you what the controller is and is not on the critical path for.

With no active controller, existing leaders keep serving produce and fetch requests normally, because serving a partition does not require the controller's involvement. Replication continues. Consumers commit offsets, because the group coordinator is a broker role rather than a controller one.

What stops is every change. No topic can be created or deleted. No leader can be elected, so a broker failure during this period leaves its partitions unavailable indefinitely rather than for seconds. No ISR change is recorded, so a replica falling behind is not removed and acks=all keeps waiting for it. No configuration change applies.

That combination is what makes a controller outage dangerous rather than merely inconvenient: the cluster looks healthy by every traffic metric and has silently lost its ability to recover from anything. It is the reason ActiveControllerCount is the first alert to create, and the reason that metric is more important than its unexciting name suggests.