Every replicated system has one configuration setting that decides what happens when its safety assumptions are exhausted. In Kafka that setting is unclean.leader.election.enable, and it is the only place in the broker where you explicitly authorise the loss of acknowledged data.

The state that forces the question

A partition has three replicas. Two fall out of the in-sync set, so the leader is the only member. Then the leader fails. The partition now has three replicas, all alive or recoverable, and none of them in the in-sync set.

The controller's normal rule, choose the first assigned replica that is in the in-sync set, has no candidate. So the partition has no leader. It accepts no produce requests and serves no fetch requests, and it stays that way indefinitely, counted in OfflinePartitionsCount.

There are exactly two ways out. Wait for a replica that was in the in-sync set to return, which keeps every committed record and leaves the partition unavailable for as long as that takes. Or elect one of the out-of-sync replicas, which restores availability immediately and discards every record the chosen replica is missing.

The setting, and what changed about its default

unclean.leader.election.enable is a broker setting, overridable per topic, and its default is false since version 0.11. Before that it defaulted to true, which means clusters built on early Kafka silently chose availability over correctness, and a surprising number of those clusters still carry the old explicit setting in their configuration files because someone copied it forward.

False means the controller never performs an unclean election automatically. A leaderless partition stays leaderless until an in-sync replica returns or an operator intervenes.

True means the controller elects an out-of-sync replica as soon as one is available, with no notification beyond a metric and a log line. The decision is made in milliseconds, by software, at whatever hour the failure happened, and the data is gone before anyone knows a choice was made.

Why the default changed, and the honest framing

The change to false was not a judgement that availability does not matter. It was a judgement about which failure a human should be present for.

Unavailability is loud, bounded and reversible. Producers get errors, consumers stop, dashboards go red, someone is paged, and when the replica returns everything resumes exactly where it stopped. Nothing has to be reconstructed.

Data loss is quiet, unbounded and irreversible. Producers received successful acknowledgements for records that no longer exist. Consumers that read those records have processed events the cluster now denies. There is no log of what was lost and no way to reconstruct it from the cluster, because the only copies were on the replicas that are gone.

So the default encodes the position that an operator should choose to lose data, with knowledge of what is at stake, rather than discover afterwards that a setting chose for them. That is a defensible default rather than an obviously correct one, and the next section is about what the choice actually costs so the decision can be made numerically.

What the elected replica is missing

Precision matters here, because the quantity lost is not arbitrary.

The out-of-sync replica fell behind at some point and stopped reaching the leader's log end offset. Everything the leader appended after that moment is absent from it. If it fell out thirty seconds before the leader failed and the partition was taking 10,000 records per second, it is missing roughly 300,000 records. If it fell out two hours earlier, it is missing two hours.

All of those records were committed. The leader was the whole in-sync set, so the high watermark equalled the leader's log end offset, so every one of them was acknowledged to an acks=all producer as durably stored.

This is the part worth stating without softening: acks=all plus min.insync.replicas of 1 plus unclean election enabled means the strongest acknowledgement Kafka offers can be followed by unbounded loss of the records it acknowledged.

Where this sits in the usual framing

It is tempting to call this a CAP tradeoff and stop, but the framing is more specific than that and the specificity is useful.

CAP's partition case asks whether to serve requests that might be inconsistent. Unclean election asks something narrower: whether to continue a log whose recent history has been destroyed. The surviving replica is not holding a stale but valid view; it is holding a prefix, and continuing from a prefix means reusing offsets that already named different records.

That is why the consequences reach further than a stale read. A stale read becomes correct again when the partition heals. A reused offset never becomes correct, because two different records now share a name and anything that recorded one of those offsets, a consumer's committed position, a downstream system's idempotency key, an audit record, is now wrong in a way nothing will fix.

When enabling it is actually right

There are legitimate cases, and treating the setting as universally wrong makes the discussion worse rather than better.

A topic carrying metrics, logs or traces, where the value of a record decays in seconds and the consumer is a dashboard, should prefer availability. Losing five minutes of CPU samples costs nothing; a leaderless metrics topic blinds an operator during an incident, which costs a great deal.

A topic that is a cache or materialised view rebuilt from an authoritative source should prefer availability, because the loss is repairable by replaying the source.

The distinguishing question is whether the records exist anywhere else. If they do, availability is nearly free. If this log is the only copy, which is true of any topic used as a system of record, an event store or a payment stream, then enabling the setting is authorising unrecoverable loss and the answer is no.

Setting it per topic, which is the useful granularity

Because the determining question is about each topic's data rather than about the cluster, the correct configuration is usually heterogeneous.

unclean.leader.election.enable as a broker default of false, overridden to true on the specific topics whose records are disposable or reconstructible, is the shape that matches the reasoning. Setting it cluster-wide in either direction applies one answer to topics with opposite requirements.

The operational cost of doing this is that the override has to be a documented decision with a named owner rather than a setting someone added during an incident. A topic whose configuration says true and whose owner cannot say why is the worst of both: the loss is authorised and nobody chose it.

How a partition reaches the state, step by step

The sequence is worth tracing because each step is individually unremarkable, which is why the end state surprises people.

Three replicas, all in sync. Broker C's disk begins failing, its appends slow, and after thirty seconds without reaching the leader's log end offset it is removed from the in-sync set. UnderReplicatedPartitions goes to a nonzero value on broker A, which leads. Nobody is paged, because the alert has a five-minute window and this is one broker.

Broker B is restarted as part of a routine rolling deployment. controlled.shutdown.enable hands over its leaderships cleanly, but it was a follower here, so it simply stops fetching and after thirty seconds is removed from the in-sync set too. The set is now broker A alone.

With min.insync.replicas at its default of 1, the partition keeps accepting acks=all writes, and every one of them is acknowledged as durable on exactly one machine. Then broker A fails.

Nothing in that sequence involved an error anyone would investigate, and the whole thing takes under two minutes. That is the shape of most real instances, and it is why the mitigations in the last section are about preventing the second step from mattering rather than about the election itself.

The ZooKeeper-era behaviour, for clusters that remember it

One historical detail resolves a persistent confusion about where the setting applies.

Under ZooKeeper, the controller kept the in-sync set in a znode per partition, and an unclean election rewrote that state. Operators sometimes recovered from a leaderless partition by editing the znode directly, which worked and which has no equivalent today.

Under KRaft the in-sync set lives in the metadata log as PartitionRecord and PartitionChangeRecord entries, which are append-only and have no edit path. The only supported route is kafka-leader-election.sh --election-type unclean, and that is an improvement rather than a loss of capability, because the znode edit was unlogged and unaudited.