The idempotent producer is one setting that has been true by default since Kafka 3.0 and whose mechanism almost nobody has read. It is worth reading, because its guarantee is narrower than the name suggests and the shape of that narrowness explains most of the surprises it produces.

The problem being solved

A produce request that times out has an unknown outcome: the broker may have appended it and lost the response, or never received it. Retrying therefore risks a duplicate, and not retrying risks a loss.

Without protection, the producer's choice is between at-least-once with duplicates and at-most-once with gaps, and neither is what an application wants. The duplicates also appear specifically during failures, which is when a consumer is least able to deal with them.

So the goal is for a retry of a request the broker already appended to be recognised and discarded, making the retry safe and the delivery exactly once from the producer's side.

The mechanism is a pair of identifiers attached to every batch: a producer id identifying the producer session, and a sequence number per partition counting batches within it. The broker remembers the last sequence it accepted for each producer and partition, and that memory is what lets it recognise a repeat.

What the guarantee covers, precisely

Four qualifications, and each one eliminates a common misreading.

It is per partition. The sequence number is scoped to a producer and a partition pair, so the guarantee is about that stream and says nothing about ordering or deduplication across partitions.

It is per producer session. The producer id is obtained at initialisation and lost when the producer is closed or restarted, so a restarted producer re-sending the same records is a new producer with a new id, and every record is a new record to the broker.

It applies to retries by the producer, not to resends by the application. If application code calls send twice for the same event, those are two different batches with two different sequence numbers and both are appended.

And it is about batches, not records. The sequence number is assigned per batch, and a batch is accepted or rejected as a whole, which is why the mechanism composes cleanly with compression and with the record format.

So the accurate statement is that the idempotent producer eliminates duplicates introduced by its own retry logic within one session for one partition, which is exactly the problem the previous lesson created and no more than that.

What it is not

Two things it is frequently confused with, and distinguishing them now saves confusion in the Transactions section.

It is not exactly-once processing. A consumer that reads a record, acts on it, and crashes before committing its offset will read it again, and nothing on the producer side affects that. Exactly-once end to end requires the consumer side too, which is what transactions provide.

It is not application-level deduplication. If an upstream system delivers the same event twice, or a service retries a request that produces, those are distinct events as far as Kafka is concerned, and only an application-level identifier can connect them. Which is why the deduplication advice from the data-loss lesson stands: deduplicate on a payload event id, never on partition and offset.

The practical division of labour is that the idempotent producer makes the transport reliable, transactions make the read-process-write cycle atomic, and the application still owns deduplication of events it may genuinely receive twice from outside.

What it costs

Almost nothing, which is why the default changed.

The batch header grows by 12 bytes: 8 for the producer id, 2 for the epoch and 2 for the base sequence. On a batch of a thousand records that is unmeasurable, and on a batch of one it is a few percent.

The broker keeps a small amount of state per producer per partition, five sequence entries, which is the subject of a later section and is the one real cost.

One extra round trip at producer startup, an InitProducerId request, which adds a few milliseconds once.

And three configuration constraints the producer enforces: acks must be all, retries must be greater than zero, and max.in.flight.requests.per.connection must be at most 5. Those are the material cost, and only the first changes behaviour that anyone notices, by making produce latency include replication.

Why the default changed in 3.0

Worth understanding as a design judgement rather than a fact to memorise, because the reasoning generalises.

Before 3.0, enable.idempotence defaulted to false, and the result was that almost every Kafka deployment had a producer capable of silently duplicating and reordering records, with a one-line fix nobody applied because nobody had read far enough to know it was needed.

The objection to changing it was the acks=all constraint, since that makes produce latency include a replication round trip and was perceived as a throughput cost. The previous lesson's arithmetic is why that objection failed: the latency is per batch and pipelined across five in-flight requests, so at any real throughput it is negligible, and the throughput that matters comes from batch size.

So the change traded a few milliseconds of per-batch latency for the removal of a silent correctness failure, and the general form of that trade, paying a small measurable cost to eliminate an unmeasurable one, is the right default almost every time.

The consequence for reading existing systems is that enable.idempotence: false or acks=1 in a configuration file is now a deliberate downgrade from the client's own default, and should be treated as one even when it was actually just inherited.

The duplicate window, drawn precisely

Worth being exact about when a duplicate can appear without idempotence, because the window is narrower than people assume and that narrowness is why the problem went unnoticed for years.

A duplicate requires three things to happen in order: the broker appends the batch, the response fails to reach the producer, and the producer retries before delivery.timeout.ms expires. Any one of those not happening means no duplicate.

So in a healthy cluster the rate is effectively zero, which is exactly the trap. The rate is not zero during a leader election, a broker restart, a network blip or a GC pause, which is to say it is zero in testing and nonzero in every incident.

The second property is that the duplicates are correlated: a 30-second network problem produces duplicates from every producer writing through the affected broker, all at once. A downstream system that handles an occasional duplicate gracefully may not handle thousands in a minute.

And the third is that there is nothing to detect them with. The duplicate records are valid, well-formed, at distinct offsets, with distinct timestamps. No broker metric counts them and no log line mentions them, so the only evidence is a downstream inconsistency discovered later, usually by someone reconciling totals.

Reading enable.idempotence in a real configuration

Four states show up in practice, and they are not equally meaningful, so the audit question is which one a given producer is in.

Absent, on a client at 3.0 or later. Idempotence is on, acks is all, and the producer is correct by default. This is the good case and it is the most common one now.

Absent, on a client older than 3.0. Idempotence is off, acks is 1, and the producer duplicates and reorders under retry. The fix is a client upgrade or one explicit setting, and finding these is the highest-value part of a producer audit.

Explicitly true. Someone read the documentation, which usually means the rest of the configuration was considered too. Worth noting as a positive signal.

Explicitly false, or acks=1 with idempotence unset on a modern client. This is a deliberate downgrade from the client's default, and almost always an old setting carried forward through a dependency upgrade rather than a decision anyone made. The producer will have been written before 3.0 and the configuration copied since.

The practical form of the audit is to collect every producer's effective configuration rather than its file, since a setting's absence means something different per client version, and the effective value is what the broker sees.