Exactly-once in Kafka is a property of a closed system, and the useful skill is drawing the boundary of that system precisely for a pipeline in front of you. Almost every production disappointment is a boundary drawn wrongly.

The closed system

Three conditions, all of which must hold, and the guarantee is exactly as strong as the weakest.

Every write is to Kafka, performed by a transactional producer whose transactional.id is stable and tied to the work rather than the process.

Every read's offsets are committed through that same producer with sendOffsetsToTransaction, and no other offset commit path exists in the application.

Every reader of every intermediate and final topic uses isolation.level=read_committed.

Inside that boundary, each input record contributes to the output exactly once no matter how many times processing is retried, how many times a process dies, or how many rebalances occur. Outside it, nothing is guaranteed by any of this machinery.

The three boundaries that leak

Where a pipeline stops being closed, in the order they appear in real systems.

The ingress. The data got into Kafka somehow: an HTTP handler, a CDC connector, a file reader. If that producer is not idempotent, the topic already contains duplicates, and an exactly-once pipeline processes each duplicate exactly once, faithfully producing duplicate outputs. Exactly-once processing of duplicated input is duplicated output, which is the single most common misunderstanding in this whole area.

The egress. The final stage writes somewhere that is not Kafka: a database, a search index, an HTTP endpoint, an email. That write is at-least-once, because no Kafka transaction covers it, and the only resolutions are an idempotent sink or offsets stored in the sink's own transaction.

Side effects inside processing. A transform that charges a card, sends a notification or increments an external counter has done that work whether or not the transaction commits. An abort discards the Kafka writes and nothing else.

The honest statement to make about a pipeline

One sentence, which is what to write in a design document instead of the phrase exactly-once.

Each record that reaches topic A is reflected in the final sink exactly once, provided the producer into A deduplicates on a business key and the sink upserts on that key. The Kafka stages contribute no duplicates and lose nothing.

That sentence names the two external conditions rather than hiding them, and it is falsifiable. The phrase exactly-once on its own is neither.

It also makes the design review productive, because the two named conditions are where the actual engineering is, and the Kafka configuration is the part that is already solved.

Idempotence as the usual alternative

Worth stating before the mechanics, because for many pipelines it is the better answer and the comparison should be made deliberately.

If every record carries a stable business key, the final sink upserts on it, and the intermediate stages are stateless or keyed the same way, then duplicates are harmless at every step and at-least-once is sufficient. No transactions, no visibility latency, no head-of-line blocking, no abort-rate bandwidth.

That design is also more robust in a way transactions are not: it survives a replay, a backfill, a migration and a bug-fix reprocessing, none of which transactions help with at all, because all of them legitimately reproduce records.

Transactions earn their cost when the output has no natural key to deduplicate on, most characteristically an aggregation or a count, where a duplicate is not a repeated row but a wrong number. That is the case the next sections assume.

Why the ingress deserves the most attention

Because it is the boundary least often examined and the one where the cost of a mistake compounds.

A duplicate introduced at the ingress is indistinguishable from legitimate data by every downstream stage. It passes through an exactly-once pipeline, is counted in every aggregation, and arrives in the sink as a real record. No amount of transactional configuration downstream can remove it, because nothing downstream knows it is a duplicate.

The defences are all at the ingress itself. enable.idempotence=true on the ingress producer, which is the default since 3.0 and eliminates retry-induced duplicates within a producer session. A business key on every record so that downstream deduplication is possible at all. And for a retrying client, an idempotency key carried from the client through the handler to the record, so that a client's retry of an HTTP request does not produce a second record.

That last one is the only defence against a duplicate that originated outside the system entirely, and it is an application design decision rather than a Kafka setting, which is why it is so often missing.

Where duplicates actually come from

Five sources, which is worth enumerating because Kafka transactions address exactly one of them and the other four are routinely attributed to the one.

A client retry. The caller did not see a response and sent the request again. Two legitimate records. Only an idempotency key supplied by the caller prevents this, and nothing inside Kafka can.

A non-idempotent ingress producer retrying. The broker received the record, the acknowledgement was lost, the producer resent. enable.idempotence=true, the default since 3.0, closes this one entirely within a producer session.

A consumer reprocessing after a crash. The batch was produced and the offsets were not committed. This is the one Kafka transactions solve, and it is the only one.

A replication hop. A cross-cluster replication tool is a plain consumer and a plain producer, so a retry inside it duplicates, and the destination has no transaction covering it.

A manual replay or backfill. Somebody reset a group's offsets or reran a job. Entirely legitimate, entirely duplicating, and no configuration prevents it because preventing it would be wrong.

Why the guarantee does not cross a cluster boundary

A specific case of the egress boundary that deserves naming, because it is invisible in the pipeline diagram.

A Kafka transaction is coordinated by one cluster's transaction coordinator and its markers are written into that cluster's partitions. Nothing about it spans two clusters, and there is no protocol that would let it.

So a replication tool reading cluster A and writing cluster B is an ordinary consumer and an ordinary producer. It can read read_committed, and it must, but it produces into B non-transactionally, and the control batches it read are not copied because they are not records it can produce.

Which means an exactly-once pipeline with a replication hop in the middle is at-least-once across that hop, and B's consumers have to tolerate duplicates regardless of how A's producers were configured. Any claim of exactly-once in a multi-cluster topology needs the hop pointed at explicitly.

Reading the guarantee out of a running pipeline

Four questions, answerable from configuration and code without running anything, which is the fastest way to find out whether a pipeline that claims exactly-once has it.

Which producers have a transactional.id, and is each one stable across restarts? An id containing a pod name, a hostname or a generated UUID means fencing never happens.

Where are offsets committed? If enable.auto.commit is unset anywhere, it defaults to true and there is a second commit path outside the transaction.

What is every reader's isolation.level, including sink connectors, ad-hoc consumers and anything in another repository? The default is read_uncommitted and the setting is usually absent rather than wrong.

And what happens at the ingress and at the egress? If the answer to either is "it is fine because the pipeline is exactly-once", the boundary has not been examined.