How Kafka Transactions Work: Atomicity by Visibility, Not Rollback

Kafka transactions confuse people because they borrow database vocabulary that does not quite fit. There is no rollback. There is no undo. The bytes hit the log immediately and they stay there forever. What Kafka does instead is control what consumers are allowed to see.

Start with the actors. A producer declares a transactional.id and calls initTransactions. That binds it to a specific transaction coordinator, which is just a Kafka broker chosen by hashing the id. The coordinator owns a special internal topic, __transaction_state, where it journals the lifecycle of every transaction. When the producer calls beginTransaction, the coordinator records that a transaction is open. The producer then writes records to as many partitions as it wants. Each of those records is appended to its partition log right away with a transactional marker on it. Other producers keep writing to those partitions in parallel.

Now the commit path. The producer calls commitTransaction. The coordinator runs a two-phase commit across every partition the producer touched. Phase one: write a prepare-commit entry to __transaction_state. Phase two: append a control record (the COMMIT marker) to each partition the transaction wrote to. The ABORT path is the same shape with an ABORT marker.

The records between the start of the transaction and the marker still sit in the log. They were never removed. What changed is that a consumer reading with isolation.level=read_committed knows to skip everything that is part of an aborted transaction and to wait on records from open transactions until it sees a marker. The skip is driven by an index of aborted offsets the broker keeps per segment. A read_uncommitted consumer ignores the markers entirely and sees the raw log.

This is why the design is called atomicity by visibility. Either every COMMIT marker lands across every partition (and a read_committed consumer sees the full transaction), or every ABORT marker lands (and the consumer sees none of it). There is no intermediate state where two partitions disagree.

Exactly-once semantics composes from three pieces. Idempotent producers prevent retries from creating duplicate writes on a single partition. Transactional writes give you atomic visibility across partitions. Offset commits performed inside the same transaction tie the consume side to the produce side, which is the trick Kafka Streams uses to make read-process-write loops exactly-once.

The failure mode worth knowing: if a stream processor crashes mid-transaction, the next instance with the same transactional.id fences the old one through an epoch bump, and the coordinator aborts the in-flight transaction. The log keeps the partial writes. The markers hide them. Nothing leaks.