Kafka transactions are narrower than the word suggests and more useful than their reputation. Being precise about the scope is most of the work, because almost every disappointment with them comes from expecting a database transaction.

The guarantee, stated exactly

A Kafka transaction makes a set of writes to Kafka, plus a set of consumer offset commits, atomic with respect to a consumer reading with isolation.level=read_committed.

That is the whole statement. Every word in it is load-bearing: writes to Kafka, so nothing outside Kafka participates; plus offset commits, which is what makes the consume-transform-produce loop work; and with respect to a read_committed consumer, so the atomicity is a property of how the data is read, not of how it is stored.

The last point is the one that surprises people. An aborted transaction's records are written to the log and stay there. They are physically present, they occupy offsets, they are replicated, and a read_uncommitted consumer sees them. Atomicity is enforced on the read path by having the consumer filter them, which is the subject of the next lesson.

So a transaction does not undo anything. It appends a marker saying how the preceding records should be interpreted, and readers that care do the interpreting.

What it does not give you

Five things, each of which somebody has shipped a design around.

No atomicity with an external system. Writing to a database and producing to Kafka cannot be made atomic by a Kafka transaction, because the database is not a participant. That is what the transactional outbox is for.

No isolation between concurrent transactions. There is no read-your-writes, no snapshot, no conflict detection. Two transactions writing the same key both succeed and the later one wins, exactly as without transactions.

No rollback of side effects. If the processing sent an email, aborting the transaction does not unsend it. Only the Kafka writes and the offset commits are in scope.

No cross-cluster atomicity. A transaction is scoped to one cluster, since the coordinator is a broker in it.

No ordering guarantee beyond the per-partition one. Records in a transaction are not delivered as a unit in any ordering sense; they are ordinary records in their partitions that become visible together.

The shape it exists for

One pattern, which is the reason the feature was built and the only one it fits cleanly.

Consume from a topic, transform, produce to another topic, and commit the input offsets. Without transactions this has an unavoidable gap: whichever of the produce and the commit happens second can fail after the first succeeded, giving either duplicates or losses.

A transaction closes the gap by putting the produce and the offset commit in the same atomic unit. The offsets are committed through the producer, not through the consumer, which is the structural change that makes it work and the detail most often missed when adapting existing code.

Everything else people want transactions for is either this pattern in disguise or outside the scope. Recognising which of the two is the first question to ask about any proposed use.

The API, in the order it is called

Six calls, and the order is not optional.

initTransactions() once at startup. It registers the transactional.id with the coordinator, obtains a producer id and an epoch, and blocks until any transaction left open by a previous incarnation of this id has been completed or aborted. That blocking recovery is the whole reason the call exists.

beginTransaction() to open one, which is purely client-side bookkeeping and sends nothing.

send() for each record, where the first send to each partition causes the producer to issue AddPartitionsToTxn so the coordinator knows which partitions to write markers to.

sendOffsetsToTransaction(offsets, groupMetadata) to include the input offsets, which sends AddOffsetsToTxn to add the __consumer_offsets partition to the transaction and then a TxnOffsetCommit to the group coordinator.

commitTransaction() or abortTransaction() to close it, both of which block until the coordinator has durably recorded the outcome.

The consumer's half of the arrangement

Two settings on the consumer side that are easy to omit and that silently break the guarantee.

enable.auto.commit=false, mandatory, because the whole point is that offsets are committed through the producer. An auto-commit left on commits offsets outside the transaction, which reintroduces exactly the gap the transaction was added to close, and nothing errors.

isolation.level=read_committed on whatever reads the output, which defaults to read_uncommitted. A downstream consumer left at the default sees aborted records, so the pipeline has a transactional producer, a correct offset commit, and no exactly-once property at all.

That pair is the most common misconfiguration in the whole feature, and it fails silently in both directions: the producer side looks correct in the code review, and the consumer side is in a different repository.

Why it is called exactly-once and what that means

The phrase is defensible for a specific claim and indefensible for the general one.

For a consume-transform-produce pipeline whose every stage is transactional and whose readers are read_committed, each input record contributes to the output exactly once, no matter how many times processing is retried. That is a real and strong property.

It is not a claim that a record is delivered once. Records are redelivered freely; what the transaction guarantees is that the effects of the discarded attempts are never visible. The honest name is the one the previous lesson used, effectively once, and the mechanism here is the second of the two ways to get it.

And it says nothing about effects outside Kafka. A pipeline that writes to Kafka transactionally and also calls a payment API is exactly-once for the Kafka part and at-least-once for the payment, which is the part that matters.

The gap it closes, with the failure spelled out

Worth walking once concretely, because the abstract statement of the problem is easy to nod along to and hard to act on.

A consumer reads offsets 1,000 through 1,499 from an input partition, transforms them, produces 500 records to an output topic, and commits offset 1,500 on the input. Two writes to two different places: the output records to the output partitions, and the input offset to __consumer_offsets.

Produce first, then commit, and a crash in between leaves the output written and the input offset at 1,000. The replacement reprocesses the same 500 records and produces them again. The output now contains each result twice, and nothing downstream can tell the copies apart, because they are byte-identical records produced by two legitimate attempts.

Commit first, then produce, and a crash in between leaves the input offset at 1,500 with no output at all. The replacement starts at 1,500 and those 500 inputs are never processed again. That is silent data loss, which is the worse of the two failures and the reason almost everyone picks the duplicate-producing order.

There is no ordering of two independent writes that closes this, which is the general two-systems problem. The only resolution is to make them one write, and sendOffsetsToTransaction is what does that: the offsets become ordinary records inside the transaction, landing in __consumer_offsets and becoming visible only when the commit marker is written.

How the offset commit gets inside the transaction

The mechanism is worth knowing because it explains why the group coordinator has to participate.

sendOffsetsToTransaction sends AddOffsetsToTxn to the transaction coordinator, which adds the relevant __consumer_offsets partition to the transaction's partition set, so a marker will be written there too. Then it sends TxnOffsetCommit to the group coordinator carrying the actual offsets.

Those offsets are appended to __consumer_offsets as transactional records like any other. The group coordinator reads its own topic with read_committed, so offsets from an aborted transaction are never observed as the group's committed position, and a restart resumes from the last committed transaction's offsets.

The groupMetadata overload, added by KIP-447, passes the group's generation and member id so the group coordinator can refuse a commit from a member whose generation it has already superseded. The older form, which took a bare consumer group id, could not do that, which is why it is deprecated and why it matters for the fencing discussion later in this lesson.