Every partition carries three distinct high-water positions, and almost every confusing observation about transactional consumption comes from conflating two of them. They are worth separating precisely before anything else.

Log end offset

The offset that will be assigned to the next record appended to the leader.

It counts everything physically in the log: committed records, records of transactions still open, records of transactions that were aborted, and the control batches holding markers. It is what --describe reports as LOG-END-OFFSET and what every lag calculation subtracts from.

It is a property of the leader's local log and says nothing about replication or visibility.

High watermark

The offset up to which every in-sync replica has the data, which is the boundary of what is durable.

Records above it exist on the leader and may not exist anywhere else, so exposing them would mean a reader could see a record that a subsequent unclean failover erases. That is why a read_uncommitted consumer is bounded by the high watermark rather than the log end offset.

It advances when followers fetch and acknowledge, so the gap between it and the log end offset is a replication-lag measure, which the ISR lesson developed.

Last stable offset

The offset up to which the outcome of every transaction is known, and the boundary for a read_committed consumer.

Its definition is a minimum of two things. If the partition has no open transactions, the last stable offset equals the high watermark. If it has open transactions, it is the first offset of the oldest of them, because no record at or after that point can be interpreted yet.

So the ordering is always last stable offset <= high watermark <= log end offset, and the two gaps mean completely different things. High watermark to log end offset is a replication gap, fixed by followers catching up. Last stable offset to high watermark is a transaction gap, fixed only by some producer committing or aborting.

How the fetch is bounded

The mechanism is a single field, which is a pleasant piece of design.

A FetchRequest carries an isolation_level field, 0 for read_uncommitted and 1 for read_committed. The leader uses it to pick the upper bound of the returned data: the high watermark for 0, the last stable offset for 1.

That is the whole enforcement. The consumer is not trusted to self-limit, the broker simply does not send bytes beyond the bound, and a read_committed consumer therefore cannot see an undecided record even if its client code is wrong.

One consequence is that isolation is per fetch, not per consumer group or per topic. Two consumers in the same group with different isolation.level settings both work and see different data, which is a configuration drift that is possible and that nothing warns about.

Why the oldest open transaction, and not each one individually

The question people ask once they see the definition, and the answer is about the log being a sequence.

Records are delivered in offset order, so a consumer cannot skip offset 1,000 and deliver 1,001. If the transaction that owns offset 1,000 is still open, the consumer cannot know whether to deliver 1,000, and therefore cannot proceed past it, even if everything from 1,001 onward belongs to transactions that have already committed.

That is head-of-line blocking, and it is per partition. One producer's slow transaction holds back every other producer's committed records behind it in that partition, which is the single most operationally important consequence in this lesson.

The alternative, delivering out of order and filling gaps later, would mean abandoning the per-partition ordering guarantee that everything else in Kafka rests on, so it was never on the table.

What the LSO means for lag and for endOffsets

Two API behaviours that are documented and routinely missed.

endOffsets() on a read_committed consumer returns the last stable offset, not the log end offset. So a consumer computing its own lag from endOffsets() minus position() measures lag against the visibility boundary, which is the number it should care about, while a broker-side exporter measuring against LOG-END-OFFSET measures a different and larger quantity. Two lag dashboards for the same group legitimately disagree, and both are right about different things.

And position() after processing everything available sits above the offset of the last delivered record, because control batches consumed offsets that were never surfaced. Any code that stores lastRecord.offset() + 1 as a resume point is correct, and any code that asserts position() == lastRecord.offset() + 1 is wrong on a transactional topic.

The last stable offset is per partition

An obvious-sounding point with a non-obvious consequence, so it is worth stating plainly.

Each partition computes its own last stable offset from its own set of open transactions. A transaction spanning 10 partitions pins the last stable offset in all 10, but only at the offset where it began in each, which differs from partition to partition.

So a stalled transaction does not stall a topic, it stalls the specific partitions that transaction had enrolled. A consumer group reading 24 partitions with one stalled transaction keeps 23 partitions flowing, which is why the symptom is a single partition's records-consumed-rate at zero rather than a dead consumer.

This also means the per-partition view is the only useful one. A topic-level or group-level aggregate of any of these quantities hides the stall completely, in the same way an aggregate lag figure hides one hot partition, which the lag lesson argued at length.

And it is why the resolution tooling takes a topic and a partition rather than a transaction id: the abort has to be applied where the marker is missing.

Where the last stable offset is tracked

The implementation detail that explains why the bookkeeping is cheap.

Each partition's leader maintains a ProducerStateManager holding, among other things, the set of transactions currently open in that partition with their first offsets. The last stable offset is derived from that set on demand, as the minimum first offset, or the high watermark when the set is empty.

That state is kept in memory and periodically written to a .snapshot file per partition, so a broker restart or a leader election does not have to replay the whole log to know which transactions were open. The snapshot plus the tail of the log is enough.

Followers maintain the same state from the records they replicate, which is what allows a new leader to compute the correct last stable offset immediately after an election rather than deriving it from scratch.

The cost is therefore proportional to the number of concurrently open transactions per partition, not to the number of transactions ever run, which is why a workload with many short transactions is cheap for this machinery and one with a few very long ones is not.

A worked example

Numbers make the three positions concrete, and the arithmetic is the one to carry into the triage at the end of this lesson.

A partition has a log end offset of 900,000. All replicas are in sync and have fetched everything, so the high watermark is also 900,000. One producer opened a transaction whose first record landed at offset 880,000 and has not yet committed. A second producer opened and committed a transaction between 885,000 and 890,000, and a third wrote non-transactionally throughout.

The last stable offset is 880,000, because that is the first offset of the only open transaction. A read_committed consumer can read to 880,000 and no further.

So the second producer's records at 885,000 to 890,000 are committed, durable, replicated, and invisible. The third producer's non-transactional records above 880,000 are equally invisible, which is the part that surprises people: a plain producer with no transactional configuration at all has its records withheld because somebody else's transaction is open in the same partition.

Broker-side lag for a caught-up group reads 20,000 and is not falling. The group's own endOffsets() minus position() reads 0. Both are correct.

If the open transaction then commits, the last stable offset jumps from 880,000 to 900,000 in one step, and 20,000 offsets' worth of records become visible at once. That jump is the normal shape of recovery from a stall, and a consumer sized for the steady rate will see a burst equal to the whole withheld backlog, which is the capacity consideration the autoscaling lesson would recognise.