Kafka's unit of everything is the record batch, not the record. A producer sends batches, the broker appends batches, a follower replicates batches, the disk stores batches, and a consumer fetches batches and splits them apart locally. Almost every performance property Kafka has follows from that one choice, and the only way to see why is to read the bytes.

The current format is version 2, introduced in Kafka 0.11 alongside the idempotent producer and transactions, and it is the format every modern cluster writes. Its header is a fixed 61 bytes:

1baseOffset             int64   8   assigned by the leader
2batchLength            int32   4   bytes that follow
3partitionLeaderEpoch   int32   4   which leader wrote this
4magic                  int8    1   format version, 2
5crc                    int32   4   CRC32C of everything below
6attributes             int16   2   codec, timestamp type, txn, control
7lastOffsetDelta        int32   4   offset of last record minus base
8firstTimestamp         int64   8
9maxTimestamp           int64   8
10producerId             int64   8   idempotence and transactions
11producerEpoch          int16   2
12baseSequence           int32   4
13recordCount            int32   4

Why the CRC sits in the middle

Notice where the checksum is. It is the fifth field, not the first and not the last, and it covers only the bytes from attributes onward. That placement is deliberate and it is the format's cleverest detail.

A producer computes the CRC before it knows the batch's offsets, because offsets are assigned by the leader. If the checksum covered baseOffset, the broker would have to recompute it after assigning offsets, which means reading and hashing every byte of every batch on the write path. By excluding baseOffset and batchLength from the covered range, the broker can stamp the offset into the header and append the batch without touching the payload at all. The producer's checksum survives all the way to the consumer, so the integrity check is genuinely end to end, and the broker's append is a memory copy rather than a computation.

The algorithm changed in v2 as well, from CRC32 to CRC32C, the Castagnoli polynomial, for the unglamorous reason that modern x86 processors implement it as a single instruction. A per-batch checksum computed in hardware is cheap enough to leave on; the same checksum per record in software would not be.

What the attributes bits carry

The 16-bit attributes field is where several features hide. Bits 0 to 2 hold the compression codec, so compression is a property of the batch rather than of the record. Bit 3 is the timestamp type, recording whether the timestamps are producer-supplied CreateTime or broker-stamped LogAppendTime. Bit 4 marks the batch transactional. Bit 5 marks it a control batch, which is how commit and abort markers are distinguished from data, and bit 6 indicates the presence of a delete horizon used by compaction when handling tombstones.

Two things follow from compression living here. First, you cannot have a batch of records with mixed compression, so a producer's compression.type applies to whole batches. Second, a consumer reading a compressed batch must decompress the entire batch to read any record in it, which is a real cost when a consumer only wants one key out of a thousand and a further reason there is no server-side filtering.

The record inside the batch

Individual records are deliberately minimal, because everything they could share with their neighbours has been hoisted into the header. A record is:

1length         varint     total bytes of this record
2attributes     int8       unused, reserved
3timestampDelta varlong    added to the batch firstTimestamp
4offsetDelta    varint     added to the batch baseOffset
5keyLength      varint     -1 means null
6key            bytes
7valueLength    varint     -1 means a tombstone
8value          bytes
9headerCount    varint
10headers        [(keyLen, key, valueLen, value)]

Both the timestamp and the offset are deltas, which is why they cost one or two bytes each instead of eight and four. A batch of a thousand records written in the same millisecond has a thousand timestamp deltas of zero, each a single byte, and offset deltas running 0 to 999, most of which fit in two bytes. The absolute values exist once, in the header.

This also explains a field that looks redundant. lastOffsetDelta is stored in the header even though it could be derived by reading the last record, and it is there so that the broker can compute the batch's next offset, and the consumer can compute the batch's offset range, without decompressing the records section. The same reasoning puts maxTimestamp in the header: time-based retention and the time index need it, and neither should have to decompress anything.

Reading real bytes

You do not have to take any of this on trust, because the broker ships a tool that dumps the format. Running kafka-dump-log.sh --files 00000000000000000000.log --print-data-log against a segment prints one line per batch with its baseOffset, lastOffset, count, producerId, producerEpoch, baseSequence, isTransactional, isControl, compression codec and CRC, followed by the records if you asked for data.

It is worth doing once on a topic you understand, because several things that are abstract until you see them become obvious immediately: that a "message" you sent one at a time became a batch of one with 61 bytes of header; that a transactional topic's log contains control batches with isControl: true holding no data records; that a compacted partition's batches have gaps in their offset ranges; and that a batch's producerId is stable across a producer's lifetime, which is the entire basis of the deduplication the next lessons rely on.

What came before, and what the change bought

The formats this replaced are worth one paragraph, because several of v2's oddities are reactions to their specific failures. Version 0 and version 1, the message set formats, had no batch header. Each message carried its own absolute offset, its own CRC, its own magic byte, its own attributes and, in v1, its own absolute timestamp. A "batch" was just a message set, and compression was implemented by wrapping a message set inside the value of a single outer message, which is why compressed batches in those formats were nested structures with offsets at two levels.

That design cost in three places. Per-record overhead was large, because nothing was shared and every record paid 8 bytes of absolute offset plus a 4-byte CRC plus header bytes. The broker had to rewrite compressed message sets to assign offsets, because the inner offsets were relative in a way the outer wrapper did not express cleanly, so compressed produce requests were never a pure memory copy. And there was no place to put producer identity, so idempotence and transactions were simply not expressible; that is the real reason v2 and the idempotent producer arrived in the same release. Every field in the v2 header that looks like overhead, producerId, producerEpoch, baseSequence, partitionLeaderEpoch, exists because a feature needed somewhere to live.

The batch is the unit of atomicity

One consequence of batching deserves to be stated on its own, because it governs error handling. A batch is appended or it is not. There is no partial append, so a produce request's response is per partition, not per record, and a failure means every record in that batch failed. The futures returned by send for all the records in one batch therefore complete together with the same outcome.

This is convenient and occasionally surprising. It means a single oversized or malformed record can fail the batch it is in, taking perfectly good neighbours with it, which is why max.request.size violations are detected at send time rather than batched and discovered later. It also means retries retry whole batches, which is what makes the idempotent producer's sequence numbering work at batch granularity: the baseSequence in the header identifies the batch, and the broker's deduplication compares batch sequences rather than tracking individual records.