A partition is a log, and a log is too large to be one file. Kafka splits it into segments, and almost every storage behaviour that seems arbitrary, retention granularity, compaction's refusal to touch recent data, the cost of a broker restart, is a consequence of how segments work.

One file per range of offsets

A segment is a .log file holding a contiguous range of the partition's record batches, named by the base offset of its first batch, zero-padded to 20 digits. A partition with three segments might hold 00000000000000000000.log, 00000000000048219114.log and 00000000000097331802.log, and from those names alone you know that offsets 0 through 48,219,113 are in the first file.

Exactly one segment at any moment is the active segment: the newest one, the only one open for append, and the only one whose contents can still grow. Every other segment is immutable and will never be written again. That split is more important than it sounds, because it is the premise for almost everything else. Immutable segments can be memory-mapped without synchronisation, sent to a network socket without being copied through user space, compressed once, deleted whole, or shipped to object storage. The active segment can do none of those things, which is why it is excluded from compaction, from deletion and from tiering.

What makes a segment roll

A roll closes the active segment and opens a new one at the next offset. Four conditions trigger it, and knowing all four saves a lot of confusion about why a segment did or did not close.

The first is size: the segment reaches log.segment.bytes, 1,073,741,824 or 1 GB by default. The second is age: the segment's first record is older than log.roll.ms, which if unset derives from log.roll.hours at 168 hours, or seven days. The third is that the segment's offset index or time index has filled, bounded by log.index.size.max.bytes at 10,485,760 or 10 MB. The fourth is that appending the incoming batch would push the offset range beyond what the index can express, which is the subject of the next section.

The age condition is the one that surprises people. A low-volume topic with 1 GB segments and 7-day rolls produces one segment per week regardless of how little data it holds, and because retention deletes whole segments, a topic configured with one-day retention can hold data for eight days: seven while the segment is still active and therefore undeletable, and then up to one more before the now-closed segment ages out. If you want retention to be approximately honest on a low-volume topic, the segment roll interval must be shorter than the retention period, which usually means setting segment.ms on the topic rather than leaving the week-long default.

log.roll.jitter.ms exists for a related reason. Without it, segments created at the same time, which happens on every topic created in one script, roll at the same time, producing synchronised bursts of file creation, index flushing and retention work across the whole cluster. The jitter spreads them.

Why retention is granular

Retention deletes segments, never records. A segment becomes eligible when every record in it is outside the retention window, which is determined by the segment's largest timestamp, and the active segment is never eligible. This has three practical consequences worth stating plainly.

A topic's on-disk size is always somewhat larger than retention implies, by up to one segment per partition plus whatever the active segment holds. For a 1 GB segment size and 200 partitions that is up to 200 GB of unaccounted-for data per replica, which is a real number in a capacity plan and is the reason the arithmetic in the sizing lesson should use segment-rounded figures.

Retention has a resolution equal to the segment size in time. If a segment covers four hours of traffic, retention can only act in four-hour steps, so a one-hour retention setting on such a topic is not meaningfully different from a four-hour one. Shrinking segments buys finer retention and costs more files, more index memory and more open file handles.

And deletion is cheap. Removing a segment is unlinking three or four files, not rewriting a data structure, which is why retention has essentially no performance impact while compaction, which does rewrite, is a background process with its own throttles and its own lesson.

Choosing a segment size

The default 1 GB is a reasonable middle and the two directions it can move have clear motivations.

Smaller segments, say 128 or 256 MB, give finer retention resolution, faster recovery after an unclean shutdown since recovery works per segment, finer-grained tiering, and quicker compaction cycles. They cost more files: a 500 GB partition at 128 MB per segment is about 4,000 segments and therefore about 12,000 files once indexes are counted, and each open segment consumes a file descriptor and index memory. Multiply by partitions per broker and the nofile limit becomes a real constraint, which is why a Kafka broker is routinely configured with a file descriptor limit of 100,000 or more.

Larger segments, 4 GB or beyond, reduce file count and metadata overhead at the cost of everything in the previous paragraph. They also run into a hard ceiling described in the next section, so 2 GB is effectively the upper bound regardless of what you would prefer.

The honest default advice is to leave log.segment.bytes alone and set segment.ms per topic to something shorter than the retention period on any low-volume topic, because that is the setting whose default is wrong far more often.

What a partition directory actually contains

Opening a partition directory makes the structure concrete, and every file in it has been named somewhere above or will be below.

1orders-7/
2  00000000000000000000.log            closed segment, 1.0 GB
3  00000000000000000000.index          sparse offset index, 2.0 MB
4  00000000000000000000.timeindex      sparse time index, 3.0 MB
5  00000000000000000000.snapshot       producer state as of this segment
6  00000000000008393102.log            active segment, growing
7  00000000000008393102.index          preallocated to 10 MB
8  00000000000008393102.timeindex      preallocated to 10 MB
9  leader-epoch-checkpoint             epoch to start-offset mapping
10  partition.metadata                  topic ID under KRaft

Three of those are derived and recomputable, the .index, .timeindex and .snapshot. Two are durable metadata that is as important as the data, leader-epoch-checkpoint and partition.metadata. And one is the data. Knowing which category a file falls into tells you immediately whether losing it is an inconvenience or an incident, and the answer is that only the .log files and the epoch checkpoint matter.

You may also see files with the suffixes .deleted, .swap and .cleaned. The first is a segment that retention has renamed and will unlink after file.delete.delay.ms, 60,000 by default, which exists so that a fetch already in flight against that file is not failed by its disappearance. The second and third are compaction's working files, discussed in the compaction lesson. Stale .swap or .cleaned files left by a crash are cleaned up on startup.

Preallocation, flushing, and what Kafka does not do

Two settings in this area are worth knowing mostly so that you leave them alone.

log.preallocate, false by default, preallocates the full segment file on creation rather than letting it grow. It helps on filesystems where extension is expensive and on Windows; on ext4 or XFS the default is fine and preallocation mainly makes disk usage reporting confusing.

The flush settings, log.flush.interval.messages and log.flush.interval.ms, both effectively unbounded by default, control how often Kafka calls fsync on a segment. The default is to never explicitly fsync and to let the operating system decide, which is the single most counterintuitive durability decision in Kafka's design. The reasoning is the one the replication lesson develops: the durability guarantee comes from acks=all across replicas on independent machines, not from bytes reaching one machine's platter, and an fsync on the append path would cost most of the write throughput for a guarantee that is weaker than the one replication already provides. Setting these to force frequent flushes is almost always a mistake, and the correct lever for durability is replication factor and min.insync.replicas.