A log that only grows is not a storage system, so every Kafka topic has a cleanup policy. There are two, delete and compact, and they answer different questions: deletion answers "how far back can I read", compaction answers "what is the current value for this key". Getting the wrong one is one of the more expensive topic-level mistakes because changing it later is not free.

Three settings, and which one wins

cleanup.policy defaults to delete, and under that policy three limits can each make a segment eligible for removal.

retention.ms, broker default log.retention.hours of 168, is the age limit. retention.bytes, broker default log.retention.bytes of minus one meaning unbounded, is the size limit per partition. And log.retention.check.interval.ms, 300,000, is how often the broker looks.

Both limits apply simultaneously and a segment is eligible if either says so, which is the sensible reading but worth stating because people often assume one overrides the other. The practical consequence is that retention.bytes is a safety valve: set the time limit to the business requirement and the size limit to what the disk can actually hold, and a traffic spike then costs you history rather than the broker.

The per-partition framing of retention.bytes is the detail that causes incidents. A topic with 100 partitions and retention.bytes of 10,737,418,240 holds up to 1 TB, not 10 GB, and multiplied by replication factor three it occupies 3 TB of cluster storage. Writing the setting as a per-partition number and the capacity plan as a per-topic number is how a cluster fills up with everyone having configured things correctly.

Why retention is segment-granular, and what that costs

Retention never deletes a record. It deletes a segment, and only when every record in that segment is eligible, which means eligibility is determined by the largest timestamp in the segment.

So the effective retention is always longer than configured, by up to one segment's worth of time. On a topic with one-hour retention and one-gigabyte segments that take six hours to fill, data is retained for seven hours, and the surprise version of this is a topic with a day of retention whose low volume means a segment takes a week to roll, so nothing is ever deleted. The fix is segment.ms, which forces a roll on time regardless of size, and it is the single most commonly missing setting on a low-volume topic with short retention.

The storage consequence is that unaccounted overhead is roughly the segment size times the partition count. At 200 partitions and one-gigabyte segments that is up to 200 GB per replica that your capacity model did not include, and at 2,000 partitions it is 2 TB, which is why high partition counts argue for smaller segments independently of any other reason.

What deletion actually does

A segment becoming eligible does not immediately free space. The broker renames the files with a .deleted suffix and schedules the actual unlink after file.delete.delay.ms, 60,000 by default.

The delay exists because a fetch may be in flight against that segment, and a reader holding an open file descriptor keeps the inode alive even after the name is gone, so the rename makes the segment invisible to new readers while letting existing ones finish. This is also why a du immediately after a retention pass shows the space still occupied, which looks like retention failing and is not.

The other consequence is that the partition's log start offset advances. A consumer whose committed offset is now below the log start offset gets OFFSET_OUT_OF_RANGE on its next fetch, and what happens next depends on auto.offset.reset: latest silently skips everything between, earliest reprocesses from the new start, and none raises. A consumer group that was down for longer than retention is the common cause, and the silent data loss under latest is the reason that default deserves a deliberate decision rather than inheritance.

The log start offset can also move on purpose

Two mechanisms advance the log start offset without retention being involved, and both are worth knowing because they look like data loss when encountered unexpectedly.

The admin API supports deleteRecords, which sets the log start offset to a given offset and makes everything before it unreadable. The segments are then removed by the normal mechanism. This is the supported way to delete data from a Kafka topic, which matters for deletion requests under data protection regulation, and it works at partition and offset granularity rather than at record granularity, so deleting one user's record means deleting everything before it in that partition.

And kafka-delete-records.sh is the command-line front end for the same operation, taking a JSON file of partitions and offsets. The operational caution is that it is not reversible and it does not ask twice.

Choosing retention, and the question people skip

The retention decision is usually framed as a storage cost question and the more useful framing is a recovery question: how far back would a consumer need to read to rebuild its state after a bug?

For a topic feeding a derived store, the answer is "to the beginning", which argues for compaction rather than a long deletion window. For an event stream feeding an analytics pipeline, the answer is the longest plausible outage of that pipeline plus the time to notice, which is usually days rather than hours and is why seven days is a reasonable default. For a topic carrying short-lived operational signals, hours are fine.

The arithmetic is simple and worth doing explicitly: retention in seconds times ingest rate per partition times partition count times replication factor is the cluster storage for that topic. A topic ingesting 20 MB/s with seven days of retention and replication factor three needs about 36 TB, and a team asking for thirty days is asking for 155 TB, which is a conversation worth having before the setting is applied rather than after.

Three sources of slack, and what they add up to

It is worth collecting the delays in one place, because a question like "how long after the limit is the data actually gone" has three terms and people usually know only one of them.

The segment must roll, which is up to segment.ms or until it reaches log.segment.bytes, whichever comes first. The retention check must run, which is up to log.retention.check.interval.ms at 300,000, so five minutes. And the unlink must happen, which is file.delete.delay.ms at 60,000.

So on a topic with one-gigabyte segments, one hour of retention and no segment.ms, the honest answer is "one hour, plus however long a segment takes to fill, plus up to six minutes". On a high-volume partition filling a segment in ten minutes that is about 76 minutes. On a low-volume partition it is unbounded.

This matters operationally in one specific situation: a disk is filling and someone lowers retention.ms to recover space. The space does not come back for at least the check interval plus the delete delay, and it only comes back at all for segments that have rolled, so the active segment's data stays regardless. Lowering retention is a slow lever, and the fast levers are a partition reassignment off the broker or kafka-delete-records.sh.

Retention, replicas, and the config that is not actually shared

Retention runs independently on every replica. Each broker evaluates its own copy of the partition against the configuration and deletes its own segments, so there is no coordination and no leader involvement in deletion.

That works because the configuration is topic-level and therefore identical everywhere, with one exception that bites. If a topic has no explicit retention.ms, it inherits the broker's log.retention.hours, and if brokers have different values for that, replicas of the same partition retain different amounts of data. The symptom is a consumer whose readable range changes when leadership moves, which is one of the stranger things to debug, and the fix is to set retention explicitly at the topic level on every topic rather than relying on broker defaults.

The log start offset itself is replicated: the leader's value is carried in fetch responses and followers do not advance past it, so a follower will not make data unreadable that the leader still serves. Deletion of the underlying segments is still local, which is why the disk usage of three replicas of the same partition is similar but not identical.