Sizing a Kafka cluster is arithmetic, and the arithmetic is simple enough that there is no excuse for not doing it. What makes clusters wrong is not the difficulty of the calculation but the three multipliers people leave out of it.

The disk formula

Five terms, and the last two are the ones omitted.

Steady-state disk is the ingress rate times the retention period times the replication factor, divided by the compression ratio, plus headroom.

Written out for a concrete case: 50,000 records a second at 1 KB each is 50 MB a second of uncompressed ingress. Seven days of retention, the log.retention.hours default of 168, is 604,800 seconds, so 30.2 TB of logical data. Replication factor 3 makes it 90.7 TB on disk.

Compression is the term that rescues this. Typical JSON-shaped records compress 4 to 6 times with lz4 or zstd, so a ratio of 5 brings 90.7 TB down to 18.1 TB. That is not a rounding adjustment, it is the difference between a six-broker cluster and a thirty-broker one, which is why the compression type is a capacity decision and not a client detail.

Headroom is the fifth term and should be at least 30 percent, which gives 23.6 TB of provisioned disk. The reason is not caution for its own sake: a broker that fills its disk stops accepting writes for every partition it leads, and recovering requires either deleting data or moving partitions, both of which take time you do not have at that moment.

Why compression has to be measured, not assumed

Because the ratio varies by more than any other term in the formula.

Compression happens on the producer, controlled by compression.type, and the broker stores the compressed batch as it arrives when its own compression.type is producer, which is the default. So the ratio achieved is a property of the producer's batching, not of the broker.

Batch size is what determines it. Compression operates on a whole record batch, so a producer with linger.ms at its default of 0 sends batches of one or two records and compresses almost nothing, while the same producer with linger.ms at 20 accumulates batches of hundreds and gets the full ratio. A cluster sized on an assumed ratio of 5 and fed by producers with default linger will be out by a factor of three.

The practical check is to produce a representative sample through the actual producer configuration, read the resulting topic's on-disk size with kafka-log-dirs.sh --describe, and divide. That measurement takes an hour and is worth more than any estimate.

zstd compresses meaningfully better than lz4 for structured records at a higher CPU cost on the producer, and gzip is almost always the wrong choice, costing far more CPU for a small gain over zstd.

Retention, and the fact that it is per topic

Which turns one number into a distribution.

log.retention.hours is a broker default of 168 hours, and every topic can override it with retention.ms. On a real cluster the retention is therefore a mix, and summing the formula per topic rather than applying one retention to the whole ingress is the only way to get a useful total.

log.retention.bytes, default -1 meaning unlimited, is a per-partition size cap and is the safety valve worth setting. With retention by time alone, a topic whose volume grows tenfold silently consumes ten times the disk and the first symptom is a full broker. With a byte cap, the topic starts losing its oldest data instead, which is a far better failure for most topics and a terrible one for a few.

Compacted topics do not obey either setting in the same way, since their size is driven by the key cardinality rather than by the message rate. A compacted topic's steady-state size is the distinct key count times the average record size times the replication factor, plus whatever the cleaner has not yet processed, and the dirty portion can be large on a cluster whose log.cleaner.threads, default 1, is saturated.

So a cluster's disk total is three sums: time-retained topics by the formula, byte-capped topics by their caps, and compacted topics by cardinality. Treating them as one number is where estimates go wrong by large factors.

Segments, files and the per-broker limits

Because disk capacity is not the only disk-related limit.

log.segment.bytes defaults to 1,073,741,824, one gigabyte, and each segment is three files on disk: the .log, the .index and the .timeindex. A broker holding 18 TB across 1 GB segments therefore has around 18,000 segments and 54,000 open file descriptors for data alone, before sockets.

The default ulimit -n of 1,024 on many distributions is therefore catastrophically low, and 100,000 is the usual recommendation. The failure mode is a broker that runs for weeks and then fails to roll a segment or accept a connection, which reads as a mysterious outage rather than as a limit.

log.roll.hours, default 168, forces a segment roll even on a low-volume partition, which matters because retention deletes whole segments and never parts of one. A partition receiving 1 MB a day with 1 GB segments and time-based retention keeps data for far longer than its retention setting, because the active segment is never eligible for deletion.

That is the usual explanation for a topic that refuses to shrink despite a short retention, and the fix is a smaller segment.bytes or a shorter segment.ms on that topic rather than anything about retention.

Internal topics, which nobody puts in the estimate

Three of them, and one is not small.

__consumer_offsets has 50 partitions by default from offsets.topic.num.partitions, is compacted, and holds one record per group per partition per commit. Its steady-state size is small because compaction collapses it to the latest offset per key, but its dirty size is not: a group committing every 5 seconds across 500 partitions writes 100 records a second, and until the cleaner runs, all of them are on disk. A cluster with a hundred such groups writes 10,000 records a second to a 50-partition topic, which is real traffic that appears in no topic inventory.

__transaction_state has 50 partitions from transaction.state.log.num.partitions and behaves the same way, with transactional.id.expiration.ms at 604,800,000 governing cleanup. A platform with many short-lived transactional producer ids accumulates state for a week.

__cluster_metadata under KRaft is the metadata log itself, with a record per partition and per configuration change, and metadata.log.max.record.bytes.between.snapshots at 20,971,520 governing how often it snapshots. It is small in bytes and large in consequence, since every broker replays it at startup.

None of these appears in a capacity request, and together they are typically a few percent of disk and a noticeable share of request rate. The reason to know it is that a cluster sized exactly to its business topics has no room for its own bookkeeping.

The whole calculation, written down

Because a sizing conversation goes better with the arithmetic in front of everyone rather than in someone's head.

1def size_cluster(records_per_sec, avg_record_bytes, retention_days,
2                 replication_factor, compression_ratio, headroom=0.30,
3                 disk_per_broker_tb=4.0):
4    ingress = records_per_sec * avg_record_bytes          # bytes/sec
5    seconds = retention_days * 86400
6    logical = ingress * seconds                           # before replication
7    on_disk = logical * replication_factor / compression_ratio
8    provisioned = on_disk * (1 + headroom)
9    brokers = provisioned / (disk_per_broker_tb * 1e12)
10    return {
11        'ingress_mb_s':   round(ingress / 1e6, 1),
12        'logical_tb':     round(logical / 1e12, 1),
13        'replicated_tb':  round(on_disk * compression_ratio / 1e12, 1),
14        'on_disk_tb':     round(on_disk / 1e12, 1),
15        'provisioned_tb': round(provisioned / 1e12, 1),
16        'min_brokers':    max(3, -(-int(brokers * 100) // 100)),
17    }
18
19size_cluster(50_000, 1024, 7, 3, 5.0)
20# ingress_mb_s 51.2, logical_tb 31.0, replicated_tb 92.9,
21# on_disk_tb 18.6, provisioned_tb 24.1, min_brokers 7

The thing to notice is the ordering of the terms rather than the result. Replication multiplies, compression divides, and headroom multiplies again, so the three of them together move the answer from 31 TB to 24 TB by way of 93 TB. Any one of them left out gives a number that is wrong by a factor rather than by a percentage, and the compression term is the one with real uncertainty in it because it is measured rather than chosen.

The floor of 3 brokers in that function is not arithmetic, it is the replication factor: a cluster cannot host replication factor 3 on fewer than three brokers, so disk may say two and the durability requirement says three.

Disk type, which is a throughput decision

Briefly, because the reasoning is counterintuitive.

Kafka's write pattern is sequential appends and its read pattern is mostly sequential from the page cache, which is why Kafka ran well on spinning disks for years and why the usual advice was that it does not need SSDs.

That holds while consumers read recent data. It stops holding the moment a consumer reads from the beginning of a large topic, which turns the workload into random reads competing with sequential writes, and on spinning disks that collapses throughput for every producer on the broker.

So the disk decision is really a question about whether historical reads happen: a cluster serving only real-time consumers can use cheap bulk storage, and one where consumers routinely replay needs SSDs or a lot more page cache. Tiered storage changes this calculation substantially, moving the historical reads to object storage entirely.

RAID 10 versus JBOD is the other question, and Kafka's own replication already provides redundancy, so JBOD with multiple log.dirs gives more usable capacity for the same disks. The cost is that a single disk failure takes down the partitions on that disk rather than nothing, which is acceptable precisely because replication exists.