Tiered storage, specified in KIP-405 and production-ready since Kafka 3.9, moves old log segments to object storage and serves reads from there transparently. It is the largest change to Kafka's storage model since the log was designed, and understanding what it is for requires being precise about which problem it addresses, because the obvious answer is only part of it.

The coupling it breaks

Without tiering, a Kafka broker's local disk holds every byte within the retention window, which couples two things that have no reason to be coupled: how much data you keep, and how much compute you run.

A cluster needing 500 TB of retention needs enough brokers to hold 500 TB locally, times the replication factor, so 1.5 PB of attached storage. If the throughput requires only six brokers, you are buying 250 TB per broker, which means either very large instances with mostly idle CPU or many more brokers than the workload needs. Either way the bill is set by the retention requirement rather than by the work.

Tiering decouples them. Local disks hold a small recent window, object storage holds the rest, and broker count is chosen for throughput. The same 500 TB becomes perhaps 10 TB of local storage across six brokers plus 500 TB in object storage at roughly a tenth to a twentieth the cost per byte, and crucially without the replication multiplier, since object storage does its own redundancy.

The operational problem, which matters more

The cost argument is the one usually made and it is the weaker one. The stronger argument is about what large local disks do to cluster operations.

Consider adding a broker to a cluster where each broker holds 20 TB. The new broker must receive its share of the data, which means copying terabytes over the network while the cluster serves traffic, at a rate slow enough not to degrade it. That is hours to days, during which the cluster is in a transitional state and partially under-replicated. The same applies to replacing a failed broker, to a rolling restart where a broker misses enough data to need substantial catch-up, and to any rebalance.

With a 1 TB local window, those operations become twenty times smaller. A new broker is in service in minutes rather than days, a failed broker is replaced quickly, and a reassignment is a small operation. This changes a cluster from something you resize carefully a few times a year into something that can be resized on demand, which is a qualitative change rather than a cost saving.

The cross-tenant problem it removes

The third benefit is the one that follows directly from the page cache lesson, and it is the one most likely to be felt on a bad day.

Without tiering, a consumer reading historical data reads it from the broker's local disk, which evicts the page cache that live consumers depend on. One batch job starting at 02:00 degrades every other consumer on those brokers for as long as it runs, and the only remedies are to throttle it, to reschedule it, or to accept it.

With tiering, that historical read is served from object storage. It does not touch the local log, does not read the local disk, and does not evict page cache. The page cache then holds the local retention window, which is a configured quantity rather than a consequence of the worst-behaved consumer. The cross-tenant interaction is gone, and the sizing rule from the page cache lesson collapses from "maximum consumer lag times ingest rate" to "the local retention window".

What it does not solve

Three things, and being clear about them prevents the disappointment that follows overselling.

It does not make historical reads fast. An object storage read has tens of milliseconds of latency against a local disk read's single-digit milliseconds, and the throughput for a single stream is lower. A consumer reading from the remote tier will be slower, sometimes substantially, and the compensation is that it is no longer anyone else's problem.

It does not reduce write cost or ingest capacity requirements. Every byte still lands on local disk first and is still replicated, so the write path is unchanged and the broker count still has to handle the ingest rate.

And it does not apply to compacted topics. Tiering requires immutable segments and compaction rewrites them, so cleanup.policy of compact and remote.storage.enable are mutually exclusive. This is a real limitation because state topics are exactly the topics with long-lived data, and the workaround is the unsatisfying one of keeping them local.

When it is worth enabling

A short test, because tiering adds operational surface and is not free to run.

If retention is driven by a requirement longer than a few days and the data volume is meaningful, yes, because the cost and operational arguments both apply. If the cluster has consumers that regularly read historical data and degrade their neighbours, yes, because removing that interaction is worth it independently of cost. If broker count is set by storage rather than by throughput, yes, because that is the coupling it exists to break.

If retention is hours, no, because there is nothing to tier. If every consumer reads the tail, no, because the remote tier would be write-only and you would be paying for an archive nobody reads, which a sink connector does more cheaply. And if the topics in question are compacted, it is not available.

The arithmetic, worked

Numbers make the coupling concrete, so here is a cluster worth sizing both ways.

Ingest is 150 MB/s of post-compression data, retention is 30 days, replication factor is 3. Thirty days at 150 MB/s is about 389 TB of logical data, which at RF 3 is 1.17 PB of attached storage. If you size brokers at 30 TB of local disk each, which is already large, that is 39 brokers. The throughput, 150 MB/s in and perhaps 450 MB/s out, needs maybe six brokers of CPU and network. So 33 of the 39 exist to hold disks.

Tiered: set local retention to 12 hours, which is 150 MB/s times 43,200 seconds, about 6.5 TB of logical data, 19.4 TB at RF 3, so roughly 3 TB of local disk on each of six brokers. Object storage holds the 389 TB once, not three times. The storage volume you pay for drops from 1.17 PB of block storage to 389 TB of object storage, which at roughly a fifth the per-byte price is a factor of fifteen, and the broker count drops from 39 to six.

The part of that arithmetic worth noticing is that the replication factor disappears entirely from the retained volume. Object storage does its own redundancy, usually to a higher durability figure than three local replicas, so a tiered segment is stored once. That factor of three applies before any price difference, and it is why the saving is larger than the per-byte comparison suggests.

Where the saving does not materialise

The counter-cases are worth naming because the arithmetic above is the best case.

If local retention has to stay long for a reason other than consumers, usually replica catch-up, the local footprint is larger and the broker count does not fall as far. A cluster that needs two days local rather than twelve hours has four times the local footprint, which is 12 TB per broker rather than 3, and the broker count might be eight rather than six.

If a large fraction of the data is on compacted topics, that fraction cannot be tiered at all and stays entirely local, so the saving applies only to the remainder. In a cluster that is half state topics, the arithmetic halves, which can move the decision.

And if the read pattern is heavy on history with small fetches, object storage request charges eat into the storage saving. The charges are per request rather than per byte, so a consumer reading 100 TB in one-megabyte fetches generates on the order of a hundred million requests, and at typical pricing that is a real number rather than a rounding error. The storage price is the favourable part of the model and the request price is the part to measure.