Most data systems maintain their own cache. Postgres has shared buffers, MySQL has the InnoDB buffer pool, Cassandra has a key cache and a row cache. Kafka has nothing of the kind, and that absence is a design decision rather than an omission.

What the broker does not keep in memory

A Kafka broker holds no cache of record contents. When a produce request arrives, the broker appends the batch's bytes to the active segment's file and that is the end of its involvement with those bytes. When a fetch request arrives, it resolves an offset to a byte range and asks the kernel to send that range. At no point is there a broker-managed structure holding records for fast access.

What makes this work is that the kernel already has one. Every write to a file lands first in the page cache, the kernel's cache of file pages in RAM, and stays there until memory pressure evicts it. A record written a moment ago is in RAM, so a consumer reading the tail is served from RAM, and the broker gets a memory cache without writing one.

Why an application cache would be worse here

Three reasons, and they are specific to this workload rather than general.

The first is double buffering. A broker-managed cache holding records that the page cache also holds wastes half the memory spent on caching, and since the broker writes through the filesystem, there is no way to avoid the page cache copy. The only way to have one copy in an application cache is direct IO, which Kafka does not use, and which would mean reimplementing read-ahead, write-back and eviction in the broker.

The second is garbage collection. A JVM heap holding tens of gigabytes of cached records is a JVM with tens of gigabytes to trace, and the pause behaviour of a large heap full of medium-lived objects is exactly the pathological case for a generational collector. The page cache lives outside the heap entirely, so it costs the collector nothing regardless of size, and a broker caching 200 GB of data can run with a 6 GB heap.

The third is that the kernel's eviction policy is already right for this workload. Access is overwhelmingly recent, the kernel's approximate least-recently-used behaviour matches that, and read-ahead on sequential access is something the kernel does well and an application cache would have to reimplement. There is no insight about Kafka's access pattern that the broker has and the kernel lacks.

Heap sizing, and the mistake of making it large

The practical consequence is that broker heap should be small. Six gigabytes is the usual recommendation, and brokers serving very large clusters sometimes go to eight or ten, but the pressure is always downward rather than upward, which is the opposite of the instinct most people bring from other JVM services.

What is actually in the heap is request objects in flight, the accumulated metadata for topics and partitions, producer state for idempotence, index objects, the replica fetcher state, and the network buffers. All of that scales with partition count, connection count and producer count rather than with data volume, which is why heap sizing is a function of cluster shape and not of throughput.

Every gigabyte given to the heap is a gigabyte taken from the page cache, so an oversized heap directly reduces the amount of data that can be served without disk access. A broker with 64 GB of RAM and a 32 GB heap has halved its cache for no benefit, and the symptom is increased disk read throughput and higher fetch latency on a cluster that was previously comfortable.

The collector choice is G1, the default since Kafka 2.0 on modern JVMs, and it needs no special tuning at these heap sizes. MaxGCPauseMillis at 20 and InitiatingHeapOccupancyPercent at 35 are the commonly cited settings, and the honest advice is to set the heap size correctly and then leave the collector alone, because GC tuning is almost never where a Kafka performance problem lives.

Sizing the page cache against consumer lag

The useful question is not how much RAM a broker should have in the abstract but how much it needs to serve its consumers without disk reads, and that has a clean answer: the page cache must hold the data written during the interval that consumers lag by.

Work an example. A broker ingests 100 MB/s across its leader partitions. Consumers are typically within two seconds of the tail, so the working set for the common case is 200 MB, which any broker has. But one consumer group runs a batch job that polls every five minutes, so when it polls it reads 30 GB of data that was written since its last poll. If the broker has 32 GB of RAM with a 6 GB heap, leaving roughly 25 GB of usable page cache, that batch consumer's read does not fit, and its fetches come from disk while evicting everything the other consumers were being served from.

The arithmetic generalises to a rule: take the maximum lag in seconds across all consumer groups, multiply by the ingest rate, and compare against available page cache. Replication counts too, since followers are consumers, and a follower that is catching up is a consumer with large lag.

This is also why consumer lag is a capacity metric and not only a correctness metric. A consumer group whose lag grows from two seconds to five minutes has not merely fallen behind; it has changed the broker's workload from cache-resident to disk-bound, and the effect is visible to every other consumer on that broker.

What ends up in the page cache besides records

Two other things, and both are small but worth accounting for.

The index and time index files are memory-mapped, so their pages occupy page cache like any other file pages. At roughly 2 MB of offset index and 3 MB of time index per gigabyte of log, a broker holding 4 TB has about 20 GB of index files, though only the recently accessed portions are resident. This is the one place where lowering log.index.interval.bytes has a memory cost worth noticing.

And the producer state snapshots are read on startup and then held in heap rather than page cache, so they belong to the heap sizing discussion rather than this one. Their size is proportional to the number of distinct producer IDs, which is the reason the per-request-producer antipattern shows up as heap pressure.

The heap, itemised

Saying that heap scales with cluster shape rather than data volume is only convincing if you can say what is in it, so here is the itemisation for a broker with 500 leader partitions and 2,000 client connections.

Per-partition metadata, the Log object with its segment map, the leader epoch cache and the in-memory index references, is a few tens of kilobytes per partition, so 500 partitions is low single-digit megabytes. The producer ID map is per-partition and holds an entry for every active producer that has written to it, roughly a hundred bytes each, so a thousand well-behaved producers across 500 partitions is tens of megabytes and a million distinct producer IDs is gigabytes, which is the mechanism behind the per-request-producer antipattern being a heap problem.

In-flight requests are the volatile part. A produce request's batch sits in heap while being validated, bounded by socket.request.max.bytes at 104,857,600 multiplied by the number of concurrently processing requests, which queued.max.requests at 500 bounds. That product is far larger than any sane heap, which is why the practical bound is the much smaller typical request size rather than the configured maximum, and why a client sending unusually large requests can produce heap pressure that no partition count explains.

Network buffers are direct rather than heap memory, and MaxDirectMemorySize defaults to the same value as -Xmx, which is worth knowing because the two are accounted separately and a broker's resident size exceeds its heap by that amount plus the mapped index files.

Why not direct IO

The alternative to leaning on the page cache is to bypass it with O_DIRECT and manage a buffer pool in the broker, which is what a relational database does. Kafka does not, and the reason is specific rather than general.

Direct IO means reimplementing read-ahead, eviction and write-back inside the broker, which is a large amount of code to write in exchange for a policy no better than the kernel's for this access pattern. But the decisive problem is that direct IO and sendfile are mutually exclusive: sendfile transfers from the page cache to the socket, so a broker holding its data in its own buffers rather than in the page cache would have to copy those buffers into the socket itself, which reintroduces exactly the copies the design exists to avoid.

So the choice is not between two caching strategies but between two architectures: own the cache and copy the bytes, or borrow the kernel's cache and never touch them. A database makes the first choice because it must transform data on the way out, and Kafka makes the second because it must not.