Most data structures are defined by the questions they answer. A hash map answers "what is the value for this key." A B-tree answers "which keys fall in this range." A priority queue answers "what is the smallest element." A log answers exactly one question: "what happened after position N." That narrowness is not a weakness Kafka works around. It is the single property that every other Kafka behaviour is derived from, and if you understand why, most of the rest of this course stops feeling like a pile of configuration settings.

A log is an append-only, totally ordered sequence of records. Its entire interface is three operations:

• append(record) returns the position the record landed at
• read(offset, maxBytes) returns records starting at that position, in order
• truncate(offset) discards everything before that position

There is no update. There is no delete-by-key. There is no insert-in-the-middle. A record, once appended, is byte-for-byte immutable until the whole prefix containing it is dropped. Kafka exposes exactly this interface over the network, and the parts of Kafka that look like exceptions (compaction, transactions, tiered storage) are all implemented on top of this interface rather than by weakening it.

Immutability is what makes the structure cheap

Think about what a mutable record would cost. If a record could be rewritten in place, then its size could change, which means the broker would need free-space management inside the file. Readers would need a lock or a snapshot to avoid seeing a half-written record. The file's byte offsets would stop being stable, so any index pointing into the file would need updating on every write. Replicas would need to agree not just on "which records exist" but on "which version of each record is current," which is a consensus problem per record rather than per position.

Immutability deletes all of that work. Because a record never changes, a byte position in the file is a permanent name for it. Because a byte position is permanent, an index can be a simple append-only array. Because the index is append-only, a follower can replicate the log by copying bytes rather than by applying operations. Because the bytes are identical on every replica, a replica's correctness check reduces to comparing one number: how far it has copied.

A partition is the log; a topic is a namespace

In Kafka, the log is the partition. A partition lives on one leader broker at a time, is replicated to a configured number of followers, and has its own independent sequence of offsets starting at 0. A topic is a name attached to a set of partitions. The topic itself has no ordering and no offsets. This distinction causes more production confusion than any other single fact about Kafka: engineers say "Kafka preserves order" when what Kafka preserves is order within a partition, which is a much narrower promise.

Offsets are assigned by the partition leader at append time. They are 64-bit signed integers, they start at 0 for a new partition, and within a partition they are gapless and monotonically increasing: if the log contains offsets 0 through 1,000, every integer in that range names exactly one record. There is one wrinkle worth knowing early, because it bites people reading offsets in production: when transactions are in use, the leader also writes control records (commit and abort markers) into the log, and those consume offsets that no consumer application ever sees. A read_committed consumer can therefore jump from offset 904 to offset 906 with nothing missing. The log is still gapless; the gap is a record you are not allowed to read.

Writers never coordinate with readers

Here is the property that matters at scale. An append needs to know one thing: where the tail is. A read needs to know one thing: the offset to start from. Those two pieces of state are disjoint. A writer can append while a thousand readers are reading, and no reader's progress affects the writer, and no writer's progress affects any reader except by making more data available.

Compare that with a classic message queue. In a queue, a read is destructive: the broker hands a message to a consumer and removes it, or marks it in-flight and removes it on acknowledgement. Because the read mutates shared state, every delivery requires the broker to coordinate: pick a consumer, update the message's state, handle the case where the consumer dies holding it. The broker owns each consumer's progress, so the broker's write path is on the critical path of every delivery.

In a log, the broker owns none of that. The consumer owns its own position. Delivery is "copy bytes from offset N," which is a read against an immutable file. This is why a Kafka broker can serve far more consumers than its disk write throughput would suggest: consumers are not doing writes.

Writing the structure out

The whole abstraction fits in about twenty lines, which is worth reading once because it makes the later chapters concrete. This is not how Kafka is implemented, but every behaviour in it has a counterpart in the real broker.

1class Log:
2    def __init__(self):
3        self.records = []        # a real log is a list of segment files
4        self.base = 0            # offset of records[0]; rises on truncate
5
6    def append(self, value):
7        offset = self.base + len(self.records)
8        self.records.append(value)   # never mutates an existing entry
9        return offset
10
11    def read(self, offset, max_records):
12        if offset < self.base:
13            raise OffsetOutOfRange(offset, self.base)
14        start = offset - self.base
15        return self.records[start:start + max_records]
16
17    def truncate(self, new_base):    # retention, not deletion of a record
18        self.records = self.records[new_base - self.base:]
19        self.base = new_base

Two details in that code are load-bearing. First, append returns the offset rather than taking one, because the leader is the only thing allowed to name a position; this is what makes a single partition totally ordered without any distributed agreement on ordering. Second, read raises on an offset below base rather than silently clamping. Kafka does exactly this, with the error OFFSET_OUT_OF_RANGE, and the consumer's auto.offset.reset setting (earliest, latest, or none) decides what happens next. A consumer that fell far enough behind that retention deleted its position has not lost its place by accident; it has asked for a record that no longer exists, and whether it then skips data or stops is a policy you choose.

Total order comes from having exactly one appender

A partition is totally ordered, and total order in a distributed system is normally expensive. Lamport clocks give you a partial order. Vector clocks give you a partial order plus the ability to detect concurrency. Getting a genuine total order usually means consensus, and consensus means multiple rounds of messages per decision.

Kafka sidesteps all of it with an unglamorous trick: a partition has exactly one leader at a time, and only the leader appends. Ordering is then not agreed, it is observed. Whatever sequence the leader happened to write is the order, and because the leader assigns the offsets as it writes, no second party ever needs to be consulted about what position a record occupies. Two producers racing to append do not negotiate; the leader's single-threaded append path decides, arbitrarily and instantly.

This is why the system that elects leaders is a separate, much more careful system. Kafka's ordering guarantee is only as good as the guarantee that two brokers never both believe they lead the same partition at the same time, and the whole of the cluster coordination section of this course exists to make that true. It is also why ordering is per partition and can never be global: a global order would need a single appender for the entire cluster, which is the throughput ceiling the partition model exists to remove.

What Kafka adds to "just a file"

If the log is this simple, a reasonable question is what the broker is for. Four things, and each one is a chapter later in this course.

Replication. A file on one machine dies with the machine. Kafka keeps N copies, tracks which copies are current, and decides which one takes over, and the subtlety is entirely in the last two.

Offsets as a protocol. A byte offset into a file is not a stable public name, because it changes if the segment layout changes and it means nothing across replicas. A Kafka offset is a logical position that every replica agrees on, and the mapping from logical offset to byte position is an index the broker maintains per segment.

Retention as a policy rather than an operation. Nobody deletes records. You declare a rule, by time, by size, or by key with compaction, and the broker enforces it in the background by dropping whole segments. This is why deleting one record is not a thing you can do, and why GDPR erasure against Kafka is a design problem rather than a command.

Group membership. A single reader of a file needs nothing. A set of cooperating readers that must not overlap, must cover every partition, and must survive one of their number dying needs a membership protocol, failure detection and a reassignment algorithm. That machinery is most of what people mean when they say Kafka is complicated, and none of it is about the log.

The log's shape is not just conceptually tidy. It happens to be the exact access pattern that storage hardware is fastest at, and it stays the fastest pattern across three generations of storage technology that are otherwise nothing alike. This section is the hardware argument, because without it the design looks like an arbitrary restriction rather than an exploitation.

The number that motivated the design

The Kafka design document makes the case with a measurement: on a six-disk 7200rpm SATA RAID-5 array, linear writes ran at about 600 MB/s while random writes ran at about 100 KB/s, a difference of more than 6,000 times. That ratio, not any particular absolute number, is the design input. The absolute numbers have moved enormously since; the ratio has not gone away.

The reason is mechanical for spinning disks and architectural for everything after. On a 7200rpm drive, one rotation takes 8.33 ms, so the average rotational latency is about 4.2 ms, and a seek adds roughly another 4 to 10 ms. A random 4 KB write therefore spends on the order of 10 ms getting the head into position and microseconds transferring data. The transfer is free; the positioning is the entire cost. A sequential write pays the positioning cost once and then streams.

The ratio survives flash

It is tempting to think NVMe erased this. It narrowed it and changed its cause. A modern enterprise NVMe drive will do several hundred thousand random 4 KB IOPS, so random access is no longer catastrophic. But three effects keep sequential ahead:

Write amplification. NAND is erased in blocks of megabytes and written in pages of a few kilobytes. A random write pattern scatters valid pages across blocks, so garbage collection has to relocate live data to reclaim space. The write amplification factor on a purely random workload is commonly 2x to 4x or worse, while a sequential workload that fills and retires whole blocks approaches 1x. That amplification is paid in both device endurance and sustained throughput, which is why a drive's random write performance falls off after its spare area saturates and its steady-state number looks nothing like its burst number.

Per-operation overhead. Each I/O operation carries a submission, a completion, an interrupt or polling cycle, and a block-layer trip. At 4 KB per operation you pay that overhead every 4 KB. At 1 MB per operation you pay it every 1 MB. The data path is identical; the overhead per byte differs by two orders of magnitude.

Readahead and cache behaviour. The kernel detects sequential access and prefetches. A log reader scanning forward gets its next pages fetched before it asks, so a disk read that does happen is overlapped with processing rather than serialised with it. Random access defeats readahead by construction, and worse, pollutes page cache with pages that will never be touched again.

Kafka's write path never does a random write

Follow a produce request. The producer has already batched records client-side, so what arrives is a block of records, commonly tens to hundreds of kilobytes, in the exact wire format that will sit on disk. The broker validates it, appends it to the active segment of the partition, and appends a sparse entry to the offset index if the configured byte interval has elapsed. That is the whole write. Every component of it is an append.

Crucially, the broker does not fsync on each append by default. It writes into page cache and lets the kernel flush. The settings that control this, log.flush.interval.messages and log.flush.interval.ms, default to effectively never, and the Kafka documentation explicitly recommends leaving them alone. Durability comes from replication instead: a record is considered safe when enough replicas have it in their page caches, not when any one of them has it on platters. This is a deliberate trade, and it has a sharp edge worth naming: a simultaneous power loss across all replicas in the ISR can lose acknowledged data, which is why replicas should not share a power domain and why broker.rack and rack-aware replica assignment exist.

What the consumer path does with the same property

Reads are sequential too, and Kafka leans on that twice. Because a consumer reads forward, its working set is a moving window near the tail, and because producers just wrote that window, it is already in page cache. A consumer reading at the tail typically performs zero disk reads. And because the bytes on disk are already in the exact format the consumer expects, with no broker-side transformation, Kafka can hand the kernel a file range and a socket and let it copy directly, using sendfile. We will take that apart properly in the page cache lesson; the point here is that it is only possible because the log is stored as an immutable byte sequence rather than as a mutable structure.

The arithmetic an operator actually does

The practical consequence is that you size a Kafka cluster against sequential throughput and network egress, not against IOPS. Work an example. Suppose you ingest 100 MB/s of application data with replication factor 3. Disk write throughput is 300 MB/s because every byte lands on three brokers. Network inbound to the cluster is 100 MB/s from producers plus 200 MB/s of replication traffic between brokers. If three consumer groups all read at the tail, network egress is 300 MB/s, served from page cache. Across a three-broker cluster, each broker handles roughly 100 MB/s of writes, which is comfortable for any modern disk, while egress is the number that gets close to a 10 Gb/s NIC's practical limit of roughly 1.2 GB/s.

Now change one thing: a fourth group starts a replay from seven days ago. Its reads miss page cache, so they become real disk reads, and they evict tail pages that the three healthy groups were being served from. Those groups start missing cache too. The failure is not a disk throughput failure, it is a cache eviction cascade, and the symptom an on-call engineer sees is rising fetch latency on consumers that did not change anything.

Batching is what makes the append large enough to matter

None of the above helps if the write arriving at the broker is 200 bytes. A sequential pattern made of tiny writes still pays per-operation overhead on every one, and the broker would pay a network round trip per record as well. So the producer batches, and the two settings that control it are the first thing to look at on any throughput complaint.

batch.size defaults to 16,384 bytes and is a per-partition ceiling on how much the producer will accumulate before sending. linger.ms defaults to 0, meaning send as soon as there is anything to send. With the defaults, a low-rate producer sends one record per request, which is correct for latency and terrible for throughput. Raising linger.ms to 5 or 10 and batch.size to 64 KB or 128 KB typically multiplies throughput several times over, at the cost of adding at most that many milliseconds of latency. The reason the gain is so large is that batching compounds: a larger batch means fewer requests, better compression ratios because compression is applied per batch, fewer appends on the broker, and fewer replication fetches carrying the same bytes.

Compression deserves its own note here because it interacts with the storage argument directly. A batch is compressed by the producer, stored compressed, replicated compressed, and handed to the consumer compressed. The broker does not decompress it in the normal path. That means compression reduces disk bytes, network bytes in, replication bytes, network bytes out and page cache pressure, all from one client-side setting, which is why compression.type=lz4 or zstd is close to free throughput on any text-shaped payload. The exception that forces broker-side decompression is a version or format mismatch between producer and broker, and it is worth knowing because the symptom is broker CPU rising sharply for no visible reason.

The filesystem and mount choices that actually matter

Since Kafka's write path is appends into page cache, the operating system is doing more of the work than the broker is, and a few choices are worth getting right.

Use XFS or ext4, and prefer XFS, which handles large sequential files and concurrent appends well and is what most large deployments run. Mount with noatime, because without it every read updates an inode timestamp and turns reads into writes. Avoid RAID-5 and RAID-6 for Kafka data: parity means a small write becomes a read-modify-write across the stripe, which is exactly the random pattern the design is built to avoid, and Kafka already has replication, so array-level redundancy buys little. Either use JBOD with multiple log directories, letting Kafka spread partitions across disks, or RAID-10 if you want the array to absorb a single disk failure without a partition reassignment.

Give the broker a modest JVM heap, commonly 6 GB and rarely more than 8 GB, and leave the rest of the machine's memory to page cache. This is the opposite of the instinct most engineers bring from other JVM services, and it follows directly from where the data lives: Kafka's working set is in the kernel's cache, not in the broker's heap, so heap given to the JVM is cache taken from the data path.

The log hands one decision to the application that every other broker family keeps for itself: where you are. That decision looks like a small bookkeeping detail and it determines your delivery guarantee, your replay story, your failure modes, and how much of your consumer you have to write yourself.

An offset is a position, not a receipt

A committed offset is a statement of the form "this group has handled everything in this partition strictly below offset N." It is not a per-message acknowledgement, and it cannot be. Because the commit is a single number per partition, it cannot express "I finished 100, 101 and 103 but not 102." There is no way to acknowledge out of order. If you need that, you need a different broker or an application-level structure on top.

That restriction is the price of the cheap read, and it has a direct consequence that surprises people moving from a queue: a single slow or poisonous message blocks the commit of everything behind it in that partition. In a queue, a message that fails goes back to the broker and the next one is delivered; progress continues around the problem. In a log, the cursor cannot step over the problem without claiming to have handled it. The standard answers are to commit past it and route a copy to a separate error topic, or to pause that partition and retry while other partitions continue, and both are things you implement rather than configure.

Where the number lives, and why that matters

Kafka stores committed offsets in an internal compacted topic called __consumer_offsets, which has 50 partitions by default (offsets.topic.num.partitions) and replication factor 3 (offsets.topic.replication.factor). A group's offsets all land in one partition of that topic, chosen by hashing the group ID, and the broker that leads that partition is the group's coordinator. Because the topic is compacted, only the latest offset per (group, topic, partition) key survives, so the topic's size is bounded by the number of distinct groups and partitions rather than by how often you commit.

This design puts offset storage inside Kafka's own replication machinery, which means your consumer's position is as durable as your data. It also means committing has a cost, and that cost is why auto.commit.interval.ms exists with a default of 5 seconds rather than committing on every record.

You are not obliged to store offsets in Kafka at all. A consumer can call assign instead of subscribe, skip the group protocol entirely, and keep its position wherever it likes. The reason to do that is the one case where Kafka's own offset store is not good enough: if you write your consumer's output to a database, you can store the offset in the same transaction as the output, which makes the two atomic. That turns at-least-once into effectively-once for that sink without Kafka transactions at all, and it is the pattern most data pipelines should reach for first.

1# offsets stored with the output, not in Kafka: the sink transaction is the commit
2consumer.assign([TopicPartition("orders", p) for p in partitions])
3for tp in consumer.assignment():
4    row = db.query_one("select offset from sink_progress where partition = %s", tp.partition)
5    consumer.seek(tp, row.offset if row else 0)
6
7while True:
8    batch = consumer.consume(num_messages=500, timeout=1.0)
9    if not batch:
10        continue
11    with db.transaction():
12        for msg in batch:
13            apply(msg)                      # the business write
14        db.execute(                         # and the position, atomically
15            "insert into sink_progress(partition, offset) values(%s, %s) "
16            "on conflict (partition) do update set offset = excluded.offset",
17            batch[-1].partition(), batch[-1].offset() + 1)

Notice the + 1. A stored offset is the next position to read, not the last one processed, and off-by-one errors here produce exactly one duplicate or one lost record per restart, which is the hardest class of bug to notice because it looks like noise.

Three positions, not one

Reasoning about a partition requires three numbers, and conflating them produces most lag confusion:

• log start offset, the oldest surviving record, which rises as retention deletes segments
• log end offset, one past the newest record the leader has accepted
• committed offset, the group's claimed prefix

Lag is log end offset minus committed offset. But the broker also exposes a high water mark, the highest offset replicated to all in-sync replicas, and consumers are not allowed to read past it, because data not yet on every ISR member might be lost in a leader election. So a consumer can show nonzero lag on a perfectly healthy cluster purely because the newest records have not finished replicating. And with transactions in play there is a fourth number, the last stable offset, which is the highest offset before the earliest open transaction; a read_committed consumer stops there. A consumer showing permanent lag of a few thousand records on a transactional topic is very often not slow at all, it is being held behind a long-running transaction.

Commit cadence is a choice about the size of the duplicate window

With at-least-once delivery, a crash replays everything processed since the last commit. So the commit interval is not a performance knob, it is the width of the window you must be idempotent across. Commit after every record and the window is one record, at the cost of a coordinator round trip per record. Commit every 5 seconds, the auto.commit.interval.ms default, and the window is 5 seconds of throughput, which at 10,000 records per second is 50,000 records that may be reprocessed.

That arithmetic is the real argument against auto-commit, and it is a different argument from the usual one. Auto-commit is dangerous not only because it can commit records that have not finished processing, which it can, but because it sets the replay window implicitly. Manual commit at batch boundaries ties the window to something you chose: the batch is the unit you made idempotent, and the commit happens exactly when the batch is durable at the sink.

The operations that move a cursor

Because position is client state, moving it is a local operation, and this is one of the places where the log model gives you capabilities a queue cannot. The four you will actually use:

• seek(partition, offset) sets the position directly, and is how a replay from a known point is implemented
• seekToBeginning and seekToEnd jump to the current log start and log end offsets, which are resolved against the broker rather than assumed to be 0 and infinity
• offsetsForTimes(partition -> timestamp) asks the broker to translate a wall-clock time into an offset using the sparse time index, returning the first offset whose timestamp is greater than or equal to the target. This is how "replay from 02:00 yesterday" works, and note that it is approximate in the sense that it lands on a record boundary, and that it depends on which timestamp the topic stores, controlled by message.timestamp.type being CreateTime or LogAppendTime
• committed(partition) reads the group's stored position back, which is what tooling uses to compute lag

None of these require broker-side state changes except the last two, and that asymmetry is the log model in miniature: the broker is asked questions about an immutable file, and the answers do not depend on who is asking.

Membership and position are separate problems

One more distinction that saves a lot of confusion later. A consumer group solves two problems at once, and they are independent. Membership decides which consumer owns which partition. Position decides where each partition's reading resumes. You can have either without the other: calling assign gives you position without membership, and static membership gives you stable membership across restarts without changing anything about position.

Static membership is worth knowing now because it addresses the most annoying operational cost of the group protocol. Setting group.instance.id to a stable identifier makes the coordinator remember the member across a restart, so a rolling deploy does not trigger a rebalance per pod as long as each instance comes back within session.timeout.ms. Without it, restarting twelve consumers produces up to twelve rebalances, each pausing the group.

Once you have an ordered, replayable, immutable sequence of facts, several systems you would otherwise build separately turn out to be the same system with different readers attached. This is why "Kafka is a message queue" undersells it so badly, and why Kafka ends up load-bearing in architectures that never intended to adopt a messaging system.

Replication is a log being shipped

Every replicated system needs the replicas to end up in the same state. Shipping the resulting state is wasteful and shipping the operations is fragile unless they are deterministic and ordered. Shipping an ordered log of the operations is the standard answer, and it predates Kafka by decades: Postgres ships a write-ahead log, MySQL ships a binlog, etcd and ZooKeeper replicate a Raft or Zab log, and a state machine replication proof is essentially the claim that identical logs plus deterministic application yields identical state.

Kafka's own replication is exactly this, which is worth internalising because it explains the failure modes later in the course. A follower is not applying operations; it issues fetch requests against the leader's log just like a consumer does, and appends the bytes it receives. "In sync" means its log is a sufficiently recent prefix of the leader's. Leader election then becomes "pick a replica whose log is long enough," and the entire question of data loss becomes a question about log prefixes, which is why the unclean.leader.election.enable setting is as consequential as it is.

Change data capture is a database's log, re-exposed

If replication is log shipping, then a database already maintains the stream you want. Change data capture reads that existing log, the Postgres WAL through logical decoding, the MySQL binlog in row mode, the MongoDB oplog, and turns it into a topic. This is why CDC is both so powerful and so sharp-edged. It is powerful because it is complete and ordered: every committed change, no dual-write race between your database and your event publisher, no application code to forget to emit an event. It is sharp-edged because the database's log is an internal representation, so you inherit its vocabulary, its schema changes arrive as DDL events you have to interpret, and the replication slot that keeps your position will retain WAL segments indefinitely if your consumer stops, which is a well-known way to fill a production database's disk.

Event sourcing makes the log the system of record

Event sourcing inverts the usual relationship. Instead of storing current state and deriving a history, you store the history and derive current state by folding over it. The derived views are caches; the log is the truth. This buys properties that are difficult to retrofit: you can rebuild a view by replaying, you can add a view that answers a question nobody asked when the data was written, and you have an audit trail by construction rather than by discipline.

It also buys obligations that teams routinely underestimate. Old events have to remain readable forever, which makes schema evolution a correctness requirement rather than a convenience, and that is why the schema registry lesson in this course is not optional reading. Every bug you ever shipped is preserved in the history, so a replay reproduces it faithfully unless you version the fold. A regulatory erasure request is genuinely hard against an append-only store, which is what pushes teams toward either key-based compaction with tombstones or crypto-shredding, where the event holds a reference and the key is what gets destroyed.

Stream processing state is a log with an index in front of it

A stateful stream processor, Kafka Streams or Flink, keeps local state: a running count, a join buffer, a windowed aggregate. If that state only lived in memory, a crash would destroy it. The standard solution is that every state change is also written to a changelog topic, a compacted Kafka topic whose latest value per key is the state. The local store, usually RocksDB, is then just a fast index over that log, and recovery is replaying the changelog into a fresh store.

Said plainly: the durable thing is the log and the queryable thing is a materialised view of it. That is the same relationship as a database's WAL and its B-tree, with the two halves pulled apart and placed on different machines. Noticing that correspondence is what makes the Kafka Streams lesson feel like a consequence rather than a new system.

The architectural pattern underneath all four

In each case the log is doing the same job: it decouples the production of facts from their interpretation, in time as well as in space. In space, because producers and consumers do not know each other and a new consumer needs no change on the producing side. In time, because the facts persist, so a consumer can be written after the facts were produced and still see them. That second kind of decoupling is the one a request-response architecture cannot offer at all, and it is the reason the question "can we add a new service that needs the last 30 days of orders" has a boring answer in a log-centric architecture and a project plan in a database-centric one.

The database, inside out

There is a framing from Jay Kreps that makes the pattern click, and it is worth stating directly. A database is internally four things: a write-ahead log that makes writes durable and ordered, indexes that make reads fast, a cache that makes recent reads faster, and a replication stream that keeps copies in step. All four are views over the log, built and maintained inside one process on one machine.

A log-centric architecture takes those four components and puts them on the outside, with the log in the middle. Kafka is the write-ahead log. A search index is one view. A read-optimised Postgres table is another. A Redis cache is another. A stream processor's state store is another. Each one is built by consuming the log, each one can be thrown away and rebuilt, and each one can be optimised for its own queries without negotiating with the others.

The real benefit of this inversion is not performance, it is independence. In a single database, adding a view means a migration, a new index on a shared resource, and a conversation with everyone else who uses that table. In the inverted version, adding a view means starting a new consumer group from the earliest offset and waiting. The cost is that you have given up the database's strongest property: you no longer get a consistent read across views, because each view is at its own position in the log. Every log-centric architecture is therefore an eventually consistent architecture, and pretending otherwise is the source of the "the dashboard disagrees with the order page" class of bug.

Audit trails and the compliance friction

The log gives you an audit trail for free, which is genuinely valuable and comes with an obligation that is easy to miss until a lawyer asks. An append-only store of facts about people is in tension with any right-to-erasure requirement, because the whole design premise is that records are not edited or removed individually.

There are two workable answers and it is better to pick one before you need it. The first is key-based compaction with tombstones: key the topic by the subject identifier, publish a null-valued record for that key, and configure compaction so that the tombstone eventually removes all prior values for the key. The timing is governed by delete.retention.ms, which defaults to 24 hours and controls how long the tombstone itself survives so that every consumer sees it, so erasure is bounded but not instant, and it only works if your partition key is the thing being erased. The second is crypto-shredding: store the personal data encrypted, keep the key in a separate store that does support deletion, and put only the ciphertext and a key reference in the log. Destroying the key makes every copy of the event, including the ones in backups and downstream views, permanently unreadable. The second approach is more work upfront and the only one that works when the data is spread across many topics with different keys.

A course that only sells the abstraction leaves you unable to recognise the problems it is wrong for, and the log is wrong for a lot. Each limitation below follows directly from a property that earlier sections presented as an advantage, which is the honest way to look at any design: the strengths and the weaknesses are the same decision seen from two sides.

No random access by key

The log has no index on content. Finding the current value for key user-42 means scanning, and there is no operation that does it for you. Kafka gives you one narrowing tool, offsetsForTimes, which uses the sparse time index to seek to approximately a timestamp, and that is the whole extent of lookup. Log compaction helps by retaining the latest value per key, but a compacted topic is still a log: there is no "get key" call, and reading the latest value for one key means reading the whole compacted log.

This is why log-centric architectures always have a materialised view somewhere. The log is the write path and the system of record; the thing that answers queries is a database, a key-value store, or a stream processor's state store built by consuming the log. If a design has a log and no materialised view, either the queries are not happening yet or someone is scanning in production.

Partition count is a hard ceiling on consumer parallelism

Because exactly one consumer in a group owns a partition, the partition count caps the group's parallelism. Twelve partitions means at most twelve working consumers, and the thirteenth is a standby. You can raise partition count at any time, but you cannot lower it, and raising it breaks key-to-partition stability: the default partitioner is murmur2(key) % numPartitions, so changing the divisor sends existing keys to new partitions. Any consumer whose correctness depends on all records for a key arriving in one partition, which includes every stateful aggregation and every ordering guarantee you have promised, is affected.

The practical consequence is that partition count is closer to a schema decision than a tuning knob. The usual advice is to over-provision modestly at creation, since idle partitions are cheap in bytes but not free in metadata, file handles and rebalance time.

Ordering is per partition, and that is narrower than it sounds

"Kafka preserves order" is true only within a partition. Across partitions there is no ordering at all, and no global timestamp you can trust. Two records produced a millisecond apart to different partitions may be consumed minutes apart. Any invariant spanning two keys that hash to different partitions is not protected by Kafka, and the common fix, choosing a partition key that puts all related records together, immediately introduces the next problem.

Key skew becomes partition skew

Partition assignment is a hash of the key, so a skewed key distribution becomes a skewed load distribution, and one partition's consumer becomes the bottleneck for the whole group while the rest sit idle. This is the multi-tenant failure everyone meets eventually: partition by tenant_id, acquire one tenant that is 40% of traffic, and now a single consumer carries 40% of the load with no way to split it without giving up the per-tenant ordering that motivated the key. There are real answers, including composite keys that split a hot tenant across several partitions while preserving ordering within a sub-key, and we spend a whole lesson on them.

Per-message operations the log cannot express

Several things queues do routinely have no log equivalent, because they all require per-message state that only the broker could hold:

• Per-message acknowledgement and out-of-order completion. A prefix claim cannot express it.
• Priority. A log is strictly positional, so an urgent record appended after a thousand ordinary ones is read after them. Priority means separate topics plus consumer-side scheduling.
• Delayed or scheduled delivery. Nothing makes a record invisible until a time. Implementing it means delay topics per tier, or consumer-side waiting that also stalls everything behind it.
• Selective consumption. A consumer cannot ask for only the records matching a predicate; it reads everything and filters, paying the full bandwidth.

Retention is a deadline on your consumers

Data is not kept forever. The default retention.ms is 7 days, and when a consumer's position falls off the front, its offset becomes invalid and auto.offset.reset decides whether it silently skips a backlog or stops. Replay is therefore bounded by retention and by disk, which is the constraint tiered storage exists to remove, and planning a replay means knowing both numbers before you need them.

Choosing between a log and a queue

Strip away the branding and the decision comes down to a handful of questions. Do multiple independent consumers need the same data, and might there be more of them later? Do you need to replay history? Is per-partition ordering sufficient, and does your key distribution cooperate? Is your throughput high enough that non-destructive reads matter? Those favour a log. Conversely: do you need per-message acknowledgement, priority, scheduling, or routing decided by the broker? Is the work a set of independent tasks with awkward individual failures? Is the volume modest and the number of consumers small? Those favour a queue, and that is exactly the territory the RabbitMQ course covers.

Partitions are not free, so "just add more" has a ceiling

Since partition count is the parallelism knob and it only goes up, the obvious move is to create topics with a very high partition count and stop worrying. That runs into real costs, all of them per partition rather than per byte.

Each partition is at least one directory with several files, and the broker holds open file handles for the active segment plus the segments being read, so a broker with thousands of partitions needs its nofile limit raised well above the default. Each partition on a producer gets its own accumulator buffer, bounded by batch.size, so a producer writing to 5,000 partitions reserves far more memory and sends smaller batches to each, which quietly undoes the batching argument from earlier. Each partition is an entry in cluster metadata that every broker and every client caches and refreshes. Each partition is a unit of work in a rebalance, so rebalance duration scales with partition count, and a group with 50 consumers and 5,000 partitions has a measurably worse failover than one with 500.

Leader election is where this used to bite hardest. Under ZooKeeper the controller had to write per-partition state during a failover, which is why the old guidance capped a broker at a few thousand partitions and a cluster in the low tens of thousands. KRaft changed the shape of that cost by making metadata itself a log that brokers replay, which is why modern clusters can carry hundreds of thousands of partitions. The per-partition costs on the client side did not go away, so the practical advice is unchanged: pick the partition count from your target throughput and consumer count with modest headroom, not from a wish to never think about it again.

No server-side filtering, and the bandwidth it costs

A consumer cannot ask the broker for only the records it cares about. There is no predicate in the fetch request, no header-based selector, nothing. If a topic carries events for 200 event types and your service needs one of them, you fetch all 200 and discard 199.

This is a direct consequence of the properties that made reads cheap. Filtering would require the broker to inspect and interpret record contents, which breaks the zero-copy path because bytes can no longer go straight from page cache to socket, and it would mean the broker understanding your payload format. Kafka declines both. The architectural answer is to decide topic granularity with this in mind: a single giant topic is cheap to produce to and expensive to consume selectively, while splitting by event type or by subscription shape moves the filtering to produce time where it is paid once. The intermediate option is a stream processor that reads the wide topic once and writes several narrow ones, which pays the bandwidth once per cluster instead of once per consumer.

Small records pay a fixed overhead

One last practical limit. Every record carries a header, and every batch carries a batch header of 61 bytes plus per-record varint-encoded fields. For a 1 KB payload that overhead is noise. For a 20-byte payload it is most of the bytes on disk and on the wire. Workloads made of very small records, such as per-tick telemetry, are therefore much better served by application-level batching, packing many logical events into one Kafka record, than by relying on the producer's batching alone. The trade is that a Kafka record is the unit of offset and the unit of a retry, so packing 500 events into one record means they succeed or fail together and the consumer must unpack them, which is usually fine and occasionally exactly wrong.