Kafka has three levels of naming and engineers routinely collapse them into one, which is the root of a surprising number of production misunderstandings. A topic is a name. A partition is a log. A replica is a copy of a partition on a particular broker. Only the middle one is a real thing with bytes in it; the other two are a label and a placement.

Write that out as an address. The unit everything in the protocol actually operates on is a TopicPartition, written orders-7, and the unit the cluster places is a replica of orders-7 on broker 3. A produce request targets a partition. A fetch request targets a partition. An offset is meaningful only within a partition. A consumer is assigned partitions. The topic appears in exactly two roles: as the thing you create and configure, and as the thing a consumer subscribes to so that the group protocol can hand it partitions.

A topic has no ordering, no offsets, and no bytes

Because the topic is only a label, several things people say about topics are category errors. A topic does not have an offset, so "the topic is at offset 4,000,000" means nothing; each of its partitions has its own independent offset sequence and partition 3 being at 4,000,000 tells you nothing about partition 4. A topic is not ordered. A topic has no size of its own beyond the sum of its partitions' sizes, and those sizes are routinely uneven.

The topic does own configuration, and that is its most practical role. Retention, cleanup policy, segment size, minimum in-sync replicas, maximum message size and compression are all set per topic and inherited by every partition. These are overrides on top of broker-level defaults, and the naming differs between the two levels in a way that catches everyone once: at the broker it is log.retention.ms and log.segment.bytes, at the topic it is retention.ms and segment.bytes. A topic-level override, once set, is sticky; it does not follow the broker default when you later change the broker default, which is how a cluster ends up with one topic quietly retaining data for the thirty days somebody set two years ago.

Topic names were an unstable identity, and now are not

There used to be a sharp edge here worth knowing even on a modern cluster, because it explains why the metadata you see has a field you might not expect. Historically a topic was identified only by its name, so deleting a topic and recreating it with the same name produced something that every client and broker believed was the same topic. Any client holding stale metadata, or any broker that was offline during the delete, could reattach to the new topic with old positions, and the result was a log that looked corrupt.

KRaft assigns each topic a UUID topic ID at creation, recorded in the partition's partition.metadata file on disk and carried in modern protocol requests. A deleted and recreated topic has a new ID, so a stale participant is now told the topic ID does not match rather than being allowed to proceed. If you ever see UNKNOWN_TOPIC_ID in a client log, that is this mechanism firing, and it is doing you a favour.

The partition on disk

The partition is a directory, and looking inside it once removes most of the mystery from the storage lessons that follow. For each directory in the broker's log.dirs, a partition gets a subdirectory named <topic>-<partition>, so orders-7 is literally a folder called orders-7. Inside it:

1orders-7/
2  00000000000000000000.log          # segment: the records themselves
3  00000000000000000000.index        # sparse offset -> byte position
4  00000000000000000000.timeindex    # sparse timestamp -> offset
5  00000000001048576241.log          # the next segment, named by its base offset
6  00000000001048576241.index
7  00000000001048576241.timeindex
8  leader-epoch-checkpoint           # which leader wrote from which offset
9  partition.metadata                # topic ID and version
10  00000000001048576241.snapshot     # producer id state, for idempotence

Three details in that listing are worth noticing now. First, the segment files are named by the base offset of the first record they contain, which means a lookup for an arbitrary offset starts with a binary search over filenames and no index at all. Second, the indexes are sparse, controlled by log.index.interval.bytes at a default of 4096, so they hold roughly one entry per 4 KB of log rather than one per record, which keeps them small enough to memory-map. Third, the on-disk layout contains no per-consumer state whatsoever. Nothing in that directory knows who has read what.

Replicas, and what "in sync" means

Each partition is replicated to replication.factor brokers. One is the leader and serves all reads and writes for that partition by default; the others are followers that fetch from the leader. The set of replicas considered current is the in-sync replica set, the ISR, and membership is decided by one setting: replica.lag.time.max.ms, default 30,000. A follower is in sync if it has either caught up to the leader's log end offset or fetched within that window. Note that the criterion is time, not offset distance, which is why a follower can be thousands of records behind and still be in the ISR provided it is fetching steadily.

Replica placement is a cluster-level concern with one knob you should always set. With broker.rack configured on each broker, Kafka distributes a partition's replicas across racks, so a rack losing power does not take all three copies. Without it, replica assignment is round-robin over broker IDs and can place all three replicas in one failure domain, which is the configuration that turns the "durability comes from replication, not fsync" trade from reasonable into reckless.

Leadership is per partition, and that is the load balancing story

Each partition has its own leader, and the cluster tries to spread leadership so that every broker leads roughly its share. The assignment algorithm encodes this: when a topic is created, the replica lists are built so that the first replica of each partition, the preferred leader, rotates across brokers. Partition 0's preferred leader is broker 1, partition 1's is broker 2, and so on, with the follower positions also rotated so that no pair of brokers is always paired.

Leadership drifts away from that layout during ordinary operation. When a broker restarts, every partition it led is led by someone else, and it does not take them back automatically unless auto.leader.rebalance.enable is true, which it is by default, with the controller checking every leader.imbalance.check.interval.seconds, 300, and acting when a broker's share of preferred leaderships exceeds leader.imbalance.per.broker.percentage, 10. The practical consequence is worth internalising: immediately after a rolling restart, the last broker restarted leads almost nothing and the first leads too much, and the cluster fixes itself over the following few minutes. A broker with high CPU and low leadership right after a deployment is usually this, not a problem.

The internal topics, which are partitions too

Three internal topics follow all of the above, and knowing their shapes makes their behaviour predictable. __consumer_offsets holds committed offsets, with 50 partitions by offsets.topic.num.partitions and replication factor 3 by offsets.topic.replication.factor; a group's partition is chosen by hashing the group ID, which is why one group's offset commits are serialised through one broker. __transaction_state holds transaction metadata with the same shape and the same hashing on transactional ID. __cluster_metadata is the KRaft metadata log, a single partition replicated by the controller quorum rather than by brokers.

Two of those settings bite on a new cluster. The offsets topic is created on first use with whatever replication factor the setting then says, so if the cluster had one broker up at that moment the topic can be created with replication factor 1 and stay that way forever, silently making every consumer group's progress dependent on one machine. Checking the replication factor of __consumer_offsets is a reasonable first move on any cluster you inherit.