Requirements

Functional Requirements:

1. Producers publish messages to topics - messages -> named topics, optionally with a key (for partitioning)
2. Consumers subscribe to topics and consume messages - plus consumer groups and offset tracking
3. Delivery guarantees - at-least-once, at-most-once, exactly-once.
4. Persistence & durability - messages survive broker crashes and can be replayed later

Non-Functional Requirements:

1. High throughput - millions of messages/sec. Why: that's what forces partitioning, batching and sequential I/O later
2. Fault tolerance / high availability - no single point of failure, no message loss when a broker dies why: this drives replication and leader election in your design.
3. Low latency - sub~10ms delivery under normal load.
4. Horizontal scalability - add brokers, scale out.

The two nice-to-haves (pub/sub + point-to-point patterns, dead letter queues)

Capacity Estimation

API Design

1. Producer API (write path)
   1. produce(topic, key, value) -> { partition, offset }
   2. POST /topics/{topic}/publish
      1. Batching - producers can send batches of messages
      2. Configurable acks - acks=0 (fire-and-forget), acks=1 (leader wrote it), acks=all (fully replicated)
      3. Returns partition + offset - so the producer knows where its message landed
2. Consumer API (read path)
   1. Fetch messages - pull-based - consumers poll
      1. consume(topic, partition, offset) -> messages
      2. response: { topic, partition, offset, key, value, timestamp }
      3. consumers control their own pace
   2. Commit offsets - this is the subtle one that makes it distributed
      1. commitOffSet(group, topic, partition, offset)
      2. if a consumer crashes, the group resumes from the last committed offset.
      3. connects to your at-least-once guarantee from Requirements
      4. auto-commit vs manual commit (auto is simpler, manual is safer
3. Admin API
   1. createTopic(name, partitions, replication_factor)
   2. listTopics() / getClusterStatus()
   3. horizontal scalability and fault tolerance get configure through this surface

High-Level Design

A cluster of brokers sits between producers and consumers; messages land in partitioned, repliecated logs; and a coordination service keeps the whole cluster's metadata consistent.

The Components:

1. Broker cluster (the heart) - brokers receive, store and serve messages. Each broker hosts some partition leaders and some followers.
2. Producers & consumers - Producers publish to topics; consumers pull from them, organized into consumer groups.
3. Coordination service (ZooKeeper/KRaft) - The part people forget. Tracks which broker is the leader for which partition, handles leader election wehn a broker dies, and keeps consumer gorp offsets. Source of truth.
4. Storage layer - Append-only log segments per partition on disk. Sequential I/O = throughput promised in reqs.
5. Monitoring - broker health, consumer lag, replication lag - the 'is this thing alive' service.
6. hot partition / traffic spike handling (a flash-sale topic flooding one partition without starving others)

What makes it distributed? Partitioning + replication. Each topic is split into partitions and each partition is replicated across brokers (fault tolerance).

Database Design

1. The commit log (message storage)

The primary store is an append-only commit log, not a traditional DB:

• Messages are written sequentially to log segments per partition — sequential writes are what get you the millions of msgs/sec throughput you promised in Requirements.
• Each message gets a monotonically increasing offset — this is your primary key and your ordering guarantee.
• Retention policy controls when segments are deleted or compacted.

2. The metadata store (cluster coordination)

A separate small store (ZooKeeper or an internal KRaft quorum) holds the state about the cluster:

• Broker registry (who's alive)
• Topic → partition → leader/replica assignments
• Consumer group offsets (how far each group has read)

Important distinction to make: message data lives in the log; metadata lives in the coordination store. Interviewers love hearing that separation.

3. The data model (entities)

Sketch these as an ER-ish set:

Topics(id, name, partitions, replication_factor, retention_ms) Partitions(id, topic_id, partition_index, leader_broker_id, replicas) Brokers(id, host, port, status, last_heartbeat) ConsumerGroups(id, name) Offsets(group_id, topic_id, partition_id, offset) Messages(topic_id, partition_id, offset, key, value, timestamp, headers)

Note the relationship that matters: a partition belongs to a topic, and offsets belong to (group, topic, partition) — that composite key is what lets consumers resume exactly where they crashed.

Detailed Component Design

1. The write path (replication + durability)

When a producer sends a message:

1. Producer → leader. The message goes to the leader broker of the partition (determined by key hash).
2. Leader appends to its log, then pushes the message to its in-sync replicas (ISR).
3. Ack policy decides the response: with acks=all, the leader only acknowledges after all ISR followers confirm — this is your "no message loss on broker failure" requirement made real. Replication factor 3 is the standard.
4. Idempotency guard: each producer attaches a producer ID + monotonically increasing sequence number. If the producer retries (network timeout, broker failover), the broker dedupes by rejecting any sequence number it's already seen. This is the mechanism that makes exactly-once possible — and it's a major point interviewers check.

2. The read path (offset management + rebalancing)

• Consumers poll the leader (or any replica) for messages at a given offset.
• They commit offsets back to the coordination service — the source of truth for "how far has this group read."
• On crash, a consumer resumes from the last committed offset. At-least-once semantics mean it may reprocess a few messages — so consumers must be idempotent on the processing side.
• Consumer groups + rebalancing: each partition is consumed by exactly one consumer in a group. When a consumer joins/leaves, the coordinator reassigns partitions. Modern systems use incremental cooperative rebalancing — only affected partitions get revoked, avoiding the stop-the-world thundering-herd problem when many consumers restart at once.

3. Failure handling (the interviewer's favorite)

• Leader failure: coordination service elects a new leader from the ISR. Producers/consumers redirect. Brief partition unavailability, no data loss (as long as you replicated before ack).
• Hot partitions: one key generating disproportionate traffic → one broker overloaded. Mitigations: salt the key, split the partition, or custom-partition high-cardinality keys. Monitor per-partition consumer lag to catch it early.

The threading theme

Notice the pattern: every major point is a Requirement made concrete — replication = fault tolerance, acks = delivery guarantees, ISR + sequence numbers = exactly-once, rebalancing = availability.