Design a Distributed Messaging System

A distributed messaging system looks like a simple publish/subscribe API, but the real challenge is durability without sacrificing throughput. How do you guarantee that no message is ever lost while processing millions per second? The answer shapes every decision in this design.

Functional

• Publish messages to topics: Producers send messages to named topics. Each message carries an optional key, a value (payload), and metadata like a timestamp. The key is not just metadata: it determines which partition receives the message. Two messages with the same key always land on the same partition, which is how you guarantee ordering for related events. Batch publishing is supported to amortize network overhead when throughput matters more than individual message latency.
• Subscribe via consumer groups: Consumers form groups. Each partition within a topic is assigned to exactly one consumer in the group, so messages are processed in parallel without duplication. If a consumer crashes, its partitions are reassigned to surviving members. The pull-based model lets consumers control their own consumption rate, a slow consumer does not block a fast one.
• Configurable delivery guarantees: The system supports three modes. At-most-once (fire and forget) is fastest but risks loss. At-least-once (retry until acknowledged) guarantees delivery but may produce duplicates. Exactly-once (idempotent producers with transactional commits) eliminates duplicates at the cost of additional coordination. The choice depends on the use case: clickstream analytics tolerates duplicates; financial transactions cannot. The delivery guarantee is configured per-producer, not per-topic, so different producers writing to the same topic can operate at different guarantee levels based on their individual requirements.
• Message persistence with configurable retention: Messages are written to disk and retained for a configurable period (typically 7 days). Unlike traditional message queues that delete messages after consumption, this system keeps them. Consumers can replay from any offset: useful for debugging, backfilling a new service, or recovering from a processing bug. This replay capability is what makes the system suitable as the backbone of an event-driven architecture: services can be added, removed, or rebuilt without losing historical context.

The numbers drive every architecture decision. When we see that 100MB/s of writes need to land on disk reliably, we know a single broker cannot handle all traffic: we need partitioning. When we calculate 60TB for 7-day retention, we know replication planning is critical.

Traffic

• 100K messages per second
• 1KB average message size
• 100MB/s write throughput
• 300MB/s read throughput (3x from multiple consumer groups reading the same data independently)

Three operations cover 90% of usage. But the parameters encode critical decisions: acks controls the durability-latency trade-off, key determines partition placement, and offset enables replay from any point in the log.

Producer API

produce(topic, key, value, acks)

Kafka achieves millions of messages per second not despite writing to disk but because of it. The key insight: sequential I/O on an append-only log achieves 600MB/s on modern disks, versus 2MB/s for random I/O, a 300x difference. This is why an append-only commit log outperforms databases that use B-trees for write-heavy workloads.

Append-Only Commit Log

Each partition is stored as a sequence of log segments on disk. A segment is a fixed-size file (typically 1GB) containing messages in arrival order. When a segment fills up, a new one is created. Old segments are deleted or compacted based on the retention policy.

The architecture has four major components: producers, the broker cluster, consumer groups, and the coordination service. Each plays a specific role, and a critical design decision is what is not in the architecture: there is no load balancer between producers and brokers.

Producers

Producer Write Flow

1. Serialize: The producer serializes the key and value into bytes using the configured serializer (Avro, Protobuf, JSON, or raw bytes).
2. Partition: If the message has a key, hash(key) % num_partitions determines the target partition. If no key, round-robin selects the next partition. The producer looks up the partition leader from its cached partition map.
3. Send to leader: The message is sent directly to the leader broker for the target partition. Messages are typically batched: the producer collects messages for the same partition and sends them together to amortize network overhead.
4. Leader appends: The leader broker appends the message to the end of the partition's commit log segment. This is a sequential write. no seeking, no random I/O. The message is written to the OS page cache first (not immediately flushed to disk), which is why writes are so fast. The OS flushes dirty pages to disk in the background. Even if the broker crashes before flushing, the replicas have the data: durability comes from replication, not from fsync (with acks=all). This is a deliberate design choice: calling fsync on every write would reduce throughput by 10-100x. Instead, the system trusts replication for durability and lets the OS manage disk flushing at its own pace.
5. Replicate to ISR: The leader sends the message to all in-sync replica (ISR) followers. Followers append to their own commit logs.
6. Acknowledge: With acks=all, the leader waits for all ISR followers to confirm their writes, then sends a success response to the producer. With acks=1, the leader responds immediately after its own write.

ISR Replication: The GATE

The core durability mechanism is In-Sync Replica (ISR) replication with acks=all. This is the single most important concept in the entire system. It is how we guarantee that no committed message is ever lost.

Acks Trade-Off With Concrete Numbers

The acks parameter is the most important trade-off in the entire system. Here are concrete latency numbers for a typical production cluster:

• acks=0: Approximately 0.5ms. Producer sends and immediately moves on. No confirmation. Messages can be silently lost if the broker crashes or the network drops a packet. Use case: metrics collection where losing a few data points is acceptable.
• acks=1: Approximately 2-5ms. Producer waits for the leader to write to its local log. If the leader crashes before replication, the message is lost. Use case: logging pipelines where low latency matters more than perfect durability.
• acks=all: Approximately 5-15ms. Producer waits for the leader AND all ISR followers to write. No committed message is ever lost (with min.insync.replicas=2). Use case: financial transactions, order processing, anything where data loss is unacceptable.

Broker Failure and Leader Election

When a broker fails, the coordination service detects it via missed heartbeats (typically a 10-second timeout). For each partition where the failed broker was the leader, the controller selects a new leader from the ISR set. The new leader begins accepting reads and writes immediately. Producers and consumers discover the change when their next request returns NOT_LEADER_FOR_PARTITION, triggering a metadata refresh.

Schema Registry

A schema registry stores message schemas (Avro, Protobuf, JSON Schema) and enforces compatibility rules. Producers register a schema before sending messages; consumers validate incoming messages against the registered schema. This prevents breaking changes: for example, a producer cannot remove a required field that downstream consumers depend on. Evolution rules (backward compatible, forward compatible, full compatible) define what changes are allowed. Without a schema registry, incompatible message format changes break consumers silently at runtime.

In interviews, mentioning schema registry demonstrates production experience. Most candidates describe the happy path (producers send, consumers receive). Few address what happens when a producer changes its message format, which is a common real-world problem that causes outages in microservice architectures.