System requirements

Functional:

• Topic & Partition Management — create, configure, and scale topics/partitions.
• Producer API — publish messages with batching, retries, compression, and idempotence.
• Consumer API — subscribe, poll, and commit offsets; support consumer groups.
• Messaging Patterns — pub/sub, queue, request/reply, streaming.
• Delivery Guarantees — at-most-once, at-least-once, and idempotent/exactly-once.
• Replication & Failover — multi-replica quorum with automatic leader election.
• Retention & Compaction — time/size-based retention, log compaction by key.
• DLQ & Retries — dead-letter queues and configurable retry policies.
• Admin & Monitoring — APIs for topics, partitions, offsets, health; metrics & tracing.
• Security — TLS, authentication, authorization (ACLs/RBAC), audit logging.

Non-Functional:

• Throughput — handle millions of msgs/sec cluster-wide.
• Latency — <10 ms.
• Availability — 99.95% per region (multi-AZ replication).
• Durability — quorum commits guarantee no data loss.
• Scalability — horizontal scale via partitions/brokers.
• Recoverability — failover in <30s; full broker recovery in <15 min.
• Observability — metrics, logs, traces, consumer lag tracking.

API design

Define what APIs are expected from the system...

High-level design

Producers — publish messages to topics/queues.

Brokers — cluster of servers responsible for storing and serving messages (log-based storage).

Metadata/Coordination service — leader election, cluster membership, topic metadata (can be consensus-based: Raft/ZooKeeper/etcd).

Consumers — pull or receive messages; support consumer groups (competing consumers).

Coordinator/Controller — manages partitions, rebalances consumers, enforces quotas, controller leader.

Storage layer — append-only log files per partition with segmenting, compaction, retention policies.

Replicator — intra-cluster replication (sync/async) for high availability.

Delivery & offset manager — track per-consumer offsets/ACKs.

Admin API & Management UI — create topics, set retention, view status.

Observability stack — metrics, tracing, logging, alerting.

Clients SDKs — for major languages, supporting batching, compression, retries, transactions.

Detailed component design

Producers

Responsibilities:

• Serialize messages (JSON, Avro, Protobuf).
• Partitioning logic: decide which partition a message goes to (hashing by key or round-robin).
• Retry and batching:
  • Buffer messages for batch writes to reduce I/O.
  • Retry failed messages until acknowledged (if at-least-once delivery is desired).
• Optional compression to reduce network/broker load.

Brokers

Brokers are the heart of the system. Each broker stores partitions, handles replication, and serves consumers.

Responsibilities:

• Accept messages from producers.
• Append messages to partition logs.
• Replicate partition logs to follower brokers.
• Serve consumer requests (read messages from offset).
• Maintain metadata (leader/follower role for partitions).
• Garbage collection: clean up expired messages (retention policy).
• Provide fault tolerance: detect leader failure and promote followers.

Consumers

Consumers read messages from brokers and acknowledge progress.

Responsibilities:

• Fetch messages from brokers for assigned partitions.
• Commit offsets to mark processed messages.
• Support consumer groups for load balancing:
  • Each partition is consumed by only one consumer in a group.
  • Rebalancing when consumers join/leave.
• Handle retries/failures and optional idempotency.

Coordination / Metadata Service

Purpose: Maintain cluster state and enable consistent partition leadership.

Responsibilities:

• Track cluster nodes (brokers, consumers).
• Manage leader election for partitions.
• Store topic configurations and partition mappings.
• Track consumer offsets (if centralized).

Typical Implementations:

• ZooKeeper, etcd, Consul.
• In modern Kafka versions, KRaft mode replaces ZooKeeper with internal metadata quorum.

Data Stored:

• List of brokers and their status.
• Partition → broker mapping (leader + replicas).
• Topic metadata (name, replication factor, retention).
• Consumer group membership and last committed offsets.

Storage Layer

Purpose: Efficiently persist messages with durability and fast access.

Design:

• Partitioned log segments on disk:
  • Each partition is split into multiple log segments (e.g., 1GB files).
• Index files:
  • Offset index: maps offset → byte position in log file.
  • Timestamp index: maps timestamp → offset.
• Replication:
  • Each partition has a leader broker and multiple follower replicas.
  • Followers replicate leader logs asynchronously.
  • Quorum-based acknowledgment for durability.

Monitoring & Observability

Responsibilities:

• Collect metrics: throughput, latency, consumer lag.
• Alert on failures: broker down, replication lag, consumer stalled.
• Logging: message failures, retries, and offsets.
• Tools: Prometheus, Grafana, ELK stack.

Database design

Database primarily needs to store:

• Messages.
• Topics and Partitions.
• Consumers and Consumer Groups.
• Offsets / Delivery tracking.
• Broker / Cluster metadata.

Schema:

Messages (the actual payloads) - store in append-only log (custom file storage) or distributed commit log (Cassandra).

Metadata - store in relational DB (PostgreSQL, MySQL) or Key-Value store (etcd, Consul, ZooKeeper).

Offsets / Consumer Tracking - store in Key-Value store (Redis).

Monitoring & Logs - Time-series DB (Prometheus, InfluxDB).