Requirements

Functional Requirements:

• Guarantees at least once delivery
• Supports pubsub, request-reply
• persists messages
• tracks message status (received, processed, in-flight)
• producers can publish to named spaces for later consumption
• consumers can subscribe and pull or poll for changes

Non-Functional Requirements:

• low latency
• high throughput - can handle high volume
• fault tolerant (node failures shouldn't bring down system)
• can be support multiple connections
• consumers should be able to track progress via offset or similar
• consumers should be able to be grouped for parallel processing

Capacity Estimation

Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...

API Design

Minimally. We'd need a few more APIs at a low level but this is a good simple first approach.

Control:

POST /topics/ - create a topic. should take a parameter for partitions on topic (unit of parallelism)

POST /topics/repartition - change partition count

GET /nodes/ - health and node information

Producer APIs

POST /topics/$topic - write event to topic

Consumer APIs

POST /topics/subscribe - subscribe to topic via pubsub. Ownership of given partition handled by system.

GET /topics/$topic?offset[optional] get a given event from a topic at offset if given

Consumer Group APIs

POST /groups/$topic/register - register as consumer group for topic

POST /groups/$topic/node - register as consumer group node. rebalance consumption

High-Level Design

Message Throughput:

• Producer(s) writes to topic
• Topic determines partition via hash key
• Topic writes to partition leader, which adds it to append only log on disk
• Partition leader syncs to follower
• Partition acks when complete
• Partition sends notify to pubsub subscribers
• Consumers that don't subscribe via pubsub can query at offset

Coordinator:

• Use some kind of distributed state service, eg Zookeeper
• Elects leaders using Raft (or similar), maintains set of leader and follower nodes
• Tracks consumer group state, registers new groups or consumer nodes, reassigns nodes to new topics as they change
• Handles node failures, adds new nodes
• On node failure, only partitions on that node should be affected. Determines if new leader is needed and elects one if it is.

Monitoring:

• Alert on hot partitions - is a single partition disproportionately handling load?
  • Mitigate with a higher cardinality key
• Alert on consumer lag when consumption rate has been reduced significantly
  • Mitigate by adding more consumers.
  • If repartitioning is necessary, take care to ensure that the new partitions don't break ordering
• Alert on bursts (increase in writes), correct action is to
  • On failure, have a policy to either
    • scale out
    • throttle producers (back-pressure)
    • drop and dlq

Database Design

Message logs are appended onto disk. Because they're append only, order is guaranteed and can quickly be looked up via an offset.

Offsets and consumer state live in the coordination service (a znode somewhere, ultimately, in this design).

Detailed Component Design

Broker replication protocol:

In order to not lose data, the partition leader should start by writing the event to its own disk, then sending that data to each of the followers to write. the followers can then write the data and ack the leader. once _all_ followers have acked, the leader can then ack the producer. at this point, even if the leader fails right after, the data has been durably written elsewhere.

On leader failure during this process, the producer can backoff and retry as new leaders are elected. it would not have acked the producer.

on replica failure, the leader can retry, check node status, or tell the producer it failed and have the producer handle it while other components try to elect a leader or otherwise mitigate it.

Less stringent durability requirements might call for fewer acknowledgements (Eg just the leader) or none at all. this means data may be lost in the event of a leader failure, which is acceptable under certain conditions.

Because these are append only writes to disk, ordering is guaranteed within a partition. care has to be taken when repartitioning to avoid ordering issues.

Consumer group coordinator:

Consumer groups should be able to commit their offsets in the coordinator so they can resume processing after failure. A consumer should commit an offset once it is sure it has durably processed the event it just picked up for handling.

The coordinator should handle mapping partitions of a topic to a given consumer in a group, and rebalance them when a consumer joins/leaves/becomes unhealthy. Because offsets are stored on the coordinator, consumers can easily fetch/be told where to start processing on join.

Rebalancing can be handled cooperatively by gradually handing over new partitions to avoid a thundering herd.

Idempotent producer:

The producer should be able to send as many of the same event as necessary without creating duplicates. it can do this by assigning a producer ID and monotonically increasing sequence number to each event. this means that if it sees a number it's already processed, it doesn't need to write it. this process should happen on a per-partition basis and can be handled in the client library.