Loading...
[Senior-level deep dive topic]
This Messaging Service is designed to be in the same data center with Providers and Consumers.
Providers, Consumers, and Messaging Service all rely on Coordination Service for partitioning configuration. More about this later.
Provider finds which Messaging server to push messages to, given a topic, using the Consistent Hashing configuration.
Likewise, Consumer finds which Messaging server to pull messages from.
[Mid-level deep dive topic]
As the volume of messages that goes through this system is substantial, we need to partition messages.
Topics create a natural opportunity to partition. For now, let’s assume topics are small enough (in terms of messages it receives), and one Messaging server can handle all the messages to each topic easily. (If this assumption is not true, we would find a way to partition one topic to multiple servers. But that would be a next step.)
Then, the next question is, how to distribute topics across servers? We want to make sure load is distributed evenly in terms of storage and traffic.
Consistent Hashing provides a good way to distribute the load. In Consistent Hashing, each Messaging server is assigned a range in a ring. Topics fall into a range assigned to one of the servers. This is how a client knows which server to talk to for a particular topic.
We can try using another algorithm, e.g., Rendezvous hashing, which is simpler to implement than Consistent Hashing. However, Consistent Hashing minimizes data movement in case of group membership change, compared to Rendezvous hashing. This property is important because in our Messaging System, messages persist (e.g., for 7 days) on disk.
[Senior-level deep dive topic]
When a group membership of Messaging servers changes, e.g., a server is removed due to crash, or a server is added for additional capacity, we need to coordinate and adjust the Consistent Hashing configuration. We need to make sure this configuration stays consistent for all the clients and servers. This involves a consensus problem.
Distributed coordination service, such as ZooKeeper, provides a solution. It stores a configuration, in this case ranges assigned to each Messaging server. Clients and servers watch this configuration. When the configuration changes, clients and servers get a notification (via long-polling mechanism). ZooKeeper makes sure that this configuration is stored in a consistent and fault-tolerant way.
Every service in this system (Provider, Consumer, Messaging servers) make a session with Coordination Service. It sends a periodic keep-alive message to Coordination Service.
If Coordination Service does not receive keep-alive for a set duration of time from a Messaging server, it decides that server is down. It would remove that server from Consistent Hashing configuration, and reassigns its topics to another server.
When a new Messaging server is added to the system, it connects to Coordination Service, and becomes responsible for some topics.
[Mid-level deep dive topic]
We have decided to store messages on a permanent storage (e.g. a disk or SSD) in memory for fault tolerance. Even if a Messaging server crashes, messages would not get lost.
However, there are situations where data in a permanent storage is lost (e.g. disk crash / corruption) or temporally unavailable (e.g. network partition). Therefore, it is important to take a backup of messages.
Consistent Hashing provides a way. If a topic is stored by server N, we can assign some other servers as back up servers, e.g., server N+1 and server N+2. This way, data are backed up and can be recovered.
If a topic receives too many messages and one Messaging server cannot handle them, we would have to find a way to partitioning a topic into smaller units.