Scalability: The system should be able to scale up when traffic increases
Consistency: All users should view comments consistantly
Fault tolerance: 1 component failure during runtime should not affect the overall performance
Observability: The status of the system should be monitored, dashboard should be included, when a metric behaves abnormally SREs need to be alerted.
Capacity estimation
1 million user
2 million comments/minute
512 bytes payload
1GB/ minute traffic
API design
Post comment
Request:
int user_id
int channel_id
string content
Response
status
View comment
Request:
int channel_id
Response
status
bytes content
Database design
Channel
primary_key int channel_id
User
primary_key int user_id
string name
Post
primary_key int post_id
string content
foreign_key int user_id
datetime timestamp
foreign_key channel_id
High-level design
API gateway: routes API, load balancing, rate limiting, authenticating
Post server: uses websocket to connect to clients. Each server hosts several connections. Data is transferred through websocket.
MessageQueue: After a comment is posted, send it to the MQ. MQ is configured to push model, to push to the subscriber that are in the same chat room.
DB: stores DB tables we mentioned above. Post needs to be partitioned because the size is very large, we can shard it based on channel_id. To improve fault tolerance, shards should also be replicated. The synchronous between replicas should be done through leader-follower pattern. Write is sent to the leader, and read from the followers to support high write traffic with low read traffic.
Request flows
User enter a channel, send connect request through API gateway
API gateway verifies rate limit, authentication etc. transfers the request to a server
establishes the connection to the server
When the new post show up in the MQ, it pushes to the client side of the post server, post server then push the update to user client.
When a user send post to post server, post server does verification deduplication to allow retries, add timestamp, send it to MQ with a channel id bounds to it.
Meanwhile, the post is sent to DB for persistent storage.
Detailed component design
DB replication need to be managed through coordination service, including leader election, health monitoring etc.
websocket provides live update for both sides
Trade offs/Tech choices
MQ pull model vs push model: with push model, less resource will be used as client don't need to keep asking for updates. The disadvantage of push model is that there could be message loss. It can be mitigated through ACK and retry.
Failure scenarios/bottlenecks
DB node might fail, in that case, coordination service monitors for health, if it verifies the failure, through leader election, select one of the replica as the new leader and receive write request again.
MQ broker might fail, so distributed MQ broker should be used. Keep update the offset of Messages consumed/sent to consumer
Post server might fail, that case, the user need to talk to API server again to establish a new websocket connection.
Future improvements
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?