Loading...
Lets say we have 1B Active users, and every users add one comment per day. And every user read 100 comments per day.
Every comment is 100 bytes.
Read daily: 100 * 1 B days = 100 B requests per day.
Read QPS: 100 B / 24 / 3600 = 1 M
Peak read QPS = 5M
Write QPS: 1B / 24 /3600 = 10 M
Peak write QPS = 50 M
Storage:
daily: 100 bytes * 1B = 100GB
monthly: 100GB * 30 = 3TB
Year: 3TB * 12 = 30TB
5 year = 150 TB
If we say every year we increase 20% DAU. In the next 5 year we will need around 300TB storage.
Write API:
POST /api/post_id={post_id}&&comment_id={comment_id}
with user authentification token
{"user_id": user_id;
"comment": "{comment text}",
"{time_stamp}": timestamp}
{comment_id} could be null
Read API:
GET /api/post_id={post_id}&&comment_id={comment_id}
{comment_id} could be null
User infomation table:
(user information we could put them into relational db)
user_id: int
user_name: str
user_email: str
Post information:
(relational db, it is better to use non SQL, easy to scale, dynamo db could be a good choice)
Post_id: int
post_title: str
post_content: str
Post_owner: user_id
post_timestamp: timestamp
Comments:
(also dynamo db)
Comment_id:int
comment_content: str
comment_owner:str
Post comments relationt db, we could use graph db to store those information.
one post could consist of several comments, one comments could consist of sever comments, we will put id only in graph db.
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
As you could see, the flow
write flow, when the client tried to comment, we will authenticate user information, make sure this user have the write right. And also we add load balancer, to make sure the traffic will distributed to go to different webservice, rate limitor will stop some users or machine comments to many times. And then the comment request went to webserver, webserver will check user information, we use the SQL as database, because user information is structure data. And we could add cache between user information and webserver, it will be good for the read performance. And we will transfer the request to post service, post service will add the comment in to comment info table, post service will need to generate the comment id as the key, comment id should be ordered by time. So we could use the way how twitter generate unique id to generate the key. And then we save the data in the comment info, the partition key should be the post id, and the sorted key could be the comment key. And update the graph DB. After this we publish the comment. Notification service will notify the user, who has been commented.
Read flow. The client call the read API, read API will user the post id or comment id to find the comment. First, we will read the graph DB, find the post and all comments. And then we will put the post, all comments together return to webserver, and then send back to client. Redering in frontend. We should also have some cache, cache some hot post and comments.
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
write flow, when the client tried to comment, we will authenticate user information, make sure this user have the write right. And also we add load balancer, to make sure the traffic will distributed to go to different webservice, rate limitor will stop some users or machine comments to many times. And then the comment request went to webserver, webserver will check user information, we use the SQL as database, because user information is structure data. And we could add cache between user information and webserver, it will be good for the read performance. And we will transfer the request to post service, post service will add the comment in to comment info table, post service will need to generate the comment id as the key, comment id should be ordered by time. So we could use the way how twitter generate unique id to generate the key. And then we save the data in the comment info, the partition key should be the post id, and the sorted key could be the comment key. And update the graph DB. After this we publish the comment. Notification service will notify the user, who has been commented.
Read flow. The client call the read API, read API will user the post id or comment id to find the comment. First, we will read the graph DB, find the post and all comments. And then we will put the post, all comments together return to webserver, and then send back to client. Redering in frontend. We should also have some cache, cache some hot post and comments.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
For webserver, post server, notification service, they are stateless service. So it is easy to scale. And we could use some coordinate service like zookeeper to control the service status. And avoid to send traffic to dead service. Also scale out.
For post info , comment info we use dynamo db, which support automatically scale up. it use consist hashing strategy. It support replicas and failure tolerate.
for user info, we could add some replicas, we use normal SQL, could could choose master, slaves mode, to make sure, if one db dead we will not lose data.
Explain any trade offs you have made and why you made certain tech choices...
We add cache in the API gateway, to cache some pages which are frequently be visited. This make user have good performance. (when user only need to read instead of write any comments). However, we could only cache some information.
Try to discuss as many failure scenarios/bottlenecks as possible.
We write the information, to graph db, and comment info. we need to make sure this could be a transaction, which means. this need to all successful or not. If webserver or post service fail, we will lose the write request, this will cause the bad user experience, in this situation, we may need to write the message into the message queue, to avoid and write request loss. Or we could add the message into WAL log.
The message queue increase the latency, however, it could increase the latency. However, it increase whole system reliability.
all the webservice are stateless, easy to scale. And the database, comment info, post info is dynamo db, which is easy to scale too. when any node in dynamo db crash, it will automatically redirect the traffic to next node. ANd the read requests will also go to the other nodes also.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
For the search the comment, we could add some document database, for example elasticsearch, which is inversed index database, could help use to find the comment easily.