Loading...
1- Users should be able to create posts.
2- Users should be able to follow and unfollow others
3- Users should be able to view a feed of posts from people they follow, in chronological order.
4- Users should be able to like and comment on posts
5- Users should be notified of a new post from the following accounts.
6- A feed may contain photos, videos or just text.
-DAU: 2.11 billion MAU: 3.07 billion
-Throughput:
Write throughput: Let's say only %10 of the users create a post daily. 10% of 2.11b = 211m write requests.
Read throughput: Let's say a normal user opens his feed 10 days a day and 10 posts in that feed. 2.11b x 10 x 10 = 211b read requests.
-Storage:
For creating posts:
Video, image, text
20MB, 0.5MB, 0.1MB
%20, %60, %20
211m * 0.2 * 20MB, 211m*0.6*0.5MB, 211m*0.2*100KB
844 TB, 633 TB, 4.22 TB
In total=1.5 PB/day
For the following (16 bytes/follow):
Let's say each user follows 1 new user every week
Total storage in 10 days = (2.11b/7 million requests per day X 16 bytes) X 365 day X 10 year
For like/comment (216 bytes per activity):
Let's say each user performs 3 activities (like/comment) per day
Total storage in 10 days = ( 2.11b X 3 activities/day X 216 ) X 365 day X 10 year
-Memory:
Let's say we cache 10% of the data everyday.
1.5 PB/day X 1% = 15 TB/day memory needed
-Network
1.5PB stored/day (required for ingress calculation)
211billion read requests/day (required for egress calculation)
The average size of a post = 0.2 * 20MB + 0.6*0.5MB + 0.2*0.1MB = 4.32MB
INGRESS: 1.5PB / (24*60*60) = 17.36 GB/second
EGRESS: 211billion * Average size of a post = 211billion * 4.32MB = 911.52/(24*60*60) = 10.55TB/second
-Create a text post
Endpoint: api/v1/posts
Method: POST
Body: {"userId": 1234, "text":"it's my message"}
-Create an image/video post
Endpoint: api/v1/posts
Method: POSTR
Body: {"userId":1234, "mediaUrl":"https://www...", "description":"relaxing in the beach"}
-Like/Comment a post
Endpoint-comments: api/v1/comments
Method: POST
Body: {"userId":1234, "postId": 1234, "comment": "beautiful, great spot"}
Endpoint-likes: api/v1/likes
Method: POST
Body: {"userId":1234, "postId": 1234}
-Follow/Unfollow another user
Endpoint-follow: api/v1/follow
Method: POST
Body: {"followerId":1234, "followeeId":1432}
Endpoint-follow: api/v1/unfollow
Method: POST
Body: {"followerId":1234, "followeeId":1432}
-Read the news feed
Endpoint: api/v1/feeds/{userId}
Method: GET
There will be 5 different databases in my design.
-Feeds DB structure
```
{
"userId": "unique_user_id",
"feedItems": [
{
"postId": "13452",
"userId": "84231",
"text": "sample post text 1",
"timestamp": "1720748111"
},
{
"postId": "86475",
"userId": "12394",
"text": "sample post text 2",
"mediaUrls": ["url1", "url2", ...],
"timestamp": "1720713459"
}
// More posts...
]
}
```
-Posts DB structure
```
{
"postId": "unique_post_id",
"userId": "user_id",
"text": "post text",
"mediaUrls": ["url1", "url2"],
"timestamp": "1720748150"
}
```
-Follow DB
```
{
"userId": "
"followers": [
{
"followerId": "
"timestamp": "1720748150"
},
// More followers...
],
"followees": [
{
"followeeId": "
"timestamp": "1720748150"
},
// More followees...
]
}
```
-Likes DB
```
{
"likeId": "unique_like_id",
"userId": "user_id",
"postId": "post_id",
"timestamp": "1720741345"
}
```
-Comments DB
```
{
"commentId": "unique_comment_id",
"userId": "user_id",
"postId": "post_id",
"comment": "comment text",
"timestamp": "1720748150"
}
```
-Follow/Unfollow mechanism
1.Client sends request to API Gateway
2.Api gateway sends request to load balancer
3.load balancer sends request to Follow Service
4.Follow service handles the request and puts a record inside the Follow DB (Graph db)
-Create a text post mechanism (I've made a diagram for this part)
1.Client sends request to API Gateway
2.Api gateway sends request to load balancer
3.load balancer sends request to PostWriter Service
4.PostWriter Service handles the request and creates a record inside the Posts db
5.PostWriter Service also puts the {userid,postid} inside a message queue
6.NewsFeed Generator service gets and item from the message queue, gets the post from the Posts Db, gets the followers from the follow db, generates the feed for the all of the followers of post owner and creates a record inside the Feeds DB, and also caches it inside the Feeds Cache.
-Create an image/video post mechanism
1.Client sends request to API Gateway
2.Api gateway gets a presigned URL from the PresignedURL Generator service and returns it back to the client
3.Client saves the image or video to the Object storage using the presigned URL and gets the uploaded item's URL back.
4.Client prepares the payload with the item's URL and sends request to API gateway
5.Api gateway sends request to load balancer
6.load balancer sends request to PostWriter Service
7.PostWriter Service handles the request and creates a record inside the Posts db
8.PostWriter Service also puts the {userid,postid} inside a message queue
9.NewsFeed Generator service gets and item from the message queue, gets the post from the Posts Db, gets the followers from the follow db, generates the feed for the all of the followers of post owner and creates a record inside the Feeds DB, and also caches it inside the Feeds Cache.
-Read the news feed
1.Client sends request to API Gateway
2.Api gateway sends request to load balancer
3.load balancer sends request to NewsFeed Reader Service
4.NewsFeed Reader Service sorts the feed according to created date of the posts, or simply use a edgerank algorithm for sorting.
4.NewsFeed Reader Service reads the user feed from the feed cache and returns it to the user
5.User gets images from the CDN using the item URLs inside the returned feed body and renders the content
-Comment on a Post
1.Client sends request to API Gateway
2.Api gateway sends request to load balancer
3.load balancer sends request to Comment Service
4.Comment service creates a record inside the comments db
5.Comments service puts a record inside a message queue
6.Third party notification service gets the record from the message queue and sends the notification to the post owner
-Like on a Post
1.Client sends request to API Gateway
2.Api gateway sends request to load balancer
3.load balancer sends request to Comment Service
4.Comment service creates a record inside the Likes db and likes cache
5.Comments service puts a record inside a message queue
6.Third party notification service gets the record from the message queue and sends the notification to the post owner
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
I used pre-signed URLs when users wanted to create a video or image post because resigned URLs give temporary access to the storage bucket and users will get the URL, and be able to upload the video/image. After uploading these items, the uploaded URL will be returned to the user and the user will send a POST request to api/v1/posts with that URL inside.
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?