Loading...
2 billion DAU
100 to 1 read to write ratio
post 1KB
ratio of images 0.6 and average size of 1MB
ratio of videos 0.1 and average size of 10MB
Posts uploaded a day: 10^9
Posts read a day: 100* 10^9
Storage capacity:
Posts: 10^9*1KB = 1TB a day -> 365TB a year -> 1.8PB 5 years
Images: 0.6*10^9*1MB = 0.6PB a day -> 219PB a year -> 1EB 5 years
Images: 0.1*10^9*10MB = 1PB a day -> 365PB a year -> 1.8EB 5 years
Total Storage: 585PB a year
Bandwidth
2PB a day -> 2PB/(24*60*60) * 8 = 184Gbps
Server for peak load:
assuming dau can be peak load and each server can handle 64000 requests
2*10^9/64000 = 31250
POST /api/v1/post
{
auth_token,
content,
image_ref,
video_ref,
}
Get /api/v1/posts/{post_id}
{
content,
image_ref,
video_ref,
}
Get /api/v1/newsfeed?limit={x}&since={last_check_time}
({user_id} is fetched from auth_token)
[post_ids]
POST /api/v1/posts/{post_id}/like
POST /api/v1/posts/{post_id}/unlike
POST /api/v1/posts/{post_id}/comment
{
comment_content
}
GET /api/v1/posts/{post_id}/comments
[comments_id]
GET /api/v1/comments/{comment_id}
{
sender_id,
comment_content
}
POST /api/v1/follow
{
followee_id
}
(follower_id is fetched from auth_token)
users table SQL
{
pk: user_id,
email,
creation_time,
name,
}
user_followers table SQL
{
pk: follower_id, followee_id
}
Both tables will be sharded by user_id (follower_id in user_followers table). We need sql server since we need to have acid properties like unique email and user_id, foreign_key for user_followers follower_id and followee_id, and relational queries to find followers.
The rest of the data will be stored in lsm-tree DB like dynamodb or cassandra since we need to support high throughput.
post_id is user_id#sequential_id
posts table NOSQL
{
pk: user_id,
sk: sequential_id,
creation_time,
content,
video_ref,
image_ref
}
like table NOSQL
{
pk: post_id,
user_id,
creation_time,
}
comments table NOSQL
{
pk: post_id,
sk: comment_id (sequential id)
user_id,
creation_time,
content,
video_ref,
image_ref
}
The posts table will be sharded by the user_id. The In case of a hot shard for some users, we can further shard them by using sub paritions in the pk like USER123#0 and USER123#1.
The other tables will be shared by their primary key, which is post_id
Graph DB to query friends of friends
Nodes:
{user_id, name}
Edges:
{
label: follows,
}
Blob Storage:
Images and videos
Load balancer is used to distribute users requests
api gateway routes requests to the right service and also performs authentication and authorization to requests and also rate limitining if users performs too many requests
post service is responsible to storing and fetching posts, likes, and comments.
Kafka events store is used to store user's events like posting, adding a like or a comment. Kafka is used in order to decouple these events processing from the critical path and enable low latency for users' requests.
newsfeed generator is generating newsfeed by ranking posts using the ranking service and then storing the posts in an in-memory db like redis so users can quickly fetch newsfeeds. the newsfeed generator is used both in response when users upload new posts and for celebrities posts newsfeed generator service will rank and push celebrities posts while their followers request a newsfeed.
The newsfeed service will fetch newsfeed from the newsfeed cache and by generating on the fly ranking for posts from celebrities.
the notification service will notify users on viral posts.
Posts requests:
like/comment requests:
Fecthing a newsfeed:
Post service
the post servers are stateless; therefore, we can autoscale them based on cpu load. They store posts/likes/comments in a nosql db like (like dynamodb) that supports high throughput write. The DB is replicated for fault tolerence and auto-scales based on user_id sharding. in case of a user with a lot of posts and like/comments traffic, we can futher shard by adding sub partitions to the user_id.
Newsfeed service
The servers are stateless so they could support scaling based on cpu load. The posts for non-celebrities are already ranked in the newsfeed cache so they will be fetched quickly to support low latency for requests. Celebrities posts will be ranked on the fly. The newsfeed cache will be sharded by user_id, and since newsfeed requester only requests posts for himself, no join in needed between the shards. The posts will be stored in redis in a sorted set data structure based on their rank score, so we can query for top N ranked posts quickly in log(n) time.
The Kafka event store is used since it's a highly available and scalable component. It's used to decouple the user's events processing from the user requests processing. Thus, making our system more available and fault tolerent to partial failures to outages and network partitions in the newsfeed generator and notification services. The kafka event store will be partitioned based on topic per user.
Pull vs. push model
there is a trade off between saving storage space by using pull for every newsfeed request to using less compute and lower latency by using the push model. Since we want our system to have low latency to improve user experience, we will use a hybrid approach: push model for regular post events and pull model for celebriteis post events. When a user requests a newsfeed, it will fetch ranked posts from the cache for regular users and will compute on the fly posts from celebrities that the user follows. ranking posts for celebrities for all of their followers will require a high amount of compute which will create a spike everytime a celebrity uploads a new post.
There is a trade off between consistency to availability and lower latency. Since users can tolerate a few seconds until they see a friend's post, the system can tolerate an eventual consistent DBs and we favour availability and low latency to enhance user's experience.
Storage cost vs. performance: storing newsfeeds in cache will make read faster and improve users experince by providing newsfeed faster. This will be at a cost of RAM storage. We will need to balance that by storing and ranking posts of users without a lot of followers; thus, minimizing the amount of compute and cache storage that is required.
DB failure or overload
The DBs are sharded by user_id and replicated. That will reduce the load on each DB instance since each instance only handles a subset of users and in case one instance is overloaded, the data can be read from another replica. The DBs are also replicated, so in case of failure, the data is stored in other instances and automatica failover will pick up new master or instances as replicas.
Newsfeed cache failure: in case of newsfeed cache failure, the notification service can rank posts on the fly. If rankning takes too long, we can use a lighter ML model to rank posts and return posts quickly. In case ranking service is down as well, we can return posts based on chronological ordering. Therefore, for each failure, we degrade gracefullly while still returning a response to the user.
network partitions and outages: in case of network partitions or outstages of the notification or newsfeed generator services, the events are stored in kafka and decouples from the post service requests; therefore, it will not impact the user request path and we can use the kafka as a buffer in case of traffic spikes. In case of certain of unhealthy servers, the load balancer and service discovery should pick them up and route traffic to healthy services.