Loading...
User able to see tweets from people they follow
User able to post tweets
User able to follow or unfollow other users
Availability over consistency
It's okay if we don't see tweets from those we follow for a few seconds
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
GET feed(user_id, before_tweet_id?)
before_tweet_id for pagination to show some tweets before a certain tweet when scrolling
POST tweet(user_id, text, media?: image/video)
POST follow(follower_id, followee_id, unfollow?: bool)
Many active users so simply storing tweets and following relationships in separate tables and joining the on feed requests is not tenable/way too slow
Instead we should also store data in the way it is going to be queried. We should store each users feeds. When a user tweets we should asynchronously insert that tweet into each of their followers feeds. If in the meantime while this is happening in the background that follower requests their feed it is okay if the new tweet hasnt hit the feed yet and their feed is slightly stale
This introduces the celebrity problem -- when users with many followers tweet many feeds need to be updated. Will discuss that in detailed component design section. We will store Celebrity tweets in Redis (and may also store a copy of them in DynamoDB for redundancy)
There will also be a CDN for additional static data such as images and videos
DynamoDB
Tweet table
tweet_id, user_id, timestamp, tweet_text, media_id?
Follower table
follower_id, followee_id
Fan table
follower_id, celebrity_id
Feed table
user_id, tweet_id, timestamp, tweet_text, media_id?
Celebrity_table
celebrity_id, tweet_id, tweet_text, media_id?
Redis
Celebrity table
celebrity_id, tweet_id, tweet_text, media_id?
When user requests feed
App server makes 2 requests to DB: 1. for feed table for some small number of rows (we could say 50). 2. for Fan table. Now we know what celebrities this user follows. Then we pull the celebrity tweets from the Redis cache, and combine those with the feed table results to return to the user.
When user tweets
App server makes request to celebrity table. If user is celebrity we store their tweets in tweet table and celebrity table. If user is not celebrity, we store their tweet in tweet table and publish the tweet event to Kafka Queue.
In the background, a worker will soon pick that event up from the queue. That worker will get all the followers of the user from the DB from the Follower table. It will then make another request to the DB to insert the tweet into the Fan table for all of the followers.
When user becomes celebrity
When a user reaches a certain number of followers, we designate them a celebrity. We push this event into Kafka under a different topic. We have background workers reading from this topic. This worker goes to Redis and DynamoDB to populate the Celebrity table and Fan table. We could reuse the Feed worker pool for this purpose, or use a different worker pool if we find it is useful to scale and operate these independently.