Loading...
List the key functional requirements for the system (Ask the AI for hints if stuck)...
List the key non-functional requirements (performance, scalability, reliability, etc.)...
Each tweet will be sized at 15KB. We will be having around 500 million tweets per day. That would mean 500 * 15 KB = 7.5 TB of storage per day
Each tweet fans out to followers. 300 Avg followers 500 million tweets per day -> 150B timeline inserts ops/day
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
POST /api/v1/createTweet
POST /api/v1/shareTweet
GET /api/v1/getTweet/{id}
GET /api/v1/getTweets?limit=30
DELETE /api/v1/deleteTweet/{id}
POST /api/v1/createUser
GET /api/v1/getUser/{id}
GET /api/v1/getUsers
DELETE /api/v1/deleteUser
POST /api/v1/followUser/{id}
POST /api/v1/unfollowUser/{id}
GET /api/v1/feed
POST /api/v1/LikeTweet/{tweetID}
POST /api/v1/UnlikeTweet/{tweetID}
CDN: It is used to host the frontend of the service. The response is fast because it is in edge locations. Most of the time it handles 80% of the reads since it can cache as well.
API Gateway: To route traffic based on the endpoint getting hit. Used for authentication and for rate limiting the requests.
Kafka: It is used as a high throughput queue to store the requests that are received. With this even if there is a down time it is easy to recover as kafka will have the missed requests. User will receive 200 OK
User service: Will handle all the apis related to users.
Tweet service: Will handle all the apis related to tweets
Feed service: Will handle all the apis related to feed.
Search service: handles the search functionality. It uses elastic search for this.
Redis:To cache the requests and take care of concurrent redundant requests and reduce load on the db and response time. and to store sorted sets so that delivery is fast.
Postgres - persistent storage to have all the data of the invocations, statuses and events that are taking place. Every invocation will have idempotency key to avoid redundant runs. We will be sharding the DB to avoid hot keys.
We will be going with postgres with shards/partitions for this application. It will have the following tables
User:
id, name, username, hashed_password, verfied
id, username being the composite index
Tweet:
tweet_id, user_id, content, interaction_id
tweet_id, user_id - composite key
user_id, likes - composite key
Tweet_Interaction:
interaction_id, likes, dislikes, shares, comments
interaction_id - index
follow graph lives in a separately sharded table (graph db cassandra)
Kafka: It is used as a high throughput queue to store the requests that are received. With this even if there is a down time it is easy to recover as kafka will have the missed requests. User will receive 200 OK immediately in case of failure the request is enqueued safely.
The messages will be partitioned based on userID to let it scale horizontally. We can have priority queues as well so that important alerts like OTPs and security alerts are let out first.
Postgres: persistent storage to have all the data of the tweets, likes. Every interaction will have idempotency key to avoid redundanancy. We will be sharding the DB to avoid hot keys.
We will be using exponential backoff for retries to handle the failed events. In case the max retries are used up we will send it DLQ so that it can be investigated.
When a cache miss happens we will be reading directly from the DB and then cache the entry into redis so that subsequent requests hit the cache.
We will be using precomputed feed for normal users and build on read for celebrities.
Recency with a recency-score boost.