Requirements
Functional Requirements:
- Allow users to tweet messages up to 140 characters.
- Enable users to follow other users.
- Allow users to like tweets from other users.
- Display tweets from followed users in the home feed.
- Show top K popular tweets in the home feed based on likes and followers.
Non-Functional Requirements:
- Fetching the feed for an user should be low-latency, <3s is acceptable
- Availability is crucial and is preferred to consistency.
- Have an eventual consistent system, ideally with RYOW guarantee.
- Scalable system
API Design
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
POST /api/twitter/tweet
Authorization: Bearer
{
content: str[140]
} -> Tweet
POST /api/twitter/follow
{
user_id: str
}
POST /api/twitter/like/:tweet_id
{
}
GET /api/twitter/feed?count=...
-> List[Tweet]
Tweet {
tweet_id: str
content: str[140]
like_count: int
}
High-Level Design
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
For the design we need a DNS server to provide a user-friendly address and also to geoproximity routing and load balancer failover.
I have devided the designs into 3 main services:
Follow service receives requests for following a friend and adds it to Neo4j Graph database.
Tweet Service is used for posting a tweet, it uses a distributed ID Generator (I assumed we have it) to generate the id for the tweet the posts the tweet in a NoSQL database such as MongoDB. It also pushes the tweetId and content to Kafka. Tweet Service only pushes to Kafka if the user is not a celebrity.
Workers listen to Kafka and take the tweetId and content and push them to Redis.
Redis is sharded by user_id, so every user will have a cached list of the tweets of their friends.
Feed Service is used to display the top K tweets. It fetches the data from redis then combines it with the data fetched directly from MongoDB which corresponds to celebrities.
Detailed Component Design
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
Follow Service:
- It's purpose is to add new friends to Neo4j, the Graph database.
- We will have a Neo4j cluster with one Leader accepting writes and multiple Read Replicas.
- If Neo4j Leader fails, we elect a new Leader via Consensus algorithm
Tweet Service:
- Uses a distributed ID Generator to get a unique tweet id. This ID Generator can use the Twitter Snowflake algorithm to work in distributed env.
- Determines if the caller user is a celebrity or not by querying Neo4j.
- If not a celebrity, push the tweet to Kafka queue. Workers regularly pull from Kafka and add this tweet to a Redis cache for the friends of the user.
- If a celebrity, only push directly to the main MongoDB as the cost for pushing to potential millions of followers is too large.
- This balanced approach assures low latency (push model) and also reduces the time having stale data(if we used push model for celebs then the workers would take a very long amount of time, time in which the user would not have actual tweets in their feed).
Feed Service:
- It takes the data from Redis for the person the user is following. For celebrities it takes the data from MongoDB synchronously.
- To fetch the top K, it combines the data from Redis with the data from MongoDB. Redis holds only the IDs while MongoDB also holds the like count. We assume there is a like service that increase the like count of a tweet and pushes to MongoDB.