System requirements
Functional:
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
1. A user should be able to post a tweet with media content
2. A user should be able to see a chronological order of tweets starting from the most recent.
3. A user should be able to follow other users.
Non-Functional:
List non-functional requirements for the system...
1. The system should have high availabilty to keep users engaged and provide a good customer experience.
2. The system should have good maintainability to catch errors before they occur and ensure the system keeps running.
3. The system should be built to scale so it can handle many users at once.
4. The system should be able to handle certain users that have many followers such as the "celebrity problem"
Capacity estimation
Estimate the scale of the system you are going to design...
300M Total users
60M DAU
Each tweet is 140 bytes + 1000 bytes for compressed media content
Each user can tweet or read up to 20 posts a day.
Total Data throughput is for one tweet per user
60 * 10^6 * 1140
65000 * 10^6
65 * 10^9
65,000,000,000 billion bytes = 65 GB
Total data per day would be 65 * 20 * 2 (data replication) = 2600 GB = 2.6 TB daily
Total transactions per second:
(60 * 10^6) / 10^5 = 60 * 10^1 = 600 transactions per second
600 * (65* 10^9 / 10^5 = 65 * 10^4 = 650000)
600 * 650000
6 * 10^2 * 65 * 10^4
6 * 65 * 10^6
6 * 65000000
240,000,000 240 million bytes = 240 MB max per second
API design
Define what APIs are expected from the system...
https://getNewsFeed/userID (GET) (get the tweets for the logged in user)
user_id, timestamp
https://getTweet/post_id (GET) (view a specific tweet)
user_id, timestamp, post_id
https://createTweet/post_id (POST) (create tweets)
user_id, timestamp, post_id, content, media_content
https://commentTweet/post_id/messageID (POST) (comment on a tweet)
user_id, timestamp, post_id, content, media_content, message_id
https://getCommentOfTweet/post_id/message_id (GET) (view a comment of a tweet)
Database design
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
users
user_id: string PK
user_pw_hash: string
timestamp: string
index: user_id
follower_table:
follower_id: string
followee_id: string
Tweets:
timestamp: string
post_id: string PK
user_id: string FK
content: string
media_content_url: string
index: timestamp'
comments:
timestamp: string
post_id: string FK
user_id: string FK
comment_id: string PK
objects
media_content: blob (raw binary data)
media_url: string
High-level design
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Request flows
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
1. The client retrieves data from the Media CDN Cache if applicable.
2. The Client communicates with the Load balancer which evenly distributes the load to the API Gateway(s).
3. The API Gateway manages authentication, rate limiting and other security checks. After the request is verified it is forwarded to the approriate comment/tweet/celeb/follow service.
4. The Tweet service handles the creation of tweets and commenting of tweets.
5. The Home Feed service manages the timeline of the users tweets from the users they follow.
6. The follow service manages the following of users
7. The celeb service manages the users who have an incredible amount of tweets associated to them.
8. All services and databases communicate with the Coordinator. The Coordinator actively monitors all databases and services to ensure they are working and to route the request to the proper location./
8. Creating/updating regular tweet requests goes through the Primary Write DB. The write DB is sharded by user_id and replicated to the follower read DBs.
9. If a user is trying to read a tweet they will first hit the TWeet cache otherwise they hit the follower read DB.
10. Media content is not store in the database, but instead stored in the object storage DB. The UrL for the object storage is stored in the database.
11. the NoSqL database is used for the Celeb service which needs an extra performance boost due to millions of associated records.
12. Tweets are written to the Tweet Cache if they are not there or out of date and the content is returned to the client via the TWeet cache.
Detailed component design
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Caching:
1. The Media CDN cache stores pictures or videos in memory for fast retrieval. The CDN Server is also close to the client, this greatly improves latency since other servers are typically in different regions and not close to the client.
2. The Tweet Cache stores regular tweets.
3. The write Database uses the write-around to cache technique where data is first written to the database and then it periodically pushes the data to the cache.
4. Caches will have a time to live property where data is set to be expired have a set amount of time.
Pros:
- Improves latency and user experience which helps provide high availability, this is mostly because reading from memory is much faster than reading from a database.
- Read requests are prioritized and their performance is increased.
- Helps the system scale because reading from the cache lessons the amount of requests other components in our database needs to handle. This helps us achieve our Scailability requirement.
- The caches are their own micro service, which allows us to scale them independently because they're not "coupled" with other micro services.
Cons:
- Cache misses are a possibility. We use the write-around cache technique which currently does not guarantee the data will be completely up to date. We accept this trade off since we're not dealing with overly critical information such as financial information.
- Complexity. This adds another layer of complexity to our architecture that now we have to manage and keep up to date and maintain. Meaning we have to handle nodes being promoted/demoted due to failures or overall scailability issues.
- Can slow down writes. This has the potential to slow down our write requests, espeically if we start to implement a feature where we don't actually write to the database unless it is written to the cache first.
API Gateway:
1. First the gateway checks the parameters of the request. For example it verifies the params are valid: https://createTweet/post_id. Here, we're expecting a createTweet param and a post_id param.
2. Second the gateway checks its allow-list /deny-list. The gateway holds a security list of ip addresses it allows to enter the system and ip addresses it denies from entering the system.
3. Authorization. The API gateway authorizes the request by either signing in the user creating a stateless session or using the stateless session or verifying that the stateless session is valid. The stateless session is retrieved from the database.
4. Rate Limiting. Rate limiting is used to ensure the system does not get over loaded with too many requests which is common in denial of service requests. Rate limiting can be achieved by tracking the IP addresses of a certain request and only allowing a maximum of lets say 5 requests per minute. If more than 5 requests are picked up per minute we deny the request.
5. The request is routed to the proper service. The API uses the structure of the API request to route the request to it's proper request. For example the API https://createTweet/post_id tells us to go to the tweet service and to retrieve the post_id from the database server.
6. The API Gateway helps manage logging and perform monitoring by using heart beats to verify services are still up and running. This is critical for us being able to achieve our goal of Maintainability. The API Gateway also does load balancing since it has information on nodes that are down/up at any given time it can route request to the proper location. The API Gateway stores logs to help the system identify errors and bottlenecks and react accordingly.
Home Feed Service:
1. The home feed service retrives data from the Tweet cache if possible. If it's not in the cache the tweet service communicates with the coordinator which in turn communicates with the various partitions to pull all applicable tweets for the user. The user table and the tweet table are joined together by the user_id, the new joined table is joined together with the follower table via the followee_id. All tweets are sorted in chronological order via timestamp where the first tweet is the most recent. We also only take the first 100 tweets to ensure our system is not overloaded with millions of old tweets. The new home feed is stored to the cache and the cache returns the information to the user
Trade offs/Tech choices
Explain any trade offs you have made and why you made certain tech choices...
Caching:
We implement the write-around cache technique and accept cache misses. We're okay with eventual consistency which means eventually all caches will have the most up to date information. We do this for performance reasons and because most users read tweets and do not write tweets.
Writes:
We prioritize a SQL table with a B-tree data structure behind the scenes which prioritizes read requests. This is because B-trees are primarily stored on disk in a binary tree page like structure which allows fast read requests using binary search. This helps achieve logarithmic time complexity. However, choosing a B-tree like data structure does not prioritize writes becausing writing to disk is slow. This is because disks store data in a ring type of format where data is not sometimes close to one another. When we perform a write request we may need to update various parts of our B-tree data structure which can be slow since it's on disk.
Failure scenarios/bottlenecks
Try to discuss as many failure scenarios/bottlenecks as possible.
The Celeb Service may need it's own cache since it likely will have so much information it may overload the normal tweet cache. This is because some users could have 60 million followers and it's tweet cache could be very large
Future improvements
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
A New celeb cache will be implemented and will be only for the celeb service. This should decouple this issue from the main tweet cache and lessen the chance of our main cache having issues which could impact all users in the system. Decoupling will also allow us to scale this indepently which is good for scailability. The new tweet cache will use a write-through cache technique which means the cache and the database will be in-sync. Since the system does not expect many celebs to be on the celeb service, taking a little more time to write should not be an issue. Also this will help with read requests