System requirements


Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...


  • Posting tweets
  • News feed
  • Notifications
  • Follow/Unfollow
  • Search
  • Liking tweets


Non-Functional:

List non-functional requirements for the system...


  • Highly available
  • Partition tolerant
  • Eventually consistent





Capacity estimation

Estimate the scale of the system you are going to design...


  • DAU = 20 million
  • MAU = 200 million
  • 5 tweets per day for DAU = 20 mill * 5 = 100 mill tweets per day




API design

Define what APIs are expected from the system...


  • GET /feed - fetches the feed for a user
  • POST /tweet - allows user to create a tweet
  • POST /like - allows user to like a tweet
  • POST /follow - allows user to follow
  • POST /unfollow - allows user to unfollow
  • POST /update-profile - allows user to update their profile
  • GET /search - allows user to search for tweets



Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...


  • Tweets will be in an nosql db. This is because the data can be dynamic. We have different forms or media and we'd want this to be extensible in the future
  • We don't need things to be immediately consistent. For example, we don't need a new like on a tweet to be replicated across all db servers
  • Allows for high read/write load, and we need both of those
  • Eliminate the need for joins which allows it to be more efficient
  • Each tweet would have an id, tweet field which could store many types of media, like count, deleted_at, created_at
  • The feed service would have a cache of users feeds



High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...


  • The Client initiates requests to the Feed Service to retrieve timelines and interacts with the Post Service to create new tweets.
  • Before fetching data, the Feed Service checks the Cache for any frequently accessed tweets, enhancing performance.
  • When posting, the Client sends the request to the Fanout Service, which places the tweet in a Message Queue.
  • Both the Post Service and the Feed Service subscribe to this queue, allowing them to process new tweets as they come in, ensuring real-time updates.






Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...






Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

  • The fanout service puts tweets into a queue. The post service and news feed service subscribes to them and processes them. So the post service will write to the db. And then the news feed service will fetch following ids for each user and modify the user news feed accordingly.




Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...


  • I'm using nosql db for availability and partition tolerance. We don't need consistency because it's not that important for certain things like if someone likes a tweet. We also need high read/write and nosql will scale better for that in some of the complex relationships we have with followers and getting their tweets and building their news feeds. We'd have to do joins and that wouldn't scale well with millions of more followers added on.



Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.


  • Feed service goes down. Users would still be able to tweet but wouldn't be able to see their feed. Users could also still search tweets.
  • Post service goes down. Users can still get their feed but can't post or like.
  • I can't think of any bottlenecks right now.




Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?


  • We could decouple the logic to display the feed from the data layer into a separate service if the feed service goes down.