System requirements


Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...

  • post a tweet
  • get timelines (reversed chronological order)
  • can follow up to 5000 users
  • 2400 tweets per day

Non-Functional:

List non-functional requirements for the system...

  • 100,000 DAU
  • 2400 tweets per day
  • 280 characters per tweet
  • text only (TBD: image/video)
  • support mobile/web





Capacity estimation

Estimate the scale of the system you are going to design...

2400*100k tweets per day

write qps: ~3000/s

read to write ratio: 10:1

read qps: 30000/s

storage:

100k users *2400 tweets *280 byte* 365 days = ~ 28 TB / year




API design

Define what APIs are expected from the system...

POST /v1/me/tweet

param: content, auth token

return:

{

code:200

status: "success"

}


GET /v1/me/timeline

param: auth token

return:

{

tweets:[

{

"id":1

"author":"user1"

"content":"hi"

"post_time" "mm-dd-yyyy"

}

]

}



Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

tweets

id

user_id

content

post_time



High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

Tweet Posting:

  • post service: persist tweets in db and cache.
  • fanout service: push tweets to users' timelines. For fast retrieval, tweets are stored in cache.
  • notification: notify users when a user they followed tweets, by sending a notification.
  • load balancer: necessary given a large user base (100k DAU)

Timeline building :

  • timeline service: fetch tweets from cache




Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...






Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...


  • auth: only users who sign in can tweet
  • rate-limiting: restrictions on the number of tweets a user can post
  • mq: decoupling fanout service and fanout workers, given huge traffic is expected



Fanout services: two types of fanout: push/pull





Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...


push: timelines are pre-computed

pros:

  • timeline generated in real-time. Tweet pushed to users immediately.
  • fetching tweets is quick

cons:

  • hotkey: computation-intensive/slow/time-consuming
  • waste of computing resources if a lot of inactive users

pull: timelines generated during read time

pros:

  • save computing resource for inactive users
  • no hotkey problem

cons:

  • slower timeline loading


we can use both strategies for different types of users:

For users having a lot of followers: pull model

for active users: push model



Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.


consistent hashing for hotkey problem



Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?


support image/video

master-slave replication

db sharding