System requirements

Functional:

• allow users to search tweets based on tags/time/location etc.
• return 100 results in reverse chronological order
• results don't need to be personalized

Non-Functional:

• highly available
• low latency
• eventual consistency

Capacity estimation

500M DAU, assume each posts 1 tweet, search 30 tweets daily.

Write

500M/10^5 = 5000 TPS

Read

500M * 30 / 10^5 = 150k TPS

Storage

500M * 1KB * 365 = 1.82PB/year

API design

• post tweets
  • create_tweet(title, content, user_id)
• search tweets
  • search(key_words, limit, offset)

Database design

Tweet

• id
• user_id
• title
• content
• url
• created_at
• last_updated_at
• tag
• location

Partition key: user_id + location for the following reasons:

1. Search often involves user, for example, see all the tweets for one of my friends.
2. location is another important factor when searching, most of the case we search tweets with the same country/city we are in.

Choose NoSQL given the huge amount of storage we need. Also we don't need strict consistency in this use case. NoSQL can provide better read/write performance.

High-level design

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

We will have two main service, app server and the search service. The former handles tweets creation, and forward all search requests to search service.

When a tweet is created, app server inserts it into the databse/cach, then puts the id to message queue. An index service will pull it from the queue, creates reverse index (index to document), and inserts into elastic search for future search query.

When a search request comes, search serivce first check if the request is stored in cache, if not, it invokes elasticsearch to get the results.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?