System requirements

Functional:

1. User can tweets.
2. User can search tweets based on text.
3. When searching, user's following's tweets should return first.

Non-Functional:

1. High availability
2. Eventual consistency. To achieve high availability, it's ok to wait for two or three seconds to allow new tweets to become searchable.
3. Security.
4. low latency.

Capacity estimation

Assume 1B daily active users for twitter.

Read QPS: 1B / 100k = 10k

peak read qps have twice traffic: 2 * 10k = 20k

Assume 1% user post 10 tweets per day.

Write QPS: 1B * 1% * 10 / 100k = 1k

peak write QPS have twice traffic: 2 * 1k = 2k (high write throughput)

Storage usage estimation:

Assume one tweet has average of 10kb storage. 90% tweets are text tweets. 10% tweets are image/video tweets.

Daily storage: 1B * 1% * 10 * 10kb = 1TB

Yearly storage: 400TB

We need 3 replica for the db. So yearly storage: 1.2PB

API design

GET searchTweets(userId, text)

return the list of tweets

POST postTweet(userId, tweetInfo)

return success/error

Database design

NoSQL database to store the tweets information. I choose NoSQL database because the tweets information is flat data and the write throughput put is very high. NoSQL db could handle high write throughput.

Tweets Table

TweetId (primary key)

tweet content

tweet media

tweet user (reference key to user id)

Document DB is used to store text information for tweets. Document DB is easy to do multiple index on the text information for fast search.

High-level design

Please see the diagram

Request flows

Please see the diagram

Detailed component design

To make search

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?