System requirements

Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...

Users should be able to create tweets

Users should be able to search their own tweets

Users should be able to search texts from other users (does not have to be their followers)

Non-Functional:

List non-functional requirements for the system...

Availability - The application should be highly available given that twitter is a global service

Latency - We want our search latency to be relatively low. It does not need to be immediate but should be

Consistency - We are a social media application where users do not need to see the most immediate information. We will aim for eventual consistency

Scalability - Our application should be scalable

Capacity estimation

Estimate the scale of the system you are going to design...

Assuming we have 100M daily active users.

Each user makes at least 1 tweet per day

Each tweet is on average 1KB

Each day, we expect to use 1KB * 100M = 100GB of storage per day for tweets.

For a year we will need 40TB of storage per year. Assuming 3x replications and indexing we will need about 150TB for a year

For qps

Assuming at peak, 1/5 of our user base will be using our search. 20M search requests / 24hrs / 60m / 60 s = 20M / 100,000 = 200 q/s (read)

For write TPS we are expecting 100M / 100,000 = 1000 q/s (write)

Given the capacity estimation we will need a database that will can scale horizontally and handle higher throughput.

A NoSQL database will filfull these requirements better than a SQL database. We can use a column db such as Cassandra

API design

Define what APIs are expected from the system...

POST createTweet(user_id, text) -> returns status code

POST follow(user_id, follow_user_id) -> returns status code

GET search(user_id, text)

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

tweets table

{

tweet_id (primary key)

user_id, (references who wrote the tweet)

message, (text of the message)

timestamp, (when the tweet was created)

hashtags, (hashtag mentions of this tweet)

media_links (media links)

}

as our database grows, we can split our data into different shards using the user_id

High-level design

For frequently accessed contents, such as tweets from popular users, images and videos can be cached in a CDN, in a geo location close to the large population of users.

We will use API Gateway to rate-limit requests and prevent DDos attacks and smooth out surge requests beyond

I will split the application server into two servicees, a Tweet Service and a Search service for separation of responsibilities.

The Tweet Service receives a tweet and stores the message in a DB. At the same time, it creates a message in the Message Queue like Kafka. The indexer pulls this message, reads tweet text, and creates index. The indexer also extracts hashtags and users mentioned.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

1. After a tweet has been posted, we will store the tweet into the tweets database and asynchronously push the tweet in a message queue for the indexer to consume.
2. The indexer will consume the tweet from the message queue and then reads the tweet from the database while creating an inverted index, which is a hashmap which contains keys from search words in a document.
3. Whenever a user searches for key words, hashtags or people mentioned in the tweet, we will check the the cache to see if the same or similar search terms were queried.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

1. Since we are opting for eventual consistency, users might not see the tweet from the users it follows immediately, there will be slight delay there

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

1. Allowing searches to be more granular, allowing for more filters and letting the user sort on the basis of certain fields
2. Allow users to block certain users
3. Handling deleting tweets and archiving tweets
4. A service that vets tweets for harmful speech and content