System requirements

Functional:

• create tweets
• share links to tweets
• go to a users profile to see their tweets
• A feed of the users followed twitter accounts
• notifications of the tweets for users they follow

Non-Functional:

• We want to be able to update the tweets in real time and have the users tweets be highly available.
• We do not want to lose tweets from users and want to make sure our system is fault tolerant.
• Make sure that we are available

Capacity estimation

API design

Post: "/tweet/:id" user sends a tweet that goes to the db

Get: "/tweet/:id" user gets a tweet and the information associated with it

Get: "/user" gets a users tweets that they have submitted

Get: "/feed" get the feed of all tweets that the user subscribes to

Database design

The database will have a table for the tweets and the users. The users table will have a list of tweets and who the user follows.

Tweets(text, images, videos, metadata, creation time, location)

Users(tweets[], username, who they follow, account_age, bio)

High-level design

A web front end framework that talks to the server. User can upload or view tweets from here.

A server that takes in the information and stores or looks up tweets from specific users.

A load balancer that will distribute request between multiple servers so that no one server gets over whelmed.

We will have a mysql store that will store the tweets and users.

We will have a cache that returns the most recent tweets, such as redis.

Request flows

The user can upload a tweet and this is sent to the server. The server publishes the tweet and store it in the database and tells the user the tweet has been posted. It also tells the notification service that we have a new tweet.

The notification service will get a new tweet and send this to users who have subscribed to the user.

The user can go to a page for a user and get all the tweets that that user has tweeted. This shows as a list and if there are new tweets the user will get a notification from the notification service via a websocket that updates the tweets they have.

Detailed component design

Notification Service: The notification service will be listening to the post service. There will be websockets that connects it to users who are currently active and also to a service for pushing notifications to users who are offline. When a new post is made the post service will inform the notification service and the notification service will look at the database to see who all follows this user and will send a notification to them in real time using websockets. The websockets will keep a realtime connection to these users so that they do not have to poll or periodically look up new tweets and will get real time information.

User subscription: Users can subscribe to a user. This will conenct them to a message queue that will be listening to the notification service. When the notification service informs the message queue that there is a new post the user will be informed directly and sent this new tweet from the user that they follow.

Database Storage: We will have a MySql database that stores the above information. We will have an index on who a user follows. That way when we look at a user we can quickly get a list of all the users who follow them for the notification service.

Trade offs/Tech choices

We want to replicate the information so that the system is fault tolerant. We do not want one failure to cause issues. This also means that we will have to have a system that is eventually consistent. But this is alright so that one failure does not knock out the entire system.

The use of websockets is the best choice to give realtime notifications and prevent us from having to send polling constantly from the users to the servers which would cause more use of resources.

Failure scenarios/bottlenecks

If the notification service fails we will have a follower that will take up the case in the event that one server fails. We have a load balancer that keeps track of the state of the machines and sends information to a machine that can handle the failure.

If our data store fails we will have replicated data storage. That way we can have backup in the case of a failure.

The user subscription service is connected to the user and the notification service. It is an in between, but if one goes down we will have multiple that can take it up in case of a failure.

Future improvements

In the future we might want to find a way to make the database better at handling replication and sharing data. We might even change the storage to be have a leader follower model where if there is a failure another database is elected as the leader and the system is more fault tolerant.