System requirements

Functional:

1. User Registration:
2. Users can create accounts with unique usernames and passwords and log in securely to access their accounts.
3. Compose Tweets:
4. Users can create tweets with a character limit.Tweets can include text, media, and hashtags.
5. Follow/Unfollow Users:
6. Users can follow/unfollow other users. The system should provide suggestions for users to follow.
7. Like Tweets:
8. Users can like tweets, each tweet should show count of likes.
9. Timeline:
10. Users have a personalized timeline displaying tweets from users they follow.
11. The timeline should show tweets in chronological order.
12. Search:
13. Users can search for other users by username.
14. Users can search for tweets based on keywords or hashtags.
15. Notifications:
16. Users receive notifications for likes, retweets, and new followers.
17. Notifications should be timely and customizable.

Non-Functional:

1. Scalability:
2. The system should handle a large number of users and tweets.
3. It should scale horizontally to accommodate increasing user activity.
4. Performance:
5. The response time for actions like tweeting and liking should be fast.
6. The system should minimize downtime and ensure high availability.
7. Security:
8. User data should be encrypted during transmission and storage.
9. The system should protect against common security threats, such as unauthorized access.
10. Reliability:
11. The system should recover gracefully from failures to prevent data loss.
12. Backup and recovery mechanisms should be in place.
13. Privacy:
14. Users' personal information should be handled with strict privacy measures.
15. The system should comply with relevant data protection regulations.

Capacity estimation

Assumptions

• Total user base of 1 billion users.
• Monthly active users 200 million.
• Average number of tweets per user - 5
• Each tweet has an average size of 500 bytes
• 10% of these tweets contain images or videos
• Each media tweet will have a size of 500KB

With the above assumptions, we can say the below

Total number of tweets per day : 200M * 5 = 1 billion tweets / day

Tweets with media = 10 million tweets / day

Number or write requests per second : 1 billion / (24 * 3600) = 12k requests / sec

Storage Estimations 

Text Tweets : 1 billion * 500 bytes = 500 GB / day => ~ 950 TB for 5 years

Media Tweets : 10 million * 500 KB = 5 TB / day => ~ 1 Petabyte for 5 years

API design

1. User Registration API: Manages user registration and authentication, using the provided username, password, and email to return the user authentication token.
2. Tweet Composition API: Enables users to compose tweets, taking tweet content and the user authentication token as input and providing details of the posted tweet as output.
3. Follow/Unfollow API: Handles user follow/unfollow actions, taking the target user ID, follow/unfollow action, and user authentication token as input, and returning a success/failure status.
4. Like Tweet API: Manages like/unlike actions on tweets, taking the Tweet ID, like/unlike action, and user authentication token as input, and returning a success/failure status.
5. Timeline Retrieval API: Retrieves a user's personalized timeline, utilizing the user authentication token as input and returning a list of tweets in chronological order as output.
6. Search API: Facilitates user and tweet searches, taking the search query and user authentication token as input, and returning a list of matching users or tweets.
7. Notification API: Delivers real-time notifications to users, using the user authentication token as input and returning the list of unread notifications for the user, if any..

Database design

For the tables required in this design, refer to the class diagram, the list of classes is not exhaustive but this is a good number of tables to start with.

Class Diagram

Database Choices:

• User Data and Authentication 
• CAP Theorem: Consistency-focused
• Database Type: Relational Database (e.g., MySQL, PostgreSQL)
• Reasoning: User registration and authentication demand strong consistency. All nodes in the system must have consistent and up-to-date user data.

• Tweets and Interactions
• CAP Theorem: Availability-focused
• Database Type: NoSQL Database (e.g., Cassandra, MongoDB)
• Reasoning: The primary purpose of the system is to allow users to compose and share tweets, like tweets, and follow/unfollow other users. NoSQL databases can provide high availability and scalability for such read and write-heavy operations, even during network partitions.

• Timeline and Notifications
• CAP Theorem: Consistency-Availability Balance
• Database Type: A combination of Caching (e.g., Redis) and NoSQL or Search Engine (e.g., Elasticsearch)
• Reasoning: Personalized timelines and notifications require a balance between consistency and availability. Caching mechanisms can enhance read performance, while a NoSQL or search engine database can handle real-time indexing and querying.

Data Partitioning:

• Which Partitioning Strategy should we apply here?
• Strategy: Hash-based partitioning.
• Reasoning: Hash-based partitioning provides a uniform distribution of data across shards, minimizing hotspots and balancing the load. For example, tweets and interactions can be hashed based on TweetID, and user data can be hashed based on UserID.
• Do we need Regional or Geographical Partitioning?
• Decision: No need for regional or geographical partitioning based on country or continents.
• Reasoning: Since social networking services typically have a global user base and interactions occur worldwide, regional partitioning might lead to uneven distribution and complexity.

Sharding:

• Which Sharding Strategy works the best in this scenario?
• Strategy: Range-based sharding for tweets and interactions, and hash-based sharding for user data.
• Reasoning:
• Range-based for Tweets: Tweets are often time-sensitive, and range-based sharding based on timestamps can facilitate efficient querying for recent tweets.
• Hash-based for User Data: Hashing UserID for user data can distribute users uniformly, preventing hotspots and ensuring balanced loads.
• As Tables Grow in Size:
• Adjustment: Regularly monitor the size and growth of tables, and consider dynamic sharding strategies based on factors like tweet creation time or user activity. For example, monthly or yearly sharding for tweet data can help manage its size.

Replication:

• Which Replication Strategy?
• Strategy: Master-slave replication for critical data (e.g., user authentication data).
• Reasoning: Ensures data redundancy and fault tolerance. Reads can be distributed among slave databases, reducing the load on the master database.
• Consistency Level for Replicas:
• Adjustment: Consider eventual consistency for non-critical data like tweets and interactions. This allows for improved availability and responsiveness.

High-level design

1. User Interface (UI):
2. Responsibilities:
3. Allow users to register, log in, and navigate through the application.
4. Provide a user-friendly interface for composing and viewing tweets.
5. Display personalized timelines, notifications, and search results.
6. Authentication and Authorization:
7. Responsibilities:
8. Manage user registration, login, and account settings.
9. Authenticate user credentials securely.
10. Authorize users based on their roles and permissions.
11. Tweet Composition and Interaction:
12. Responsibilities:
13. Handle tweet composition, including character limit enforcement.
14. Manage interactions such as liking and unliking tweets.
15. Facilitate following and unfollowing other users.
16. Timeline and Feed Generation:
17. Responsibilities:
18. Generate personalized timelines for users based on the people they follow.
19. Aggregates tweets, interactions, and updates from followed users.
20. Search Engine:
21. Responsibilities:
22. Enable users to search for other users, tweets, and trending topics.
23. Implement search indexing for efficient and fast retrieval.
24. Notification Service:
25. Responsibilities:
26. Deliver real-time notifications for activities like likes, retweets, and new followers.
27. Manage user preferences for notification settings.
28. Database Management:
29. Responsibilities:
30. Store and retrieve user data (relational database).
31. Manage tweets, interactions, and timelines (NoSQL database).
32. Utilize caching mechanisms for performance optimization.
33. Load Balancer:
34. Responsibilities:
35. Distribute incoming user requests among multiple application servers and ensure even distribution of load and prevent overloading of specific servers.
36. Application Servers:
37. Responsibilities:
38. Handle user requests for various functionalities.
39. Communicate with the database, search engine, and other services.
40. Content Delivery Network (CDN):
41. Responsibilities:
42. Cache and deliver static assets like images, stylesheets, and scripts.
43. Optimize content delivery for improved performance.

Request flows

Detailed component design

1. User Interface (UI):

• Responsibilities:
• Render the user interface for account registration, login, tweet composition, and navigation.
• Communicate user actions (like tweets, follows, etc.) to the backend services.
• Display personalized timelines, notifications, and search results.
• Interaction:
• Sends HTTP requests to backend APIs based on user actions.
• Receives and displays data retrieved from backend services.

2. Authentication and Authorization:

• Responsibilities:
• Manage user registration, login, and account settings.
• Authenticate user credentials securely using hashed passwords.
• Authorize users based on their roles and permissions.
• Interaction:
• Receives requests from the UI for registration and login.
• Communicates with the Database to verify user credentials.

3. Tweet Composition and Interaction:

• Responsibilities:
• Handle tweet composition, enforcing character limits.
• Manage interactions such as liking and unliking tweets.
• Facilitate following and unfollowing other users.
• Interaction:
• Receives requests from the UI for tweet creation, liking, and following.
• Communicates with the Database to store and retrieve tweet data.

4. Timeline and Feed Generation:

• Responsibilities:
• Generate personalized timelines for users based on the people they follow.
• Aggregate tweets, interactions, and updates from followed users.
• Interaction:
• Utilizes user data and follows information from the Database.
• Applies algorithms to generate personalized timelines.

5. Search Engine:

• Responsibilities:
• Enable users to search for other users, tweets, and trending topics.
• Implement search indexing for efficient and fast retrieval.
• Interaction:
• Receives search queries from the UI.
• Communicates with the Database for user and tweet data.

6. Notification Service:

• Responsibilities:
• Deliver real-time notifications for activities like likes, retweets, and new followers.
• Manage user preferences for notification settings.
• Interaction:
• Monitors user interactions and communicates with the Database for activity data.
• Sends notifications to users based on their preferences.

7. Database Management:

• Responsibilities:
• Store and retrieve user data (relational database).
• Manage tweets, interactions, and timelines (NoSQL database).
• Utilize caching mechanisms for performance optimization.
• Interaction:
• Receives requests from various components for data storage and retrieval.
• Communicates with caching mechanisms and the NoSQL database for tweet and user data.

8. Load Balancer:

• Responsibilities:
• Distribute incoming user requests among multiple Application Servers.
• Ensure even distribution of load and prevent overloading of specific servers.
• Interaction:
• Monitors the health of Application Servers.
• Routes incoming requests to the least loaded servers.

9. Application Servers:

• Responsibilities:
• Handle user requests for various functionalities.
• Communicate with the Database, Search Engine, and other services.
• Interaction:
• Process incoming requests from the UI and distribute tasks to backend services.
• Communicate with the Database and other services to fulfill requests.

10. Message Broker (Optional):

• Responsibilities:
• Manage asynchronous processing of tasks like notifications.
• Decouple components for improved scalability and fault tolerance.
• Interaction:
• Receives tasks (e.g., notification delivery) from components.
• Distributes tasks to appropriate worker processes.

11. Content Delivery Network (CDN):

• Responsibilities:
• Cache and deliver static assets like images, stylesheets, and scripts.
• Optimize content delivery for improved performance.
• Interaction:
• Receives requests for static assets from the UI.
• Delivers cached assets to users based on their geographical location.

12. Caching Mechanisms:

• Responsibilities:
• Cache frequently accessed data for improved read performance.
• Reduce the load on the primary database.
• Interaction:
• Receives requests from components for data.
• Stores and retrieves cached data for rapid access.

Future improvements

1. Multi-Region Replication: Implement multi-region replication for databases to enhance fault tolerance and reduce latency for users in different geographical locations.
2. Content Delivery Network (CDN) for Media: Introduce a CDN for media storage to optimize the delivery of images and other media content, improving overall system performance.
3. Microservices Architecture: Break down monolithic components into microservices to enhance scalability, maintainability, and independent deployment of different functionalities.
4. Real-time Analytics: Integrate a real-time analytics system to gain insights into user behavior, trends, and system performance, aiding in informed decision-making and optimizations.