System Requirements

Functional

1. User Registration and Authentication
   1. Users can create accounts and delete accounts
   2. Users can log in and log out
   3. Password can be reset and account recovery mechanisms are supported
2. Posting Content
   1. Users can submit text posts to links to various topics, called subreddits
   2. Support for formatting options like Markdown or Rich Text
   3. Users can also posts images, videos and GIFs
3. Voting System
   1. Users can upvote or down-vote posts or comments
   2. Vote fuzzing is implemented to combat maniplulation
4. Commenting and Discussion
   1. Users are allowed to make comments on posts and they can also reply to other comments
   2. Users can tag users, the tagged user will get a notification that they were mentioned
   3. Discussions are threaded to combat manipulation
5. Subreddit Management
   1. Users can create new subreddits based on different topics
   2. Users can subscribe to a subreddit to get posts from that subreddit in their feed
   3. Subreddits can appoint moderators that can moderate subreddits, allowing the moderator to delete comments, posts etc.
6. User Profile
   1. Users can set profile pictures
   2. Users can follow other users
   3. Posts, comments, replies made by a user are displayed in their post history on their profile
   4. Sorting is allowed based on the most number of likes by time period
   5. Users can follow other users
   6. Users can send each other private messages
7. Content Moderation
   1. A system is set in place to detect and remove spam, offensive content and ensure content quality
8. Recommendation Engine
   1. An engine handles content recommendation based on user activity and likes
   2. Users can report inappropriate content
9. Notification
   1. Users receive notifications for new comments, likes and other relevant activities
   2. Real time notifications are added
10. Home Feed
    1. A user's home feed is filled with content based on the recommendation engine and content from subreddits and users they follow
    2. The home feed can be customized to show popular posts made in the last hour, day, week, month, year etc.

Non-Functional

1. Scalability
   1. The system should be able to handle a large number of posts, users effectively
   2. Horizontal scaling should be supported to accommodate a growing service
2. Performance
   1. Response times for users for posting content, voting, and commenting should be minimal
   2. The platform should be responsive even during peak hours
3. Reliability
   1. The system should be resilient to network failures
   2. Any content posted by the user cannot be lost
   3. Data should be replicated multiple times over to protect against failures
4. Security
   1. User data should be encrypted and protected against unauthorized access
   2. Only authorized users should be able to post content
5. Consistency
   1. We aim for eventual consistency here, we are okay with users seeing posts from a user they follow after a while
6. Usability
   1. User experience should be intuitive and smooth
   2. Easy to navigate
7. Compliance
   1. We need to comply with privacy laws based on regions, such as Europe's GDPR
   2. Content moderation policies should adhere to community guidelines and legal requirements

Capacity Estimation

1. We assume the site has a total of 500 million users, of which 100 million are active daily
2. We assume a total of 2 million subreddits

API Design

1. User Management APIs
   1. /register Creates a new account
   2. /login Login account
   3. /logout Logout of account
   4. /user/profile Fetch user profile
   5. /user/update Update user profile
2. Post Management APIs
   1. /post/create Create post
   2. /post/:postID Get a specific post
   3. /post/:postID/update Update a post
   4. /post/:postID/delete Delete a post
   5. /post/:postID/upvote Upvote a post
   6. /post/:postID/downvote Down-vote a post
   7. /post/:postID/comments Fetch comments for a post
3. Sub-reddit Management APIs
   1. /subreddit/create Create a new subreddit
   2. /subreddit/:subredditID Fetch details of a subreddit
   3. /subreddit/:subredditID/delete Delete a subreddit
   4. /subreddit/:subredditID/update Update a subreddit
   5. /subreddit/:subredditID/posts Fetch posts from a subreddit
4. Comment Management APIs
   1. /comments/:commentID Retrieve details of a specific comment
   2. /comments/:commentID/update Update a specific comment
   3. /comments/:commentID/delete Delete a specific comment
   4. /comments/:commentID/upvote Upvote a comment
   5. /comments/:commentID/downvote Downvote a comment
   6. /comments/create Write a new comment
5. Moderation and Reporting APIs
   1. /moderation/posts Retrieve posts reported for moderation
   2. /moderation/comments Retrieve comments reported for moderation
   3. /moderation/action Take action on a post
6. Search & Discovery APIs
   1. /search/posts Search for posts with a keyword
   2. /search/users Search for a specific user
   3. /trending/posts Search for trending posts
   4. /recommend/posts Get recommended posts
7. Notification APIs
   1. /notifications Retrieve notifications for the user
   2. /notification/markread Mark notifications as read
8. Misc APIs
   1. /health Check the health status of the system
   2. /metrics Retrieve system metrics for monitoring and analytics

Database Design

Below is the diagram that describes the tables.

classDiagram class User { + userId: string + username: string + email: string + passwordHash: string } class Post { + postId: string + userId: string + subredditId: string + title: string + content: string + createdAt: date + updatedAt: date } class Comment { + commentId: string + postId: string + userId: string + content: string + createdAt: date + updatedAt: date } class Subreddit { + subredditId: string + name: string + description: string + createdAt: date + updatedAt: date } User "1"--o "*" Post : creates User "1"--o "*" Comment : writes User "1"--o "*" Subreddit : moderates Post "1"--o "*" Comment : has Subreddit "1"--o "*" Post : contains Comment "1"--o "1" Post : belongs to

1. User Data
   1. Stores user profile data
   2. Relational Database
   3. User data is well structures and generally requires ACID transactions. Relational databases provide strong consistency and transactional support making them suitable for user management, authentication and activity logs.
   4. We prioritize consistency and partition tolerance
2. Posts and Comments Data
   1. Stores posts and comments data
   2. Non-relational database
   3. Post and comments data is semi-structures and can benefit greatly from sequential reads and writes, thus, we choose a non-relational database here. They are suitable for read and write heavy workloads with loose consistency requirements. Here, since we aim to store comments and post data, we can use a document style store as that suits that kind of data well
   4. We prioritize availability and partition tolerance here while aiming for eventual consistency
3. Subreddit Data
   1. Stores subreddit related data
   2. Relational Database
   3. Data is structures and should thus be stored in a relational database to provide relational modeling capabilities
   4. We prioritize consistency and partition tolerance here
4. For partitioning our relational databases, we can use user IDs to partition data
5. For partitioning non-relational databases, we can use subreddit ID or post IDs. This ensures faster reads and writes
6. Geographical partitioning is not necessary for a service like reddit since it is able to serve all users globally without any geographical constraints
7. If there are regulatory or compliance guidelines based on region, then we need to partition data based on geographical locations using geographic data centers
8. We scale all our data stores horizontally to ensure performance is not compromised. Also, we replicate data to ensure data reliability
9. For storing media files like images, videos etc. we can use an object store
10. Content Delivery Networks are utilized to cache static content like thumbnails, HTML, CSS etc. to users globally, reducing latency and bandwidth usage

High-Level Design

graph TD; subgraph Client_Applications WebApp(Web Application) MobileApp(Mobile Application) end subgraph Frontend_Services LB1(Load Balancer) Auth(Authentication Service) Notif(Notification Service) LB1 --> Auth LB1 --> Notif end subgraph Backend_Services API(API Gateway) User(User Service) Post(Post Service) Comment(Comment Service) Subreddit(Subreddit Service) Search(Search Service) Recommend(Recommendation Service) Moderate(Moderation Service) Notify(Notification Service) Analytics(Analytics Service) API --> User API --> Post API --> Comment API --> Subreddit API --> Search API --> Recommend API --> Recommendation end subgraph Infrastructure_Components LB(Load Balancer) Cache(Caching Layer) CDN(Content Delivery Network) DB(Database Clusters) ObjStore(Object Storage) end Client_Applications --> |Routes Requests| API API --> Frontend_Services Backend_Services --> LB Client_Applications --> CDN DB --> ObjStore Monitor --> Backend_Services Security --> Backend_Services

1. Client Applications (Web & Mobile)
   1. Responsible for providing user interfaces for interacting with the platform, allowing users to browse content, submit posts, comment and interact with other users
2. Frontend Service
   1. Authetication service handles user authentication and authorization, ensuring secure access to the platform
   2. Notification Service sends real time notifications to users for activities such as new comments on their posts or replies to their comments
3. Backend Service
   1. API Gateway acts as a single entry point for client applications to access backend services, managing requests and routing them to the appropriate service
   2. User Service handles making changes to user accounts
   3. Post Service facilitates in the creation of new posts, updating posts, etc.
   4. Comment Service is responsible for everything related to comments
   5. Subreddit Service manages everything related to subreddits
   6. Search Service helps users search for posts based on keywords
   7. Recommendation Service recommends posts to a user based on their interests, followed subreddits etc.
   8. Moderation Service handles content moderation, including detecting and removing spam, offensive content and enforcing community guidelines
   9. Notification Service sends notifications to a user when someone comments on their comment, replies to them etc.
   10. Analytics Service collects and analyzes data on user interactions, content popularity and platform usage to provide insights for optimization and decision making
4. Infrastructure Components
   1. Load Balancer handles server related load and routes traffic appropriately
   2. Caching Layer stores frequently accessed data to reduce database load and improve response times
   3. Content Delivery Network delivers static content to users
   4. Database Clusters store data effectively
   5. Object Stores store media files
   6. Messaging Queues support asynchronous communication between components of the system, ensuring reliable message delivery and decoupling services
   7. Monitoring and Logging Services monitor system health and performance, providing insights for troubleshooting, optimization and compliance
   8. Security and Firewall Components protect the system from unauthorized access, malicious attacks and data breaches

Request Flows

sequenceDiagram participant ClientUser as Regular User participant ClientMod as Moderator participant API as API Gateway participant Post as Post Service participant Mod as Moderation Service participant DB as Database ClientUser ->> API: Create New Post API ->> Post: Create New Post Request Post ->> DB: Save New Post DB -->> Post: Post Saved Post -->> API: Post Saved Response API -->> ClientUser: Post Saved Response ClientMod ->> ClientMod: Review Pending Posts ClientMod ->> API: Retrieve Pending Posts Request API ->> Mod: Retrieve Pending Posts Request Mod ->> DB: Retrieve Pending Posts DB -->> Mod: Pending Posts Retrieved Mod -->> API: Pending Posts Retrieved API -->> ClientMod: Pending Posts Retrieved ClientMod ->> ClientMod: Review and Approve Post ClientMod ->> API: Approve Post Request API ->> Mod: Approve Post Request Mod ->> DB: Update Post Status to Approved DB -->> Mod: Post Status Updated Mod -->> API: Post Status Updated API -->> ClientMod: Post Status Updated

1. User Creates a New Post:
   1. The client application (web or mobile) sends a request to the API Gateway to create a new post.
   2. The API Gateway routes the request to the Post Service.
   3. The Post Service validates the request, creates a new post object, and saves it to the database.
   4. The Post Service triggers notifications to subscribers of the subreddit where the post was created.
2. Moderator Reviews and Approves the Post:
   1. The moderator accesses the moderation dashboard in the client application.
   2. The client application sends a request to retrieve pending posts to the API Gateway.
   3. The API Gateway routes the request to the Moderation Service.
   4. The Moderation Service retrieves pending posts from the database and presents them to the moderator.
   5. The moderator reviews the post and decides to approve it.
   6. The client application sends a request to approve the post to the API Gateway.
   7. The API Gateway routes the request to the Moderation Service.
   8. The Moderation Service updates the status of the post in the database to "approved".

Detailed Component Design

1. User Management Component
   1. User Roles & Permissions
      1. Regular users have basic privileges such as creating posts, commenting and voting
      2. Moderators have additional rights within specific subreddits where they can moderate posts
      3. Administrators have full control over the platform, including user management, system configuration and policy enforcement
2. Threaded discussions are displayed hierarchically, with nested replies indented to visually indicate their relationship to parent comments
3. Recommendation engine can recommend content based on what the user follows, we can use machine learning modes to deduce what other content the user might like
4. We cache trending posts completely, for other posts we can cache post metadata since a lot of times, users will read the post title and not open the actual post. If the user wants to see the post, we fetch the complete post then
5. We can use Least Recently Used (LRU) as our caching strategy as using this strategy, only the popular posts will remain in memory and the lesser popular ones will, over time, be evicted from the cache. We can configure the cache to store N posts, where N is a parameter than is configurable
6. We implement automated spam detection algorithms to identify and flag suspicious user activities, such as posting repetitive or low quality content, excess link sharing, such posts can be marked as 'pending moderation'
7. We can utilize machine learning and natural language processing (NLP) models to analyze posts for hate speech and harmful content
8. We enforce rate limiting and throttling mechanisms to restrict the frequency and volume of user actions, such as posting, commenting etc.
9. Set such limits based on average user behavior
10. Reporting tools can help users to report content that slips away from our moderation service

Trade Offs/Tech choices

1. For our relational database of choice, we choose MySQL
2. For our non-relational document store of choice, we choose MongoDB
3. For our cache, we use in memory key value store Redis which will provide us with fast reads and writes
4. For our object store, we use Amazon S3
5. For our CDN, we can use Cloudfare or Akamai
6. For our messaging queue, we use Apache Kafka

Failure Scenarios/Bottlenecks

1. Hot Spot can happen when a celebrity makes a post and that post has very high traffic. That will concentrate a large number of requests to a very small part of our service

Future Improvements

1. Implement distributed counters to count upvotes on posts, comments etc.
2. Offline support
3. Let users give awards to other users