Loading...
Problem
Develop a scalable and efficient system for managing nested comments, enabling users to engage in discussions by posting comments and replying to other comments in a hierarchical manner. The system should support real-time updates, preserve the order and structure of the conversation, and ensure a seamless user experience across various devices.
Functional Requirements
1. A user logs in
2. A user logs out
3. A user posts a comment (optionally a comment as a response to another comment)
4. A user deletes a comment
Non-functional
1. 1e9 users
2. 1% users daily
3. an daily active user reads 10 comments
4. an daily active user writes a comment
5. comments are stored for 10 years
6. Latency: [read,p50,100ms],[write,p50,150ms],[read|write,p99,1s]
7. Availability: [read|write,99.99%]
8. Reliability: [read|write,99.99%]
9. Durability: [p99.99%]
Estimate the scale of the system you are going to design...
API
RESTful, all through HTTPS
1. create accout - /login/create - POST, post params: {id='userid',password='password'} - returns 200 with a jwt if successful, or 400 with bad request if problems like with userid or password
2. loging - /login/ - POST, post params: {id='userid',password='password'} - returns a jwt token
3. logout - /logout/ - GET/POST, no params
4. Read comments tree - /post/{id} - GET - url params id - returns a json with structured comments
5. delete comment - /post/{id} - DELETE
6. create/update comment - /post/create - POST - param: {id='id of the post to be updated or null for new posts', parentid='id of the parent, null for new posts', comment='text of the post'}
7. search for some text
8. Get comments by a user: /user/posts/ - GET - param {userid=''} - returns a list of post by the user, paginated
Tradeoffs
1. We could add 3rd party authentication such as with OATH (google, etc.)
Model
User
id - long, 8B
username (email) - nvarchar (50) (400B)
password hash - nvarchar (260) (512 B) (bcrypt hash)
total ~ 1KB per entry
Post
id - long, (8B)
parent_id - long, (8B)
created_on: datetime (8B)
text - varchar 40000 chars (max 320KB, average @200 char => 1.6KB)
tota: max:340KB, ave:1.7KB ~2KB (less if compressed)
Capacity estimate
Users
storage: 1e9 * 1e3B = 1e12B = 1e6MB = 1e3GB = 1TB
reads: 1e9 / 100 reads per day = 1e7 rpd => 115 rps (low)
writes: << reads (new accounts per day, negligible compared to reads)
Posts
reads: 1e9 / 100 reads per day => 115 rps
reads size: 115 * 2KB rps ~256 KBps ~.25 MBps (low)
writes: 1e9 / 1000 new comments = 1e6 comments per day = 11.5 wps
writes size: ~25KBps
storage: 1e6 comments per day => 365e6 per year => 3.65e9 per 10 years
@200KB ave => ~7e11KB = 7e8MB = 7e5GB = 700 TB (very large) (much less with compression ~15th => ~140 TB)
DB
Users -
(recommended) MySQL (OLTP + RDBMS).
Pros: ACID transactions ensuring consistency and durability, handles the load well, solid systems, easy to query and admin
Cons: Could have a hard time handling spikes, or scaling to beyond 10x the number of users (fake accounts, ai bots?)
(alternative) Other (OLTP + RDMS), or some key/value store configured for strong consistency.
Posts -
(recommended) - ElasticSearch (OLTP + search engine).
Pros: Handles the load requirements very well, supports complex text search queries, supports querying by user with appropriate index on the userid.
Cons: Consistency needs to be ensured at application level
(alternative) - Another OLTP such as RDMS with partitioning. Cons: Support for text search is more limited and slower.
Entities
U - User UI (mobile/web)
AG - API Gateway
LB - LoadBalancer
BE - Backend Servers
MYSQL - MySQL DB with Users table
ES - ElasticSearch with Posts table and search indices
REDIS - Redis layer for caching Posts
AXIOM - AXIOM service for Log Management (alternatives ELK, AWS Cloudwatch, Datadog)
INFLUX - Influx service for metrics and observability (alternativs: Prometheus, Datadog Metrics, AWS Cloudwatch Metrics, etc)
U --> AG
AG --> LB
LB --> BE
BE --> MYSQL
BE --> ES
BE --> REDIS
REDIS --> ES
AG|LB|BE|ES|MYSQL|REDIS --> AXIOM
AG|LB|BE|ES|MYSQL|REDIS --> INFLUX
Example
Post comment, happy path
Precondition: user is logged in (expressed by a coockie with a valid JWT token), the request is all good.
1. User sends a request to the service which arrives at the API Gateway (after getting to a Routing service such as AWS Route 53)
2. The APIG forwards the POST HTTP request to the LoadBalancer
3. The LB chooses a backend server and forwards the request to it
4. The BackEnd server handles the request. Verifies that the JWT token is valid and has not expired, validates the POST params (valid information), and then sends a query to the elasticsearch service to create a new entry with the provided text, userid.
5. The ES service executes the query which creates a new entry in the database, and replies success to the BE server.
6. The BE server creates an HTTP response with 200 and the id of the new post.
7. The LB forwards the response to the client
8. The client receives the HTTP 200 response and notifies the user of the success.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Tradeoffs
1. using AXIOM may be more costly than a tool like elastic search with the elk stack (elastic search + kogstash + kibana) but the later requires more configuration and complexity.
Failure scenarios
1. API Gateway failure. Other API servers are proving the main APIG server. If it fails, another server is promoted to the master role and takes over the failed server.
2. LB fails. Similar to API Gateway.
3. Backend server fails. The LB detects this and stops using the failed server. With alarms, the infrastructure can span new servers if some failed or if load increases.
4. MySQL DB fails. We configure the SQLDB with read replicas which get promoted to master if the master fails. If a replica fails, a new one is created.
5. ElasticSearch fails/offline. ??
6. Redis layer fails. The requests get directed to the ES service. This potentially increases the load on it. The ES is configured to scale horizontally and handles high write/read loads. Some read requests will be slower.
7. Read/Writes spikes. Read spikes are handled by the Redis cache, and if needed by the ES server which handles very large loads.
8. DDOS attacks - Use a service that protects agains this like Cloudflare (mitigates DDoS in layers 3, 4 y 7), rate limiting. Autoscaling and load balancers. Firewalls for IP ranges or regions. Alerts for unusual spikes, CPU etc. Define a DDOS plan. Perform DDOS testing.
9. SQL injection attacks - Prametrized queries to ensure that user input is handled as data and not code. Some tools such as Spring ORM handle this automatically. Minimum req. permits in the DB. Third party like Cloudfare can detect some SQL Injection patterns. Configure alerts in the log management to detect failed queries. Perform Penetration Testing.
10. Other security risks (mitigation) - Cross-Site Scripting (XSS) (Content Security Policy (CSP) etc.), Cross-Site Request Forgery (CSRF) (MFA), Credenciales expuestas o autenticación débil (strong passwords, MFA, code scanning with tools TruffleHog, monitor failed logins), Ataques de inyección en APIs (API Abuse) (OAuth2, rate limiting at API Gateway, metrics monitoring), Errores de configuración (infrastructure scan tools), Ataques de bots (CAPTCHAs, CDN, traffic pattern monitoring)
Future improvements
1. Add support for 3rd Party auth via OAUTH
2. Add reputation system for users, granting them more priviledges (edit other users' posts, delete)
3. Add upvoting/downvoting to posts
4. Add post flagging (users can flag a post for review by trusted users)
5. Add some sort of moderation to prevent illegal content (violent, etc.)
6. Add ai process to flag posts as illegal and flag for review
7. Add filters such as 18+, an other categories
8. Add images to posts
9. Add failure test with tools like Chaos Monkey