Loading...
DAU = 100M
New comments / day = 10M / day
Writes: 100 * 10^6/day ~ 100/sec
Peak Writes = 350 writes/sec
Reads: 100:1 ratio -> 10,000/sec
Peak Reads = 35000/sec
Storage: Each comments would be around 200 bytes, on an average, including metadata.
Storage in a day = 10*10^6 * 200 bytes/day = 2GB/day
Storage in year = 730GB
Storage in 5 years = 3.65TB
Throughput:
Write = 100 * 200bytes/sec = 20KB/sec
Read = 10000 * 2000 bytes/sec = 20 MB/sec
Comments:
POST /v1/comments/
Header: Idempotency-Key: uuid
Body: {
user_id: uuid,
comment: string,
post_id: uuid,
parent_comments: None # List of parent uuids, potentially empty for new comments
}
Response:
Status: 202 Accepted
{
success: True,
curr_likes: int,
edited: False,
deleted: False
parent_comments: uuid1,
comment_id: uuid
}
PUT /v1/comments/{comment_id}
Body: {
user_id: uuid,
comment: string,
post_id: uuid,
}
Response:
{
success: True,
parent_comments: uuid3
curr_likes: int,
edited: True,
deleted: False
}
DELETE /v1/comments/{comment_id}
Status: 204
Response:
{
success: True,
parent_comments: uuid2
curr_likes: int,
edited: True,
deleted: True
}
GET /v1/comments/{post_id}?limit=20, cursor=abcd
Response:
{
comments: [{
comment: string,
parent_comments: uuid1,
curr_likes: int,
edited: True,
deleted: False
}, {}, {}, ....],
success: True,
has_more_comments: True,
next_cursor: "efgh"
}
Likes/Dislike:
POST /v1/comments/{comment_id}/like
Header: Idempotency-Key: uuid
Response:
Body: {
user_id: uuid,
comment: string,
post_id: uuid,
parent_comments: [uuid1, uuid2] # List of parent uuids, potentially empty for new comments
curr_likes: int,
edited: True,
deleted: False
}
DELETE /v1/comments/{comment_id}/like
Header: Idempotency-Key: uuid
Response:
Body: {
user_id: uuid,
comment: string,
post_id: uuid,
parent_comments: [uuid1, uuid2] # List of parent uuids, potentially empty for new comments
curr_likes: int,
edited: True,
deleted: False
}
POST /v1/comments/{comment_id}/dislike
Header: Idempotency-Key: uuid
Response:
Body: {
user_id: uuid,
comment: string,
post_id: uuid,
parent_comments: [uuid1, uuid2] # List of parent uuids, potentially empty for new comments
curr_likes: int,
edited: True,
deleted: False
}
DELETE /v1/comments/{comment_id}/dislike
Header: Idempotency-Key: uuid
Response:
Body: {
user_id: uuid,
comment: string,
post_id: uuid,
parent_comments: [uuid1, uuid2] # List of parent uuids, potentially empty for new comments
curr_likes: int,
edited: True,
deleted: False
}
A user can have only 3 states for a comment - Like, Neutral, Dislike
If an user likes a comment (previous state was dislike) -> the dislike is removed and like is added and vice versa
Clients request are processed via Load Balancer and then routed to Gateway, where we have Rate Limiting, Authentication etc.
We have 3 core services:
Read, Write and Like Services
Writes:
We write the comments in Postgres DB for persistence and emit an event in the Kafka topic to update the Redis cluster for caching.
We store all the metadata about the comments, including status of it getting edited, deleted, likes count in Postgres as a source of Truth.
Likes:
Similar to Comments, the Writes DB serves as the source of truth about the user's like/dislike status of a comment. It also emits an event to Kafka to update the counter in Redis Cluster, which is ultimately persisted in the Comments DB
Reads:
Read is served primarily from Redis for both the comments and likes.
We follow Cache Aside Pattern.
If Redis is down, the reads fall through to Postgres, which is the source of truth.
For live updates, we will use SSE to send updates to the clients, to render the latest updates.
We also have a Notification Service, which will notify the parent comment's user about a new reply. If notification services is down, it can re-read the Kafka logs to replay the event from the last known position (metadata stored separately in a Redis)
Postgres:
User Table:
All sign up info about the user, including the user id
Posts:
All metadata about a post, including post_id
Comments DB:
(user_id, post_id, parent_comment_id, comment, edited, deleted, total_likes, created_at, updated_at), indexed at post_id, parent_comment_id. (post_id, created_at), (post_id, total_likes)
Likes DB:
(user_id, comment_id) -> indexed at both
Redis:
like_status:{comment_id}:{user_id}: LIKE/DISLIKE/NEUTRAL
like_count:{comment_id}: 100
comment:{comment_id}: stirng
We will use sorted set of Redis to store the top N top comments
post:{post_id}:top_level -> list of top N comments
post:{post_id}:{page_num} -> list of comment id on this page_num
comment:{comment_id}:replies -> list of comment ids which are direct child of this comment
Live Comments:
Each client sends a request to the server and the server keeps the connection open.
While an user is subscribed to a post (i.e. he has the post page opened), his connection is kept open and the SSE server keeps listening to the kafka topic for that comment_id. Kafka is partitioned by the comment_id, so the comments are processed serially.
SSE pushes the updates to the clients and the client renders the comment.
GET /v1/post/{post_id}/comments/subscribe -> We will have this API to push the live updates.
Likes Counting System: We will store the current status of the like/dislike of the user in Redis for a quick read (if cache miss, read from Likes DB). If it is a valid update, we will update the DB and add a kafka event.
For counting, we aggregate the updates and update the Redis Counters, and then it is flushed into the Comments DB as a source of truth.
For high volume, we can shard the likes counter in Redis -> likes:{comment_id}:count:{shard_no.} = 50. Shard no can be user_id % N (total shards) -> This will remove the bottle neck
Failure Handling:
Server Disconnection: Client can send a new connection to establish an SSE connection and pass the last known state id and the SSE can start reading the Kafka from that known point. On refreshing a page, we can fetch the latest state from redis and then establish the connection for future live updates.
On throttle due to high load, we can stop the non-critical section like the analytics or notification services.
If the backpressure starts building up, we can add more consumers to process the kafka events.