System requirements

Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...

• Users can publish new comments.
• Users can reply to other users' comments
• The system guarantees to show the same comment order for every user
• The user gets a notification if another user replies to his comments
• Let's restrict the one comment thread, which may have 1000 items and only 4 nested depths. (It would ignore the comments if its depth exceeds 4).
• Users can't delete or change the comments; they can only append new ones.

Non-Functional:

List non-functional requirements for the system...

• This service should be highly available.
• This service should be highly scalable.
• When users publish comments, the new comments appear with a delay of 10 milliseconds.
• The service should be highly reliable.
• Only registered users can append comments, others may only read.
• The users can use the mobile or web applications.
• The comments will be shown in chronological order
• The comment can have only text.

Capacity estimation

Estimate the scale of the system you are going to design...

1 million daily active users.

Every user leaves 5 comments per 5 minutes.

Every user reads 10 comments per 5 minutes.

We are going to use RDMS to store user and comments information:

The table Users:

id string (100 bytes)

name string (100 bytes)

email string (100 bytes)

createdAt timestamp (8 bytes)

active boolean (1 bytes)

The table Comments:

id string(100 bytes)

userId string (100 bytes)

parentId string (100 bytes)

level byte (1 byte)

createdAt timestamp (8 bytes)

content bytes (400 bytes)

type byte (1 byte)

Users write 17000 new comments per second or 34 MBytes per second.

Users read 34000 comments per second or 34 MBytes per second.

The system persists 1 Petabyte of comments for 1 year.

The system persists 5 Petabytes of comments for 5 years.

API design

Define what APIs are expected from the system...

We use REST API:

add_new_comment(user_id, comment_payload, timestamp): add new comment by the registered user. It returns id of the new comment and HTTP code 201

add_reply(user_id, comment_id, comment_payload, timestamp): Add a new reply to a comment by a registered user. It checks if the depth's comment is less than or equal to 4 and the comment length is less than 400 bytes; if it fails, it returns HTTP code 400, otherwise is HTTP code 201

get_comments(api_key, comment_parent_id, startDate,endDate, offset, page_size) return JSON with found comments. If comment_parent_id is -1, it returns the comment of level 0. The api_key is to throttle the user request frequency. StartDate and endDate to filter the comments by createdAt. offset and page_size regulate the number of reply comments and allow scrolling up and down by comment list.

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

We are going to use RDMS to store user and comments information:

The table Users:

id string (100 bytes)

name string (100 bytes)

email string (100 bytes)

createdAt timestamp (8 bytes)

active boolean (1 bytes)

The table Comments:

id string(100 bytes)

userId string (100 bytes)

parentId string (100 bytes)

level byte (1 byte)

createdAt timestamp (8 bytes)

content bytes (400 bytes)

type byte (1 byte)

The table Comments has the foreign key of the user of the table Users. In order to optimize the search an user or a comment, we will create the indexes:

(id, createdAt,active) for the table Users;

(id, userId) and (id, parentId,level,createdAt) for the table Comments.

We will use Postgres as the RDMS. It supports replication ,indexing, and transactions.

High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

See the diagram from the high-level architecture.

API Gateway provides DDoS protection and TLS termination and forwards requests to the right service nodes.

We have three microservices. User service manages the adding, updating, or deleting of the users. Message service caters to adding new messages and replies. The notification service updates the mobile or web application if other users reply or add new messages.

The message service prefetches user messages and builds the comments in the cache. Also, the messages service sends updates to Kafka for the notification service.

The User service caches the frequently used users.

The Notification service caches the user application connection information to update mobile or web applications. The microservices use HTTP REST for communications. The web or mobile application may use the long polling to get the message update. The cache uses many eviction strategies: LRU, LFU, FIFO. Let's use LRU for all caches.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

add_new_comment(user_id, comment_payload, timestamp):

The user using the web or mobile applications sends a new comment to API Gateway. API Gateway calls the User service to check user by user_id and get user permissions. The User service finds the user by user_id in the cache; if it misses, the User service downloads the user from Postgres. If the user is not found, the API Gateway returns the HTTP 403, otherwise, the request is directed to the Message service. The message service persists the message and adds the new message to the cache in parallel. This cache (Redis) of the Message service contains a few structures that allow to appending or deleting messages quickly. The Message service sends update to Kafka. Later, the notification service consumes the update from the Kafka and using the user connection information from the cache(Redis), sends the new messages, replies to the mobile or web applications.

add_reply(user_id, comment_id, comment_payload, timestamp):

The user using the web or mobile applications replies new comment through API Gateway. API Gateway calls the User service to check user by user_id and get user permissions. The User service finds the user by user_id in the cache; if it misses, the User service downloads the user from Postgres. If the user is not found, the API Gateway returns the HTTP 403, otherwise, the request is directed to the Message service. The message service persists the message and adds the new message to the cache in parallel. This cache (Redis) of the Message service contains a few structures that allow appending or deleting messages quickly. The Message service sends updates to Kafka. Later, the notification service consumes the update from the Kafka and using the user connection information from the cache(Redis), sends the new messages and replies to the mobile or web applications.

get_comments(api_key, comment_parent_id, startDate,endDate, offset, page_size):

The user using the web or mobile applications gets the list of messages through API Gateway. API Gateway calls the User service to check user by user_id and get user permissions. The User service finds the user by user_id in the cache; if it misses, the User service downloads the user from Postgres. If the user is not found, the API Gateway returns the HTTP 403, otherwise, the request is directed to the Message service. The message service checks its cache if the messages are in the cache, if not found messages in the cache, the Message Service downloads them from Postgres and puts to the cache, and return them back to the client applications. The message service uses the index by timestamp and comment_parent_id to find requested messages. It supports the offset and page_size for message list.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Let's discuss how to organize the Message service and its cache in Redist to append, read, and find the messages fast.

When the Message service starts, it prefetches messages to the Redis cache. The Redis cache of the message service includes the following structures:

• map1 is the map with user_id as key and ordered set with message_id as value, the ordered set is filled up with messages this user_id created on level 0;
• map2 is the map with message_id as key and ordered set with message_id as value, the ordered set contains messages whose parent_message_id is the key.
• map3 is the map with message_id as key and the message JSON object as value. The JSON object contains: user_id who created this message, message content, message level, and parent_message_id.

If a user appends a new message, the message service goes to the map1 and appends new message_id and the current timestamp to the ordered set, appends to map3 the JSON object: own user_id, message content, 0 and parent_message_id equals to -1. The ordered set sorted message_id by its timestamp in increasing order and it gives to keep the message in chronological order.

Also the message service persists new message.

If a user replies to a message, the message service goes to map3 and takes the parent message JSON object, and if the parent message level is 3, the user exceeds the permitted nested message level and the message service returns HTTP code error 404. Otherwise, the message service serves the user request: find the ordered set by parent_message_id in map2 and add there, the current timestamp and new message_id, adds to map3 new message JSON object: user_id, message level + 1, parent_message_id, message content.

and set of message_id and adds new message_id for the user. The user updates this message to the ordered set level_0 with the current timestamp and message_id, adds to map this message_id and the message JSON: user_id who created this message, message content, message level, and parent_message_id.

If a user replyis some message, the message service take this message_id and find its level, then

If a user requests the message within time interval, the message service finds the ordered set in map2 by parent_message_id and applies the time interval, it gets the list of message_ids.The list of message_id would be paged with offset and page_size. The offset and page_size would be passed in by the client web or mobile applications.After that, the message content is taken from map3 by the message service.

The client web or mobile application would use the long polling to check if new messages appear. It would allow to save the server resources. The notification service stores the client connection information in the cache: client IP, lastUpdateTime. One user_id may have a few client application connections. Because user_id is passed in HTTP request as a header, API Gateway may restrict the connection number for one client_id, by implementing the rate limit.

When the message service updated the catch with new messages, it sent the update to the notification service with Kafka. The notification service reads the update from Kafka and uses the client application connection information to send the last update to the client applications.

The client web or mobile application use the HTTP to communicate with backend.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

We use Redis to cache the messages because Redis is the scalable and fault-tolerant solution of our system.

In order to connect the message service and the notification service, we use Kafka. Kafka is a scalable and fault-tolerant event system with high message throughput.

The client mobile or web application uses the HTTP longh polling because it's simple and reliable way to connect client and backend, if the system is read-heavy and we want to save the server resources.

All our services are stateless, and they would be deployed and managed by Kubernetes. We may deploy our microservices to AWS EKS. Kubernetes uses auto-scaling for picky loads and has a comfortable AWS CloudWatch as a logging system to watch the system events.

The user service would be use JWT to propagate user authentication and authorization in microservices.

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?