Loading...
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
GET /api/v1/counters/{entity_id}
POST /api/v1/counters/{entity_id}/increment
POST /api/v1/counters/{entity_id}/decrement
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
API Gateway: To route traffic based on the api that is getting hit and to authenticate and throttle requests.
Load Balancer: To distribute the traffic received from the API Gateway in a balanced manner so that the traffic is evenly distributed
Rabbit MQ: To decouple the components and have buffer so that heavy traffic can be handled effectively.
Redis Cluster: group of redis containers which do approximate and fast counter increments and decrements.
Mongo DB: To store the counter data for long time. Data is periodically flushed from redis cluster to Mongo
Counter Service: To increment and decrement the counter.
Retrieve Count: To get the counts from redis and from mongo if redis fails.
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
API Gateway: Helps in routing traffic to the components based on the api that is getting hit. Apart from that we can authenticate and throttle the requests to avoid any DDoS attacks.
Load Balancer: o distribute the traffic received from the API Gateway in a balanced manner so that the traffic is evenly distributed
Rabbit MQ: To decouple the components and have buffer so that heavy traffic can be handled effectively.
Redis Cluster: group of redis containers which do approximate and fast counter increments and decrements.
Mongo DB: To store the counter data for long time. Data is periodically flushed from redis cluster to Mongo
Counter Service: To increment and decrement the counter.
If we did not use rabbitmq or any message queue then it would have been fine if the traffic is small but for larger traffic the system would not scale as the load is too much. By using rabbitmq we are able to buffer there by decoupling the components and reducing overall failures
We can have shard counter where we will be having N sub-counters and each increment randomly picks a shard. and every count will have an ideompotency key so that the counts are idempotent which prevents double counting.
We will then aggregate the counts from these shards and every 30 mins we will flush the data into Mongo DB. There by persisting the data.