System requirements

Functional:

List functional requirements for the system (Ask interviewer if stuck)...

1. Different rate limits for different API endpoints or user groups.
2. Burst rate limiting to allow short bursts of requests exceeding the rate limit.
3. Customizable rate limit quotas based on user roles or subscription plans.

Non-Functional:

1. Low latency around 100 ms
2. Highly available

List non-functional requirements for the system...

Capacity estimation

Estimate the scale of the system you are going to design...

For 100M requests per day, which means we will have a QPS of 1200 Requests per second

API design

Define what APIs are expected from the system...

Rest APIs:

1. GET API : getAPIRateLimit -> /:apiname -> {user id and roles will be assumed by a JWT token}
2. Header with developer token for each user
3. PUT API: modifyAPIRateLimit -> /:apiname -> payload will be some thing like {"apiname: "", rate: 100, unit: "minute""}

Database design

We can have a Redis cluster to store the recent number of calls to each api from each developer token

Our cache will be LRU in Redis, evicted items are aggregated and persisted in the database for analysis.

Our persistance layer can be a relational DB like postgres, where we can store relations like

User - > userId, name, developerToken, allowedRate

Rate: apiId, userId, limit

High-level design

1. Each request is directed to a LoadBalancer, where the LB will find the necessary service behind its configured group.
2. The request reaches our RateLimitService where it will apply its Token Bucket algorithm and check the existing limit from our redis cache and updates it if its under limit. if the call is over the limit the respose will be 429
3. Redis cache will be LRU, and once the item is evicted from the cache, we can add a aggregator to aggregate the removed data and persist in to the db
4. PUT api call will modify the cache and the DB directly with the limit by following cache write all strategy.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

1. LB can be used to pick servers efficiently using many techinques like sticky sessions, usage based, less busy etc. We will use usage based because we do not want to deal with session tokens which is complicated.
2. We have caching on the RateLimitService by using redis. We improve this caching by LRU method. We will have this service in multiple data centers and configure auto scaling to add more when there is peak demand.
3. There are multiple ways to cache the data but redis is better since I use it regularly and its optiomized for heavy read and writes and support various different data structures to store the cache.
4. Then the data is aggregated and persisted in postgres for analytics. I picked postgres because our data is aggregated and easy to form relations using joins with it. postgres is also easy to sclae and we can indexes on top of our tables.
5. Our DB can be scaled by using sharding and we can select the shards based on the api type and popular apis can have more shards/instances if required

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

1. writes are written to cache and db first, our writes will be slow but our system is mostly read heavy and we are ok to make the tradeoff.
2. OUr system is eventually consistent since we are aggregating the data

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

1. data center going down
2. cache layer not updated correctly which is causing to limit the requests more than usual.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?

1. I would use replication the data across multiple data centers.
2. Setup proper usage metrics to monitor our cache aand scale them when needed based on peak/non peak traffic.