Requirements

Functional Requirements:

• per-user / per-API rate limits.
• Admins can add/update/delete/view rate-limit rules via an API or console.

Non-Functional Requirements:

• low latency should be achieved, responses should be returnes in less than 200ms. also the rate limiting checks shouldnt significantly increase the response times for other requests. regarding the rate limiting operations (add, update, delete, get) those should also be done in less than 200ms
• high availability is a must, rate limiting should be applied to all operations done by users. the up time should be 99.99% or greater
• the responses should be consistent, so the rate limiter always knows if a response should be limited and denied
• the deny-allow path should take less than 10ms
• the system should be able to scale up in case of high spikes in order to handle all requests from users, meaning at leat 20000 operations a second

API Design

all requests will first go through the rate limiter using this endpoint

PUT /limit/user/{userId}/api/{apiId} - this endpoint will check if the user has any left tokens for that api, if so it will allow the call and increase its used tokens. if the token limit is already reached, it will deny the request. we could use the token bucket method, storing the users allowed number of tokens per minute and resetting every minute the number of consumed tokens

for admins there are these endpoints:

POST /rules body:{limit:number, userId:string, apiId:string} - this creates a rule

PUT /rules/{ruleId} body:{limit:number, userId:string, apiId:string} - updates a rule by its id

DELETE /rules/{ruleId} - deletes a rule

GET /rules/{ruleId} - fetches a rule

High-Level Design

the client makes firstly a call to DNS in order to get the correct IP address for a domain name.

afterwardds the client makes a call, which reaches the load balancer.

I'd recommend adding a load balancer in order to be able to scale up in case of high spikes by increasing the number of instances. also te load balancer will evenly distributes all incoming calls.

the next hit component is the rate limiter - this one checks in a cached DB if the user still has available tokens (it computes based on lastRefill and the refillRate how many available tokens there are, afterwards it checks what's the maximum number of tokens and if it exceeds it, it uses the max number of tokens) in order to receive a response for this call, if not it denies it, if so it continues the flow and decreases the number of available tokens from rules service. when it reaches 0, all requests are denied.

after the rate limiter, API gateway is the next one in order, this one makes calls to rules service, eiter to update/delete/create rules which update the db (performed by an admin), either updates the number of used tokens for that user. after this the call is made to the service where users wait for their resonses (service here has the logic for any other api call the user want to do which is NOT related to rules). it can also use a cached memory in order not to be that heavy on the db.

Each backend instance has a lightweight rate limiter sidecar (container alongside the main service). The sidecar intercepts requests, checks Redis, and either forwards to the backend or returns 429.

If a user's rate limit entry doesn't exist in Redis (first request or cache eviction):

• We treat it as a fresh user with full tokens (maxTokens)
• Create new entry with tokens = maxTokens - 1, lastRefill = now

Detailed Component Design

the rules db will store tokens, refillRate, lastRefill, maxTokens. on each request, done by a user, the number of tokens will be recomputed, it will check lastRefill time, the current time, and multiply it with refillRate then choose the minimum between this result and maxTokens. from this number we will substract one for each request. if the number is 0 before substracting, then the request is denied.

Admins can update the maxTokens and refillRates for users and APis

the results will be cached in Redis for quicker responses