Requirements

Functional Requirements:

• per-user / per-API rate limits.
• Admins can add/update/delete/view rate-limit rules via an API or console.

Non-Functional Requirements:

• Allow bursts
• Immediate response if too many requests
• Allow request if below the limit set

API Design

This API design applies to the Admin user:

• POST /rule: Adds a per-user or per-api rate limit rule, based on number per requests in a given period of time. Stores the rule in PostgreSQL (user, api name, number of tokens (requests) per time unit, token capacity in the bucket).
• GET /rule: Gets a rule by user id or api name. Returns the current rule (number of tokens -requests- per time unit, capacity of the token bucket)
• PUT /rule: Allows you to update the user or api rule. The main parameters to update are token capacity of the bucket, time unit and number of requests (tokens).
• DELETE /rule: With this you can remove an user or api rule.

High-Level Design

There are 2 mains flows, from Admin and User perspectives.

User:

1. Wants to call one of the endpoints executed by the API servers, and call it.
2. The API Gateway gets the request, and checks against the Counter using the cached rules.
3. The Counter (Redis) updates the token bucket for that user/api by consuming tokens (one by request + all tokens according to the rate since the last request).
4. If there are no tokens available, the Counter returns a reject with a Retry-After header informing about the seconds needed to retry again.
5. If there are tokens available, the Gateway redirects to the endpoint that the user wanted to hit in the first place
6. The API gateway keeps the cached rules up-to-date by hitting the rules db.

Admin:

1. Add/update/view/delete the Rules DB by using the Admin API.
2. The main parameters to update are the capacity of the token bucket but also the refill rate for the bucket. This applies to user and API.
3. Configuration rollout: When an Admin modifies a rule in one of the ways above, the API gateway updates the rules within 5 minutes/the time we want. For urgent changes we can publish to Redis using Pub/Sub.

Detailed Component Design

The main components are:

1. API gateway: In charge of caching the rate limiter rules from PostgreSQL refreshing after a given cadence (for instance 5 minutes since rules should not change very often). For every request from a non-admin user, it would use those cached rules as input parameters for Redis that will respond with 429 and a Retry-After header specifying the seconds where the user should retry or allowing the request to the API servers.
2. Counter (Redis): Implements the token bucket algorithm on demand. This means that for every request received checks the number of tokens allowed for user/API that is the minimum between the capacity of the token bucket and number of remaining tokens plus the refill rate (if we have 100 req per minute, the rate is 100/60 per second, so we calculate the number of seconds that have passed since the last timestamp and now). If the number of tokens after this operation is greater than one we allow the operation so the gateway can redirect to the API servers. Otherwise it is 429 after saving in the counter the current number of tokens and now (will be used in the next iteration)
3. Rules DB: SQL db that will store the rules to be applied by the rate limit. It is modified by the Admin via API. This DB is consulted directly by the API gateway to keep the cache up-to-date so it can be used as input parameters for the counter.

High QPS and Fast Decision Path: The rate limit decision must happen in under 5ms to not add significant latency. The entire check runs as a Lua script on Redis, making it atomic and avoiding round-trips. The gateway does NOT query the Rules DB per request — rules are cached locally in memory and refreshed every 5 minutes. This means the fast path is: read rule from local memory, execute Lua script on Redis, return allow/reject. No database involved in the hot path.

Burst Traffic and Local Aggregation: Each gateway instance maintains a local in-memory counter per user. If a user has already been rejected in the last N seconds locally, the gateway rejects immediately without consulting Redis. This serves as a first layer that absorbs burst traffic and reduces Redis load by up to 90% for repeat offenders. Only requests that pass the local check reach Redis.

High Availability and Partial Outages: Redis runs in replica mode with Redis Sentinel for automatic failover — if primary fails, a replica promotes within seconds. If Redis is completely unavailable, gateways fail open (allow requests) and alert the operations team, since blocking all traffic is worse than temporarily exceeding rate limits. The Rules DB has read replicas. If the DB is down entirely, gateways continue using their local cache. Cache misses (new user with no cached rule) fall back to a default rate limit until the cache refreshes.

If Redis and Rules DB are completely unavailable, we have the cache of the API gateway with the responses for the requests.

Configuration Rollout Without Downtime: Admin updates write to the Rules DB. Gateways refresh cache every 5 minutes via polling. For urgent changes, the Admin API publishes an invalidation message via Redis Pub/Sub, forcing all gateways to refresh immediately. Rules propagate gradually — no restart or downtime required.

Handling burst traffic at the gateway: Use a local aggregation and handling traffic using the Load Balancer so we can add more instances at ease.

Handle very high QPS and fast decision paths: Adding more API gateway instances, adding more Redis replicas using Redis Sentinel. Fast decision paths will be related to the cache in the API Gateway that will have the rules for the last 5 minutes.