Detailed Component Design
The main components are:
- API gateway: In charge of caching the rate limiter rules from PostgreSQL refreshing after a given cadence (for instance 5 minutes since rules should not change very often). For every request from a non-admin user, it would use those cached rules as input parameters for Redis that will respond with 429 and a Retry-After header specifying the seconds where the user should retry or allowing the request to the API servers.
- Counter (Redis): Implements the token bucket algorithm on demand. This means that for every request received checks the number of tokens allowed for user/API that is the minimum between the capacity of the token bucket and number of remaining tokens plus the refill rate (if we have 100 req per minute, the rate is 100/60 per second, so we calculate the number of seconds that have passed since the last timestamp and now). If the number of tokens after this operation is greater than one we allow the operation so the gateway can redirect to the API servers. Otherwise it is 429 after saving in the counter the current number of tokens and now (will be used in the next iteration)
- Rules DB: SQL db that will store the rules to be applied by the rate limit. It is modified by the Admin via API. This DB is consulted directly by the API gateway to keep the cache up-to-date so it can be used as input parameters for the counter.
In terms of capacity and scalability:
- The API Gateway is stateless so we can bring up as many instances as required, since all consult the same Redis. If Redis is the bottleneck, we can have a Redis Cluster using the user_id to shard.
- Connection pool saturation: Limit the number of connections that a Gateway can make to Redis since it is single-threaded. We could introduce a local rate limiter as a first layer to avoid calling Redis.
- The Admin can increase the rate limit per user and API using the Admin API. The system would allow potential bursts since it consumes tokens, avoiding common issues with other rate limit approaches (like fixed windows where a user might not fit in the right interval).
- Clock skew: We use the Redis clock instead of the gateway clock. Since it is distributed we would avoid this problem.
If the Rules DB goes down, the API Gateway can still function using the cache. For scalability in this sense I would have read replicas, write sharding would not apply here since the rules are just updated by Admin users.