Loading...
The rate limiter can be placed along with api gateway or load balancer or above of both.
Our primary api will call redis with a lua script and decide whether request is allowed or not.
If not allowed then return 429.
Handle burst at api gateway:
Once request got accepted, we will check the leaky queue and place in queue. Soon the request will be out of queue on regular intervals and got executed.
We can use a token bucket algorithm to handle the api requests to handle the burst.
For database, redis is best option to get atomic completion of operations.
We will have a bucket with a capacity of 50 tokens and 5 token/sec.
Whenever an api request came , one token will be consumed.
Since we are using redis, consistency will be achieved. For scalability we have to follow sharding and separate keys in different shards based on hash.
For Observability and audits, we can use either OpenTelemetry and Prometheus to analyze. We can send data token consumption data along with api ,so that we can handle the scalability and bucket capacity for keys.
To handle, burst at api gateway end, we can leaky bucket which will work to handle burst along with queue.
But there can be latency issues to get response. So we have to maintain the token and leaky bucket limiters in a way to handle latency issues.
We can use Redis to handle the bucket capacity and incrementing and decrementing. Since we are using token bucket algorithm, we need the database operations to be performed as atomic and single threaded. So Redis is the best option for that.
We can define bucket as below
capacity (bucket capacity): number
key (Redis key):
refill-token-limiter: number
lastRefillTimeInMs:
We have to define another bucket to handle burst:
capacity: max number of queued requests
leakRate: how many requests needs to be sent to backend
lastLeakTimeinMs: When the last leak happened
Redis Execution:
We can use Lua Script to perform the required actions to define whether the request is allowed or denied
Sample Lua Script:
GET Key
IF not exists then add key with ttl and tokens number
If exists
check tokens present in bucket , if available then decrement 1
return allow/deny
Based on allow/deny , we can either pass the request to backend service or deny it or kept in queue.
To handle burst at api gateway level, we will use kafka queue, to maintain a rate at which requests got passed to api gateway and remaining will be sent to a queue.
We will maintain a queue with constant size which is equal to leaky bucket capacity. Once the request is allowed from token bucket, then we will place it in queue. Soon the queue will be processed and sent to api gateway.