Loading...
A rate limiter service for a distributed environment. It serves as a reverse proxy and either forwards the traffic to the destination traffic or returns 429, too many requests. The rate limiting may be by user ID, ID address, or API key. The allowed configurations are requests per X seconds/minutes/hours/days.
The limiter should handle 1 million requests per second. The system should easily scale to supporting more requests. Consistency is not critical because if one of many requests is missed it's not critical. The service should be highly available and in case one instance of it is not responsive the other will handle the traffic. The service should have very low latency, < 50 ms.
We may introduce a hard limit for each of the categories and persist up to the max duration, namely:
This serves as a retention period for our data, which strictly limits our memory requirements.
We'll use token buckets for simplicity and since it's an idiomatic solution for this type of issues. The alternatives such as a leaky bucket or a sliding counter will also do the job yet may be more resource intensive.
So the schema for the in-memory cache such as memcached or redis will be:
key - service:service_id:instance:instance_id:limiting_by:type:bucket_type:type
value - counter
So storage requirements are moderate with the count of entries being service count * instance per service count * number of configurations per service where a configuration is what to limit by what.
Say, 1000 services * 2 instances per service * 10 configurations per service = 20 000 rows where one row is the storage for the key, say, 200 bytes and the value is an int, 4 bytes, so we need 204 * 20 000 bytes, which is approx 3 000 000 bytes, 3mb
but the 1m requests per second is heavy
The server expects a configuration which states there to rate limit which paths, for instance /v1/users/*; api_key; 10 r/s; https://api.users.com, which means call api.users.com in case not more than 10 r/s and limit per api_key
As discussed above for redis/memcached
key - service:service_id:instance:instance_id:limiting_by:type:bucket_type:type
value - count: counter
We also need
value - last_refilled: timestamp
The service supports multi-tenant deployments. It's excellent for complex systems with independent services. E.g., we may deploy the rate limited to each of our K8s clusters.
The main components are
A requests gets rate limited from the client to one of api service instances. Each instance is stateless and depends on a static configuration. The configuration is mounted and optionally re-read periodically. On receiving a request, the service batches writes to the in-mem cache and either rate limits or executes the API call and forwards the response.
Since we have a lot of requests the api service will do great with batching writes across a short period of time, e.g., 100ms. But it depends on the bucket. The token bucket will read "count" and "last_refilled" from the corresponding key and decrement the used requests and increment the token generation.
N/A
N/A
N/A