Loading...
100k reqs/min
144M reqs / day
1k client
each client config will have in average 1 Kbyte
so, for config storage we will need 1 MB
so, a relational database would be enough and it can handle well this structured data
we also need to account that depending on the algorithm choosed we will store the requests per time window on a cache, like Redis, in memory
if we go for a token bucket algorith we will store the requests in memory and number of tokens available also in memory. Assuming, that this will store 1 Kbyte for each client we will need 1 gb 1 MB of memory on our redis.
If in the future the number of clients increase and only one instance we can increase and start using a Redis cluster to account for more memory, we can also enable high availability on the cluster
if we use another algorithm like a sliding window the amount of memory used will increase as we will have to store more items per client
the apis here will follow the REST architecture and they will provide the ability to configure clients
all endpoints will have an authorization token
Authorization: {{token}}
{
"algorithm": "LEAKY_BUCKET"
"threshold": "",
"window": ""
}
{
"algorithm": "LEAKY_BUCKET"
"threshold": "",
"window": ""
}
the client id can be omitted and we can infere from the token
We will store the configurations for each client on a table
Configuration
Everything related to configuration will be stored on a Relational database, as the data is very structured.
Based on the estimations, for now only one database would be sufficient, but if we scale it in the future, we could shard the data on multiple databases per client.
As the data should be fast accesible we will maintain the data on a cache, like Redis.
Then there is the issue of invalidate the cache, we could do this in a cache-aside, where we update the database and then we update the cache, as this doesn't require to be very consistent.
The components system will have:
The system will communicate through REST apis and it will use a microservices architecture to better scalability and the single responsability approach
There are two types of requests:
The requests that go to the configuration service for modification, they follow the path client -> load balancer -> configuration service -> database -> cache. On these requests after the database is updated we update the cache of configurations.
The requests that go to the rate limiter, they are put on a queue in memory, then the rate limiter checks the configurations and the limits for this client and apply the rule, allowing or blocking the request to continues to the downstream service.
If the requests is blocked we return a 429 error code and we can return some headers with more details about the limits reached and when the client should try again.
The Rate Limiter Service is the main piece of the design where all the logic occurs. It receives a request, check the configurations for the client request and check based on the algorithm, Leaky Bucket, Token Bucket or Sliding Window, if the request can be processed. If it can be processed the request is forwarded for the destination service, if not a 429 http error code is returned with some headers, like the next time to retry. The Rate Limiter works behind a load balancer that routes the request to a one of the servers on a round robin fashion way, This makes the Rate limiter scalable as we just need to add more instances to account for more requests. The rate limiters servers use a redis instance, where the limits for each client is stored in memory for fast access. Redis provide a very good service at maintaning data structures in memory and handling concurrent access. To account for an increase in clients we can use a Redis Cluster, where multiple isntances work together to provide the data. IN case we need to scale even more, we could shard the clients, and each client will be routed to a collection of rate limiters and redis clusters.
The database is a relational database due to the low volume of storage, if the data incereases we can shard the database in the same way as the redis cluster to store only the data for a list of clients.
The configuration service is also behind a Load Balancer so we can scale them indepentently if needed.
If the rate limiters are down on a region, we can have another rate limiter cluster on a different region and route all requests to this fallback cluster. Also, in case the redis cluster is down or we can´t connect to the database we prioritize forward the requests to the downstream services, so we dont lose anything, or we can block until the system recovers, depending on the specifics of the services. This can also be configurable.
Because we need the data fast, we choose to have a cache layer on top of the configuration database and for the client limits. However this is kept in memory, Redis is a very good option due to its data structures.
We use a relational data because the data is structured and it will ok to use it, and we can add indexes to improve the queries.
We also allow multiple types of algorithms but this comes with the cost of complexity and more memory on Redis. IN a system we could only start with one algorithm and start doing others in future.
If the rate limiters are down on a region, we can have another rate limiter cluster on a different region and route all requests to this fallback cluster. Also, in case the redis cluster is down or we can´t connect to the database we prioritize forward the requests to the downstream services, so we dont lose anything, or we can block until the system recovers, depending on the specifics of the services. This can also be configurable.
We could have a fallback cluster in a different region in case our system goes down.
If we start with only one algorithm we can provide others in the future.
We could add metrics on how much requests are denied and use some event streaming system to process them and analyse further to help improve the system.