Requirements


Functional Requirements:


  • per-user / per-API / per ip address rate limits.
  • Admins can add/update/delete/view rate-limit rules via an API or console.



Non-Functional Requirements:


  • Server should not crash because of an api over load.
  • fast decision design to allow/deny a request.
  • scalable to all services, and their api's


API Design

1.) decision check

POST : v1/rate-limit/api/{api_id}

request {user_id , ip_address}

2.) create or update

POST : v1/rate-limit/api/{api_id}

request: {total , per_user , }

3.) DELETE : v1/rate-limit/api/{api_id}

request : {}

4.) audit purpose

GET : v1/rate-limit/api/{api_id}

respose: {api_id, total_request, r_p_u, ....}



High-Level Design

flowchart TD

 B["client"];

 C{"server"};

 D["Database"];

 cdn["cdn"];

 ld["load balancer"];

 rd[("redis cache")];

 rls["rate limit service"];

 B --> cdn;

 cdn --> ld;

 ld --> rls

 rls --> rd

 rls --> C

 C --> D;


  • we have a rate limiting service which is like a wrapper to all the other backend services for the rate limiting logic
  • The gateway should first check incoming requests against the rate limiter before forwarding them to backend services. This can be achieved through a token bucket or leaky bucket algorithm, which allows for bursts of traffic while maintaining an overall limit. The rate limiter can be implemented as a centralized service or as a sidecar pattern, depending on your architecture.
  • In scenarios where the rate limiter becomes slow or unavailable, consider implementing a fail-open policy that allows requests to pass through with a warning, or a fail-closed policy that blocks all requests until the rate limiter is restored. This ensures that your system can handle failures gracefully.
  • To achieve statelessness, you can leverage shared data stores, such as Redis or a distributed cache, to maintain the current state of rate limits.




Detailed Component Design

  • Using the leaky bucket algorithm
  • use a centralized service with distributed caching to manage state efficiently.
  • Distributed Locks
  • Leverage Local Caching
  • Atomic Operations