System requirements

Internal service API Rate Limiter named Rate Limiter Service API (RLS)

Functional:

• Policy
  • Set Rule/Registration:
    • Which source services are allowed to call at what TPS
    • Which destination services will be received the requests
  • Set Rate
    • How much traffic allowed before throttling
  • Change Rate
    • Ability to change rate
  • Set Status
    • Active: rule being enforced
    • Terminated: rule no longer being enforced
• Throttling
  • Should be able to throttle request (returned exception) when exceeding the threshold

Non-Functional:

• Reliability
  • Consuming services of this API should not be impacted when other services being throttled
• Availability
  • Available 24/7 with 99.999% up time
• Scalability
  • Should handle total of 10,000 requests per seconds (10k tps)
  • Should be able to scale down (to save resources) or scale up (to handle exceptional peak traffic or new consumer services)
• Security
  • Only registered services can send the request
• Fault
  • When this Rate Limiting is down, consumer traffics would still be allowed with very limited rate.
• Monitor/alarm
  • When requests are being throttled, sufficent monitoring fault/throttled rate needed for each service and overall Rate Limiter service health.

Capacity estimation

• Support up to 1,000 services with default/average of 1,000 tps limit and maximum of 10,000 tps limit
• Each server can handle 50k request, 20 servers and scale to 200 servers when peak.

API design

Restful API

POST /throttleRequest

Description:

Consumer service calling this API upon first intercepting the external requests to validate its caller service with allowed tps limit. exception will be thrown if limit being reached.

Input:

SourceService: string

Response:

200 Status

Error:

Exceed TPS Limit

GET /getThrottleRule

Input:

SourceService: string

DestinationService: string

Output:

Tps: number

Error:

ServiceNameNotFound

PUT /setThrottleRule

Input:

SourceService: string

DestinationService: string

Tps: number

Output:

200 Status

Error:

ServiceNameNotFound

Database design

DynamoDB for storing policy and tps rate between services

Policy_Service tbl

1. SourceService(Primary Key): string
2. Status

TPS_Service_Relation tbl

1. SourceService (Primary Key): string
2. DestinationService (Sort Key): string
3. Rate: Number
4. Status: string

High-level design

When serviceA received requests from serviceB, serviceA will invoke Rate Limiter Service API (RLS). The RLS will determine if serviceB either exists in the policy and if the current traffics request from B exceed the rate limit.

Because this is for internal service, there is not required for API gateway/Load Balancing

HLD:

• ServiceB -> serviceA
• ServiceA -> RLS /throttleRequest
• RLS -> look up in-memory cache
  • if missed policy cache -> check Policy_Service DynamoDB if policy has ServiceA + get TPS rate from TPS_Service_Relation for (sourceService = ServiceA and destinationService = ServiceB)
  • if hit cache, return rate
• RLS -> in-memory TPS table to validate if current TPS count for (source = serviceA and destination = serviceB) exceeded the rate
• RLS -> service A either 200 status or throw exception if request exceeded the limit

Request flows

Detailed component design

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?