Design An API Rate Limiter - System Design

Requirements

Functional Requirements:

per-user / per-API rate limits (meaning that an endpoint within an API can have stricter rate limits).
Admins can add/update/delete/view rate-limit rules via an API or console.

Non-Functional Requirements:

Calling our API rate limiter should not affect external APIs. We expect low latency.
System should be scalable (add more machines to handle more traffic)
System should be reliable and fault tolerant

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

limitersvc API

Primary Endpoint:

POST /v1/ratelimit/check
Request:
{
  "apiKey": "string",
  "endpoint": "string",
  "requestCount": 1  // optional, defaults to 1 for batch operations
}
Response (Success - 200 OK):
{
"allowed": true,
"remainingTokens": 95,
"resetTime": "2026-01-29T10:26:00Z",
"retryAfter": null
}
Response (Rate Limited - 429 Too Many Requests):
{
"allowed": false,
"remainingTokens": 0,
"resetTime": "2026-01-29T10:26:00Z",
"retryAfter": 30  // seconds until next token available
}

Batch Check Endpoint (for API Gateway to pre-validate multiple requests):

POST /v1/ratelimit/check/batch
Request:
{
"checks": [
{"apiKey": "key1", "endpoint": "/api/users"},
{"apiKey": "key2", "endpoint": "/api/posts"}
]
}
Response:
{
"results": [
{"apiKey": "key1", "allowed": true, "remainingTokens": 50},
{"apiKey": "key2", "allowed": false, "retryAfter": 15}
]
}

Health Check:

GET /v1/health
Response:
{
"status": "healthy",
"cacheConnected": true,
"latencyMs": 2
}

configsvc API

Get Rate Limit Configuration:

GET /v1/config/limit?apiKey={apiKey}&endpoint={endpoint}
Response:
{
"apiKey": "abc123",
"endpoint": "/api/users",
"tier": "premium",
"maxRequests": 1000,
"windowMinutes": 60,
"createdAt": "2026-01-15T10:00:00Z",
"updatedAt": "2026-01-20T14:30:00Z"
}

List All Configurations for an API Key:

GET /v1/config/limits?apiKey={apiKey}
Response:
{
"limits": [
{
"endpoint": "/api/users",
"tier": "premium",
"maxRequests": 1000,
"windowMinutes": 60
},
{
"endpoint": "/api/posts",
"tier": "basic",
"maxRequests": 100,
"windowMinutes": 60
}
]
}

Create/Upgrade Plan:

POST /v1/config/plan
Request:
{
"apiKey": "abc123",
"endpoint": "/api/users",
"tier": "premium",
"maxRequests": 1000,
"windowMinutes": 60,
"payment": {
"method": "credit_card",
"token": "tok_xyz789"
}
}
Response (Success - 201 Created):
{
"status": "success",
"message": "Plan upgraded successfully",
"effectiveAt": "2026-01-29T10:25:00Z",
"config": {
"apiKey": "abc123",
"endpoint": "/api/users",
"tier": "premium",
"maxRequests": 1000,
"windowMinutes": 60
}
}
Response (Failure - 400/402):
{
"status": "failed",
"errorCode": "PAYMENT_FAILED",
"message": "Payment processing failed"
}

Update Plan:

PUT /v1/config/plan
Request:
{
"apiKey": "abc123",
"endpoint": "/api/users",
"tier": "enterprise",
"maxRequests": 10000,
"windowMinutes": 60
}
Response:
{
"status": "success",
"message": "Plan updated successfully",
"previousLimit": 1000,
"newLimit": 10000
}

Cancel Plan:

DELETE /v1/config/plan?apiKey={apiKey}&endpoint={endpoint}
Response:
{
"status": "success",
"message": "Plan cancelled successfully",
"effectiveAt": "2026-01-29T10:25:00Z"
}

Internal Endpoint (used by configsvc to notify limitersvc):

POST /v1/internal/cache/invalidate
Request:
{
"apiKey": "abc123",
"endpoint": "/api/users",
"newLimit": {
"maxRequests": 1000,
"windowMinutes": 60
}
}
Response:
{
"status": "success",
"cacheUpdated": true,
"affectedInstances": 12
}

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

Load balancer: redistribute traffic.
API gateway: abstract microservices that we have for the client. The gateway should know to where route traffic. We can have multiple instances to provide horizontal scaling. Each API gateway instance also enforces a global per instance rate limit to protect itself from overload. If incoming RPS exceeds safe capacity, the gateway sheds excess requests.
limitersvc: service that will keep track of number of requests of each api key. It will tell whether maximum limit has been reached or good to make requests
cache: distributed cache to keep track of how many requests an api key has done within a time range (token bucket)
configsvc: service to add/update/delete/view rate-limit rules via an API
database: keep state of api key rules (which plans api key or user has...). Ex: api key has this many number of requests for endpoint X in api X, api key has this many number of requests for endpoint Y in api X. Different endpoints can different rate limits

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

limitersvc: we can leverage token bucket algorithm to keep track of how many requests an api key has made within a time range. Token bucket algorithm allows a burst in traffic. Ex: assume a user can make 3 requests per minute for endpoint X (stored in database). And the bucket refills itself. 1 request = 20 seconds. Meaning that bucket will add 1 token every 20 seconds (if bucket is less than 3). limitersvc have multiple instances to scale horizontally + having high availability. If limitersvc is down (0 instances are working), we can fallback to the api gateway's local cache cache. The expectation is that limitersvc will get a bit of less request because of the local cache in api gateway.
1. Example: For each api key and endpoint pair, the limitersvc will query the configuration database to know max limit api key can query for that endpoint only in the FIRST request
  1. User has 3 tokens left and makes request at 10:25:35. In cache: Tokens left 2, last refill at 10:25:35 with plan as 3 requests/minute)
  2. User has 2 tokens left and makes request at 10:25:52. Tokens left 1
  3. time arrives at 10:25:55, 20 seconds have passed. Add one token to the bucket. So now we have 2 tokens in bucket and last refill is 10:25:55.
Cache key is api key + endpoint. Values are number of tokens + last refill time.
cache: it's a distributed cache, that way all instances of limitersvc uses the same cache and its not tied to a local cache per limitersvc instance. We can also have read replicas in case our cache fails. Why use a cache? we're storing simple key value pairs. And reason of using cache is because it is a faster retrieval and our service should be low latent. If cache is down (0 instances are working), we can fallback to the api gateway's local cache to provide high availability. But since we have leader replication (high availability), low probability of cache being unavailable. When master gets down, we can rely on one of our followers.
configsvc: for simplicity, let's assume that a user has only one api key. User onboards and signs in, selects their plan and pays for it. Here users can upgrade their plans to increase rate limiting. Let's say a user was rate limited because he was paying for 100 requests every 1 hour. Let's say that in 30 min, user already made the 100 requests and wants to update his/her membership to make 1000 requests in 1 hour. As soon as this change is made, configsvc should call limitersvc or send a notification to it via a queue, to update the cache for a particular api key + endpoint cache key (if it exists) So cache will now say 1000 requests/hour for that api key + endpoint. configsvc should continue working as is.

Tradeoffs

Cons: what if we receive two requests from the same API key+ endpoint as cache key at same millisecond, different machines? Use a locking mechanism, where one acquires a lock to write, but this is very slow. Redis has support for atomic operations. We move all logic into one atomic step inside Redis and it guarantees no race conditions and that no other request can interrupt this logic.

what if machines have different clocks? Servers can differ in time. Let's use Redis time as the single source of truth of time.

Handling High QPS

With billions of DAU and potentially millions of QPS, the system must minimize latency while preventing cache and network saturation.

Multi layer caching strategy:

API Gateway local cache: each API Gateway instance has an in memory cache. Cache stores leased tokens from Redis, not full rate limit state.
1. Token leasing algorithm: when gateway receives request, check local cache for (apikey, endpoint) pair. If cache hit and tokens available, decrement local token count and return. If cache miss or tokens depleted, lease tokens from Redis and store them in local cache.
2. Leasing formula = min(tokens available / number of gateway instances, maxLeaseTokens)
Distributed cache: Another way to handle high QPS is to partition distributed cache by pair (api key and endpoint) to distribute load. We can use consistent hashing for whenever a machines gets added/removed. Even if we parition the cache, we may get a huge number of requests for a particular pair (apikey and endpoint). We can make an exception for those of type of keys and partition even further.
Read replicas: 1 primary redis node, 3-5 replicas

Request Flow with Multi-Layer Cache

Incoming Request → API Gateway
  ↓
  Check Layer 1 (Local Cache)
  ├─ Hit + Tokens Available → [Allow Request] (0.1ms latency)
  └─ Miss or No Tokens
      ↓
      Check Layer 2 (Distributed Redis)
      ├─ Standard Key → Hash to Redis Instance
      ├─ Hot Key → Hash to Shard
      │   ↓
      │   Atomic Redis Operation:
      │   - Get current tokens
      │   - Get last refill time
      │   - Calculate refilled tokens
      │   - Decrement if available
      │   - Update last refill time
      │   ↓
      │   Lease tokens to Gateway Local Cache
      │   ↓
      │   [Allow/Deny Request] (2-3ms latency)
      └─ Redis Failure
          ↓
          Fallback to Gateway Local Cache (best effort)
          ↓
          [Allow with degraded limits] (0.1ms latency)