Design an API Rate Limiter

Every API call you make (Google Maps, Stripe, OpenAI) passes through a rate limiter before it reaches business logic. The limiter must decide allow or deny in under 10ms, at millions of requests per second, without itself becoming the bottleneck it was built to prevent.

The defining tension: a rate limiter must be accurate (never let excess traffic through) and available (never block the limiter itself). The algorithm choice and state location are the two decisions that shape everything else.

Functional

Rate limiting state is tiny per user. But multiplied across millions of users and thousands of endpoints, the aggregate memory challenges a single machine. The real bottleneck is not storage but throughput.

Traffic

• Peak decisions: 1 million per second across all APIs and users.
• Allow/deny ratio: 80% allowed, 20% denied. The allow path dominates and must be optimized.
• Config changes: fewer than 10 writes per second. Rule updates are rare compared to rate limit checks.

Two very different APIs serve this system. A blazing-fast decision endpoint called millions of times per second, and a slow management API called a few times per day. Mixing them in one service would let a slow rule update block the fast decision path.

Decision API

The core endpoint: POST /ratelimit/check. The gateway sends {"user_id": "u_42", "endpoint": "/api/messages", "scope": "user"}. On allow, the response is {"allowed": true, "remaining": 14, "limit": 100, "reset": 1710432000}. On denial: {"allowed": false, "remaining": 0, "limit": 100, "retry_after": 23}. The response takes under 1ms because it reads from Redis, never PostgreSQL.

Two data types with opposite access patterns, config that rarely changes but must be durable, and counters that change on every request but can tolerate loss. Trying to serve both from one store forces a compromise that hurts performance on the hot path.

Rule Configuration (PostgreSQL)

The rate_limit_rules table stores all rate limit configurations. Key columns: rule_id (UUID primary key), scope_type (enum of user, ip, org, api_key), scope_value (the specific user ID or IP), api_endpoint (the path being limited), algorithm (token_bucket or sliding_window), max_requests (the limit), window_seconds (the time window), and burst_size (for token bucket). A unique constraint on (scope_type, scope_value, api_endpoint) prevents conflicting rules for the same scope and endpoint.

Deceptively simple architecture. A stateless service sitting between the gateway and backends, consulting Redis for every decision. The complexity hides in what happens when Redis is slow or a single key gets millions of hits per second.

Architecture Overview

Every request follows the same outer path (check, decide, respond) but the internals change depending on the algorithm. Token bucket refills on read. Sliding window counts events in a range. Both must complete atomically in under 1ms.

Allow/Deny Decision Flow

1. Client sends a request to the API Gateway.
2. API Gateway authenticates the request and extracts user_id, IP, and endpoint path.
3. Gateway calls the Rate Limiting Service with the extracted metadata.
4. RLS constructs the Redis key: rl:{scope_type}:{scope_value}:{api_endpoint}.
5. RLS sends the Lua script to Redis with the key and current timestamp.
6. Redis executes the Lua script atomically, reads bucket state, applies the algorithm, updates state, and returns the result. RLS receives allowed/denied with remaining tokens. If allowed, Gateway forwards to Backend. If denied, Gateway returns 429 with Retry-After.

Three engineering challenges define the detailed design, making concurrent decisions atomically, choosing the right algorithm for the use case, and handling hot keys that concentrate millions of hits on a single Redis entry.

Atomic Operations With Lua Scripts

Every rate limiter design balances three tensions: accuracy versus speed, centralized state versus distributed state, and fail-open versus fail-closed. There is no universally correct answer. The right choice depends on what your system cannot tolerate.

Token Bucket vs Sliding Window

As discussed in the detailed design, token bucket is the default for most endpoints. The key trade-off decision is whether your system can tolerate burst behavior. If so, token bucket wins on simplicity. If not (login, payment), switch to sliding window.

A rate limiter that goes down is worse than no rate limiter at all, services either get flooded with unthrottled traffic or blocked from serving any requests. Every failure mode must have a pre-planned response.

Redis Failure

The basic per-user limiter handles 90% of use cases. Production systems evolve beyond this baseline into multi-scope limits, adaptive throttling, and global distribution. Each adding a layer of sophistication.

Multi-Scope Limits

Real systems apply per-IP, per-user, per-org, and per-endpoint limits simultaneously. A single request is checked against all applicable scopes in order: IP first (cheapest, no auth required), then user, then org. The first denial wins. If the IP limit is exhausted, the user and org checks are skipped. This ordered evaluation short-circuits expensive checks and ensures that the strictest applicable limit is enforced.