System requirements

Functional:

Rate limiting per user or API key (e.g., 100 requests/min).

Time-based quota tracking (fixed window/sliding window/token bucket).

Real-time enforcement of limits (must reject/allow immediately).

Custom rules (different limits for different endpoints or users).

Distributed support (should work across multiple servers).

Auditing & Logging of throttled requests.

Admin interface to update limits.

Non-Functional:

High availability – must not be a single point of failure.

Low latency – check must complete in a few milliseconds.

Scalability – must support millions of requests per second.

Consistency – decision should be consistent across nodes.

Fault tolerance – should gracefully degrade or retry on failure.

Capacity estimation

Assume:

• 10 million users.
• Each user makes up to 100 req/min.
• Peak: 100M requests/min = ~1.67M req/sec.

The rate-limiter must sustain 2M QPS at peak traffic.

API design

POST /check_rate_limit

Payload: { "user_id": "123", "api_key": "abc", "endpoint": "/upload" }

Response: { "allowed": true/false, "remaining_quota": int }

POST /update_limit

Payload: { "user_id": "123", "limit": 100, "duration_sec": 60 }

GET /limits

GET /metrics

Database design

Fast in-memory counters.

TTLs for automatic expiration.

Atomic operations (via Lua scripts or INCR + EXPIRE).

Schema (Redis Keys):

rate:{user_id}:{endpoint} => counter + TTL

rate-limit-config:{user_id} => {limit: 100, duration: 60}

High-level design

Components:

• API Gateway – intercepts all requests and queries rate limiter.
• Rate Limiter Service – core logic for tracking and evaluating limits.
• Redis Cluster – stores counters and TTLs.
• Management UI/API – to configure limits and view logs.
• Logger/Monitor – stores metrics, violations, etc.

Client → API Gateway → Rate Limiter → Redis → Decision (Allow/Deny)

Request flows

API Gateway receives request.

Gateway calls RateLimiterService.check(user_id, endpoint).

Service uses INCR and EXPIRE in Redis to count and enforce limits.

If over limit, return HTTP 429 (Too Many Requests).

If under limit, pass to downstream API.

Log rate limit usage for audit and metrics.

Detailed component design

🔹 Redis-based Token Bucket Implementation

• Use Redis INCR for atomic counters.
• Set TTL to match window duration (EXPIRE).
• Use Lua scripts to handle token refill in sliding windows.

🔹 API Gateway

• Lightweight filter to intercept requests.
• Retry on rate-limiter unavailability.
• Fallback logic for critical APIs.

🔹 Rate Limiter Service

• Stateless service.
• Pulls limit configs from Redis or config store.
• Logs usage metrics to monitoring (e.g., Prometheus + Grafana).

Trade offs/Tech choices

Failure scenarios/bottlenecks

• Redis failure: Rate limit decisions may fail open (allow) or closed (deny). Use fallback logic.
• Network latency to Redis: Could slow down decisioning. Consider co-locating Redis with service.
• Hot keys (popular users): Redis sharding to distribute keys evenly.
• Config mismatch: Use centralized config service with cache refresh.

Future improvements

Implement sliding window log or leaky bucket for smoother control.

Use local caching + syncing to reduce Redis dependency.

Add user-specific or endpoint-specific dynamic rate limits.

Expose GraphQL UI or Admin Panel for real-time updates.

Integrate AI anomaly detection on rate-limiting patterns.