1. Problem and assumptions

We need to design a system that limits how many requests a client can make in a given time window.

The system should sit in front of APIs and decide:

• allow request
• reject request with something like HTTP 429 Too Many Requests

I’ll assume:

• clients are identified by API key, user ID, IP, or token
• limits can differ by customer tier or endpoint
• the system must make decisions in real time
• low latency is critical
• exact fairness is desirable, but we can tolerate tiny inaccuracies if needed for high scale

Out of scope:

• billing
• authentication itself
• full WAF / DDoS protection, though rate limiting may help with abuse

2. Functional requirements

The system should:

• enforce rate limits per key, user, IP, tenant, or endpoint
• support policies like:
  • 100 requests per minute
  • 1000 requests per hour
  • burst + sustained limits
• return allow or reject quickly
• support distributed deployment across many API servers
• allow dynamic configuration updates
• expose metadata:
  • remaining quota
  • reset time
  • retry-after
• support whitelist / blacklist / tier-based overrides

Nice to have:

• dashboards and analytics
• per-endpoint rules
• sliding-window accuracy
• shadow mode for testing policies without enforcing them

3. Non-functional requirements

Latency

Decision should usually happen in a few milliseconds.

A good target:

• p50 under 2 ms inside the limiter layer
• p95 under 10 ms

Availability

If the rate limiter goes down, it should not bring down the whole API platform. We need a fail-open or fail-closed policy depending on the endpoint.

Scalability

Should support very high request volume, possibly:

• 100k to millions of requests per second globally

Consistency

We want reasonably accurate enforcement across distributed instances. Perfect global consistency on every request is expensive, so we may use approximate or eventually consistent strategies.

Security

Clients should not be able to spoof identifiers easily if limits are tied to user identity.

Operability

Need monitoring, debugging, and configuration rollout support.

4. Capacity estimation

Let’s assume a medium-large public API system.

• 10 million registered API keys
• 1 million daily active keys
• peak traffic 200k RPS
• average 40k RPS

Suppose each rate-limit check requires:

• read current counter
• update counter
• return decision

If done naïvely in a central DB, this is too expensive and too slow.

Storage for active counters is actually not huge if counters are ephemeral.

Example:

• 5 million hot keys active in a day
• each counter record ~100 bytes including metadata
• that is about 500 MB for current active window state

So the main challenge is not storage.

It is:

• low-latency writes
• high throughput
• coordination across distributed nodes

5. API and data model

Enforcement API

Internally, the gateway or service may call:

checkLimit(identity, resource, policyContext) -> decision

Response:

• allowed: true/false
• remainingTokens
• retryAfter
• resetAt

Config API

Admins should be able to define policies like:

• key A: 100 req/min
• tenant B: 10k req/min
• endpoint /search: 20 req/sec per user

Counter record

A logical record could look like:

• subjectId
• scope (global / endpoint / tenant)
• algorithm type
• current count or tokens
• window start or refill timestamp
• TTL

For example:

• user123:/search
• count = 57
• windowStart = 12:00:00
• expiresAt = 12:01:00

6. Algorithm choice

This is the first major deep-dive area because rate limiting depends heavily on algorithm.

Main options:

Fixed window counter

Example:

• 100 requests per minute
• reset every minute

Pros:

• simple
• cheap
• easy in Redis

Cons:

• boundary problem
• A client can send 100 requests at 12:00:59 and 100 more at 12:01:00

This allows burstiness.

Sliding window log

Store timestamps of recent requests and count those inside the last window.

Pros:

• most accurate

Cons:

• expensive in memory and writes
• not great at large scale for every key

Sliding window counter

Approximate sliding window using current + previous bucket weighted by overlap.

Pros:

• more accurate than fixed window
• cheaper than log

Cons:

• slightly more complex
• still approximate

Token bucket

Tokens refill at a fixed rate. A request consumes a token.

Pros:

• excellent for allowing controlled bursts
• widely used
• elegant
• easy to reason about

Cons:

• implementation slightly more involved than fixed counter

Leaky bucket

Smooths traffic to a constant drain rate.

Pros:

• useful for shaping traffic

Cons:

• less intuitive for quota-style API limits

Recommendation

For API rate limiting, I would usually choose:

• token bucket for enforcement
• optionally combined with a longer-term quota like fixed or sliding window for sustained control

Reason:

• allows burst handling
• easy to support limits like 10 req/sec with burst 20
• efficient in memory

7. High-level architecture

A good production design is:

Client

→ CDN / Edge / API Gateway

→ Rate Limiter

→ Application Service

Supporting systems:

• config store
• distributed in-memory counter store
• metrics pipeline
• admin control plane

Components

1. API Gateway / Sidecar / Middleware

This is where the check happens.

Best to enforce as close to request entry as possible.

Possible placements:

• inside API gateway
• sidecar per service
• shared rate-limiter service

2. Rate Limiter Decision Engine

Takes identity + policy and computes allow/reject.

3. Distributed fast state store

Usually Redis or an in-memory distributed key-value store.

Used for:

• counters
• token buckets
• TTL-based state
• atomic updates

4. Config store

Stores policy definitions:

• per-tier limits
• per-endpoint overrides
• allowlists

Could be backed by:

• MySQL/Postgres/Cassandra/etc.
• distributed config system

Frequently cached locally in limiter instances.

5. Metrics / logs

Track:

• allowed
• throttled
• top offending keys
• per-endpoint pressure
• store latency
• error rate

8. Core request flow

Let’s use token bucket.

Request path

1. Client sends API request.
2. Gateway extracts identity:
   • API key
   • user ID
   • IP
3. Gateway determines applicable policy.
4. Gateway asks limiter:
   • does this identity have at least 1 token?
5. Limiter checks state in fast store.
6. If token exists:
   • decrement token count
   • allow request
7. Else:
   • reject with 429
   • include Retry-After

9. Token bucket implementation

For each subject we store:

• availableTokens
• lastRefillTimestamp

Suppose policy:

• refill rate = 10 tokens/sec
• bucket size = 20

On each request:

1. read state
2. compute elapsed time since last refill
3. add replenished tokens up to max bucket size
4. if tokens >= 1:
   • tokens -= 1
   • allow
5. else reject
6. write updated state back atomically

This must be atomic to prevent race conditions.

In Redis, this is often done with:

• Lua script for atomic read-modify-write
• or a single atomic command pattern

TTL can be set so inactive subjects are cleaned up automatically.

10. Detailed architecture choice

Why Redis-like store?

Because rate limiting is:

• write-heavy
• low-latency
• counter-oriented
• often ephemeral

Redis gives:

• in-memory speed
• atomic operations
• TTL expiry
• replication
• clustering

Why not main SQL DB?

Because at high RPS:

• too slow for per-request coordination
• too expensive
• poor fit for ephemeral counters

11. Distributed system challenges

This is where the interview gets interesting.

Challenge 1: multiple gateway instances

Many app servers may process requests for the same client simultaneously.

If each server keeps only local memory counters, limits become inaccurate globally.

Option A: centralized distributed store

All instances update the same Redis cluster.

Pros:

• global correctness across instances
• simple mental model

Cons:

• extra network hop
• Redis hot keys can appear

This is the standard practical answer.

Option B: local token caches

Each gateway gets a lease of tokens from a central store.

Example:

• central store gives node A 50 tokens for user X
• node A serves requests from local memory
• when exhausted, fetch more

Pros:

• lower latency
• fewer central writes

Cons:

• temporary over-allocation across nodes
• more complex correctness

This is useful for very high scale.

A strong interview answer is:

• start with centralized Redis
• later optimize with token leasing if needed

Challenge 2: hot keys

A very popular client may generate huge traffic on the same key.

Solutions:

• shard by subject ID
• add local caching/leases
• separate premium high-traffic tenants onto dedicated partitions
• use consistent hashing
• possibly split counters per dimension if safe

Challenge 3: cross-region enforcement

Suppose the same API key sends requests to us-east and eu-west.

If each region rate-limits independently, the client may exceed the intended global quota.

Options:

Option A: regional limits only

Each region enforces independently.

Pros:

• simple
• low latency

Cons:

• no true global quota

Option B: global centralized store

All regions consult one global store.

Pros:

• more accurate global limits

Cons:

• cross-region latency
• blast radius

Option C: hierarchical limits

Use:

• local regional rate limit for fast path
• slower global quota reconciliation in background
• or split quota across regions

This is often the best practical compromise.

For most interviews, I’d say:

• single-region system: centralized distributed store
• multi-region system: per-region enforcement plus optional global quota layer depending on product requirements

12. What to do when the limiter store fails

This is an important trade-off.

Fail-open

Allow requests if the rate limiter cannot decide.

Pros:

• protects availability of core API
• good for non-critical APIs

Cons:

• abuse can slip through

Fail-closed

Reject requests if limiter cannot decide.

Pros:

• protects backend from overload or abuse

Cons:

• limiter outage becomes API outage

Recommendation

Make this configurable by endpoint:

• login, payments, expensive compute APIs: likely fail-closed or degraded strict mode
• normal read APIs: often fail-open with alerts

You can also keep a small local emergency fallback limiter to reduce total abuse during store failure.

13. Data partitioning

Partition counter state by hash of:

• user ID / API key / IP / route

Use consistent hashing so requests for the same subject map to the same shard.

For Redis cluster:

• key format like rate_limit:{apiKey}:{endpoint}

TTL ensures expired windows disappear automatically.

For token bucket, inactive users’ state should expire after some idle period.

14. Configuration distribution

Policy lookup cannot hit a DB on every request.

So:

• source of truth in durable config store
• push updates to limiter nodes via pub/sub or config service
• cache policies in local memory
• version policies for safe rollout

This allows low-latency checks.

Example:

• Free tier: 100 req/min
• Pro tier: 1000 req/min
• Admin updates Pro tier to 2000 req/min
• control plane publishes new config
• limiter fleet refreshes local cache

15. Example full design

Control plane

• admin UI / API
• durable config DB
• config propagation service

Data plane

• API gateway / envoy / nginx / service middleware
• local policy cache
• rate limiter module
• Redis cluster for counter state
• observability pipeline

Request path

1. request enters gateway
2. gateway identifies subject and endpoint
3. local cache fetches policy
4. Redis Lua script atomically refills/decrements token bucket
5. gateway gets decision
6. allow or 429

This is a clean production-grade design.

16. Monitoring and observability

Must track:

• total requests
• throttled requests
• top throttled tenants
• decision latency
• Redis latency
• Redis errors/timeouts
• config mismatch/version skew
• per-endpoint rate-limit hit ratio

Also useful:

• audit log for config changes
• shadow-mode analysis before rollout

17. Security considerations

Need to make sure the identity used for rate limiting is trustworthy.

Examples:

• do not trust raw client IP behind proxies unless correctly normalized
• use authenticated API key / user ID when possible
• for public unauthenticated APIs, combine IP + fingerprint + route
• guard against distributed abuse across many IPs

Rate limiting is not a full anti-DDoS solution, but it helps.

18. Bottlenecks and mitigations

Bottleneck: Redis latency

Mitigation:

• local policy cache
• colocate limiter and Redis
• shard Redis
• pipeline requests where possible
• use token leasing for very hot paths

Bottleneck: single global store

Mitigation:

• regional sharding
• hierarchical quotas
• local fallback buckets

Bottleneck: too many dimensions

If you rate limit by user, IP, endpoint, method, org, etc., state explodes.

Mitigation:

• only enforce on important dimensions
• compose limits carefully
• store only hot active counters

19. Trade-offs summary

Fixed window vs token bucket

• fixed window is simpler
• token bucket handles bursts better
• I would choose token bucket for API platforms

Centralized vs local enforcement

• centralized distributed store is more accurate
• local memory is faster but inaccurate across nodes
• start centralized, optimize later with token leasing

Strong consistency vs availability

• globally consistent exact limits are expensive
• most systems accept slight approximation for low latency and high availability