Requirements

Functional Requirements:

• Store and serve frequently accessed web content from cache.
• Support cache invalidation when origin data changes.
• Evict entries when cache reaches capacity using a replacement policy.

Non-Functional Requirements:

• Low latnecy - Cache hits served in N<5ms p99 (vs 50-500ms from origin)
• High thruput: Handle millions of req/sec across the cache flet
• High hit ratio: target >95% cache hit rate for cacheable content
• Consistency: Stale content served for at most a configurable TTL; support active invalidation for critical content
• Scalability: horizontally scalable - add nodes to increase capacity and throughput linearly
• Availability: Cache unavailability must not cause origin overload (thundering herd protection)
• Memory efficiency: Maximize useful content stored per GB of RAM

API Design

Client facing:

GET /any/path HTTP/1.1

Host: www.example.com

Response (cache hit):

HTTP/1.1 200 OK

X-Cache: HIT

Age: 45

Cache-Control: max-age=300

Content-Type: text/html

...body...

miss:

Response (cache miss)

HTTP/1.1 200 OK

X-Cache: MISS

Cache-Control: max-age=300

...body fetched from origin...

Invalidation API (origin -> cache)

PURGE /articles/123

Host: www.example.com

Response: 204 No Content

POST /v1/cache/invalidate

Body: {"patterns":["/articles/*]. "reason": "deploy-v2.3"}

Response: {"purged_count": 1482}

Admin/Observability

GET /v1/cache/stats

Response: {

"hit_ratio":

entries

memory_used_gb

memory_total_gb

evictions_per_sec

}

High-Level Design

The cache sits between clients and origin servers as a reverse proxy.

Client -> LB -> cache layer -> origin server

Key design decisions:

1. Two-tier cache: HOT content in RAM (L1), warm content on SSD (L2). Maximizes effective capacity without sacrificing latency for hot content.
2. Consistent hashing: Distribute catched objects across nodes by URL hash. Each URL lives on one node - avoids duplication and maximizes effective capacity.
3. Pull-through caching: On miss, the cache fetche from origin, stores, and serves. No pre-warming required (though supported).

Components:

1. Cache proxy nodes -- stateless HTTP reverse proxies with local RAM + SSD storage. Each node owns a hash range.
2. Routing layer -- consistent hash ring that maps request urls to cache nodes.
3. Origin shield - a second cache ltier that collapses concurrent misses for the same URL into one origin fetch (thundering herd protection)
4. Invalidation bus - pub sub system that distributes purge commands to all relevant nodes.
5. Config service - stores caching rules (TTLs per path pattern, bypass rules, vary headers)

// The HTTP Vary header tells caching servers which request headers to consider when determining if a cached response can be served. It ensures that clients—like browsers—receive the correct content (such as compressed files, mobile layouts, or specific languages) even when the URL remains the same.

Detailed Component Design

Cache lookup and stoarge engine

Datastructure: Hash Map + LRU/LFU hybrid (W-TinyLFU)

Standard LRU evicts recently inserted but infrequent items too slowly, and promotes one-hit wonders. LFU doesn't adapt to changing access patterns. We use W-TinyLFU (used by caffeine/varnish)

WIndow Cache (1% of cap, LRU) -> new entries land here

Main Cache (99% of cap, segmented LRU) - probation segment (20%), protected segment (80%) -> promoted form window if frequency > victim

TinyLFU Frequency Sketch (Count-Min Sketch)

Admission policy: When the window cache is full, the candidate for promotion to main cache is compared against the eviction victim from main cache using the frequency sketch. Higher frequency wins. This filters out scan pollution (one-hit content that woudl evict valuable entries)

Cache key: hash(method+Url+Vary). Vary heder means the same URL may have multiple cached variants (gzip vs bratli)

Storage layout:

• L1 (RAM): serialize http responses in a slab allocator (fixed size chunks to avoid fragmentation). Metadata (headers, TTL, frequency) in a separate hash map.
• L2 (SSD): Overflow for large objects (images, videos) or warm content evicted from RAM. Idnexed by the same hash map with a pointer to disk offset.

2. Consistency and invalidation

Problem: Cached content can become stale. Three mechanisms, used together:

A. TTL-based expiratoin (passive)

• Every cached entry has a TTL derived from origin's Cache-Control: max-age or configured per-path rules.
• on access, if Age > max-age: traet as miss, fetch from origin
• simple, but stale for up to TTL duraction

B. Active Invalidation (push)

• Origin publishes purge events when data changes (e.g. article edited, deploy rolled out)
• Events flow thru the invalidation bus (Kafka or Redis pubsub) -> each cache node receives and deletes matching entries.
• Supports exact keys (/articles/123) and glob patterns (/articles/*)
• Latency: entries purged within 100ms of origin change.

C. Conditional revalidation (pull)

• For expired entries, instead of fetching the full response, send a conditional request:

GET /articles/123

If-None-Match: "etag-abc"

If-Modified-Since: Tue, 19 May 2026

Origin responses 304 Not Modified (no body) -> cache refreshes TTL. Saves bandwidth.

Consistency guarantee: with active invalidation + short TTLs, worst case staleness = invalidation propagation delay (100ms). For non-critical content, TTL-only if sufficient (seconds to minutes of staleness acceptable)

3. Thundering Herd Proection (origin shield/request coalescing)

Problem: A popular URL expires or gets purged. 10k concurrent requests arrive. Without protection, all 10k hit the origin simultaneously - potentially killing it.

Solution: Request coalescing (also called collapsed forwarding)

Req 1 for /popular-page -> cache miss -> lock acquired -> fetch from origin. Req 2 -> wait, etc.

...

origin responds -> store in cache -> wake all waiters -> serve from cache

Implementation:

• Per key mutex (in-memory hash map of cache key -> promise)
• first req creates the promis eand fetches. Sub reqs await the same promise.
• Timeout - if origin doesn't respond in 4s, release waiters with an error

Origin shield(optional second tier)

• For CDN-scale deployments with multiple cache PoPs, an origin shield is a designated cache cluster that sits between edge caches and origin
• Reduces origin load from N_edges x requests to 1 x requests per URL

Key Tradeoffs

Consistent hashing: higher per node miss impact on failure vs no duplication, maximum effective capacity

W-TinyLFU vs plain LRU: slightly more cpu per access vs sig higher hit ratio 5-10% improvement

Two-tier RAM + SSD: complexity of tiered storage vs 10-50x more cap at marginal latency cost for warm content

Active invalidation via pubsub: Operational complexity vs sub-second consistency

Request coalescing: Added latency for waiters (they block) vs origin protection

Conditional revalidation (ETags): Extra round trip on expiry vs bandwidth savings (no body transfer on 304)

Consistent hash with virtual nodes: Rebalancing moves 1/N content on node add/remove vs full reshuffle