Design a Web Cache

A web cache sounds simple: store a copy of something so you do not have to fetch it again. The real challenge is what happens when the cache is full, when the original data changes, and when one server is not enough. The system is not just a dictionary; it is a distributed, fault-tolerant data layer that sits between your services and the origin, absorbing the majority of read traffic.

Functional

Store and serve frequently accessed content: The cache stores static or dynamic web content (HTML pages, images, API responses) keyed by URL or content hash. Repeat requests are served directly from fast in-memory storage, completely bypassing the origin server. This is the fundamental value proposition: trade memory for speed.

The capacity math for a web cache is dominated by one insight: you do not need to cache everything. The 80/20 rule applies aggressively. A small fraction of content generates the vast majority of requests. Sizing the cache to hold this working set is the key decision.

Traffic

100K requests per second: A large web application serving millions of users. Not all requests hit the cache; some are cache misses that go to the origin. The target is a 95% cache hit ratio, meaning 95K requests per second are served from cache and only 5K reach the origin.

The API surface for a web cache is deliberately minimal. Three core operations cover the entire functionality: store an entry, retrieve an entry, and remove an entry. A small API reduces integration complexity and limits the attack surface.

Cache Write

1PUT /cache
2Headers: Content-Type, X-Cache-TTL (seconds)
3Body: {
4key: string (URL or content hash),
5value: bytes (response body),
6headers: map (response headers),
7ttl: number (optional, seconds until expiry)
8}
9Response: 200 OK { key: string, stored_at: ISO-8601, expires_at: ISO-8601 }

The storage layer for a web cache is straightforward: the primary store is RAM. The interesting decisions are which in-memory technology to use, how to structure the cache entries, and how to partition data across nodes.

Redis vs. Memcached: Primary Cache Store

1Cache Entry
2key STRING URL or content hash
3value BYTES Response body
4headers MAP Original response headers
5created_at NUMBER Unix timestamp (ms)
6ttl NUMBER Seconds until expiry

The architecture places a cache proxy between client services and the origin server. The cache proxy handles key routing, cache lookups, and miss handling. A configuration manager (ZooKeeper or etcd) maintains the consistent hashing ring so all proxies agree on which cache node owns each key.

Two request flows define the system: the cache hit (fast path) and the cache miss (slow path). Understanding the exact sequence of operations, and where failures can occur at each step, reveals why the architecture is structured the way it is.

Cache Hit Flow

The core engineering challenge (the GATE of this problem) is building a key distribution system that scales horizontally without invalidating the entire cache when nodes are added or removed. Consistent hashing with virtual nodes is the mechanism that makes this possible.

Consistent Hashing in Detail

Every architectural decision in this system involves a trade-off between speed, consistency, and operational complexity. Understanding these trade-offs, and being able to articulate them in an interview, demonstrates deliberate engineering rather than pattern adoption.

Write-Through vs. Write-Back Caching

Write-through writes to both cache and origin synchronously on every update. The cache is always consistent with the origin, but every write pays the latency of the slower path (the origin). For a web cache where writes are infrequent (100:1 read-to-write ratio), this overhead is manageable. Choose write-through when stale data is unacceptable, such as caching user session data or authentication tokens.

The system's reliability comes from its ability to handle failures gracefully rather than preventing them entirely. Each failure scenario maps to a specific architectural decision made earlier.

Cache Node Failure

These improvements extend the system without changing the core architecture. Each builds on the existing cache proxy and Redis cluster.

Multi-Tier Caching (L1 + L2)

The current design uses a single distributed cache tier (L2, Redis). Adding an L1 in-process cache (Caffeine or Guava, 100-500MB per proxy instance) handles the hottest keys with zero network overhead. L1 has very short TTLs (5-10 seconds) because it lacks active invalidation. This creates a two-tier hierarchy: L1 absorbs the hottest traffic, L2 handles the broader working set, and the origin handles cold misses. The L1 cache alone can reduce Redis traffic by 30-50% for workloads with high key locality.