Problem scope

Clarifying questions

• what content do we need to store? -> static content like HTML files, images, videos, other resources
• what size to expect? -> total cache size of 1TB
• for how long do we need to store it? -> depends on how frequent data is accessed but it should have a TTL
• what's the scale? -> medium scale for now
• what's the eviction policy? -> tied to the TTL, if stale just drop it

Problem statement

Design a medium scale distributed cache system to store static content and reduce backend load. Total cache size ~1TB and use TTL-based caching (stale entries are evicted).

Requests hit the cache first and on miss the content is fetched from origin and cached with TTL.

Core constraints

1. low latency reads - cache must serve hot content fast
2. correct freshness - TTL must be respected and stale content not served
3. high availability - cache misses or node failures must not break reads

Requirements

Functional Requirements:

• cache static content for services
• serve cached content if TTL is valid
• on TTL expiry, fetch fresh content from origin and update cache
• evict stale entries after TTL expiration and inactivity
• support manual or admin triggered invalidation

Non-Functional Requirements:

• low latency read (cache-first read)
• high availability (no single cache node dependency)
• correct freshness enforced by TTL
• total cache size of 1TB

API Design

The APIs are called by the API Gateway.

Read path

Write path

Invalidation

High-Level Design

Components

Client

API Gateway

Cache Server (all in memory - 1TB)

Backend Server

Read path

• the Client sends a request to the API Gateway
• the API Gateway derives the cache key
• the API Gateway quries the Cache with the cache key
• Cache hit and TTL valid -> return value to API Gateway -> Client
• Cache hit but TTL expired -> treat as miss
• Cache miss -> gateway fetches data from the Backend Server
• API Gateway returns the data to the Client
• API Gateway stores it in Cache with TTL

Write path

This is covered by the cache miss from Read path.

Failure - cache node failure

When the cache lookup fails or times out we treat it as cache miss so we fetch from backend. Optionally we can write to another healthy node.

Failure - Backend Server is down or slow

In this case the TTL is expired and the cache cannot be refreshed. We can server the data stale with a flag and let the client decide if/how they want to use it.

Admin - invalidation of TTL

In this case admins can manually trigger data refresh by invalidating the TTL for a given cache key.

Detailed Component Design

Eviction and capacity management

Problem: Cache is limited to 1TB and hot data can crowd out useful entries.

Approach: have TTL-based eviction as baseline and add LRU/LFU when capacity pressure hits.

Tradeoff: simple eviction vs optimal hit rate.

Multiple misses on the same key at once

Problem: Many clients miss the same key and the backend gets overloaded.

Approach: We can group the requests for the same key if they happen at once and do a single flight.

Tradeoff: slightly stale reads vs backend protection.

High availability and failure handling

Problem: Cache node failure or timeouts can impact reads.

Approach: Treat the failures as cache misses and fallback on the backend. Additionally we can use replication for hot keys.

Tradeoff: Lower hit rate during failure vs availability.