Caching works because of a simple observation about real-world access patterns: data that was recently accessed is likely to be accessed again soon (temporal locality), and data near recently accessed data is also likely to be accessed soon (spatial locality).

These patterns are everywhere. A user who views their profile will probably view it again within minutes. A product page that one customer visits is likely to be visited by other customers browsing the same category. A database query that ran once will likely run again with the same parameters within seconds. Caching exploits this predictability by keeping frequently accessed data in a faster storage medium.

The core mechanism is straightforward. You place a fast storage layer (memory, SSD) between your application and a slower storage layer (database, remote API). When a request arrives, you check the fast layer first. If the data is there (a cache hit), you return it immediately without touching the slower layer. If the data is not there (a cache miss), you fetch it from the slow layer, store a copy in the fast layer for future requests, and return it to the caller. The ratio of hits to total requests is your hit ratio, and it determines whether your cache is earning its keep.

A 95% hit ratio means only 5% of requests reach your database. A 50% hit ratio means your cache is barely helping: half of all requests still hit the slow path, and you have added complexity (a second data store to manage) for marginal benefit. Most production caches target 90%+ hit ratios to justify the operational overhead.

But caching introduces a fundamental problem: you now have two copies of the truth, and they can disagree. The database might say a product costs $29.99 while the cache still says $39.99. The database might say a user's account is suspended while the cache still returns their active session. Every caching decision is ultimately about managing this disagreement: how long can you tolerate stale data, how do you detect it, and how do you fix it.

Cache Eviction Policies

Every cache has finite memory. When it fills up, you must decide which entries to remove to make room for new ones. The eviction policy determines this, and the right choice depends on your access pattern.

LRU (Least Recently Used) evicts the entry that has not been accessed for the longest time. This works well when recent access predicts future access, which is true for most web applications. User sessions, product pages, and API responses all follow this pattern. LRU is the default choice for good reason. Most cache libraries (Redis, Guava, Caffeine) use LRU or an approximation of it as the default eviction policy.

LFU (Least Frequently Used) evicts the entry with the fewest total accesses. This protects popular items even if they have not been accessed in the last few seconds. It works well for content like trending posts or popular products that are accessed repeatedly over hours.

The weakness of LFU is that it can be slow to adapt when popularity shifts. An item that was popular yesterday but is irrelevant today stays cached because its historical count is high. New content that is currently trending gets evicted in favor of old content that accumulated a high count over time. Some implementations address this with a decay mechanism that gradually reduces older access counts, but pure LFU has no built-in way to forget past popularity.

FIFO (First In, First Out) evicts the oldest entry regardless of how often or how recently it was accessed. This is the simplest eviction policy to implement (just a queue), but it is wasteful for most use cases. It treats a frequently accessed entry the same as one that was never read after insertion, evicting both purely based on insertion order.

FIFO is rarely the right choice for application caches because access patterns in web applications are not correlated with insertion order. However, it works well for append-only logs, time-series data, and streaming buffers where age genuinely determines relevance and older entries should always give way to newer ones.

TTL-Based Expiration

Time-to-live (TTL) is the simplest invalidation strategy. Each cache entry gets a timestamp, and the cache automatically removes it after a fixed duration. Set a 5-minute TTL on user profiles, and every profile is at most 5 minutes stale.

TTL is a tradeoff between freshness and load. A 10-second TTL keeps data very fresh but forces frequent cache misses, meaning the database handles nearly as many reads as it would without a cache. A 1-hour TTL reduces database load dramatically but means users might see hour-old data. The right TTL depends on how stale the data can be before it causes problems.

Different data types have very different staleness tolerances. Session tokens and authentication data need TTLs of seconds because serving a revoked session is a security risk. Product prices need minutes because a stale price leads to customer complaints or revenue loss. Static configuration and feature flags can tolerate hours because they change infrequently and are rarely time-sensitive.

The danger of TTL alone is that it is time-based, not change-based. If a product price changes, the cache serves the old price until the TTL expires. Users see stale data for the entire remaining TTL duration, with no mechanism to correct it earlier.

For data where correctness matters more than performance (prices, permissions, account status), you need event-based invalidation, which we cover in a later section. For data where approximate freshness is acceptable (recommendations, analytics summaries, content feeds), TTL is the right choice because it is simple, requires no additional infrastructure, and provides a predictable upper bound on staleness.

Cache Sizing

The size of your cache relative to your working set determines your hit ratio more than any other factor. The working set is the total amount of distinct data actively accessed by users during a time window matching your TTL. If your TTL is 10 minutes, the working set is all unique keys accessed in the last 10 minutes.

When the cache is larger than the working set, the hit ratio approaches 100% because every active entry fits in cache without eviction. When the cache is much smaller than the working set, LRU eviction becomes aggressive: entries are evicted before they can be accessed again, and the hit ratio drops significantly.

The relationship between cache size and hit ratio is not linear. There is typically a knee in the curve where a small increase in cache size produces a large improvement in hit ratio. Beyond that knee, additional memory yields diminishing returns because the cache already holds the full working set and extra space sits unused. Finding this knee through access log analysis or simulation is the most effective way to right-size your cache. A common approach is to replay production access logs against simulated caches of different sizes and plot the resulting hit ratios to find the optimal point.

Two formulas make that curve easier to reason about before you have the logs. The latency a user actually experiences is a weighted average of the two paths,

which is why a hit ratio is not a score to maximize but a term in a budget: with a 1 ms cache and a 40 ms origin, going from 90% to 95% removes 2 ms, while going from 95% to 99% removes another 1.6 ms. The first number sounds like a smaller win and is a larger one.

The shape of the knee comes from the access distribution. Real request traffic is close to Zipfian, where the $k$-th most popular item is requested with probability

with $\alpha$ typically near 1 for web workloads. That distribution is the reason a cache holding 1% of the keys can serve half the requests, and equally the reason the last few percent of hit ratio are unreachable: the tail is long, flat, and individually cold.