ElastiCache and Caching Tiers

Topics Covered

What a Cache Buys

Latency and load

Hit ratio arithmetic

What is worth caching

The cost of a cache

Caching as capacity planning

Where caches sit

A cache keeps copies of data somewhere faster or cheaper to read than its source. On AWS that somewhere might be an ElastiCache cluster a millisecond away, a CloudFront edge near the user, a DAX cluster in front of DynamoDB, or memory inside the application process. Every cache trades freshness and complexity for speed and reduced load on the source, and the decision to add one should start from what it buys and what it costs rather than from the assumption that caching is always good.

Latency and load

A cache helps in two distinct ways. It cuts latency, because reading a value from memory takes well under a millisecond while computing it from a database query, an API call or a rendering step might take tens or hundreds. And it cuts load on the source, because every hit is a request the database or downstream service did not have to serve. Often the second benefit matters more: a database that would need an instance three times larger to serve peak reads can stay small when most reads are absorbed by a cache. Knowing which benefit is the goal decides how the cache is judged.

Tail latency is often where a cache earns most. A database whose median query takes eight milliseconds may have a ninety ninth percentile of two hundred, because of lock waits, cold pages or a busy neighbour. A cache hit has a much tighter distribution, so moving most reads to the cache flattens the tail that users actually notice. When evaluating a cache, compare percentiles before and after rather than averages, and remember that the misses still carry the source's tail, so a page that makes ten cached calls will still meet the slow path on some requests.

Hit ratio arithmetic

The value of a cache depends on its hit ratio, the fraction of reads it serves. Average latency is roughly the hit ratio times cache latency plus the miss ratio times source latency plus cache latency. With a one millisecond cache, a fifty millisecond database and a ninety percent hit ratio, average latency is about six milliseconds. At ninety nine percent it is about one and a half. Load on the source is the miss ratio times total reads, so going from ninety to ninety nine percent cuts database reads tenfold. Small changes in hit ratio at the high end have large effects, which is why cache sizing and key design matter.

Going from 90 to 99 percent hits cut database reads tenfold. The last few points of hit ratio were worth more than the first fifty.

The arithmetic also says where to stop. Each extra point of hit ratio near the top usually needs a disproportionately larger cache, because the remaining misses are for keys read rarely, the long tail of the access distribution. A cache sized to hold the hottest ten percent of keys might reach ninety percent hits, while reaching ninety nine might need half of all keys in memory. Plotting hit ratio against cache size from a sample of real traffic, which tools can simulate from access logs, shows where the curve flattens and further memory stops paying for itself.

What is worth caching

Good candidates are read far more often than written, expensive to compute or fetch, and tolerant of being slightly stale: product catalogue entries, user profiles, configuration, rendered page fragments, results of expensive aggregate queries, session data, and responses from slow third party APIs. Poor candidates change on almost every read, must always be exactly current, such as an account balance used to authorise a payment, or are read once and never again. Measuring access frequency per key before caching, from logs or database statistics, often shows that a small fraction of keys gets most reads, which a modest cache can serve.

Shared versus per user data matters too. A value read by every visitor, such as a category page, gets a high hit ratio from a small cache. A value that only one user ever reads, such as their own recommendations, gets a hit only when that user returns within the TTL, so its hit ratio depends on session behaviour and the cache must hold an entry per active user. Per user caching can still pay off for expensive computations, but its sizing follows active users rather than catalogue size, and it raises privacy questions that shared caching does not.

The cost of a cache

A cache is another system to run, monitor, secure and pay for. It introduces stale reads, so every cached value needs an answer to how stale it may be and how it is refreshed. It introduces new failure modes, such as a cold cache after restart or a stampede of misses when a hot key expires. It can hide database problems until the cache fails, at which point the database meets traffic it was never sized for. And it adds code paths where bugs live, especially around invalidation. A cache that buys five milliseconds on a request nobody waits for may not be worth these costs.

There is a subtler cost in correctness reasoning. Once a value can come from a cache, every reader must tolerate the staleness that cache allows, and code that reads a value, decides something, then writes based on it can make decisions on old data. Teams should name which reads may come from the cache and which must go to the source, and enforce it in the data access layer, rather than letting every developer decide case by case. A short written policy per entity, stating the maximum staleness and the invalidation path, prevents most surprises.

Caching as capacity planning

Once a cache absorbs most reads, the system's capacity depends on it. If the cache fails or is flushed, the source sees the full read load, which can be ten or a hundred times its normal traffic, and may fall over. Teams should know what the source can handle with the cache gone, and either size for it, add protection such as request coalescing and rate limiting, or treat the cache as critical infrastructure with replicas and failover. A cache that the system cannot survive losing is no longer an optimisation but a tier with its own availability requirements.

A useful exercise is a cache off test in a staging or load test environment: disable the cache and drive production level traffic at the source. The result shows the real dependency, often surprising, and gives numbers for the protections. If the database saturates at forty percent of normal traffic, the system needs a coalescing layer, a rate limit at roughly that level, and a degraded mode for the rest. Repeating the test after major changes keeps the answer current, since traffic growth quietly turns a cache that was optional into one that is essential.

Where caches sit

Caches can sit at several layers, and the closer to the user, the more load they remove from everything behind them. A browser cache avoids the request entirely. A CDN serves from an edge near the user. An API gateway can cache responses. The application can cache in process, in a shared cache such as ElastiCache, or in front of a specific database with a purpose built cache such as DAX. Each layer has a different scope of sharing, a different invalidation story and a different cost, and many systems use two or three together.

Layering has a cost. Each layer adds its own TTL to the staleness a user can see, so a value cached for five minutes in the application and five minutes at CloudFront can be ten minutes old. Invalidating one layer leaves the others serving the old value. Designs should add up the TTLs along the path to find the worst case staleness, and invalidate layers in order from the source outward, so an outer layer never refills from an inner layer still holding stale data.

Five minutes in the application cache plus five at the edge meant a user could see a price ten minutes old. Staleness adds up along the path.
A cache is a tier, not a trick

Caches cut latency and offload sources, and at high hit ratios the source depends on them. Decide what may be stale and for how long, know what the source can take if the cache disappears, and judge the cache by the hit ratio and load it removes.