Introduction

An LRU cache is easy to sketch in an interview and much harder to use safely in production. The core eviction rule is simple, but real systems also care about concurrency, memory accounting, observability, expiration, warm-up behavior, and whether a custom cache is even justified.

Start by Asking Whether You Should Build One

In production code, the first best practice is to avoid implementing your own cache unless there is a clear reason. Mature libraries already solve edge cases that are easy to miss.

Typical examples are:

• Caffeine on the JVM
• 'functools.lru_cache for simple Python function memoization'
• 'cachetools in Python for configurable in-process caches'
• battle-tested distributed caches such as Redis when data must be shared across processes

A custom LRU cache is most defensible when:

• the cache is tiny and local to one component
• the eviction policy needs custom behavior
• dependency constraints rule out a library

If you do build one, keep the scope narrow.

The Standard Data Structure Pair

An efficient LRU cache usually combines:

• a hash map from key to node
• a doubly linked list ordered by recency

The map gives O(1) lookup. The list gives O(1) moves to the front on access and O(1) eviction from the tail.

Python's OrderedDict already packages most of this behavior, which makes it a good example for a small production-oriented implementation.

1from collections import OrderedDict
2from threading import RLock
3
4
5class LRUCache:
6    def __init__(self, capacity):
7        if capacity <= 0:
8            raise ValueError("capacity must be positive")
9        self.capacity = capacity
10        self.data = OrderedDict()
11        self.lock = RLock()
12
13    def get(self, key):
14        with self.lock:
15            if key not in self.data:
16                return None
17            self.data.move_to_end(key)
18            return self.data[key]
19
20    def put(self, key, value):
21        with self.lock:
22            if key in self.data:
23                self.data.move_to_end(key)
24            self.data[key] = value
25            if len(self.data) > self.capacity:
26                self.data.popitem(last=False)
27
28    def __len__(self):
29        with self.lock:
30            return len(self.data)

This code is small enough to audit and already handles the two essential operations correctly.

Production Requirements Usually Extend Beyond Pure LRU

In real systems, “LRU” is often only one part of the policy. You may also need:

• TTL or max age
• size-based eviction instead of item-count eviction
• metrics such as hit rate and eviction count
• loader behavior for cache misses
• protection against stampedes when many requests miss at once

That is why production teams often adopt a library even when they fully understand the basic algorithm. The surrounding behavior is where most complexity lives.

Concurrency Is Not Optional

A cache used by multiple threads must define its concurrency behavior. Without locking or a concurrent implementation, a cache can corrupt its recency order or even crash during simultaneous reads and writes.

The RLock in the example above is the minimal answer for thread safety in a single Python process. In other languages, you might use synchronized blocks, concurrent maps, or lock striping.

The point is not that every cache needs maximal concurrency sophistication. The point is that thread safety cannot be an afterthought.

Count Size Carefully

Using “number of entries” as the capacity is often too crude. One cached item may be a short string, while another is a multi-megabyte object.

In production, it is common to evict based on weighted size instead:

• bytes
• estimated object size
• number of rows or records
• total serialized payload

A cache that holds “100 items” may still blow the process memory budget if item sizes vary heavily.

Instrument the Cache

A production cache without metrics is guesswork. At minimum, track:

• hits
• misses
• evictions
• current size
• load failures if the cache populates on demand

Here is a simple extension of the earlier cache that tracks hit and miss counts.

1class InstrumentedLRUCache(LRUCache):
2    def __init__(self, capacity):
3        super().__init__(capacity)
4        self.hits = 0
5        self.misses = 0
6
7    def get(self, key):
8        value = super().get(key)
9        if value is None:
10            self.misses += 1
11        else:
12            self.hits += 1
13        return value

The example is intentionally simple, but the idea matters: if you cannot measure cache behavior, you cannot tune capacity or even prove the cache helps.

Understand the Failure Mode

A cache should be an optimization, not a hidden source of correctness bugs. That means defining what happens when:

• the cache is empty
• the loader fails
• an item expires unexpectedly
• the cache is disabled for debugging

Production code should still behave correctly when the cache is cold or completely bypassed.

Common Pitfalls

A common mistake is writing a perfect interview-style LRU and shipping it without concurrency protection.

Another mistake is caching unbounded values and assuming the entry count alone protects memory. It does not.

Teams also forget observability. A cache with no hit-rate metrics often survives for months without anyone knowing whether it improves latency or makes memory pressure worse.

Finally, avoid inventing a custom cache when a mature library already fits the use case. Production code should optimize for reliability, not novelty.

Summary

• In production, the first LRU decision is often whether to use a library instead of building one.
• A correct in-memory LRU combines a map with recency ordering.
• Thread safety, memory accounting, and metrics matter as much as the eviction rule.
• Real systems often need TTL, weighted size limits, and miss-handling behavior in addition to LRU.
• Treat the cache as an optimization layer that can fail or be bypassed without breaking correctness.