Hot Partition Mitigation: When One Key Brings the Cluster to Its Knees

A hot partition is what happens when a partition key concentrates traffic onto one physical shard. Cassandra calls it a hot row, DynamoDB calls it a throttled partition, Postgres simply melts the underlying disk. The cause is the same: one key on one node handles more work than the rest of the cluster combined.

There are four mitigations worth knowing, and they apply to persistent storage shards. This is a different problem from the in-memory hot-key case covered in hot-key-mitigation-in-distributed-cache, where the answer is usually request coalescing and a tiny local cache. Persistent shards cannot coalesce because writes have to land.

Salt the key. Append a small random suffix to spread writes across N synthetic partitions. Reads now have to fan out across all N suffixes and merge, which is acceptable for write-heavy logs and painful for point lookups.

Pre-split the partition. If you know in advance that one tenant is going to be loud, allocate dedicated shards for it before traffic arrives. This is how DynamoDB's adaptive capacity is supposed to work, and it is what Bigtable's pre-split feature does explicitly.

Time-bucket the key. Append a coarse time component so the hot key naturally rolls forward: user:123:2026-W18 instead of user:123. Each week opens a fresh partition. Old buckets cool down on their own.

Front the shard with a cache tier. Push hot reads into Redis with a short TTL. This only helps read-skewed hot keys. Write-skewed hot keys still hammer the shard.

The production failure I watched: a notifications service stored a per-user activity log with (user_id) as the partition key. One creator on the platform gained 2 million followers in a single week after a viral clip. Every follower interaction wrote into that one partition. The shard hit 100 percent CPU while the cluster averaged 8 percent. Write latency on that shard climbed from 4 ms to 9 seconds, and because the shard was also responsible for unrelated users assigned by hash collision, those users saw write timeouts too.

The fix was to change the key to (user_id, day_bucket) so each day spawned a new partition for the hot creator. Chronological reads ran a background merge across the day buckets, and the read latency cost was a single-digit percent. The cluster CPU evened out within an hour of the schema migration.

The lesson: if one key gets disproportionate traffic, the cluster is not your scaling unit. The partition is. Adding nodes does nothing until you split the key.