Kafka Partitioning and Ordering: Picking the Right Key

Partitioning in Kafka decides three things at once, and the decision is mostly irreversible once a topic has data in it. Engineers usually frame it as a scaling question. It is also an ordering question and a hot-spot question, and the same key has to satisfy all three.

Parallelism comes from partitions. One partition feeds at most one consumer in a group, so a topic with 12 partitions caps consumer parallelism at 12. Adding consumers past that does nothing. Adding partitions later means existing keys rehash to new partitions, which breaks any guarantee that the same key always lands on the same place.

Ordering only exists within a partition. Kafka has no global order across a topic. If you need to process all events for user_id=42 in the order they arrived, those events must land in the same partition, which means user_id is your key. If you need order across users, you need one partition, and you have given up parallelism entirely.

Hot keys are where this bites. Hash partitioning is uniform only if your keys are uniform. They almost never are.

A team I worked with partitioned an orders topic by customer_id. Looked fine in staging. In production, one Fortune 500 customer drove roughly 30% of total volume. Their customer_id hashed to a single partition, that partition lagged constantly behind the others, and the consumer group's effective throughput was bounded by that one consumer no matter how many pods they ran. Autoscaling did nothing. The bottleneck was the partition, not the worker.

There are two clean fixes and you usually need both.

For ordering within a tenant, change the key to a composite like (customer_id, sub_id) where sub_id is order_id or some other entropy source. You lose strict ordering across all of customer_id's events, but you keep ordering at the sub_id level, which is what the business usually actually needs.

For the largest tenants, give them their own topic. Top three customers each get a dedicated orders topic with their own partition count tuned to their volume. Everyone else stays on the shared topic. This is ugly but it is the only thing that fully decouples a whale from the rest of the fleet.

One more thing on partition count. Too few and you cap parallelism, as covered. Too many and you pay a different price: the controller tracks metadata for every partition in the cluster, rebalances move more state per event, and broker open file handles climb. A topic with 10,000 partitions and three replicas is 30,000 segment files per broker before you write a record. Aim for the smallest partition count that gives you the parallelism you need with a little headroom.

Pick the key first. Pick the partition count second. Test with production-shaped traffic, not synthetic load with uniform keys.