The partition a record lands in determines its ordering guarantee, which consumer processes it, and which broker's disk it occupies. The default partitioner makes that choice in three different ways depending on what the record carries, and the three behave nothing alike.

The three paths through DefaultPartitioner

If the record specifies a partition explicitly, that partition is used and no partitioner runs. This is the escape hatch for applications that compute placement themselves, and it bypasses every behaviour below.

If the record has a non-null key, the partition is murmur2(keyBytes) % numPartitions, taken as a positive value. The hash is over the serialized key bytes, not the key object, which is the source of several surprises later in this lesson.

If the key is null, the partitioner is free to choose, and what it chooses changed in Kafka 2.4 and again in 3.3. Before 2.4 it was round-robin per record. From 2.4 it was the sticky partitioner. From 3.3 the default partitioner is the uniform sticky one, and partitioner.class left unset selects it.

So "Kafka hashes the key" is accurate for keyed records and says nothing about the majority of high-volume topics, which are unkeyed.

Why murmur2 and why it is frozen

The hash is Kafka's own implementation of murmur2, not Java's hashCode, and the choice of algorithm is less important than the fact that it cannot change.

Every producer writing to a topic must compute the same partition for the same key, or records with one key split across partitions and the ordering guarantee evaporates. So the hash is part of Kafka's wire compatibility contract in the same way the record format is, and clients in every language reimplement this specific murmur2 to match.

Which has a practical consequence: a non-Java client with a subtly different hash produces a topic where some keys appear in two partitions, and the symptom is out-of-order processing for a minority of keys with no error anywhere. When debugging that, computing the expected partition by hand for a sample key is the check that finds it.

It also means hashCode intuitions do not apply. Two keys that a Java program considers equal hash identically only because their serialized bytes are identical, which is the next subsection's problem.

Hashing serialized bytes, and the bugs that follow

The partitioner sees bytes, so the serializer, not the key type, determines the partition. Three failures come from forgetting that.

A JSON-serialized key whose field order is not deterministic produces different bytes for the same logical key, so the same customer lands in different partitions depending on how the map was iterated. This is the commonest version and it is invisible until someone notices ordering violations.

An Avro or protobuf key carrying a schema id or a timestamp hashes that too, so logically identical keys differ. A key that includes anything varying, a version, a producer hostname, a serialization timestamp, is not a stable key.

And a string key with inconsistent case or whitespace, "ACME" against "acme ", is two keys, which is obvious when stated and routine in practice.

The rule that avoids all three: keys should be a canonical byte representation of an identifier and nothing else, which in practice means a plain StringSerializer over a normalised identifier string, and never a serialized object.

The partition count is in the formula

murmur2(key) % numPartitions depends on the partition count, which means adding partitions re-maps every key.

A topic with 12 partitions grown to 16 moves roughly three quarters of all keys to a different partition. Records for one key now exist in two partitions, the old one holding history and the new one receiving writes, and their relative order is undefined because no consumer reads both in sequence.

For a topic where key ordering matters, this is a breaking change dressed as a scaling operation. For a stateful stream processor keyed on the same field, it is worse: the state for a key sits in the task that owned the old partition, and new records arrive at a different task, so the state is silently wrong rather than missing.

Which is why partition count for a keyed topic is a one-time decision made with headroom, and why the remedy when it must change is a new topic with a migration rather than an --alter. Partitions also cannot be removed, so the headroom is deliberately generous and the cost of generosity is the per-partition overhead the operating lessons quantify.

What ordering the key actually buys

Worth stating precisely, because it is both stronger and narrower than people assume.

Records with the same key go to the same partition, and a partition is a totally ordered log read by one consumer in a group, so records with the same key are processed in the order they were produced. That is a real and strong guarantee, and it is the reason keys exist.

It says nothing across keys. Two records with different keys have no ordering relationship, even if produced microseconds apart from the same thread, because they may be in different partitions consumed at different rates.

It also depends on the producer not reordering, which is the previous lesson's subject: without idempotence and with more than one request in flight, a retry can reorder two records for the same key within the partition. So key-based ordering requires enable.idempotence, which is why its default being true matters more than it appears.

And it holds only while the partition count is fixed, per the previous subsection.

Computing the partition by hand

Being able to work out where a key will land, without a cluster, is the check that resolves most placement arguments, and it takes two lines.

Kafka's murmur2 is the 32-bit variant with seed 0x9747b28c, and the partitioner takes Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions, where toPositive is a bitwise and with 0x7fffffff rather than an absolute value. That distinction matters: masking the sign bit is not the same as negating, so an implementation that uses Math.abs produces a different partition for roughly half of all keys.

The practical uses are three. Verifying that a non-Java producer agrees with the Java client for a sample of keys, which is the check that finds the cross-client hash mismatch. Simulating the distribution of a candidate key over a candidate partition count, which is the measurement the design subsections call for. And locating a specific entity's records for debugging, so that a consumer can be pointed at one partition rather than the whole topic.

The last of these is worth having in a runbook: given a customer id and a partition count, knowing the partition turns "search the topic" into "read one partition from an offset", which at high volume is the difference between a minute and an hour.

Headers, and why they are not keys

Record headers carry metadata and have no effect on partitioning, which is a question that comes up often enough to answer directly.

A header is a key-value pair of bytes attached to a record, added in 0.11 alongside the record format changes, and the partitioner's interface does not receive them. Partitioner.partition() is given the topic, the key object, the serialized key bytes, the value object, the serialized value bytes and the cluster metadata. So a custom partitioner can route on the value but not on a header.

Which means headers are the right place for anything that should travel with a record without affecting placement: a trace id, a schema version, a source system, a retry count. Putting any of those in the key would both change the partition and break the key's stability.

The common mistake in the other direction is putting routing information in a header and then discovering that routing has to happen in a custom partitioner that cannot see it. The fix is to pass the routing field as part of the value, or to compute the partition explicitly at the call site, which the next section covers.