Introduction

Producing to different Kafka partitions from Python is straightforward once you separate two ideas: choosing a partition yourself and letting Kafka choose one from a message key. Both are valid, but they solve different problems.

Manual partition selection gives you full control. Key-based partitioning is usually better when you need all messages for the same entity to stay in order on the same partition.

Send to an Explicit Partition

Most Python Kafka clients let you pass a partition number directly. Here is an example with confluent-kafka.

1from confluent_kafka import Producer
2
3producer = Producer({"bootstrap.servers": "localhost:9092"})
4
5
6def delivery_report(err, msg):
7    if err is not None:
8        print(f"delivery failed: {err}")
9    else:
10        print(f"delivered to partition {msg.partition()} at offset {msg.offset()}")
11
12
13for i in range(6):
14    partition = i % 3
15    producer.produce(
16        topic="events",
17        value=f"message-{i}".encode("utf-8"),
18        partition=partition,
19        callback=delivery_report,
20    )
21    producer.poll(0)
22
23producer.flush()

This code sends messages in a round-robin pattern across partitions 0, 1, and 2.

Use a Key When Ordering by Entity Matters

If you want all events for the same customer, order, or device to land on the same partition, do not invent partition numbers in application code unless you really need to. Use a message key.

1from confluent_kafka import Producer
2
3producer = Producer({"bootstrap.servers": "localhost:9092"})
4
5orders = [
6    ("order-100", "created"),
7    ("order-101", "created"),
8    ("order-100", "paid"),
9]
10
11for order_id, status in orders:
12    producer.produce(
13        topic="orders",
14        key=order_id.encode("utf-8"),
15        value=status.encode("utf-8"),
16    )
17
18producer.flush()

Kafka hashes the key and chooses the partition. That keeps records with the same key together and preserves per-key ordering inside that partition.

When Manual Partitioning Makes Sense

Explicit partitions are useful when:

• you are replaying data into a known shard layout
• a downstream system expects a specific routing rule
• you are running a test and want deterministic placement
• partition ownership is based on an external algorithm

In ordinary business-event pipelines, key-based routing is usually easier to maintain because the producer does not need to know how many partitions exist or how they are balanced.

Handling Delivery Correctly

Kafka producers are asynchronous. That means produce queues the message locally first, and delivery happens in the background.

Two practical rules matter:

• call poll(0) periodically so delivery callbacks run
• call flush() before exit so buffered records are sent

If you skip both, your partition logic may look wrong when the real problem is that the process exited before the producer drained its queue.

Equivalent Idea with kafka-python

The kafka-python library also accepts a partition argument.

1from kafka import KafkaProducer
2
3producer = KafkaProducer(bootstrap_servers=["localhost:9092"])
4future = producer.send("events", value=b"hello", partition=1)
5metadata = future.get(timeout=10)
6print(metadata.partition)
7producer.close()

The API differs slightly, but the partitioning idea is the same.

Common Pitfalls

• Sending to a partition number that does not exist for the topic.
• Hard-coding partition counts in the producer when the topic may be reconfigured later.
• Using manual partition selection when a stable message key would preserve ordering with less coupling.
• Forgetting poll() and flush() in asynchronous producer code.
• Assuming partition assignment is a load-balancing feature only, when it also affects ordering guarantees for consumers.

Summary

• Python Kafka clients can send directly to a chosen partition by passing partition=.
• A message key is usually the better choice when related events must stay ordered together.
• Manual partitioning is appropriate when routing rules are external or deterministic.
• Producer calls are asynchronous, so poll() and flush() matter.
• Good partition strategy is about both load distribution and ordering semantics.