Introduction

In kafka-python, bytes are already the native wire format. That means if your keys and values are already bytes or bytearray, you often do not need any special serializer or deserializer at all.

The Default Behavior

Kafka transmits raw bytes over the network. kafka-python reflects that design:

• the producer can send raw bytes directly
• the consumer receives raw bytes by default

So the simplest byte-array setup is actually no custom serializer.

1from kafka import KafkaProducer
2
3producer = KafkaProducer(bootstrap_servers="localhost:9092")
4producer.send("events", key=b"user-1", value=b"raw-payload")
5producer.flush()

On the consumer side:

1from kafka import KafkaConsumer
2
3consumer = KafkaConsumer(
4    "events",
5    bootstrap_servers="localhost:9092",
6    auto_offset_reset="earliest"
7)
8
9for message in consumer:
10    print(type(message.key), message.key)
11    print(type(message.value), message.value)
12    break

The key and value arrive as bytes unless you configured a deserializer.

When To Pass An Explicit Serializer Anyway

Sometimes teams want the configuration to document intent explicitly. In that case, you can use an identity serializer.

1from kafka import KafkaProducer
2
3producer = KafkaProducer(
4    bootstrap_servers="localhost:9092",
5    key_serializer=lambda k: bytes(k),
6    value_serializer=lambda v: bytes(v),
7)
8
9producer.send("events", key=bytearray(b"k1"), value=bytearray(b"hello"))
10producer.flush()

This is useful when upstream code might hand you bytearray objects and you want to normalize everything to immutable bytes before sending.

Explicit Byte Deserialization

The consumer side usually needs no deserializer if you want raw bytes. But again, you can make that explicit:

1from kafka import KafkaConsumer
2
3consumer = KafkaConsumer(
4    "events",
5    bootstrap_servers="localhost:9092",
6    key_deserializer=lambda k: k,
7    value_deserializer=lambda v: v,
8)

That does not transform the payload. It just makes the contract obvious to readers.

When You Actually Need Serialization

If your application works with strings, JSON, or domain objects, then you do need a real serializer layer.

For JSON:

1import json
2from kafka import KafkaProducer, KafkaConsumer
3
4producer = KafkaProducer(
5    bootstrap_servers="localhost:9092",
6    value_serializer=lambda v: json.dumps(v).encode("utf-8"),
7)
8producer.send("events", value={"kind": "created", "id": 42})
9producer.flush()
10
11consumer = KafkaConsumer(
12    "events",
13    bootstrap_servers="localhost:9092",
14    auto_offset_reset="earliest",
15    value_deserializer=lambda v: json.loads(v.decode("utf-8")),
16)
17
18for message in consumer:
19    print(message.value)
20    break

This is not a byte-array serializer anymore. It is application-level encoding into bytes.

Producer And Consumer Must Agree

The important rule is not "always specify a serializer." The real rule is that producer and consumer must agree on the byte contract.

Examples:

• raw bytes on the producer means raw bytes on the consumer
• UTF-8 strings on the producer require decoding on the consumer
• JSON encoding on the producer requires JSON decoding on the consumer

If the two sides disagree, the bug usually appears as a decoding error or silently wrong interpretation of the payload.

bytes Versus bytearray

Kafka libraries conceptually send bytes. Python code may use either bytes or bytearray, but normalizing to bytes is often cleaner because it is immutable and works naturally with serializers and hashing.

That is why lambda v: bytes(v) is a useful explicit adapter when your data source emits byte arrays in varying forms.

Common Pitfalls

The most common mistake is thinking you need a special serializer just to send bytes. In kafka-python, raw bytes are already the expected payload type.

Another mistake is adding a string serializer and then forgetting to decode on the consumer, or vice versa.

Developers also sometimes use str(value).encode("utf-8") on arbitrary objects. That is usually a poor serialization format because it is not structured or stable.

Finally, be explicit about whether the application contract is raw bytes, UTF-8 text, JSON, or something else. Kafka only stores bytes; the meaning above that layer is your responsibility.

Summary

• For raw byte payloads in kafka-python, you usually do not need a custom serializer or deserializer.
• The producer can send bytes directly, and the consumer receives bytes by default.
• Use identity lambdas only if you want the contract to be explicit.
• Add real serializers only when you are encoding strings, JSON, or objects into bytes.
• Producer and consumer must agree on the payload format above Kafka's raw byte layer.