Introduction

Kafka producers send keys and values as byte arrays, so any custom object must be turned into bytes before it can be published. Built-in serializers handle simple types such as strings and integers, but domain objects need either a framework serializer or one you write yourself.

Implement Kafka's Serializer interface

At the Java API level, a custom serializer is just a class that implements org.apache.kafka.common.serialization.Serializer.

Here is a simple example that serializes a UserEvent to JSON with Jackson:

1package com.example.kafka;
2
3import com.fasterxml.jackson.core.JsonProcessingException;
4import com.fasterxml.jackson.databind.ObjectMapper;
5import org.apache.kafka.common.errors.SerializationException;
6import org.apache.kafka.common.serialization.Serializer;
7
8public class UserEventSerializer implements Serializer<UserEvent> {
9    private final ObjectMapper mapper = new ObjectMapper();
10
11    @Override
12    public byte[] serialize(String topic, UserEvent data) {
13        if (data == null) {
14            return null;
15        }
16
17        try {
18            return mapper.writeValueAsBytes(data);
19        } catch (JsonProcessingException e) {
20            throw new SerializationException(
21                "Failed to serialize UserEvent for topic " + topic,
22                e
23            );
24        }
25    }
26}

And the value type:

package com.example.kafka;

public record UserEvent(long id, String username, String action) {}

Returning null for null input is normal Kafka behavior. Throwing SerializationException is important because Kafka knows how to surface that failure meaningfully.

Register the serializer in producer configuration

Once the class exists, wire it into the producer:

1package com.example.kafka;
2
3import java.util.Properties;
4import org.apache.kafka.clients.producer.KafkaProducer;
5import org.apache.kafka.clients.producer.ProducerRecord;
6
7public class ProducerApp {
8    public static void main(String[] args) {
9        Properties props = new Properties();
10        props.put("bootstrap.servers", "localhost:9092");
11        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
12        props.put("value.serializer", "com.example.kafka.UserEventSerializer");
13
14        try (KafkaProducer<String, UserEvent> producer = new KafkaProducer<>(props)) {
15            UserEvent event = new UserEvent(101L, "maria", "signed-in");
16            producer.send(new ProducerRecord<>("user-events", event.username(), event));
17            producer.flush();
18        }
19    }
20}

From Kafka's point of view, the serializer is just the adapter between your in-memory type and the bytes sent over the network.

Think about the wire format as a contract

The serializer is not only local application code. It defines a wire format that consumers must understand. That is why format choice matters.

JSON is convenient because it is easy to inspect and debug. Binary formats such as Avro or Protobuf are usually better for strict schemas and versioning. A handwritten serializer can be perfectly fine, but you should still treat its output as a long-lived interface, not an implementation detail.

That means asking a few questions early:

• how will consumers deserialize the bytes
• what happens when fields are added or renamed
• does null carry meaning
• should invalid domain objects be rejected before serialization

If you ignore those questions, the serializer may compile while quietly creating a compatibility problem for every downstream consumer.

Test the serializer by itself

A serializer is easy to unit test because the input and output are clear.

1package com.example.kafka;
2
3import java.nio.charset.StandardCharsets;
4
5public class SerializerDemo {
6    public static void main(String[] args) {
7        UserEventSerializer serializer = new UserEventSerializer();
8        byte[] bytes = serializer.serialize(
9            "user-events",
10            new UserEvent(1L, "ava", "created-account")
11        );
12
13        System.out.println(new String(bytes, StandardCharsets.UTF_8));
14    }
15}

Even if the real consumer is another service, a local smoke test like this catches obvious mistakes before Kafka is even involved.

Pair it with a matching deserializer

In most systems, a custom serializer implies a matching deserializer on the consumer side. If the producer writes JSON and the consumer expects Avro, the integration fails no matter how correct each side looks in isolation.

So whenever you add a serializer, think in pairs:

• serializer for the producer
• deserializer for the consumer
• agreed schema or format rules between them

That mindset prevents many event-stream bugs that are wrongly blamed on Kafka itself.

Common Pitfalls

The most common mistake is writing a serializer without thinking about the consumer format at all. Bytes are only useful if another side can decode them correctly.

Another common issue is swallowing serialization errors and returning bad data or empty arrays. Throw SerializationException when encoding fails.

People also forget that changing field names or format details is a compatibility change, not just a refactor.

Finally, avoid treating serialization as the place for complex business logic. Keep it focused on encoding and lightweight validation.

Summary

• A Kafka custom serializer converts your domain object into bytes for the producer.
• Implement Serializer<T> and register the class in producer properties.
• Choose a wire format deliberately because it becomes a producer-consumer contract.
• Throw SerializationException on failure instead of hiding errors.
• Always think about the matching deserializer and schema evolution story.