Kafka 0.8.2.0
Default serializer
Data serialization
Apache Kafka
Kafka configuration

Default serializer for kafka 0.8.2.0

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

Kafka 0.8.2.0 is confusing here because it exposed both the legacy producer API and the then-new Java producer API. The answer to "what is the default serializer" depends on which producer you are using, so treating the version number alone as the whole story leads to the wrong configuration.

Legacy Producer Versus New Java Producer

In the older producer configuration, the important property was serializer.class. In Kafka 0.8.2 documentation for that legacy producer, the default value serializer is kafka.serializer.DefaultEncoder, which simply passes byte[] through unchanged. The key serializer defaults to the same value unless you set key.serializer.class.

That means the legacy producer has a default, but it is not a string serializer. It is a byte-array pass-through serializer, which is only useful if your code already produces byte[].

The new Java producer introduced around Kafka 0.8.2 is different. For that producer, key.serializer and value.serializer do not have useful implicit defaults for normal application code. You are expected to choose serializers explicitly.

Configure The New Producer Explicitly

If you are using org.apache.kafka.clients.producer.KafkaProducer, set both serializers yourself. For string keys and values:

java
1import java.util.Properties;
2import org.apache.kafka.clients.producer.KafkaProducer;
3import org.apache.kafka.clients.producer.ProducerRecord;
4
5public class App {
6    public static void main(String[] args) {
7        Properties props = new Properties();
8        props.put("bootstrap.servers", "localhost:9092");
9        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
10        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
11
12        KafkaProducer<String, String> producer = new KafkaProducer<>(props);
13        producer.send(new ProducerRecord<>("events", "user-1", "hello"));
14        producer.close();
15    }
16}

If your application already has byte[], use ByteArraySerializer instead. The important point is that the new producer wants an intentional serializer choice rather than relying on an old default.

The Legacy Producer Default

If you are using the older producer API, the configuration looks different:

java
1import java.util.Properties;
2import kafka.javaapi.producer.Producer;
3import kafka.producer.KeyedMessage;
4import kafka.producer.ProducerConfig;
5
6public class LegacyProducerExample {
7    public static void main(String[] args) {
8        Properties props = new Properties();
9        props.put("metadata.broker.list", "localhost:9092");
10        props.put("serializer.class", "kafka.serializer.StringEncoder");
11
12        Producer<String, String> producer = new Producer<>(new ProducerConfig(props));
13        producer.send(new KeyedMessage<>("events", "user-1", "hello"));
14        producer.close();
15    }
16}

Even though the legacy producer has a default DefaultEncoder, most real applications override it with something intentional such as StringEncoder or a custom encoder.

When You Need A Custom Serializer

If the message value is neither a string nor raw bytes, define your own serialization logic. In the legacy producer this meant implementing Encoder<T>. In the newer producer API it means implementing the newer Serializer<T> interface.

For example, a basic custom serializer in the newer client API could look like this:

java
1import java.nio.charset.StandardCharsets;
2import java.util.Map;
3import org.apache.kafka.common.serialization.Serializer;
4
5public class UserEventSerializer implements Serializer<UserEvent> {
6    @Override
7    public void configure(Map<String, ?> configs, boolean isKey) {
8    }
9
10    @Override
11    public byte[] serialize(String topic, UserEvent data) {
12        if (data == null) {
13            return null;
14        }
15        String payload = data.userId() + "|" + data.action();
16        return payload.getBytes(StandardCharsets.UTF_8);
17    }
18
19    @Override
20    public void close() {
21    }
22}

The point is not that this format is ideal. The point is that Kafka always transports bytes, so your application must decide how domain objects become bytes.

Common Pitfalls

  • Assuming StringEncoder is the universal default in Kafka 0.8.2.0.
  • Mixing up the legacy producer config keys with the new Java producer config keys.
  • Forgetting to set both key.serializer and value.serializer when using the new producer API.
  • Relying on byte-array defaults when the application is actually sending strings or objects.
  • Treating the version number as sufficient context without checking which producer client your code imports.

Summary

  • In Kafka 0.8.2.0, the serializer answer depends on whether you mean the legacy producer or the newer Java producer.
  • The legacy producer documents kafka.serializer.DefaultEncoder as the default message serializer.
  • The newer Java producer should be configured explicitly with serializers such as StringSerializer or ByteArraySerializer.
  • If the payload is a domain object, provide a custom serializer instead of guessing.

Course illustration
Course illustration

All Rights Reserved.