Introduction

kafka-console-consumer is useful for quick inspection of topic data, but it is not a full replacement for a custom consumer application. The important detail is that the console consumer can decode bytes only through the formatter and deserializer hooks it actually supports, and those hooks are more limited than many people expect. If you need to read JSON, Avro, Protobuf, or your own binary format, the right approach depends on whether you only need readable output or you need real object reconstruction.

Understand What the Console Consumer Really Does

The console consumer reads records from Kafka and prints them. By default, it treats keys and values as plain bytes and renders them through simple string decoding.

That works for topics containing UTF-8 text, but it breaks down when values are:

• custom binary payloads
• Java-serialized objects
• Avro or Protobuf messages
• JSON with extra framing or compression

The first design question is therefore: do you want a quick human-readable view, or do you need the exact domain object?

If you only need readable output, a custom formatter may be enough. If you need true business-object deserialization, a small Java consumer is often cleaner than stretching the console tool beyond its intended use.

Use Built-In Properties First

For plain text debugging, start with the built-in console properties before writing custom code.

Example:

1kafka-console-consumer \
2  --bootstrap-server localhost:9092 \
3  --topic demo \
4  --from-beginning \
5  --property print.key=true \
6  --property key.separator=' | '

This is useful when the producer already writes string keys and string values. It is also the fastest way to prove whether the topic contains readable text at all.

Custom Deserializer Versus Custom Formatter

A common source of confusion is the difference between a Kafka Deserializer and a console message formatter.

• A Kafka Deserializer converts raw bytes into Java objects for the normal consumer API.
• A console formatter decides how a consumed record is rendered to standard output.

For kafka-console-consumer, the display side is usually what matters. If your goal is to print decoded values, you often need a custom formatter or a formatter that internally uses your deserializer.

That means simply having a custom Deserializer class is not always enough by itself. The console consumer still needs code that knows how to print the resulting object meaningfully.

Example Custom Deserializer in Java

Here is a small JSON deserializer for the normal Kafka consumer API:

1import com.fasterxml.jackson.databind.ObjectMapper;
2import org.apache.kafka.common.serialization.Deserializer;
3
4import java.util.Map;
5
6public class JsonMapDeserializer implements Deserializer<Map<String, Object>> {
7    private final ObjectMapper mapper = new ObjectMapper();
8
9    @Override
10    public Map<String, Object> deserialize(String topic, byte[] data) {
11        if (data == null) {
12            return null;
13        }
14        try {
15            return mapper.readValue(data, Map.class);
16        } catch (Exception e) {
17            throw new IllegalArgumentException("Failed to deserialize JSON", e);
18        }
19    }
20}

This class is valid for a normal Java consumer. But the console consumer still needs a path that uses this logic when printing records.

A Small Consumer App Is Often the Better Tool

If your payload is custom and important, a short Java consumer is usually more reliable than trying to force everything through the console tool.

1import org.apache.kafka.clients.consumer.ConsumerConfig;
2import org.apache.kafka.clients.consumer.ConsumerRecord;
3import org.apache.kafka.clients.consumer.ConsumerRecords;
4import org.apache.kafka.clients.consumer.KafkaConsumer;
5import org.apache.kafka.common.serialization.StringDeserializer;
6
7import java.time.Duration;
8import java.util.List;
9import java.util.Properties;
10
11public class DebugConsumer {
12    public static void main(String[] args) {
13        Properties props = new Properties();
14        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
15        props.put(ConsumerConfig.GROUP_ID_CONFIG, "debug-consumer");
16        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
17        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JsonMapDeserializer.class.getName());
18        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
19
20        try (KafkaConsumer<String, Object> consumer = new KafkaConsumer<>(props)) {
21            consumer.subscribe(List.of("demo"));
22            ConsumerRecords<String, Object> records = consumer.poll(Duration.ofSeconds(5));
23            for (ConsumerRecord<String, Object> record : records) {
24                System.out.println(record.key() + " -> " + record.value());
25            }
26        }
27    }
28}

This is often the most honest answer to the original question: if you need a real custom deserializer, write a tiny consumer and control the output yourself.

When Schema Tools Change the Answer

If you use Confluent Schema Registry or another schema-aware stack, there may already be dedicated console tools for Avro, JSON Schema, or Protobuf topics. In those environments, using the schema-aware console utility is usually better than inventing your own ad hoc printing layer.

So the practical choice is:

• plain strings: use kafka-console-consumer
• schema-aware formats with vendor tooling: use the matching console utility
• truly custom payloads: write a short consumer application

Common Pitfalls

The biggest mistake is assuming kafka-console-consumer can magically turn any bytes into readable objects just because a deserializer class exists somewhere in your project.

Another mistake is confusing console formatting with Kafka consumer deserialization. Those are related, but they are not the same responsibility.

Teams also often try to debug custom binary formats without first verifying whether the payload is text, compressed data, or schema-managed data. Start with the actual wire format.

Finally, do not overinvest in console-consumer customization if a 30-line consumer program would be clearer and easier to maintain.

Summary

• 'kafka-console-consumer is best for quick inspection, not full custom-consumer behavior.'
• A Kafka Deserializer and a console formatter solve different problems.
• For custom payloads, a tiny Java consumer is often the simplest reliable debugging tool.
• Use schema-aware console utilities when your Kafka stack already provides them.
• Choose the tool based on whether you need readable output, schema-aware decoding, or true object deserialization.