Apache Avro is a popular data serialization system that is often used in Apache Kafka to ensure robust schema management and data integrity. Kafka's Avro messages typically utilize schemas, making it essential to have tooling that can both deserialize and display these messages directly from a Kafka topic for debugging and development purposes.

Here's how you can achieve reading Avro messages directly from a Kafka topic without using kafka-avro-console-consumer and explore some alternative methods and tools.

Using GenericRecord and SpecificRecord with KafkaAvroDeserializer

The Apache Kafka and Confluent Platform provide a tool known as kafka-avro-console-consumer, but one can also utilize programmable approaches such as using Kafka's own consumer API together with Avro deserializers.

Setup

1. Dependencies: To begin, add dependencies for Apache Avro and the Kafka clients to your project. For Maven, the dependencies would look something like this:

1<dependency>
2    <groupId>org.apache.kafka</groupId>
3    <artifactId>kafka-clients</artifactId>
4    <version>YOUR_KAFKA_VERSION</version>
5</dependency>
6<dependency>
7    <groupId>org.apache.avro</groupId>
8    <artifactId>avro</artifactId>
9    <version>YOUR_AVRO_VERSION</version>
10</dependency>
11<dependency>
12    <groupId>io.confluent</groupId>
13    <artifactId>kafka-avro-serializer</artifactId>
14    <version>YOUR_CONFLUENT_VERSION</version>
15</dependency>

1. Kafka Consumer Configuration: Configure the Kafka consumer to use the Confluent KafkaAvroDeserializer. This requires specifying the schema registry URL:

1Properties props = new Properties();
2props.put("bootstrap.servers", "localhost:9092");
3props.put("group.id", "test-group");
4props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
5props.put("value.deserializer", "io.confluent.kafka.serializers.KafkaAvroDeserializer");
6props.put("schema.registry.url", "http://localhost:8081");

1. Reading Messages: Create a Kafka consumer, subscribe to your topic, and read messages. Here's a simple example in Java:

1KafkaConsumer<String, GenericRecord> consumer = new KafkaConsumer<>(props);
2consumer.subscribe(Arrays.asList("your_topic_name"));
3while (true) {
4    ConsumerRecords<String, GenericRecord> records = consumer.poll(Duration.ofMillis(100));
5    for (ConsumerRecord<String, GenericRecord> record : records) {
6        System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
7    }
8}

Alternative Tools and Utilities

For those looking for alternatives similar to the functionality provided by kafka-avro-console-consumer, here are some options:

• Kafkacat with Avro Support: Kafkacat is a generic non-JVM producer and consumer for Kafka topics that, when built with Avro support, can consume Avro messages using schema information from Schema Registry.

kafkacat -b mybroker -t mytopic -s avro -r http://localhost:8081

• Kafka Tool: This is a GUI application that provides capabilities for managing and visualizing Kafka topics, and it can be configured to work with Avro.

Summary Table

Conclusion

Reading Avro messages from a Kafka topic in a terminal is achievable through various means, each with its benefits and drawbacks depending on your requirements and environment. Using Kafka's native APIs provides the most flexibility, while tools like Kafkacat offer simplicity and a quick setup for non-Java environments. For those who prefer a GUI, Kafka Tool provides an excellent visual approach.

This approach not only facilitates rapid development and debugging but also ensures that you can interact with your data in Kafka in a way that best fits your operational workflows.