Apache Kafka is a popular distributed streaming platform that enables building real-time data pipelines and streaming applications. Kafka is often used to process and analyze streaming data in various formats, including JSON, which is a common data format for messaging due to its simplicity and ease of integration with web applications and services.

Understanding Kafka Console Consumer

The Kafka Console Consumer is a command-line tool that comes with the Apache Kafka distribution, used for consuming messages from a Kafka topic. While the console consumer is primarily used for debugging and simple output viewing, it has limitations in handling complex structures like JSON directly in a human-readable format.

Why Pretty Print JSON in Kafka Console Consumer?

JSON messages consumed from a Kafka topic are often minified: all unnecessary spaces and line breaks are removed to reduce message size for performance optimization. This can make JSON difficult to read or debug directly from the console. Pretty printing is a technique where JSON data is reformatted with proper indentations and new lines, making it much easier to read and understand.

Steps to Pretty Print JSON in Kafka Console Consumer

Here’s how you can pretty print JSON when consuming messages from a Kafka topic:

1. Consume Messages from Kafka Topic Start by consuming messages from your specific Kafka topic using the Kafka console consumer tool:

   kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic your_topic_name --from-beginning

1. Use jq for Pretty Printing jq is a lightweight and flexible command-line JSON processor that can pretty print JSON data. You can pipe the output of the Kafka Console Consumer into jq to achieve pretty printing:

   kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic your_topic_name --from-beginning | jq .

• Here, jq . takes each string of JSON and formats it, adding necessary indentations and line breaks.

1. Handling Complex JSON Structures If your JSON messages contain nested structures, jq can still handle them effectively. You can specify filters and other transformations to selectively pretty print parts of your JSON messages:

   kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic your_topic_name --from-beginning | jq '.details'

Advanced Usage with jq

jq offers vast functionality beyond simple pretty printing, including:

• Slice Elements: You can process only a certain portion of JSON arrays.
• Filters: Apply complex filters to include or exclude specific parts based on conditions.
• Map Values: Transform JSON data by applying functions to values.

Here is a more complex command illustrating these capabilities:

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic your_topic_name --from-beginning | jq '{user: .user, details: {address: .details.address, phone: .details.phone}}'

Summary Table for Key Options and Commands

Further Enhancements

For automated data processing or integration into richer applications, consider:

• Writing a custom consumer using Kafka's Consumer API.
• Integrating with stream processing frameworks like Kafka Streams or Apache Flink.

In summary, pretty printing JSON in Kafka Console Consumer requires the auxiliary use of tools like jq, but significantly enhances the readability and usability of stream data for development and debugging purposes.