Apache Kafka is a distributed streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since version 0.11, Kafka has introduced several enhancements, one of which is the support for message headers in Kafka messages.

Understanding Kafka Message Headers

Message headers in Kafka are key-value pairs that are used to store metadata about the message. These headers are similar to headers in HTTP or AMQP messages and allow producers to send additional information that can be consumed or inspected downstream.

Headers provide a flexible and structured way to manage metadata such as tracing IDs, content encoding details, or even message routing hints, without cluttering the message payload.

Producing Messages with Headers Using the Console Producer

Kafka 0.11 ships with an updated console producer that allows users to specify headers from the command line. This is useful for testing and demonstrations. Detailed below are the steps to produce messages with headers using the console producer:

Step 1: Start the Kafka Environment

Ensure that your Kafka server is up and running. If you’re running a local setup, you typically start the Zookeeper server and then the Kafka server:

1# Start Zookeeper
2bin/zookeeper-server-start.sh config/zookeeper.properties
3
4# Start Kafka
5bin/kafka-server-start.sh config/server.properties

Step 2: Use the Console Producer with Headers

To produce messages with headers, you utilize the kafka-console-producer utility. The command below demonstrates how to send a message with headers:

1bin/kafka-console-producer.sh --broker-list localhost:9092 --topic ExampleTopic \
2--property "parse.key=true" \
3--property "key.separator=:" \
4--property "headers=headerKey1:headerValue1,headerKey2:headerValue2"

Here, you specify the broker list and the topic you’re producing to. The --property flags are used to specify the message key parsing, key separator, and headers. Note that headers are comma-separated.

Enter Your Message

After executing the above command, the console waits for input. Enter your message in the format key:value. For example:

myKey:myValue

Press Ctrl+D to send the message.

Step 3: Validating Message Reception with Headers

To verify that your message has been sent with headers, consume the messages from the topic using the updated console consumer:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic ExampleTopic --from-beginning \
--property print.headers=true

This command starts the consumer, and you should see output similar to:

headerKey1=headerValue1,headerKey2=headerValue2	myKey	myValue

Tips for Using Headers

Here are a few tips when working with headers in Kafka messages:

• Remember that headers can add to the overall message size and could impact performance if excessively used.
• Headers are not indexed, and thus searching or filtering by headers is not directly supported at the Kafka broker level.

Summary Table

Here’s a summary of key commands and properties used in this tutorial:

Conclusion

Producing messages with headers in Kafka 0.11 using the console producer is straightforward once you are familiar with the appropriate command-line options. Headers are particularly useful for passing metadata, which can be leveraged in various ways by consumers or during message processing.