Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. Kafka Console Producer is a built-in command-line tool that comes with Kafka, allowing you to send messages directly to a Kafka topic. This tool is primarily used for debugging and testing.

Kafka Console Producer

Producing messages into Kafka involves writing messages to a particular topic from which they can be consumed by Kafka Consumers. kafka-console-producer.sh is a useful script bundled with Kafka that facilitates this from the command-line interface.

Basic Usage

To run the Kafka Console Producer, use the following syntax:

kafka-console-producer.sh --broker-list <brokers> --topic <topic_name>

• --broker-list - This is a list of one or more Kafka brokers to which the producer will connect (formatted as host1:port1,host2:port2,…).
• --topic - The target Kafka topic to which the data will be produced.

For example, to send messages to a topic named "test" on the local Kafka instance running on the default port:

kafka-console-producer.sh --broker-list localhost:9092 --topic test

After executing the command, you can type your messages into the console. Each newline denotes a separate message.

Advanced Options

Some advanced options include:

• --producer.config - Path to a properties file containing Kafka producer properties.
• --property - Use this to set specific properties in the format key=value.
• --compression-codec - This specifies the compression codec to be used (none, gzip, snappy, lz4, zstd).

Example with Properties

kafka-console-producer.sh --broker-list localhost:9092 --topic test --property parse.key=true --property key.separator=,

This sets the key-value separator as a comma, so messages typed at the prompt can include keys like key1,value1.

Integrating with Bash Scripts

Kafka Console Producer can be integrated into Bash scripts for automation of message sending tasks. Here is an example where messages are composed within the script and sent to a Kafka topic:

1#!/bin/bash
2
3# Kafka broker and topic configuration
4BROKER_LIST="localhost:9092"
5TOPIC="test_topic"
6
7# Message to be sent to Kafka
8MESSAGE="Hello from bash script"
9
10echo "$MESSAGE" | kafka-console-producer.sh --broker-list $BROKER_LIST --topic $TOPIC

This script sends a single message to the Kafka topic. For more elaborate purposes, the script could generate or read messages from files, databases, or other sources.

Handling Data Streams

For dynamic input from another process or a stream, you can pipe data into the Kafka Console Producer. For instance:

tail -f /path/to/logfile | kafka-console-producer.sh --broker-list localhost:9092 --topic logs

This command tails a log file, sending new entries to the "logs" Kafka topic in real-time.

Summary Table

Conclusion

Kafka-console-producer is a powerful tool that provides great flexibility and ease of testing within Kafka environments. By integrating it with Bash scripts, its potential is significantly expanded, allowing for automated and dynamic message production workflows. This versatility makes it an essential tool in the arsenal of developers working with Kafka.