Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. An integral part of Kafka's ecosystem is its administrative client interface, which helps in managing and inspecting topics, brokers, and other configurations within a Kafka cluster. The AdminClient API, which can be interacted with via the command line using kafka-admin-client command, allows for a diverse range of administrative operations. One such command-line option is --command-config, which enables users to specify a configuration properties file containing overriding settings for the AdminClient.

Understanding the --command-config Option

When you use the --command-config option with Kafka's AdminClient commands like kafka-topics, kafka-configs, kafka-acls, kafka-broker-api-versions, and others, you can specify a properties file that includes specific client configurations. These configurations often include security credentials and other settings necessary for interfacing with a Kafka cluster securely and effectively.

Format of the AdminClient Properties File

The properties file used in conjunction with --command-config typically follows the standard Java properties file format, where each property is specified in the form of key-value pairs. Here’s an example of what this file might look like:

1bootstrap.servers=localhost:9092
2security.protocol=SASL_SSL
3sasl.mechanism=PLAIN
4client.dns.lookup=use_all_dns_ips
5sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";
6ssl.truststore.location=/usr/local/share/kafka/client.truststore.jks
7ssl.truststore.password=changeit

Key Properties Explained

• bootstrap.servers: Specifies the Kafka brokers that the AdminClient should connect to.
• security.protocol: The protocol used to communicate with brokers. Common values are PLAINTEXT, SSL, SASL_PLAINTEXT, and SASL_SSL.
• sasl.mechanism: Defines the SASL mechanism that should be used. (GSSAPI, PLAIN, SCRAM-SHA-256, etc.)
• client.dns.lookup: Adjusts how DNS lookups should be handled. Can be default or use_all_dns_ips.
• sasl.jaas.config: JAAS configuration for SASL. It specifies the login module and relevant options required for authentication.
• ssl.truststore.location: Filepath to the truststore file.
• ssl.truststore.password: Password for the truststore file.

Usage in the Command Line

When you run a Kafka AdminClient command, you typically specify the --command-config option followed by the path to your properties file. Here’s an example command:

kafka-topics --bootstrap-server localhost:9092 --command-config /path/to/adminclient.properties --list

This command lists all topics in the Kafka cluster configured at localhost:9092 using additional properties specified in adminclient.properties.

Summary Table

Conclusion

The --command-config option is crucial for administering Kafka securely and effectively, especially in environments where enhanced security configurations are necessary. Understanding how to configure and use this properties file empowers developers and administrators to manage their Kafka clusters with greater flexibility and security.