Apache Kafka is a highly popular distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. Monitoring Kafka's status is vital for ensuring that your Kafka clusters are running smoothly and reliably. Here we discuss various methods for checking the status or details of your Kafka server.

1. Using Kafka's Built-in Scripts

Apache Kafka ships with a variety of scripts that can be used for monitoring and management. These scripts can be found in the bin directory of your Kafka installation.

kafka-topics.sh

This script can be used to inspect topic details such as list, describe, create, alter, and delete Kafka topics. An example command to list all topics is:

bin/kafka-topics.sh --list --zookeeper localhost:2181

kafka-consumer-groups.sh

This script helps you to list all consumer groups, describe consumer groups, delete consumer group info, and check the consumer group lag. For example, to describe all groups:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups

2. JMX (Java Management Extensions)

JMX is a Java technology that supplies tools for managing and monitoring applications, system objects, devices, and service-oriented networks. Kafka exposes various metrics that can be accessed via JMX. Enabling JMX is straightforward:

export JMX_PORT=9999
bin/kafka-server-start.sh config/server.properties

You can then use any JMX client, like JConsole, to connect to the JMX port. This provides a rich array of Kafka metrics regarding throughput, latency, memory, garbage collection, and much more.

3. Kafka Exporter

Kafka Exporter is a tool that provides Kafka metric exports to Prometheus. The metrics exposed by Kafka Exporter can be viewed on a Prometheus or Grafana dashboard, making it easier to monitor multiple Kafka metrics effectively over a nice graphical user interface.

To run Kafka Exporter, you would typically deploy it where it can access your Kafka cluster and point it to your Kafka brokers.

docker run -d --name=kafka-exporter --network=host danielqsj/kafka-exporter --kafka.server=kafka:9092

4. Third-Party Monitoring Tools

Several third-party tools and services can monitor Kafka. Popular tools include:

• Datadog
• New Relic
• LinkedIn's Burrow

These tools provide comprehensive monitoring solutions with alerting, visualizations, and detailed insights into cluster health, consumer states, and performance metrics.

Summary Table

Here is a table summarizing the tools/methods mentioned above for monitoring Kafka:

Additional Details

Log Files

Kafka log files are also a valuable resource. Located in the directory specified by log.dirs in the Kafka configuration, these log files can give insights into the health and operations of Kafka brokers.

ZooKeeper

Since Kafka relies on ZooKeeper, you can examine ZooKeeper's status related to Kafka by using the zkCli.sh script. This can show real-time details about ZooKeeper's management of Kafka:

bin/zkCli.sh -server localhost:2181

Monitoring Kafka is crucial for maintaining the performance, availability, and reliability of your Kafka-based applications. Using the built-in Kafka scripts, JMX, Kafka Exporter, or third-party monitoring tools allows you to thoroughly inspect and effectively manage your Kafka environments.