Apache Kafka Streams is a client library for building applications and microservices, where the input and output data are stored in Kafka topics. Kafka Streams allows you to process your streams of data with ease and reliability. The logging level of KafkaStreams is an important aspect to consider for debugging and monitoring the running streams effectively. This article shows how to change the log level of KafkaStreams effectively.

Understanding Log Levels

In Kafka Streams, the log levels typically used are:

• DEBUG: Provides detailed information, typically of interest only when diagnosing problems.
• INFO: Confirmation that things are working as expected.
• WARN: An indication that something unexpected happened, or indicative of some problem in the near future (e.g., ‘disk space low’). The software is still working as expected.
• ERROR: Due to a more serious problem, the software has not been able to perform some function.

Configuring Log Levels

KafkaStreams uses the logging framework provided by Apache Kafka, which by default is SLF4J in combination with Log4j. Therefore, adjusting the log level in a Kafka Streams application generally involves configuring Log4j or whichever logging framework you are using.

Configure With Log4j

Here is a typical example of how you can configure the logger for Kafka Streams in your log4j.properties file:

1# Root logger option
2log4j.rootLogger=INFO, stdout
3
4# Redirect log messages to console
5log4j.appender.stdout=org.apache.log4j.ConsoleAppender
6log4j.appender.stdout.Target=System.out
7log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
8log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
9
10# Set the logger level of Kafka Streams to DEBUG
11log4j.logger.org.apache.kafka.streams=DEBUG

This configuration sets the global log level to INFO and specifies that logs from Kafka Streams (org.apache.kafka.streams) should be captured at the DEBUG level.

Configure programmatically in Java

Alternatively, you can adjust log levels programmatically within your Java code. Although it is less common, it is particularly useful for dynamic log level changes:

1import org.apache.log4j.Logger;
2import org.apache.log4j.Level;
3
4public class AdjustKafkaStreamsLogging {
5    public static void changeLogLevel() {
6        Logger logger = Logger.getLogger("org.apache.kafka.streams");
7        logger.setLevel(Level.DEBUG);
8    }
9}

When to Adjust Log Levels

Adjusting the log level can be useful during development or troubleshooting, to get more detailed information about what the Kafka Streams application is doing. However, keep in mind that more verbose logging (e.g., DEBUG) can have performance implications and clutter log files in production environments.

Best Practices

1. Manage Performance: Extensive logging, particularly at DEBUG or TRACE levels, can slow down an application and lead to disk space issues quickly.
2. Security Considerations: Be cautious with logging sensitive information. Always sanitize logs if they are to be stored or reviewed.
3. Use Dynamic Configuration: If feasible, configure your logging to be dynamically adjustable without requiring a full restart of your services.

Summary Table

The following table summarizes key points for configuring KafkaStreams log levels:

By carefully setting and managing the log levels, you can maintain a good balance between efficiency and the need for information, making your KafkaStreams applications well-monitored yet performant.