Introduction

Kafka relies heavily on logs for diagnosing broker startup failures, connection issues, controller activity, and consumer or producer problems. To “enable Kafka logging with log4j” in practice, you usually point Kafka at a Log4j configuration file and then adjust log levels for the broker components you care about.

Know Which Process You Are Configuring

Kafka has several Java processes, and each one can have its own logging behavior:

• broker processes
• command-line tools such as console producer and consumer
• Connect workers
• Streams or custom client applications

The core idea is the same everywhere: the JVM starts with Log4j configuration, and loggers write either to the console, files, or both. Before changing anything, decide whether you are troubleshooting the broker itself or a separate client tool.

Start with a Log4j Properties File

A basic Log4j properties file defines the root logger and at least one appender.

1log4j.rootLogger=INFO, stdout, rollingFile
2
3log4j.appender.stdout=org.apache.log4j.ConsoleAppender
4log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
5log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
6
7log4j.appender.rollingFile=org.apache.log4j.RollingFileAppender
8log4j.appender.rollingFile.File=/var/log/kafka/server.log
9log4j.appender.rollingFile.MaxFileSize=100MB
10log4j.appender.rollingFile.MaxBackupIndex=10
11log4j.appender.rollingFile.layout=org.apache.log4j.PatternLayout
12log4j.appender.rollingFile.layout.ConversionPattern=[%d] %p %m (%c)%n

This gives you both console output and a rotating file. The root level of INFO is a sensible default for normal operation.

Kafka distributions often ship with a logging configuration under config/. The exact filename and logging backend details can vary by Kafka version or packaging, so use the shipped file as the starting point when possible instead of building a config from scratch.

Point Kafka at the Configuration

A common way to supply custom logging is through JVM options. For shell-based Kafka startup, that often means setting KAFKA_LOG4J_OPTS before launching the process.

export KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:/etc/kafka/log4j.properties"
bin/kafka-server-start.sh config/server.properties

That tells the broker JVM where to find the Log4j settings. On managed platforms or service wrappers, the same idea applies, but the configuration may live in a service unit, container environment, or startup script instead of an interactive shell.

Raise Log Levels for Specific Kafka Areas

Do not switch everything to DEBUG unless you truly need it. A better pattern is to target the subsystem you are investigating.

log4j.logger.kafka.network=DEBUG
log4j.logger.kafka.request.logger=DEBUG
log4j.logger.org.apache.kafka.clients.NetworkClient=DEBUG

This keeps the log volume focused on network and request handling instead of flooding disk with unrelated details.

For a producer or consumer issue, application-side client loggers may be more useful than broker loggers. For cluster issues, broker and controller categories usually matter more.

Verify the Output Early

After restarting or launching the process, verify that:

• the log file is being created where you expect
• messages appear at the level you configured
• the process can actually write to the target directory

A quick check avoids the common mistake of editing a file that the running process never reads. In containerized environments, it is also worth confirming whether logs are expected on stdout for collection by the platform rather than in local files.

Common Pitfalls

• Editing a Log4j file that is not the one the Kafka process actually uses.
• Setting global DEBUG or TRACE and creating far more log volume than the system can comfortably retain.
• Forgetting file permissions or disk space when sending broker logs to a local path.
• Troubleshooting a client-side issue by changing only broker loggers.
• Assuming every Kafka installation uses the same logging file name or startup wrapper.

Summary

• Enabling Kafka logging with Log4j means supplying the right logging configuration to the JVM running Kafka.
• Start from the packaged config when possible and adjust it rather than rebuilding everything.
• Use KAFKA_LOG4J_OPTS or the equivalent process-level JVM options to point Kafka at your config.
• Raise logging on specific Kafka packages instead of globally when debugging.
• Always verify that the intended process is reading the intended config file.