Introduction

For local Kafka development today, the main shift is that you usually want KRaft mode, not the older ZooKeeper-based layout. A good docker-compose.yml for local use should solve two practical problems: start a single broker easily, and advertise listener addresses correctly for both containers and tools running on your host machine.

Use a Single-Broker KRaft Setup for Local Work

For local experimentation, one broker is usually enough. Confluent's current Docker documentation shows KRaft combined mode examples for local experimentation, which is much simpler than teaching a new project to juggle ZooKeeper as well.

Here is a useful single-broker compose file:

1services:
2  broker:
3    image: confluentinc/cp-kafka:8.1.1
4    hostname: broker
5    container_name: broker
6    ports:
7      - "9092:9092"
8      - "9101:9101"
9    environment:
10      KAFKA_NODE_ID: 1
11      KAFKA_PROCESS_ROLES: 'broker,controller'
12      KAFKA_CONTROLLER_QUORUM_VOTERS: '1@broker:29093'
13      KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
14      KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
15      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
16      KAFKA_LISTENERS: 'PLAINTEXT://:29092,CONTROLLER://:29093,PLAINTEXT_HOST://0.0.0.0:9092'
17      KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092'
18      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
19      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
20      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
21      KAFKA_JMX_PORT: 9101
22      KAFKA_JMX_HOSTNAME: localhost
23      CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'

This is not production topology. It is the right level of complexity for local development.

Understand the Listener Split

The most important part of local Kafka-in-Docker setup is the listener configuration.

In the example above:

• 'PLAINTEXT://broker:29092 is for other containers on the Docker network'
• 'PLAINTEXT_HOST://localhost:9092 is for clients running on your host machine'

That split matters because Kafka does not just accept a connection. It also tells clients where to reconnect. If advertised.listeners is wrong, the broker may start successfully while every client still fails in confusing ways.

Generate a Real Cluster ID

The CLUSTER_ID in examples is a placeholder. Generate your own before starting the stack.

docker run --rm confluentinc/cp-kafka:8.1.1 /bin/kafka-storage random-uuid

Copy the output into CLUSTER_ID. If you reuse the same one locally, that is fine. The important part is not leaving it blank.

Start and Test the Broker

Bring the stack up:

docker compose up -d
docker compose ps

Then create a topic and produce test data from inside the container:

1docker compose exec broker kafka-topics \
2  --create \
3  --topic demo \
4  --bootstrap-server localhost:9092 \
5  --partitions 1 \
6  --replication-factor 1
7
8docker compose exec broker kafka-console-producer \
9  --topic demo \
10  --bootstrap-server localhost:9092

This validates both the broker startup and the host-exposed listener path.

Prefer Local Simplicity over False Production Imitation

A common mistake is building an overly "realistic" local stack too early. For day-to-day development, you usually do not need multiple brokers, replication factors above 1, or a full platform stack unless the feature under test depends on them.

If you later need Connect, Schema Registry, or more brokers, add them intentionally. Do not make the basic local path harder than necessary.

Confluent also documents a confluent-local image optimized for local development. That is worth considering if you want an even faster start, but the explicit cp-kafka example above is helpful when you want to understand and control the listeners directly.

Common Pitfalls

• Using an old ZooKeeper-based compose file for a new local setup.
• Setting advertised.listeners incorrectly so clients receive the wrong hostname.
• Forgetting that one-broker local setups need replication-related settings such as KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1.
• Copying a sample CLUSTER_ID without understanding that it should be generated and managed intentionally.
• Overcomplicating local development with a production-like topology before it is needed.

Summary

• For local Kafka development, prefer a single-broker KRaft setup.
• Listener configuration is the most important part of a working Docker-based setup.
• Use separate advertised listeners for Docker-network clients and host-machine clients.
• Set replication-related internal topics to 1 in a one-broker environment.
• Keep the local compose file simple until your use case truly requires more services.