Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. Kafka is widely used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.

Connecting to Kafka running in a Docker container involves several steps, including setting up the Docker environment, running Kafka in Docker, and configuring your local system or another Docker container to connect to Kafka.

Step 1: Install Docker

Before you can run Kafka in Docker, you need to have Docker installed on your machine. Docker is available for Windows, macOS, and various distributions of Linux. You can download it from the Docker website and follow the installation instructions for your platform.

Step 2: Run Kafka in Docker

You can run Kafka in Docker either by using a Docker image that includes both Kafka and ZooKeeper or by setting up separate containers for Kafka and ZooKeeper. ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Here's how to use the wurstmeister/kafka Docker image, which is popular for running Kafka in Docker environments:

1. Create a Docker Network: This allows containers to communicate with each other more easily.

   docker network create kafka-net

1. Start ZooKeeper: ZooKeeper is required for Kafka to run. You will use the wurstmeister/zookeeper image.

   docker run -d --name zookeeper --network kafka-net -p 2181:2181 wurstmeister/zookeeper

1. Start Kafka: Link Kafka to the ZooKeeper container and expose the necessary ports.

1   docker run -d --name kafka --network kafka-net \
2       -p 9092:9092 \
3       -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
4       -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \
5       -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
6       wurstmeister/kafka

Step 3: Configuration for Kafka Clients

To connect to Kafka from a client on your host machine or another container, you should use the exposed port 9092. If you're connecting from another container within the same Docker network (kafka-net), you can refer to the Kafka container by its name, kafka.

Here's an example of how a client can produce or consume messages:

1# Example Kafka console producer
2docker run --rm --network kafka-net wurstmeister/kafka \
3  kafka-console-producer.sh --broker-list kafka:9092 --topic test
4
5# Example Kafka console consumer
6docker run --rm --network kafka-net wurstmeister/kafka \
7  kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic test --from-beginning

Key Concepts and Troubleshooting

• Networks in Docker: Ensure that all related containers are on the same network to facilitate easier communication.
• Environment Variables: Kafka uses environment variables for configuration. The KAFKA_ADVERTISED_LISTENERS variable is crucial for ensuring that Kafka can be accessed correctly from both inside and outside the Docker network.
• Logs: Check logs of the Kafka container if you face issues with Kafka not starting up or clients not being able to connect:

  docker logs kafka

Summary Table

This setup guide and summary should help you to get Kafka up and running in a Docker container and to connect clients for producing and consuming messages.