Apache Kafka is a powerful distributed streaming platform that enables you to build real-time data pipelines and streaming applications. Deploying Kafka using Docker can simplify the setup and maintenance process. In this article, we'll discuss how to start Kafka topics using Docker Compose with the Spotify Kafka image.

Overview of Apache Kafka and Docker Compose

Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. It is designed for high throughput and scalability. Docker, on the other hand, is a platform for developing, shipping, and running applications inside lightweight containers.

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you create a docker-compose.yml file to configure your application's services, networks, and volumes. Then, using a single command, you create and start all the services defined in your configuration.

Using spotify/kafka

The Spotify Kafka Docker image (spotify/kafka) bundles Kafka and Zookeeper in a single container. This configuration suits development and testing but may not be ideal for production environments, where Zookeeper and Kafka typically run on separate systems or clusters for performance and reliability.

Prerequisites

To follow this setup, you need:

• Docker installed on your machine
• Docker Compose installed on your machine

Configuration Details

First, you need to create a docker-compose.yml file that describes the Kafka service you want to run. Here is an example of such a configuration:

1version: '3'
2services:
3  kafka:
4    image: spotify/kafka
5    environment:
6      ADVERTISED_HOST: kafka
7      ADVERTISED_PORT: 9092
8      TOPICS: "my-topic"
9      PARTITIONS: 1
10      REPLICATION_FACTOR: 1
11    ports:
12      - "2181:2181" # Zookeeper
13      - "9092:9092" # Kafka

Explanation:

• ADVERTISED_HOST and ADVERTISED_PORT are the host and port that Kafka advertises to producers and consumers.
• TOPICS initializes a Kafka topic — in this case, my-topic.
• PARTITIONS sets the number of partitions within the topic.
• REPLICATION_FACTOR sets the replication factor of the topic (1 means no replication).

Starting the Container

You can start the Kafka container via Docker Compose with the below command:

docker-compose up

By running this command, Docker Compose reads the docker-compose.yml, pulls the necessary Docker images, and starts the defined services.

Creating Additional Kafka Topics

If you need to create additional topics after your container is up, you can use the Kafka topic command:

docker exec -it <kafka-container-id> kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic <new-topic-name>

Here, <kafka-container-id> should be replaced with the actual ID of the Kafka container, which can be found using docker ps.

Interaction with Kafka

To produce messages:

docker exec -it <kafka-container-id> kafka-console-producer.sh --broker-list localhost:9092 --topic <your-topic>

To consume messages:

docker exec -it <kafka-container-id> kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic <your-topic> --from-beginning

Summary Table

This table provides an overview of the key components and settings related to our Kafka deployment using Docker.

Conclusion

Setting up Kafka using Docker Compose with spotify/kafka streamlines the development and testing of Kafka applications. This consolidated approach can also aid in educational environments or when prototyping. However, for production environments, it's recommended to use separate, scalable configurations for Kafka and Zookeeper to ensure better reliability and performance.