Apache Kafka, a popular event streaming platform, traditionally requires the use of Apache Zookeeper for managing its cluster state and coordination. However, with the release of Kafka 2.8.0, it's now possible to deploy Kafka without the dependency on Zookeeper, thanks to the introduction of KRaft mode (Kafka Raft Metadata mode). This article explores the Kafka Docker image configured to run in KRaft mode, a setup that simplifies Kafka's architecture by eliminating the need for Zookeeper.

Overview of Kafka in KRaft Mode

KRaft mode configures Kafka to use its own internal consensus mechanism based on the Raft algorithm, rather than relying on Zookeeper. This simplifies operations and potentially improves performance and scalability. Setting up Kafka in KRaft mode involves configuring the server.properties and managing the Kafka cluster using new tools and commands designed specifically for this mode.

Running Kafka in KRaft Mode using Docker

The application of Docker provides an efficient and isolated environment for running Kafka in KRaft mode. Here’s how you can set up such an environment:

Docker Image Configuration

1. Base Image: You will start with a functional Kafka Docker image, which could be based on confluentinc/cp-kafka or bitnami/kafka. Ensure that the Kafka version is at least 2.8.0.
2. Configuring Kafka for KRaft Mode: Modify the KAFKA_SERVER_PROPERTIES to initiate Kafka in KRaft mode. Add the following lines to the configuration:

    process.roles=broker,controller
    node.id=1
    controller.quorum.voters=1@localhost:9093

Here, process.roles specifies the roles this Kafka instance is playing. Since it's a standalone setup without Zookeeper, it functions as both the broker and the controller. node.id is a unique identifier for the server node in the cluster.

1. Ports Exposure: Ensure that the necessary Kafka ports are exposed via Docker. The default is 9092 for Kafka broker services.

Creating the Dockerfile

To build the Kafka Docker image configured for KRaft mode, you would start with a Dockerfile like this:

1FROM confluentinc/cp-kafka:latest
2
3# Set environment variables
4ENV KAFKA_CFG_PROCESS_ROLES broker,controller
5ENV KAFKA_CFG_NODE_ID 1
6ENV KAFKA_CFG_CONTROLLER_QUORUM_VOTERS 1@localhost:9093
7
8# Expose the necessary ports
9EXPOSE 9092
10
11# Command to run Kafka
12CMD ["kafka-server-start.sh", "/etc/kafka/server.properties"]

Running the Docker Container

To run your Kafka container in KRaft mode, use the following command:

docker run -d --name my-kafka-kraft -p 9092:9092 my-kafka-kraft-image

This command starts a detached Kafka instance named my-kafka-kraft, mapping the internal port 9092 to the same external port for client access.

Table: Key Configuration Options for Kafka in KRaft Mode

Advantages of Kafka Without Zookeeper

• Simplified architecture: Removing Zookeeper simplifies the overall architecture, reducing the operational overhead.
• Scalability and performance improvements: Direct control over the metadata may translate into better performance, especially for large clusters.
• Ease of configuration and management: Fewer components to install and manage can make configuration and upkeep easier.

Conclusion

Kafka in KRaft mode represents a significant architectural shift, destined to simplify deployments by reducing dependencies. Running Kafka in Docker further encapsulates its environment, making it easier to manage and deploy. The transition away from Zookeeper is highly anticipated in environments where simplification and modernization of data architectures are critical. This shift not only aligns Kafka with contemporary distributed system design patterns but also enhances its usability and scalability.