Apache Kafka is a powerful distributed event streaming platform capable of handling trillions of events a day. Originally developed by Linkedin, Kafka is open-source and widely used for building real-time streaming data pipelines and applications. This guide will walk you through the process of installing Kafka on a Windows operating system.

Prerequisites

Before you start the installation process, you need to ensure your system meets the following requirements:

• Java: Kafka is written in Java, so you will need Java installed on your machine. Kafka 2.13 requires Java 8 or higher. You can download Java from Oracle's official site.

Step 1: Download Kafka

1. Visit the official Apache Kafka website.
2. Select the version of Kafka you want to download. It's generally recommended to download the latest stable release.
3. Click on the 'Binary downloads' section and download the binary files for Windows.

Step 2: Extract Kafka

1. Once the download is complete, extract the Kafka binaries to your desired location. For instance, you might extract it to C:\kafka.

Step 3: Set Up the Environment

1. Set JAVA_HOME Variable: You need to ensure that the JAVA_HOME environment variable is set to the path where Java is installed on your local machine.
   • Right-click on 'This PC' and select 'Properties'.
   • Click on 'Advanced system settings'.
   • In the 'System Properties' window, click on 'Environment Variables'.
   • Under 'System Variables', click 'New'. Set the variable name to JAVA_HOME and the variable value to the path where Java is installed.

Step 4: Start the Kafka Environment

Kafka uses ZooKeeper, so you first need to start a ZooKeeper server.

Start ZooKeeper Server

1. Open a new command prompt as an administrator.
2. Navigate to your Kafka directory (e.g., cd C:\kafka).
3. Run the command:

   .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties

1. Keep this command prompt open.

Start Kafka Server

1. Open another command prompt as an administrator.
2. Navigate to the Kafka directory (e.g., cd C:\kafka).
3. Run the command:

   .\bin\windows\kafka-server-start.bat .\config\server.properties

Step 5: Create a Topic

Now that you have your ZooKeeper and Kafka servers running, you can create a Kafka topic to store your messages.

1. Open another command prompt window.
2. Run:

   .\bin\windows\kafka-topics.bat --create --topic test --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

Step 6: Test Kafka

To test if your Kafka broker is working correctly:

Producer Test

1. Open a new command prompt.
2. Send some messages:

   .\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
   >Hello, Kafka!
   >This is a test message.

Consumer Test

1. Open another command prompt.
2. Start the consumer:

   .\bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic test --from-beginning

You should see the messages you typed in the producer prompt.

Summary Table

Additional Details

• Logging: Kafka logs details about system behavior, and you can find these files under <your-kafka-folder>\logs.
• Configuration Files: Kafka's configuration is held under the config folder. Key files are server.properties (for Kafka servers) and zookeeper.properties (for ZooKeeper).

Conclusion

By following these steps, you can get Apache Kafka up and running on a Windows machine. This setup is ideal for development and testing purposes. For a production environment, Kafka should ideally be run on a Linux-based system with a robust cluster configuration.