Apache Kafka is a powerful distributed streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Setting it up for a developer or integration test environment involves a series of steps that ensure it mimics as closely as possible the behavior of a production system without the associated overhead.

Prerequisites

Before setting up Apache Kafka, ensure you have the following prerequisites installed:

• Java 8 or higher: Kafka is written in Java, so Java is required to run it.
• Zookeeper: Kafka uses Zookeeper for managing and coordinating Kafka brokers.
• Apache Kafka: Download the latest version of Kafka from https://kafka.apache.org/downloads.

Step-by-Step Setup Procedure

Step 1: Install Zookeeper

First, install Zookeeper or use a Docker image. If installing manually:

1wget https://downloads.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz
2tar -xzf apache-zookeeper-3.7.0-bin.tar.gz
3cd apache-zookeeper-3.7.0-bin
4cp conf/zoo_sample.cfg conf/zoo.cfg
5./bin/zkServer.sh start

This will start Zookeeper on the default port (2181).

Step 2: Install Apache Kafka

After downloading Kafka, extract it and use the provided scripts to start the Kafka server.

tar -xzf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0
./bin/kafka-server-start.sh config/server.properties

Step 3: Create a Kafka Topic

Create a topic where you can send and receive messages.

./bin/kafka-topics.sh --create --topic your-topic-name --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

Here, replication-factor 1 and partitions 1 are typically enough for most development environments.

Step 4: Test Producer and Consumer

To ensure Kafka is running correctly, test producing and consuming messages:

1# Producer command
2./bin/kafka-console-producer.sh --topic your-topic-name --bootstrap-server localhost:9092
3# Type your messages here
4
5# Consumer command
6./bin/kafka-console-consumer.sh --topic your-topic-name --from-beginning --bootstrap-server localhost:9092

Docker Alternative

For easier setup and teardown, use Docker. You can run Kafka and Zookeeper using docker-compose:

1version: '2'
2services:
3  zookeeper:
4    image: wurstmeister/zookeeper
5    ports:
6      - "2181:2181"
7  kafka:
8    image: wurstmeister/kafka
9    ports:
10      - "9092:9092"
11    environment:
12      KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100
13      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
14    volumes:
15      - /var/run/docker.sock:/var/run/docker.sock

Run it using:

docker-compose up

Key Configurations

Configuration parameters in the server.properties and zoo.cfg files play a crucial role. Here are some important settings:

Additional Considerations

• Data persistence: In production, consider the data retention policy. In a test environment, using /tmp as log storage with a cleanup policy might be sufficient.
• Security: Setup is without security parameters such as SSL or SASL for simplicity, important in production setups.
• Monitoring: Consider implementing monitoring tools like JMX exporter for Prometheus to track Kafka’s performance even in development.

Conclusion

Setting up Apache Kafka for a development or integration testing environment provides a robust platform for developing streaming applications. By following the setup steps and configurations discussed, developers can create a functional environment that allows for the development and testing of Kafka-based applications before deployment to production.