Apache Kafka is a distributed streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being open-sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.

Scenario Overview

In this article, we will see how to set up a KafkaProducer on a local machine and send messages to an Apache Kafka instance running inside a Hortonworks Data Platform (HDP) sandbox on VirtualBox. Hortonworks Data Platform is an open-source framework for distributed storage and processing of large data sets. Running it in a VirtualBox VM is a convenient way to create a personal Hadoop environment for development and testing.

Prerequisites

• Hortonworks Sandbox: Running on VirtualBox.
• Apache Kafka: Installed and configured in the Sandbox.
• Java: Required on the local machine to run Kafka producer.
• Network Settings: Proper network settings to allow communication between your local machine and the VirtualBox.

Step 1: Configuring Hortonworks Sandbox

Ensure that Kafka is installed and running in your Hortonworks Sandbox. Kafka can usually be started through Ambari, which is the web-based management tool for Hadoop provided in the Hortonworks sandbox. Here’s how to start it:

1. Login to Ambari by accessing http://<Your-VM-IP>:8080. The default username and password are often "admin".
2. Navigate to Kafka service -> Service Actions -> Start/Restart.

Step 2: Network Configuration

Configure your VirtualBox network settings to ensure that your local machine can communicate with Kafka running on Hortonworks Sandbox:

• Go to VirtualBox settings for your Hortonworks VM.
• Set up a 'Host-Only Adapter' or a 'Bridged Adapter' for allowing network requests to and from your local machine.
• Take note of the IP address that your VM is using in this network.

Step 3: Setting up Kafka Producer on Local Machine

1. Install Java: Kafka clients require Java, so ensure Java is installed on your local machine.
2. Download Kafka: Download the Kafka binaries from the official Apache Kafka website to your local machine.
3. Setup Kafka Client:

   tar -xzf kafka_2.13-2.8.0.tgz
   cd kafka_2.13-2.8.0

1. Configure Producer: Create a Java file to set up your Kafka producer. Here’s an example producer code snippet:

1   import org.apache.kafka.clients.producer.KafkaProducer;
2   import org.apache.kafka.clients.producer.ProducerRecord;
3
4   import java.util.Properties;
5
6   public class SimpleProducer {
7     public static void main(String[] args){
8       Properties props = new Properties();
9       props.put("bootstrap.servers", "<Your-VM-IP>:9092");
10       props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
11       props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
12
13       KafkaProducer<String, String> producer = new KafkaProducer<>(props);
14
15       for(int i = 0; i < 100; i++) {
16         producer.send(new ProducerRecord<String, String>("test", Integer.toString(i), "Message " + i));
17       }
18
19       producer.close();
20     }
21   }

1. Compile and Run the Producer: Use the following commands:

   javac -cp "path/to/kafka/libs/*" SimpleProducer.java
   java -cp ".:path/to/kafka/libs/*" SimpleProducer

Step 4: Verify the Messages

You can verify that messages are being sent by using Kafka's console consumer in the Hortonworks Sandbox:

cd /usr/hdp/current/kafka-broker/bin
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Summary Table

This setup allows you to experiment with Kafka messaging by integrating a local Java application with a Hadoop-based Kafka running in a virtualized environment, providing a realistic scenario for developing big data pipelines.