Send KafkaProducer from local machine to hortonworks sandbox on virtualbox
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Apache Kafka is a distributed streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since being open-sourced by LinkedIn in 2011, Kafka has quickly evolved from messaging queue to a full-fledged event streaming platform.
Scenario Overview
In this article, we will see how to set up a KafkaProducer on a local machine and send messages to an Apache Kafka instance running inside a Hortonworks Data Platform (HDP) sandbox on VirtualBox. Hortonworks Data Platform is an open-source framework for distributed storage and processing of large data sets. Running it in a VirtualBox VM is a convenient way to create a personal Hadoop environment for development and testing.
Prerequisites
- Hortonworks Sandbox: Running on VirtualBox.
- Apache Kafka: Installed and configured in the Sandbox.
- Java: Required on the local machine to run Kafka producer.
- Network Settings: Proper network settings to allow communication between your local machine and the VirtualBox.
Step 1: Configuring Hortonworks Sandbox
Ensure that Kafka is installed and running in your Hortonworks Sandbox. Kafka can usually be started through Ambari, which is the web-based management tool for Hadoop provided in the Hortonworks sandbox. Here’s how to start it:
- Login to Ambari by accessing
http://<Your-VM-IP>:8080. The default username and password are often "admin". - Navigate to
Kafka service -> Service Actions -> Start/Restart.
Step 2: Network Configuration
Configure your VirtualBox network settings to ensure that your local machine can communicate with Kafka running on Hortonworks Sandbox:
- Go to VirtualBox settings for your Hortonworks VM.
- Set up a 'Host-Only Adapter' or a 'Bridged Adapter' for allowing network requests to and from your local machine.
- Take note of the IP address that your VM is using in this network.
Step 3: Setting up Kafka Producer on Local Machine
- Install Java: Kafka clients require Java, so ensure Java is installed on your local machine.
- Download Kafka: Download the Kafka binaries from the official Apache Kafka website to your local machine.
- Setup Kafka Client:
- Configure Producer: Create a Java file to set up your Kafka producer. Here’s an example producer code snippet:
- Compile and Run the Producer: Use the following commands:
Step 4: Verify the Messages
You can verify that messages are being sent by using Kafka's console consumer in the Hortonworks Sandbox:
Summary Table
| Component | Specification |
| VirtualBox VM IP | <Your-VM-IP> |
| Kafka Topic | test |
| Kafka Producer File | SimpleProducer.java |
| Network Adapter | Host-Only or Bridged (depending on the user's configuration) |
| Ambari Access | http://<Your-VM-IP>:8080 |
This setup allows you to experiment with Kafka messaging by integrating a local Java application with a Hadoop-based Kafka running in a virtualized environment, providing a realistic scenario for developing big data pipelines.

