Apache Kafka is a popular distributed event streaming platform capable of handling trillions of events a day. Initially conceived as a messaging queue, Kafka is based on an abstraction of a distributed commit log. Since Kafka can handle such high-throughout, low-latency messaging, it's a favorite for building real-time data pipelines and streaming applications. When deploying Kafka on Kubernetes, which provides automatable deployment, scaling, and operations of application containers across clusters of hosts, it could be beneficial to expose each Kafka broker through an AWS Elastic Load Balancer (ELB) for better load distribution and fault tolerance. Here, we discuss how to achieve this by setting up a Kafka cluster on Kubernetes and then exposing each broker via an ELB on AWS.

Kubernetes Deployment of Kafka

To begin, Kafka can be deployed on a Kubernetes cluster in several ways, including using Helm charts (like the popular bitnami/kafka chart), or using operators like the Strimzi Kafka Operator, which simplifies the management of Kafka on Kubernetes.

Here's a basic step using Helm:

1. Install Helm (if you haven't):

   curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

1. Add Bitnami Chart and Install Kafka:

   helm repo add bitnami https://charts.bitnami.com/bitnami
   helm install my-kafka bitnami/kafka

Expose Kafka Brokers Through an AWS ELB

Exposing each Kafka broker via an AWS ELB is slightly more complex than exposing a typical stateless service (like a web server) because Kafka brokers need to be individually addressable. You will need a way to assign each Kafka pod its own ELB, which requires dynamic provisioning and DNS updates.

Step-by-Step Configuration

1. Create a Headless Service in Kubernetes: A headless service in Kubernetes is required to manage the DNS records that allow us to not only discover all brokers but also address them individually.

1   apiVersion: v1
2   kind: Service
3   metadata:
4     name: kafka-headless
5     labels:
6       app: kafka
7   spec:
8     ports:
9     - port: 9092
10       name: server
11     clusterIP: None
12     selector:
13       app: kafka

1. Assign Each Kafka Pod an External IP Using ELB: Kubernetes does not natively support assigning an External IP from an ELB to a pod directly. Instead, you can create a LoadBalancer service for each broker which will be automatically assigned an ELB by AWS.
   First, ensure each Kafka pod has a stable and unique identifier, like my-kafka-0, my-kafka-1, etc., typically achieved through a StatefulSet.
   Then, for each broker, create a LoadBalancer service:

1   apiVersion: v1
2   kind: Service
3   metadata:
4     name: kafka-broker-0
5   spec:
6     type: LoadBalancer
7     ports:
8     - port: 9092
9       targetPort: 9092
10     selector:
11       statefulset.kubernetes.io/pod-name: my-kafka-0

1. Update Kafka Configuration for Custom Listener: Brokers need to be aware of their external access configuration, configured using listeners. A separate listener for each broker can be set in the Kafka broker configuration:

1   broker.id=0
2   listeners=PLAINTEXT://:9092,EXTERNAL://:9094
3   advertised.listeners=PLAINTEXT://my-kafka-0.my-kafka-headless.default.svc.cluster.local:9092,EXTERNAL://<ELB-DNS-0>:9094
4   ...

Replace <ELB-DNS-0> with the actual DNS name of the ELB assigned to the broker.

Summary Table

Considerations

• Security: Make sure to implement security groups and access controls to restrict access to the ELBs.
• Monitoring and Logging: Use tools like Prometheus and Grafana for monitoring and ELK stack or similar for logging.

Conclusion

Setting up Kafka on Kubernetes to expose individual brokers through an ELB involves configuring headless services, LoadBalancer services, and custom brokerage settings for network communication. This configuration supports high availability and fault tolerance, essential for production-grade Kafka deployments on AWS.