Amazon Managed Streaming for Apache Kafka (MSK) is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data. Apache Kafka is an open-source platform for building real-time streaming data pipelines and applications. When you create an MSK cluster, one of the essential pieces of information you need is the broker endpoints, which allow your application to connect to the Kafka cluster.

Understanding Broker Endpoints in Amazon MSK

A broker endpoint in Amazon MSK is a URL that your application uses to connect to the Kafka brokers within your cluster. Each broker in an MSK cluster has its own unique endpoint. For applications to produce or consume messages, they need to connect to these endpoints. MSK supports two types of endpoints:

• Zookeeper Endpoint: Used for administrative operations and is typically required only for applications that use Zookeeper directly for managing Kafka.
• Bootstrap Broker Endpoint: Used by Kafka clients to connect to the Kafka cluster. This endpoint is sufficient for most of the typical Kafka operations (producing and consuming data).

How to Retrieve Broker Endpoints

To obtain the broker endpoints of your Amazon MSK cluster, you can use either the AWS Management Console or the AWS CLI.

Using AWS Management Console

1. Sign in to the AWS Management Console.
2. Navigate to the MSK service.
3. Select the cluster for which you want to get the broker endpoints.
4. In the cluster details page, under "Connectivity and security," you'll find the endpoints listed.

Using AWS CLI

You can also use the AWS Command Line Interface (CLI) to retrieve the broker endpoints. Here's the command:

aws kafka get-bootstrap-brokers --cluster-arn <ClusterArn> --region <Region>

Replace <ClusterArn> with the Amazon Resource Name (ARN) of your cluster, and <Region> with the AWS region your cluster is hosted in. This command returns the bootstrap broker endpoint and, if enabled, the TLS endpoint.

Example

Suppose the ARN of your MSK cluster is arn:aws:kafka:us-east-1:123456789012:cluster/my-kafka-cluster/6bc8f9999f10. You would use the following command:

aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:us-east-1:123456789012:cluster/my-kafka-cluster/6bc8f9999f10 --region us-east-1

This command would return something like:

1{
2    "BootstrapBrokerString": "b-2.my-kafka-cluster.abcdefg.c2.kafka.us-east-1.amazonaws.com:9092, b-1.my-kafka-cluster.abcdefg.c2.kafka.us-east-1.amazonaws.com:9092, b-3.my-kafka-cluster.abcdefg.c2.kafka.us-east-1.amazonaws.com:9092",
3    "BootstrapBrokerStringTls": "b-2.my-kafka-cluster.abcdefg.c2.kafka.us-east-1.amazonaws.com:9094, b-1.my-kafka-cluster.abcdefg.c2.kafka.us-east-1.amazonaws.com:9094, b-3.my-kafka-cluster.abcdefg.c2.kafka.us-east-1.amazonaws.com:9094"
4}

Table: Summary of Commands and Endpoints

Conclusion

Understanding and accessing broker endpoints is crucial for integrating your application with an Amazon MSK cluster. Whether you use the AWS Management Console or the AWS CLI, getting this information is straightforward once you are familiar with the process. Ensure that your security groups and network settings allow traffic on the appropriate ports (default 9092 for plaintext and 9094 for TLS) to prevent connectivity issues.