Apache Kafka, developed by LinkedIn and later open-sourced under the Apache Software Foundation, is a distributed streaming platform that excels in handling high-throughput data streams. One of the fundamental components of Kafka architecture is the "topic," which acts as a category or feed for messages to be stored and published.

Understanding Kafka Topics

Kafka topics are uniquely identified within a cluster and are used to organize messages. Each topic is split into partitions which allow Kafka to scale horizontally by distributing the data across multiple brokers in the cluster. Messages within these partitions are ordered and immutable, and each message within a partition is assigned a unique sequence number known as an offset.

Creating a Topic in Apache Kafka

Creating a topic in Kafka can be achieved either using the Kafka command-line tools or programmatically using Kafka’s AdminClient API. Here are both methods detailed:

Using Kafka Command Line Tools

Kafka ships with a set of command-line tools located in the bin directory of your Kafka installation. The kafka-topics.sh script is used to create topics.

Example:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 3 --topic my-kafka-topic

In this example:

• --bootstrap-server specifies the Kafka server to connect to.
• --replication-factor specifies the number of copies of the topic to be maintained.
• --partitions defines the number of partitions for the topic.
• --topic is the name of the topic you are creating.

Using AdminClient API

For those implementing Kafka interactions programmatically, the AdminClient API provides a way to manage topics from your Java applications.

Java Example:

1Properties props = new Properties();
2props.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
3
4try (AdminClient admin = AdminClient.create(props)) {
5    NewTopic newTopic = new NewTopic("my-kafka-topic", 3, (short) 1);
6    admin.createTopics(Collections.singletonList(newTopic)).all().get();
7} catch (Exception e) {
8    e.printStackTrace();
9}

In this Java code:

• AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG defines the initial list of brokers.
• The NewTopic object details the topic name, number of partitions, and replication factor.
• createTopics is called on the AdminClient instance to create the topic.

Topic Configuration Options

Apache Kafka allows you to configure topics according to your needs. Common configurations include:

• retention.ms: Controls how long messages are retained.
• cleanup.policy: Determines whether old data is deleted or compacted.

Example Command:

bin/kafka-topics.sh --alter --bootstrap-server localhost:9092 --topic my-kafka-topic --config retention.ms=7200000

Summary

Additional Considerations

• Topic Deletion: Deleting a topic can be done similarly using both command-line and AdminClient API. However, ensure that the topic deletion is enabled in the broker settings (delete.topic.enable must be set to true).
• Monitoring Tools: Tools like Kafka Manager or Confluent Control Center can also be used for easier management and monitoring of topics.
• Security: Consider implementing proper ACLs and security practices especially in production environments.

By properly utilizing Kafka topics, developers and organizations can effectively manage the flow of data across systems, ensuring robustness and scalability in data processing pipelines.