Apache Kafka is a distributed event streaming platform capable of handling trillions of events a day. As you work with Kafka, managing topics — the categories or feeds to which records are published — becomes crucial. Sometimes, for various reasons including cleanup, testing, or reorganization, you might need to delete one or more topics from your Kafka cluster.

Understanding Kafka Topic Deletion

Before you delete topics, it's important to understand that once a topic is deleted, all the data stored in that topic is also permanently lost unless specifically configured otherwise. Kafka's topic deletion process involves marking a topic for deletion and then having the Kafka brokers remove all data associated with the topic from the cluster.

Configuration Requirements

To delete topics, the Kafka cluster needs to be configured to allow topic deletions. This can be done by setting the delete.topic.enable property to true in the broker configuration file (server.properties):

delete.topic.enable=true

This setting should be enabled on all brokers in the cluster. After making this change, you will need to restart the brokers for the configuration to take effect.

How to Delete Multiple Topics

Deleting multiple topics in Kafka can be performed using the Kafka command-line tools that come bundled with Kafka, or programmatically using Kafka's AdminClient API.

Using Kafka Command-Line Tools

The Kafka command line tool that deals with topic management is kafka-topics.sh. To delete multiple topics, you can use this tool in conjunction with shell scripting.

1. List all topics you might want to delete to double-check their names:

    kafka-topics.sh --bootstrap-server localhost:9092 --list

1. Delete multiple topics by specifying each topic name:

    kafka-topics.sh --bootstrap-server localhost:9092 --delete --topic topic-name1 --topic topic-name2

Replace localhost:9092 with your Kafka broker's addresses. If you have many topics, you might want to automate or script this process.

Using AdminClient API

For those preferring to use Java or another JVM language, the AdminClient API provides a way to manage Kafka topics programmatically.

Here’s a basic example in Java:

1import org.apache.kafka.clients.admin.AdminClient;
2import org.apache.kafka.clients.admin.DeleteTopicsResult;
3import org.apache.kafka.clients.admin.AdminClientConfig;
4
5import java.util.Arrays;
6import java.util.Properties;
7import java.util.concurrent.ExecutionException;
8
9public class DeleteTopicsExample {
10    public static void main(String[] args) {
11        Properties config = new Properties();
12        config.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
13
14        try (AdminClient admin = AdminClient.create(config)) {
15            DeleteTopicsResult result = admin.deleteTopics(Arrays.asList("topic-name1", "topic-name2"));
16            result.all().get();
17        } catch (InterruptedException | ExecutionException e) {
18            e.printStackTrace();
19        }
20    }
21}

This code configures an AdminClient instance, then deletes the specified topics and waits for the operation to complete.

Precautions and Best Practices

Conclusion

Deleting topics in Kafka is a straightforward process provided the prerequisites are met and careful planning is undertaken. Whether using command-line tools or programmatically via the AdminClient API, always ensure you are working within the bounds of your data governance and compliance requirements.