Apache Kafka is a distributed streaming platform capable of handling trillions of events a day. Utilizing Kafka, users can enable their applications with robust capabilities for handling data streams. One specific feature it offers is the creation of compacted topics. Compacted topics are particularly useful in scenarios where the latest state is more significant than the historic data, such as in event sourcing or storing the latest status of entities.

What is a Compacted Topic?

A compacted topic in Kafka is a special type of topic configuration where the Kafka broker retains only the latest value for each key within the topic. Older records with the same key are discarded during a process known as "compaction". This is ideal for reducing storage needs while ensuring that the most recent state is always available.

Creating a Compacted Topic

To create a compacted topic in Kafka, you need to set the topic's cleanup policy to compact. This tells Kafka that the topic should use log compaction to maintain only the latest record for each key.

Here is a step-by-step guide on how to create a compacted topic:

Step 1: Install and Start Kafka

Ensure that Apache Kafka and Zookeeper are installed and running. You typically start Zookeeper and Kafka server by running:

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties

Step 2: Create the Topic with Compaction

You can create a compacted topic using the Kafka command line tools. For example, to create a topic named my-compacted-topic, you would use the following command:

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic my-compacted-topic --config cleanup.policy=compact

This command sets the cleanup.policy configuration to compact, which enables log compaction. You can adjust the replication-factor and partitions based on your requirements.

Step 3: Produce Messages to the Topic

With the topic created, you can now produce messages to it. Each message should include a key and a value. Kafka will ensure that only the latest message for each unique key is retained.

1bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-compacted-topic
2>key1:value1
3>key2:value2
4>key1:value3

Step 4: Verify Compaction

Kafka periodically triggers compaction (this may take some time depending on log segment settings and the compaction configuration). After compaction, if you consume the messages, you should see that only the latest values for each key are retained:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-compacted-topic --from-beginning --property print.key=true --property key.separator=":"

You should see:

key2:value2
key1:value3

Key Configuration Parameters

When configuring a compacted topic, various parameters can affect its behavior:

Best Practices and Considerations

• Use Cases: Use compacted topics when you need efficient storage for state or are implementing patterns such as event sourcing or CQRS (Command Query Responsibility Segregation).
• Performance: Compaction may be resource intensive. Monitor performance and adjust parameters like min.cleanable.dirty.ratio and segment.ms.
• Dual Nature: It’s possible to have both deletion and compaction by setting cleanup.policy to compact,delete. This allows you to manage retention based on time and also ensure only the latest record for each key is stored.

From optimizing storage to facilitating stateful architectures, Kafka's compacted topics offer a powerful tool for data management in modern applications. By setting up and managing these topics carefully, developers can greatly enhance the functionality and performance of their Kafka implementations.