Apache Kafka, a distributed event streaming platform, enables developers to build robust messaging and streaming applications. One of its core features is the ability to configure how messages are retained and managed within a topic through the cleanup.policy setting. This policy can significantly influence disk space utilization and access to historical data.

Understanding cleanup.policy

Before discussing how to set the cleanup.policy to both delete and compact for a Kafka topic, it's important to understand what these terms mean:

• Delete Policy: This policy instructs Kafka to delete messages either after a specified retention time or after the log reaches a certain size.
• Compact Policy: Compaction ensures that a Kafka topic retains only the last message for each key. This is crucial for topics that act as a state store or changelog.

Setting cleanup.policy

To configure a Kafka topic to use both deletion and compaction, follow these steps:

1. Create a new topic with both policies: You can use the Kafka command-line tools to set the cleanup.policy to both delete and compact. This ensures that older messages are removed based on time or size, but the last message for each key is always retained.

   kafka-topics --bootstrap-server YOUR_KAFKA_SERVER --create --topic your-topic-name --partitions 3 --replication-factor 1 --config cleanup.policy=compact,delete --config retention.ms=1680000 --config segment.bytes=1073741824

• --config cleanup.policy=compact,delete sets the topic to utilize both compaction and deletion policies.
• retention.ms=1680000 configures the topic to retain messages for a specified amount of time (in milliseconds).
• segment.bytes=1073741824 sets the maximum size of a log segment file. After this size is reached, old segments are checked for eligibility for expiration or compaction.

1. Modify cleanup policy for an existing topic: If you need to modify an existing topic to use both policies, you can alter the topic configuration:

    kafka-configs --bootstrap-server YOUR_KAFKA_SERVER --alter --entity-type topics --entity-name your-topic-name --add-config cleanup.policy=compact,delete

1. Verify Topic Configuration: It's good practice to verify that the configurations have been set as expected:

    kafka-topics --bootstrap-server YOUR_KAFKA_SERVER --describe --topic your-topic-name

This command will show all the configurations for your topic including the set cleanup.policy.

Recommended Configurations and Considerations

When setting up your topic with both deletion and compaction, consider the following recommendations:

• Message Retention: Set a reasonable retention.ms to ensure that messages are not deleted too quickly, which might lead to data loss.
• Segment Tuning: segment.bytes should be tuned based on the amount of data typically written to the topic and disk space considerations.
• Performance Impact: Keep in mind that compaction is a resource-intensive process. Monitor performance and adjust configurations as necessary.

Summary Table of Key Configurations

Additional Details

• Using the Admin Client API: You can also configure these settings programmatically using Kafka’s Admin Client API.
• Impact of Misconfiguration: Incorrect settings can lead to unexpected disk usage or loss of messages. Always ensure your settings are validated in a test environment before applying in production.

Conclusion

Effectively combining the delete and compact policies in Kafka topics ensures that your application can manage storage efficiently while retaining necessary historical data. This setup is critical for applications requiring a balance between immediate data availability and long-term storage efficiency. Be sure to test different configurations to find the optimal setup for your specific requirements.