Apache Kafka is a distributed streaming platform that enables building real-time streaming data pipelines and applications. A fundamental aspect of managing Kafka involves defining optimal retention and deletion policies for messages, which can significantly affect storage management, system performance, and reliability. This article explains the detailed considerations involved in setting up these policies.

Understanding Topic Retention in Kafka

Kafka stores logs of records in topics. Each topic is a separate log and can be configured with its own retention settings. Two primary mechanisms control the lifecycle of logs within Kafka:

1. Time-based Retention: Records can be retained based on time, specified in milliseconds.
2. Size-based Retention: Records can be retained based on the total size of the log files on disk.

The appropriate settings depend on the specific requirements of your application, including how long you need to retain data and how much storage you can allocate to Kafka.

Time-Based Retention

Time-based retention (log.retention.hours, log.retention.minutes, log.retention.ms) determines how long Kafka retains messages in a topic before they are eligible for deletion. For instance, if time-based retention is set to two days, any record older than two days will be deleted during the next run of the log cleaner.

Example: Set retention for 24 hours.

log.retention.hours=24

This implies that logs older than 24 hours will be deleted, freeing up storage space.

Size-Based Retention

Size-based retention (log.retention.bytes) defines the maximum size of the log before old messages are purged. This setting is highly useful when the requirement is to use a fixed amount of disk space no matter the age of the messages.

Example: Limit log size to 1 GB.

log.retention.bytes=1073741824

Deletion and Compaction

Beyond simple retention, Kafka also supports more intricate policies like deletion and compaction:

• Deletion Policy: This is the default policy, where old records are deleted according to the retention policy.
• Compaction Policy: This ensures that the log contains at least the last known value for each key. It’s particularly useful for restoring state.

Example of Setting a Deletion Policy:

log.cleanup.policy=delete

Example of Setting a Compaction Policy:

log.cleanup.policy=compact

Combining Time, Size, and Compaction

Kafka allows combinations of these settings to realize complex retention strategies. For instance, a topic can be configured to use both compaction and deletion.

Example:

log.cleanup.policy=compact,delete
log.retention.hours=168 # retain for one week
log.retention.bytes=5368709120 # 5 GB

Managing Overloads and Performance

Care should be applied when setting retention policies as they can significantly impact system performance. Excessive compaction or very frequent deletions might lead to increased CPU usage and slower message throughput.

Best Practices

• Data Nature and Requirements: Assess the type of data and compliance needs. For sensitive data, enforcing stringent retention might be necessary.
• Monitor Disk Usage: Regularly monitor Kafka’s disk usage to adjust the size or time-based settings as the data grows.
• Consumer Speed: Ensure the retention policy accommodates the slowest consumer, to prevent loss of data before it is processed.

Summary Table: Key Retention Settings

In conclusion, setting the optimal retention and deletion policies in Kafka is critical for managing disk space while ensuring that the data remains available as needed. By carefully considering your application’s requirements and balancing them with the capabilities of your Kafka infrastructure, you can maintain efficient and reliable data streams.