Get Kafka compressed message size
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation. It is designed to handle real-time data feeds with high throughput and low latency. When dealing with large volumes of data, it is often beneficial to compress messages before sending them through Kafka to save bandwidth and increase throughput. This article explores how to get the compressed message size in Kafka, including technical explanations and practical examples.
Understanding Compression in Kafka
Kafka supports multiple compression codecs, including GZIP, Snappy, LZ4, and more recently zstd (Zstandard). Compression can be configured at the topic or producer level, meaning that messages can be compressed before being stored or sent through the network.
When a Kafka producer sends a batch of messages, they can be compressed as a single batch rather than individually. This batch compression is more efficient than compressing messages one at a time because it reduces the overhead introduced by the compression metadata and takes better advantage of the compression algorithm's pattern recognition over larger datasets.
How to Configure Compression
Compression is configured in the Kafka producer settings with the compression.type property. Here's how you might set it:
Measuring Compressed Message Size
To determine the size of a compressed message in Kafka, you can either calculate it programmatically before sending the message or measure it indirectly by observing the difference in network usage or storage. Here’s an example of how you could calculate the size in Java before sending:
This snippet will give you the size of the compressed version of a message using GZIP.
Table: Kafka Compression Codecs and Characteristics
| Compression Type | Compression Ratio | Compression/Decompression Speed | Use Case |
| None | 0% | N/A | Testing or very small datasets |
| GZIP | High | Slow | High compression, less sensitive to latency |
| Snappy | Moderate | Fast | Balance between speed and compression |
| LZ4 | Moderate | Faster than Snappy | Similar to Snappy, slightly better performance in some cases |
| zstd | Very High | Moderate | High compression with reasonable speeds |
Practical Considerations
- Effect on Performance: While compression reduces the size of the data transmitted over the network and stored, it also requires additional CPU resources. The choice of compression algorithm can impact both producer and broker performance.
- End-to-End Latency: Compressed messages take time to be compressed and decompressed. This added time can affect the end-to-end latency of message delivery, especially with algorithms like GZIP.
- Compression Tuning: Kafka 2.1 and higher versions allow producers to use
linger.msandbatch.sizesettings along withcompression.typeto optimize both compression ratio and performance. Larger batch sizes allow better compression but use more memory.
Conclusion
Compressing Kafka messages can lead to significant improvements in performance and cost savings in storage and bandwidth. However, it also introduces complexity in terms of configuration and a balance between compression efficiency and resource usage. By carefully choosing and configuring the compression codec, developers can optimize their Kafka deployment to suit their specific requirements.

