Sending large messages with Kafka (over the default limit of 15MB) involves adjustments to the producer, broker, and consumer configurations, as well as some best practices to handle large payloads efficiently. Here's how you can achieve this:

1. Increase Kafka's Maximum Message Size

Kafka has configurable limits for message size. You need to increase these limits in the producer, broker, and consumer.

Producer Configuration

Set the max.request.size to a larger value (in bytes) in the producer to allow sending large messages.

Properties props = new Properties();
props.put("max.request.size", 20000000); // 20MB

Broker Configuration

Set the message.max.bytes in the broker to a value slightly larger than the producer’s max.request.size.

• Edit the server.properties file on the broker:

  message.max.bytes=20000000

Consumer Configuration

Set the fetch.max.bytes in the consumer to a value larger than the largest expected message size.

Properties props = new Properties();
props.put("fetch.max.bytes", 20000000); // 20MB

Topic-Specific Configuration

Optionally, set the topic-specific limit:

kafka-configs.sh --alter --bootstrap-server <broker-address> \
  --entity-type topics --entity-name <topic-name> \
  --add-config max.message.bytes=20000000

2. Split Large Messages into Smaller Chunks

For very large messages, splitting the payload into smaller chunks and reassembling them on the consumer side is a better practice. Kafka is optimized for small, high-throughput messages, so handling large payloads this way improves performance.

• Producer Side: Split the payload into smaller parts and send them with a unique key or identifier.
• Consumer Side: Reassemble the chunks based on the unique key or identifier.

Example:

1// Producer: Split and send chunks
2for (int i = 0; i < parts.size(); i++) {
3    ProducerRecord<String, String> record = new ProducerRecord<>(
4        "topic", key, parts.get(i)
5    );
6    producer.send(record);
7}
8
9// Consumer: Reassemble chunks
10// Group messages by key and reassemble them.

3. Use an External Storage for Large Payloads

Instead of sending the entire payload through Kafka, you can store the large message in an external storage system (e.g., Amazon S3, HDFS) and send a reference or URL through Kafka.

Producer Side

1. Upload the large payload to external storage.
2. Send the URL or reference in the Kafka message.

Example:

1ProducerRecord<String, String> record = new ProducerRecord<>(
2    "topic", key, "s3://bucket/key-for-large-payload"
3);
4producer.send(record);

Consumer Side

1. Read the reference or URL from Kafka.
2. Fetch the payload from the external storage.

4. Optimize Compression

Enable compression in Kafka to reduce the size of messages before they are sent. Supported compression codecs include:

• gzip
• snappy
• lz4
• zstd (Kafka 2.1 and above)

Producer Configuration:

Properties props = new Properties();
props.put("compression.type", "gzip"); // Or snappy, lz4, zstd

Compression can significantly reduce the size of text-based or highly compressible payloads.

5. Monitor and Tune Performance

When dealing with large messages, ensure that Kafka’s performance and resource utilization remain optimal:

1. Monitor Broker Memory Usage: Large messages increase memory usage due to buffering.
2. Adjust replica.fetch.max.bytes: Ensure that the replica brokers can handle large messages by setting replica.fetch.max.bytes in server.properties to a value larger than message.max.bytes.
3. Increase Network Buffering: Adjust the network buffer size to accommodate larger messages:

 socket.request.max.bytes=20000000

Summary

Recommendation

For messages slightly larger than 15MB, increasing Kafka's size limits is sufficient. However, for significantly larger payloads, splitting messages or using external storage with Kafka as a metadata transport is the most robust and efficient solution.