Introduction

RecordTooLargeException means the producer tried to send a record or request that exceeded one of Kafka's size limits. The fix is rarely "raise one setting and move on" because large-message problems usually involve producer limits, broker or topic limits, consumer fetch limits, and the broader question of whether the event should be that large in the first place.

Where Kafka Enforces Size Limits

A message can fail at several points along the path:

• the producer may reject the request because it exceeds max.request.size
• the broker or topic may reject the record because of message-size limits
• the consumer may later fail to fetch the message if its fetch limits are too low

That means large-message handling is a pipeline problem, not only a producer problem.

Reproducing the Error

Here is a simple Java example that can trigger the exception if the payload exceeds the current limits:

1import java.util.Properties;
2import java.util.concurrent.ExecutionException;
3import org.apache.kafka.clients.producer.KafkaProducer;
4import org.apache.kafka.clients.producer.ProducerConfig;
5import org.apache.kafka.clients.producer.ProducerRecord;
6import org.apache.kafka.common.errors.RecordTooLargeException;
7import org.apache.kafka.common.serialization.StringSerializer;
8
9public class Main {
10    public static void main(String[] args) throws Exception {
11        Properties props = new Properties();
12        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
13        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
14        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
15
16        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
17            String payload = "x".repeat(2 * 1024 * 1024);
18
19            try {
20                producer.send(new ProducerRecord<>("big-topic", payload)).get();
21            } catch (ExecutionException e) {
22                if (e.getCause() instanceof RecordTooLargeException) {
23                    System.out.println("Record exceeded Kafka size limits");
24                } else {
25                    throw e;
26                }
27            }
28        }
29    }
30}

The actual failing limit depends on the cluster configuration.

Producer-Side Settings

The first setting to inspect is often:

props.put(ProducerConfig.MAX_REQUEST_SIZE_CONFIG, 2_500_000);

This controls the maximum size of a request the producer will send. If the serialized record exceeds this value, the producer fails before the broker even gets involved.

But increasing only this setting is not enough if:

• the broker still rejects the record
• the topic limit is lower
• consumers cannot fetch the message later

Broker and Topic Limits

Kafka brokers also enforce message-size constraints. Depending on how the cluster is configured, a large record may need a higher broker-wide or topic-specific maximum message size.

The exact property names vary by where you apply them, but the important idea is simple: the producer and the broker must agree on what size is acceptable. Raising the producer ceiling alone only moves the failure downstream.

Consumer Fetch Limits Matter Too

A surprisingly common mistake is fixing publishing while forgetting the read side. Consumers may need larger fetch settings if they are expected to read the same large records successfully.

Typical consumer-side limits to consider include:

• 'max.partition.fetch.bytes'
• 'fetch.max.bytes'

If those stay too small, consumers can lag or fail even after the producer-side exception is gone.

Compression Can Help

If the payload is text-heavy or repetitive, compression may reduce the effective request size:

props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "gzip");

Compression is useful, but it is not a design substitute. If your event is fundamentally huge, compression may only postpone the problem.

Better Design Than Giant Kafka Records

Many systems should not send large blobs through Kafka at all. Common alternatives are:

• split the data into smaller events
• store the large object in external storage
• publish a reference or object key through Kafka

For example, instead of sending a 20 MB document through Kafka, upload it to object storage and publish a message containing:

1{
2  "documentId": "123",
3  "objectKey": "documents/123.pdf"
4}

That usually scales better and avoids pushing Kafka toward roles it was not designed to serve.

A Good Troubleshooting Sequence

When you hit RecordTooLargeException, use this order:

1. measure the serialized record size
2. check producer max.request.size
3. check broker or topic message-size limits
4. check consumer fetch settings
5. decide whether the message should be redesigned instead

That sequence prevents the common mistake of raising random limits without understanding which component actually failed.

Common Pitfalls

• Increasing only the producer limit and forgetting broker and consumer settings.
• Measuring the raw object size instead of the serialized record size that Kafka actually sends.
• Treating Kafka as a bulk blob transport when the message would be better stored elsewhere.
• Assuming compression removes the need for message-size discipline.
• Solving the exception without checking whether consumers can still fetch the resulting records.

Summary

• 'RecordTooLargeException means a Kafka record exceeded one of the system's size limits.'
• Producer, broker or topic, and consumer settings all have to align.
• 'max.request.size is important, but it is only one part of the path.'
• Compression can help, but redesigning the message is often better.
• Large-message support should be an intentional architecture choice, not a last-minute config patch.