Kafka
Confluent
ProducerConfig
Docker Images
Max.request.size

Kafka/Confluent ProducerConfig change default max.request.size using docker images

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Introduction

max.request.size is a Kafka producer client setting, not a broker image default that you "change" globally just by running a Confluent Docker image. If your producer runs inside a container, you configure that producer process the same way you would outside Docker: through application code, a client properties file, or container environment passed into the producer application.

What max.request.size Actually Controls

According to the current Kafka and Confluent producer configuration docs, max.request.size is the maximum request size in bytes that the producer will send. Its default is 1048576, which is 1 MiB.

That means this setting belongs to the producer client. It does not reconfigure every producer in your cluster just because your broker happens to run in a Confluent container.

Docker Image Versus Producer Application

This distinction is the part people often miss:

  • 'confluentinc/cp-kafka is a broker image'
  • 'max.request.size is a producer client config'

So if your Java, Python, Go, or console producer runs somewhere else, changing broker container environment variables will not magically change the producer's limit.

Instead, configure the producer where it is created.

Set It In Application Code

For a Java producer:

java
1Properties props = new Properties();
2props.put("bootstrap.servers", "broker:9092");
3props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
4props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
5props.put("max.request.size", 2097152);
6
7KafkaProducer<String, String> producer = new KafkaProducer<>(props);

If that producer runs inside Docker, the code is still the place where the setting actually takes effect.

Or Mount A Producer Config File

For containerized command-line producers or custom scripts, mount a properties file:

properties
bootstrap.servers=broker:9092
max.request.size=2097152

Then pass it to the producer process:

bash
1kafka-console-producer \
2  --bootstrap-server broker:9092 \
3  --topic demo \
4  --producer.config /config/producer.properties

If you are running that command in a container, bind-mount or bake the file into the image.

Docker Compose Example For A Producer Container

Here is a simple Compose example where the producer app receives the setting through an environment variable and maps it into application code:

yaml
1services:
2  producer-app:
3    build: .
4    environment:
5      BOOTSTRAP_SERVERS: broker:9092
6      KAFKA_MAX_REQUEST_SIZE: 2097152

Then the application reads it:

java
String maxRequestSize = System.getenv().getOrDefault("KAFKA_MAX_REQUEST_SIZE", "1048576");
props.put("max.request.size", maxRequestSize);

That is the Docker-specific part: Docker passes configuration into the producer process. Docker is not redefining Kafka's producer defaults by itself.

Remember The Broker Has Its Own Limits

Even if the producer allows larger requests, the broker may still reject them if broker-side message size limits are smaller. So tuning only the producer is not always enough.

In practice, large-message workflows often require coordinated changes across:

  • producer request size
  • broker message-size limits
  • consumer fetch-size limits

If you increase only one side, the system still fails at another boundary.

Common Pitfalls

One common mistake is trying to set max.request.size as if it were a broker environment variable on the cp-kafka image and expecting all clients to inherit it.

Another issue is increasing the producer limit without checking broker and consumer size limits, which leads to confusing partial fixes.

A third problem is using very large messages instead of rethinking the payload design. Kafka works better with reasonably sized records than with huge blobs.

Finally, some teams confuse a Docker container with the Kafka client itself. The container is only the runtime environment; the client config still belongs to the producer application.

Summary

  • 'max.request.size is a producer client setting, not a global broker image default.'
  • Configure it in producer code, a producer properties file, or producer container environment.
  • Running Kafka in Docker does not automatically change client defaults.
  • Large-message workflows usually require matching broker and consumer size settings too.
  • If message sizes keep growing, revisit payload design instead of only raising limits.

Course illustration
Course illustration

All Rights Reserved.