Kafka Producer Performance Tuning: The Four Knobs That Actually Matter

When someone tells me their Kafka producer is slow, my first question is never "how many partitions?" It is "what config did you change from the defaults?" The defaults are conservative. They ship for a laptop running an integration test, not for a pipeline pushing a million events a second.

Four settings carry almost all the weight. Learn these before you touch anything else.

batch.size is the per-partition buffer size, default 16KB. That is tiny. A high-throughput pipeline wants 128KB or 256KB so each ProduceRequest ships a real payload instead of a handful of records. Headers, syscalls, and network round trips all amortize against batch size. Small batches mean overhead dominates the wire time.

linger.ms is how long the producer waits to fill a batch before sending. Default is 0, meaning ship as soon as you have anything. Set it to 10 or 20 and the producer accumulates records during that window. You pay a few milliseconds of latency. You get fewer, bigger requests, which means higher throughput and lighter broker load.

compression.type defaults to none. Flip it to snappy or zstd and your network traffic roughly halves on typical JSON or Avro payloads. Compression happens on the producer, the broker stores compressed batches as is, and replication to followers ships less data. The cost is CPU on the producer, which is almost always free relative to network.

acks is the durability dial. acks=1 waits for the leader only. acks=all waits for all in-sync replicas. The latency difference is real but smaller than people think. The throughput difference matters mostly when your batches are tiny, which loops back to batch.size.

Idempotent producers (enable.idempotence=true) are free and on by default in newer clients. Turn them on if you somehow have them off.

A team I worked with ran Spark Streaming into Kafka with stock defaults. Throughput capped at 50K messages per second per producer and they were about to triple the partition count to scale out. The fix was four lines: batch.size=131072, linger.ms=20, compression.type=zstd, acks=all (which they needed anyway). Throughput jumped to 400K messages per second on the same partitions. The cluster was never the bottleneck.

One nuance worth knowing. Batching is per partition, not per producer. If your key cardinality scatters records across 200 partitions, each partition's buffer fills slowly and linger.ms ends up gating most sends. More partitions can actually hurt throughput at low volumes for this reason. The batch size you configure is a ceiling, the linger time is the floor, and the real batch is whichever fills first.

Change settings before you change topology. Adding partitions is expensive and only reversible with pain. Editing four producer configs is a config push, a redeploy, and a Grafana dashboard refresh.