producer.send(record) returns before the record has left the process. Everything interesting about Kafka producer behaviour, throughput, latency, ordering and loss, follows from what happens between that return and the broker's acknowledgement, and almost none of it is visible from the call site.

The pipeline a record travels

Six stages, and naming them makes every configuration setting locatable.

The serializer converts key and value to bytes, using key.serializer and value.serializer. This happens on the calling thread, so an expensive serializer is latency the application pays directly.

The partitioner chooses a partition, which is the next lesson's subject.

The record accumulator appends the serialized record into a per-partition batch held in a memory pool sized by buffer.memory, 33,554,432 by default. This is where send returns from.

The sender thread, a single background thread, drains ready batches, groups them by destination broker into one ProduceRequest per broker, and writes them to the network.

The broker appends and, depending on acks, responds immediately or after replication.

The completion runs the callback and completes the returned Future, on the sender thread.

The one sender thread, and what it means for callbacks

There is exactly one sender thread per producer instance, and every callback for every partition runs on it.

A callback that blocks, on a database write, a lock, a synchronous HTTP call, stops the sender thread, which stops all sending for every partition until it returns. This is the commonest cause of a producer that mysteriously stalls under load, and it is invisible in the application's own metrics because the application's threads are idle and happy.

So the rule is absolute: callbacks do metric increments, logging and handing work to another executor, and nothing else. Anything that can block goes on a different thread.

The same thread does the network I/O, so a producer sending to many brokers at high volume can also become sender-thread bound on serialization of request headers and compression, which is one reason very high throughput applications run multiple producer instances rather than one.

Batching, and the two settings that control it

Batching is where Kafka's producer throughput comes from, and the two settings are easy to describe and easy to get wrong.

batch.size, 16,384 bytes, is the size at which a partition's batch is considered full and immediately ready to send. It is a per-partition byte limit, not a record count, and it is also the allocation unit in the buffer pool, which matters for memory.

linger.ms, 0 by default, is how long a non-full batch waits before being sent anyway. At 0 the sender sends whatever is available as soon as it is free, which means under light load every record is its own batch.

The interaction is the point. At linger.ms 0 and low throughput, batching does nothing and the producer makes one request per record. At linger.ms 5 to 20, the producer accumulates for a few milliseconds, which at any real throughput fills batches and multiplies effective throughput, at the cost of a few milliseconds of added latency per record.

The counterintuitive part: raising linger.ms often reduces end-to-end latency at high throughput, because fewer, larger requests mean less queueing at the broker. Lingering costs latency only when the producer is not busy.

buffer.memory and the block that looks like a hang

buffer.memory is the total memory the accumulator may hold across all partitions, and exhausting it is one of the few ways send itself blocks.

When the buffer is full, send blocks for up to max.block.ms, 60,000, waiting for space, then throws TimeoutException. It also blocks on the same budget waiting for metadata when sending to a topic it has not seen.

A sixty-second block inside what the application believes is an asynchronous call is the behaviour most likely to be mistaken for a deadlock, and the fix is to set max.block.ms to something the application can actually tolerate and handle the exception, rather than leaving a minute of hidden blocking in the hot path.

Buffer exhaustion means the producer is accepting records faster than the sender can deliver them, so the real causes are a slow or unavailable broker, a blocked callback, or genuine over-production. The buffer is a shock absorber, not capacity.

The fan-out arithmetic nobody does

One piece of arithmetic resolves a recurring surprise about memory, and it is worth doing before setting either value.

A producer writing to N partitions holds up to one batch.size allocation per partition it is actively writing to. At the default 16,384 bytes, a producer writing to 2,000 partitions wants 32.8 MB, which is already essentially all of the default buffer.memory.

A producer writing to 24,000 partitions wants 393 MB against a default of 33.5 MB, so it will never form full batches: each partition gets a fraction of an allocation, batches are sent small, and throughput collapses while the producer looks correctly configured.

The fix is either fewer partitions per producer, a larger buffer.memory, or accepting small batches with linger.ms raised to compensate. The diagnostic is the ratio of records to requests, available as batch-size-avg and records-per-request-avg in the producer's own metrics, and a batch-size-avg far below batch.size under load is this problem.

What the sender actually sends, and why requests are per broker

The grouping step is easy to skip over and it explains a behaviour that otherwise looks strange.

The sender does not send one request per partition. It collects every ready batch whose partition leader is the same broker and sends them as a single ProduceRequest to that broker. So a producer writing to 300 partitions spread over 6 brokers sends at most 6 requests per round, not 300.

Which is why the broker's request rate is so much lower than the partition count suggests, and why adding partitions to an existing topic does not multiply request load the way it multiplies memory pressure in the accumulator.

It also means a single slow broker holds up only its own request. The other five complete normally, so a degraded broker presents as elevated latency on a subset of partitions rather than a uniform slowdown, and that asymmetry is the signal that distinguishes one bad broker from a cluster-wide problem.

The request's total size is bounded separately by max.request.size, 1,048,576, which applies to the assembled request rather than to a batch, so a producer with many large batches destined for one broker can hit it even though no individual batch is close.

Metadata, and the refresh nobody configures

The producer needs to know which broker leads each partition, and the way it learns and relearns that is behind several confusing behaviours.

On startup it contacts a bootstrap.servers entry and fetches cluster metadata, which it then caches. metadata.max.age.ms, 300,000, is how long before it refreshes proactively, which is a long time and almost never the mechanism that matters.

The mechanism that matters is reactive: when a request fails with NOT_LEADER_OR_FOLLOWER, the producer marks its metadata stale and refreshes immediately, then retries. So during a leader election the producer discovers the new leader in one round trip rather than waiting five minutes, and the five-minute setting exists for changes nobody told it about, such as a new partition appearing.

The first send to an unseen topic blocks on this fetch, against the max.block.ms budget, which is why the very first record a service produces has latency in the tens of milliseconds while every subsequent one is microseconds. A service with a per-request timeout tight enough to be broken by that should warm the producer at startup rather than widen the timeout.

One consequence worth knowing: auto.create.topics.enable on the broker, true in many clusters, means a typo in a topic name creates a topic rather than failing, with the broker's default partition count and no replication tuning. The producer's metadata fetch is what triggers that creation, and the result is a quietly misconfigured topic that works.