Kafka exposes several hundred JMX metrics and almost all dashboards show the wrong dozen. The useful set is small, and the reason it is small is that most Kafka metrics describe activity rather than health, and activity tells you nothing about whether the cluster is in trouble.

The one metric that is always wrong when something is wrong

UnderReplicatedPartitions, on every broker, under kafka.server:type=ReplicaManager.

It counts partitions on this broker whose ISR is smaller than the full replica set. In a healthy cluster it is 0 on every broker, permanently, and it is the single highest-signal number Kafka publishes.

A non-zero value means one of three things: a broker is down, a broker is alive but failing to keep up with replication, or a disk is failing. All three are serious and all three are invisible in throughput metrics, because a cluster with under-replicated partitions serves traffic perfectly well right up until the moment it loses another replica.

The alert should fire on any non-zero value sustained for a few minutes, not on a threshold. There is no acceptable steady-state level. A brief spike during a rolling restart is expected; a persistent 40 is a cluster running without the durability it believes it has.

Its companion is UnderMinIsrPartitionCount, which counts partitions whose ISR has dropped below min.insync.replicas. Those partitions are already rejecting acks=all produces with NotEnoughReplicasException, so this metric is not a warning, it is an outage in progress.

The controller metrics, two of which are binary

Three, and the first two should never need a graph.

ActiveControllerCount is 1 on exactly one broker and 0 on every other, so the cluster-wide sum must be exactly 1. A sum of 0 means no controller and nothing can create a topic or elect a leader; a sum of 2 means a split brain, which KRaft makes far harder but which is catastrophic if it happens.

OfflinePartitionsCount is 0 in a healthy cluster. A non-zero value means partitions with no leader at all, which are fully unavailable for both produce and consume. It is the metric that corresponds to user-visible failure most directly, and unlike under-replication it is never acceptable even briefly.

LeaderElectionRateAndTimeMs is the rate of leader elections and should be near zero in steady state. A sustained non-zero rate without any broker restarts means leadership is moving on its own, which usually means brokers are being marked dead and recovering, which usually means a network or GC problem rather than a Kafka problem.

UncleanLeaderElectionsPerSec should be 0 and should be alerted on at any value, because every occurrence is silent data loss. With unclean.leader.election.enable at its default of false it cannot be non-zero, which is a good reason to verify that setting rather than assume it.

The throughput and latency metrics worth keeping

Four, and the latency breakdown is the one people do not know exists.

BytesInPerSec and BytesOutPerSec per topic and per broker are the capacity-tracking numbers, and BytesOutPerSec is the one to watch against the NIC because egress is the saturating direction.

RequestsPerSec, separated by request type, exposes the small-request problem: a Fetch rate far above the Produce rate without a correspondingly high consumer count means consumers polling an empty topic with fetch.min.bytes at its default of 1.

TotalTimeMs per request type is the end-to-end request latency, and it decomposes into five components that Kafka publishes separately: RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs and ResponseSendTimeMs. That decomposition is the most useful diagnostic in Kafka and is almost never dashboarded.

Each component names a different problem. Queue time high means the io thread pool is saturated. Local time high means disk or lock contention on the broker. Remote time high on a produce request means waiting for followers, so it is a replication problem rather than a broker problem. Response send time high means the client is slow to read or the network is congested. Reading the breakdown turns "produce is slow" into one of four distinct investigations.

Request handler and network idle, the saturation signals

Two ratios, both of which should sit near 1.0.

RequestHandlerAvgIdlePercent is the fraction of time the num.io.threads pool spends idle, and NetworkProcessorAvgIdlePercent the same for num.network.threads. Both are between 0 and 1 despite the name.

Below about 0.3 means the pool is the bottleneck, and which one it is tells you which setting to raise. Both are dynamic broker configs, so the change is a kafka-configs.sh --alter rather than a restart.

The reason these matter more than CPU utilisation is that a saturated thread pool does not necessarily show as high CPU: a request handler blocked on disk is busy from the pool's perspective and idle from the CPU's. A broker at 40 percent CPU with an idle ratio of 0.1 is fully saturated, which reads as a contradiction on a dashboard that only has CPU.

PurgatorySize for Produce and Fetch is the related number: it counts requests parked waiting for a condition, so a large produce purgatory means many requests waiting on replication acknowledgement, which points back at RemoteTimeMs and at the followers.

The client metrics nobody collects

Four, and their absence is why so many Kafka incidents are investigated from the wrong end.

Client metrics live in the application process and have to be collected from there, so they are missing from any monitoring setup built by scraping brokers. The result is a blind spot covering the half of the system where most problems originate.

On the producer, record-error-rate is the number of records that failed permanently after retries were exhausted, and it is the only metric that distinguishes data loss from latency. A producer with a non-zero error rate is dropping records, and nothing broker-side reports that. buffer-available-bytes falling toward zero means the accumulator is filling because the broker is not acknowledging fast enough, which is the leading indicator of the 60-second stall described later. And record-queue-time-avg is how long a record waits in the accumulator before being sent, which separates broker slowness from linger.ms doing its job.

On the consumer, rebalance-rate-per-hour is the one metric that identifies a rebalance storm from the client side, and commit-latency-avg rising points at the group coordinator rather than at the data brokers, which is a distinction brokers' aggregate latency hides completely.

Collecting these is awkward because it means instrumenting application processes rather than infrastructure, and it is the highest-value monitoring work available on most Kafka platforms for exactly that reason: nobody has done it.

Getting the metrics out, which is its own problem

Because the collection mechanism shapes which metrics you can actually have.

Kafka publishes everything through JMX, and JMX is a poor fit for a modern metrics pipeline. The standard answer is the JMX Exporter running as a Java agent inside the broker process, started with -javaagent:jmx_prometheus_javaagent.jar=7071:config.yaml, which scrapes the MBeans and serves them as Prometheus text.

The configuration file is where the real decision lives, because the exporter's default behaviour is to expose everything, and everything on a broker with 1,200 partition replicas is tens of thousands of time series. A whitelist is not optional, and the pattern most operators converge on is cluster and broker level metrics unconditionally plus topic level metrics for topics that matter, with per-partition series excluded by default.

The cost of getting that wrong is concrete. An unfiltered exporter on a dense broker produces a scrape that takes seconds, which both delays the scrape and burns broker CPU inside the JMX reflection path, and the metrics backend charges for every series. A cluster whose monitoring costs more than its brokers is a real and common outcome.

The second plumbing fact is that metric names differ between the broker and the clients. Broker metrics are MBeans like kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions; client metrics are a different namespace entirely, kafka.producer:type=producer-metrics,client-id=..., and they have to be collected from the application process rather than from the broker. Teams that collect only broker metrics have no visibility into the client side at all, which is where a large share of incidents originate.

What to leave out

Briefly, because dashboard noise has a real cost.

MessagesInPerSec is a record count and is less useful than bytes, because capacity is bounded by bytes and request rate rather than by records. It is worth having for the ratio it gives against bytes, which is the average record size.

IsrShrinksPerSec and IsrExpandsPerSec matter as a pair rather than individually: a steady equal rate of both is a cluster where followers repeatedly fall behind and catch up, which is a replication capacity problem, and shrinks without expands is a broker that is simply gone.

Per-partition metrics should be collected but not dashboarded by default, because a cluster with 7,000 partition replicas produces a volume of time series that costs real money in a metrics backend and is only ever read during an investigation.