There are three ways to get numbers out of RabbitMQ, they do not return the same numbers, and the differences decide what a dashboard can honestly say.

The three interfaces

A CLI, an HTTP API and a Prometheus endpoint, in the order they arrived.

rabbitmqctl and rabbitmq-diagnostics talk to the node over the Erlang distribution protocol, which means they need the right cookie and a reachable distribution port, and they return a snapshot computed when asked. They are the right tool for an investigation and the wrong tool for a scrape, because each invocation starts a short-lived Erlang VM and the startup cost is tens to hundreds of milliseconds.

The management plugin's HTTP API, on port 15672, serves the same data the management UI shows, along with rates computed by the plugin itself. The rabbitmq_prometheus plugin, on port 15692, serves an OpenMetrics endpoint and is the path the documentation now recommends for monitoring. The important difference is that the management API computes rates and the Prometheus endpoint exposes counters, so with Prometheus the rate calculation happens in the monitoring system where its window is explicit.

Why the management API is the wrong scrape target

An aggregation cost that scales with object count.

A request to /api/queues returns every queue with every field, and producing it means the node fans out to every other node in the cluster, collects per-queue state, and merges it. At a few hundred queues that is unremarkable; at tens of thousands it is seconds of work, and a monitoring system scraping it every fifteen seconds is a sustained load on the same node that serves messages.

The usual symptoms are a management UI that times out, scrapes that intermittently fail, and a node whose CPU includes a periodic spike nobody can attribute. The first mitigation is ?columns=, which restricts the response to the fields actually wanted and is the single biggest saving, since most of the cost is in assembling fields no dashboard reads. Requesting name,messages_ready,messages_unacknowledged,consumers instead of everything changes the shape of the problem.

The structural fix is to scrape the Prometheus endpoint instead, which by default returns metrics aggregated across objects rather than per object, so its cost does not grow with queue count. Per-object metrics are available from /metrics/per-object or by enabling them globally, and that choice is the one real decision in setting it up: aggregate metrics are cheap and cannot tell you which queue is backed up, per-object metrics can and reintroduce a cost proportional to the object count.

What the management plugin's rates actually measure

A sampling interval that is easy to forget and changes the meaning of every rate on the dashboard.

The management plugin collects samples on an interval, collect_statistics_interval, defaulting to five seconds, and retains them for a set of configured periods. A rate shown in the UI is computed across those samples, so a spike shorter than the interval can be invisible and a rate shown over a long retention window is an average that smooths the thing you were looking for.

This matters most when a dashboard and an alert disagree. The management UI's one-minute publish rate and a Prometheus rate() over five minutes are different quantities, and a monitoring setup that mixes both will produce incidents where two sources disagree about whether traffic stopped. Choosing one source for alerting and treating the other as an investigation tool resolves it; using both as equals does not.

The second consequence is cost. Statistics collection is work the node does whether or not anyone looks, and the plugin's retention policies hold samples in memory. On a broker with many thousands of channels and queues, reducing retention or turning fine-grained collection down is a legitimate tuning step, and the setting that controls how much is collected at all is collect_statistics, whose none, coarse and fine values trade per-channel and per-queue detail against overhead.

What each interface can and cannot see

Three scopes, and the mistake of assuming any one of them is cluster-wide.

A rabbitmqctl command runs against one node and most of its subcommands return cluster-wide results by contacting the others, which is why list_queues on a three-node cluster is slower than it looks and why it fails in an interesting way during a partition: it returns what the reachable nodes know, which is a partial answer presented as a complete one.

The Prometheus endpoint is the opposite. It is strictly local, reporting only what this node knows, which means the scrape has to hit every node and the aggregation happens in the monitoring system. That is the correct design for a scrape and it has one consequence people trip over: a queue's metrics come from the node hosting it, so a dashboard that scrapes only one node sees only that node's queues, and in a cluster where queues are spread deliberately that is a fraction of the picture.

The management API sits in the middle, fanning out per request, which is both its convenience and the reason it is expensive. The practical rule is that the CLI and the management API answer "what is true of the cluster", the Prometheus endpoint answers "what is true of this node", and a monitoring setup has to scrape all nodes and know which metrics are node-scoped and which are object-scoped before any dashboard can be trusted.

The plugins, and what enabling them costs

Two plugins, both off by default, with different overheads.

rabbitmq_management brings the UI, the HTTP API and the statistics collection that feeds them. rabbitmq_prometheus brings the metrics endpoint and is cheap, since it reads state that already exists rather than maintaining a database of samples. Enabling Prometheus and leaving management enabled for its UI and its health-check endpoints is the common arrangement, and it is a reasonable one.

The cost asymmetry is worth being precise about. The Prometheus plugin's work happens when scraped and is proportional to the metrics requested. The management plugin's work happens continuously, on its collection interval, whether or not anybody reads the result, and the samples it retains live in memory on the node. The management plugin is the only one of the two that costs something while nobody is looking, which is why a broker with memory pressure and tens of thousands of channels is a broker where the statistics database is worth examining.

There is a third option worth knowing about for large clusters: running the management plugin's UI without statistics collection, which keeps the API and health checks while dropping the rates. That is a legitimate configuration for a cluster where Prometheus owns the metrics and the UI exists only for occasional human use, and it removes an overhead that otherwise scales with the object count.

Per-object metrics and the cardinality decision

The question to settle before building dashboards, because it is expensive to change later.

With per-object metrics enabled, every queue contributes a time series per metric, and the monitoring system's cardinality is the queue count times the metric count times the label set. A broker with 20,000 queues and twenty per-queue metrics is 400,000 series from one broker, which is a real cost in the monitoring system rather than in RabbitMQ.

The usual compromise is aggregate metrics for alerting and per-object metrics for a small, named set of queues that matter. That works because most alerts are about the broker's health and most investigations are about one queue, so the cheap signal can carry the alert and the expensive signal can be scoped. Where the queue count is small, a few hundred, per-object metrics everywhere is simpler and the cardinality is irrelevant.

What does not work is relying on aggregate metrics and expecting to diagnose a per-queue problem during an incident. The aggregate rabbitmq_queue_messages_ready across all queues rising says a backlog exists somewhere, and the queue-per-class designs from the previous lesson exist precisely so that a per-class depth is readable; discovering at three in the morning that the dashboard cannot name the queue is the common way this decision gets revisited.