Introduction

Pod labels are not automatically copied onto every Prometheus metric sample. To make them available in queries, you usually transfer selected Kubernetes discovery labels into Prometheus target labels during scrape relabeling, or configure the Prometheus Operator to copy specific pod labels into scraped targets.

Understand Where the Labels Come From

When Prometheus discovers Kubernetes Pods, it gets temporary metadata labels such as:

• '__meta_kubernetes_pod_name'
• '__meta_kubernetes_namespace'
• '__meta_kubernetes_pod_label_app'
• '__meta_kubernetes_pod_label_environment'

These discovery labels exist during target relabeling. If you want them to survive into the scraped time series, you must map them to normal labels such as app or environment.

Add Pod Labels with relabel_configs

If you run raw Prometheus with kubernetes_sd_configs, add relabel rules to the scrape job:

1scrape_configs:
2  - job_name: kubernetes-pods
3    kubernetes_sd_configs:
4      - role: pod
5    relabel_configs:
6      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
7        action: keep
8        regex: "true"
9      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
10        target_label: __address__
11        regex: "(.+)"
12        replacement: "$1"
13      - source_labels: [__meta_kubernetes_pod_label_app]
14        target_label: app
15      - source_labels: [__meta_kubernetes_pod_label_environment]
16        target_label: environment

After this, every sample scraped from matching Pod targets carries app and environment labels. That makes queries like these possible:

sum by (app, environment) (rate(http_requests_total[5m]))

The important detail is that the labels are attached to the target before scraping, not added afterward inside PromQL.

Import Several Labels at Once

If you want multiple pod labels and they follow a predictable naming rule, labelmap is convenient:

relabel_configs:
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(app|environment|team)

This copies:

• '__meta_kubernetes_pod_label_app to app'
• '__meta_kubernetes_pod_label_environment to environment'
• '__meta_kubernetes_pod_label_team to team'

This is less repetitive than writing a separate rule for every label. Still, be selective. Copying too many labels increases cardinality, which can make Prometheus slower and more expensive to run.

Using the Prometheus Operator

If your cluster uses the Prometheus Operator, you often do this through a PodMonitor or ServiceMonitor instead of editing raw prometheus.yml.

For a PodMonitor, use podTargetLabels:

1apiVersion: monitoring.coreos.com/v1
2kind: PodMonitor
3metadata:
4  name: app-monitor
5spec:
6  selector:
7    matchLabels:
8      app: my-app
9  podMetricsEndpoints:
10    - port: metrics
11  podTargetLabels:
12    - app
13    - environment

This tells the operator to propagate those pod labels onto the scraped targets. It is a cleaner approach when you already manage monitoring declaratively through Kubernetes resources.

Choose Labels Carefully

Not every pod label belongs in Prometheus. Good candidates are low-cardinality dimensions such as:

• app name
• environment
• team
• region

Bad candidates include request ids, commit hashes for every deployment, or any label with extremely high churn. Those explode the number of time series and degrade query performance.

As a rule, if the label is useful for aggregation and does not change constantly, it is usually safe. If it identifies nearly every Pod uniquely, think twice before importing it.

Common Pitfalls

The most common mistake is trying to add pod labels in PromQL after scraping. If the label was not preserved during target relabeling, it is already gone from the sample data.

Another issue is confusing relabel_configs with metric_relabel_configs. Target relabeling happens before scraping and is usually the correct place to copy Kubernetes discovery labels. Metric relabeling happens after samples are scraped and is more expensive.

Cardinality is the biggest operational risk. Importing every pod label by default can produce too many unique series, especially in large clusters.

Finally, make sure the label actually exists on the Pod. If __meta_kubernetes_pod_label_environment is empty because the workload has no environment label, Prometheus cannot invent it for you.

Summary

• Kubernetes pod labels appear first as temporary discovery labels in Prometheus.
• Use relabel_configs or operator settings such as podTargetLabels to preserve selected labels.
• Copy only the labels that are useful for aggregation and low in cardinality.
• Prefer target relabeling over metric relabeling for this use case.
• Verify the source pod labels exist before debugging the Prometheus side.