Introduction

If a Prometheus pod stays in Pending after a Helm install, the issue is almost never "Prometheus is broken." Pending means the pod has not been scheduled or cannot start its required volumes yet, so the fix is to read the scheduler and storage signals first rather than guessing at chart settings.

Start With the Pod Events

The fastest way to stop guessing is to inspect the pod description:

kubectl -n monitoring get pods
kubectl -n monitoring describe pod prometheus-server-0

For kube-prometheus-stack, the pod name may look more like prometheus-kube-prometheus-stack-prometheus-0, but the debugging method is the same. The Events section usually tells you the real reason:

• 'Insufficient cpu'
• 'Insufficient memory'
• 'node(s) had untolerated taint'
• 'persistentvolumeclaim is not bound'
• 'node(s) didn't match Pod's node affinity'

Do not skip this step. Pending is a scheduler and volume state, and Kubernetes usually explains it directly.

Common Cause 1: Requests Are Too Large for the Cluster

Prometheus often requests more CPU, memory, or storage than small clusters can provide, especially when charts use production-oriented defaults.

Check the requests:

kubectl -n monitoring get pod prometheus-server-0 -o yaml | grep -A5 resources:
kubectl describe nodes

If no node can satisfy the requested resources, the pod remains pending forever. In that case, lower the requests in your Helm values or add larger nodes.

Example values override:

1prometheus:
2  prometheusSpec:
3    resources:
4      requests:
5        cpu: 250m
6        memory: 512Mi
7      limits:
8        cpu: 1
9        memory: 1Gi

This is common on Rancher-managed development clusters, where worker nodes may be small and default Prometheus charts assume more capacity than is actually available.

Common Cause 2: PVC or Storage Class Problems

Prometheus is often deployed as a StatefulSet, which means it can depend on a persistent volume claim before the pod becomes fully schedulable.

Check the claims:

kubectl -n monitoring get pvc
kubectl -n monitoring describe pvc

If the PVC is stuck in Pending, the Prometheus pod usually will be too. Common reasons include:

• no default StorageClass
• requested size cannot be provisioned
• the chart refers to a storage class name that does not exist
• the provisioner is unhealthy

A values override can make the storage dependency explicit:

1prometheus:
2  prometheusSpec:
3    storageSpec:
4      volumeClaimTemplate:
5        spec:
6          storageClassName: local-path
7          accessModes:
8            - ReadWriteOnce
9          resources:
10            requests:
11              storage: 20Gi

If your cluster has no working dynamic provisioner, you either need to configure one or switch the chart to a storage mode your cluster can satisfy.

Common Cause 3: Taints, Affinity, and Rancher Node Layout

Rancher clusters often have control-plane nodes or specialized worker pools with taints. If Prometheus has no matching tolerations, the scheduler will refuse those nodes.

Inspect taints:

kubectl describe nodes | grep -A3 Taints

If the event says untolerated taint, either schedule Prometheus on normal worker nodes or add matching tolerations intentionally:

1prometheus:
2  prometheusSpec:
3    tolerations:
4      - key: "node-role.kubernetes.io/control-plane"
5        operator: "Exists"
6        effect: "NoSchedule"

Be careful with this. Adding tolerations blindly can move a monitoring workload onto nodes you intentionally reserved. The right fix may be removing an unnecessary node selector rather than broadening tolerations.

Affinity settings can cause the same symptom. If the chart or your values file includes a node selector or hard affinity rules that match no nodes, the pod remains pending even when the cluster has enough raw capacity.

Common Cause 4: Project Quotas and Namespace Limits

On Rancher, cluster and project-level resource governance can interact with Helm chart defaults. If the namespace or project has quotas that are lower than Prometheus requests, scheduling may fail or resources may be rejected indirectly.

Inspect:

• namespace LimitRange
• namespace ResourceQuota
• Rancher project resource limits

If your Prometheus chart requests more than the project allows, the fix is either to lower requests or raise the allowed quota. This is easy to miss because users often focus on node capacity and forget the namespace policy layer.

Common Pitfalls

The biggest mistake is troubleshooting the wrong problem. Pending is not a container crash, so application logs rarely help until scheduling or storage is fixed.

Another common mistake is assuming the issue is Rancher-specific when the actual cause is a normal Kubernetes constraint such as an unbound PVC or insufficient memory. Teams also often patch tolerations first because it feels quick, but the real blocker may be a missing storage class or a node selector that matches nothing.

Finally, avoid relying on older PodSecurityPolicy advice. Current clusters are much more likely to block Prometheus at scheduling, quota, taint, affinity, or storage time than through deprecated PSP-era behavior.

Summary

• Read kubectl describe pod first and trust the scheduler events.
• Check resource requests against real node capacity.
• If Prometheus uses persistent storage, verify the PVC and StorageClass are binding correctly.
• Inspect taints, tolerations, node selectors, and affinity rules, especially on Rancher-managed node pools.
• Review namespace or project quotas when cluster capacity looks fine but the pod still stays pending.