Use Prometheus operator with DB volume for k8s

Prometheus

Kubernetes

Operators

Database

Monitoring

Use Prometheus operator with DB volume for k8s

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

By default, the Prometheus Operator deploys Prometheus with emptyDir storage, meaning all metrics data is lost when the pod restarts. To persist data across pod restarts and upgrades, you must configure a persistent volume (PV) through the storage spec in the Prometheus custom resource or through Helm values. This ensures Prometheus retains its time-series database (TSDB) on a durable disk, which is critical for production monitoring where you need historical data.

The Problem: Data Loss on Restart

bash

# Without persistent storage, restarting Prometheus loses all data
kubectl delete pod prometheus-prometheus-kube-prometheus-prometheus-0
# Pod recreates → all metrics gone → dashboards empty

Configuring Storage via Helm Values

kube-prometheus-stack (Most Common)

yaml

1# values.yaml for prometheus-community/kube-prometheus-stack
2prometheus:
3  prometheusSpec:
4    retention: 30d                      # Keep data for 30 days
5    retentionSize: "50GB"               # Or limit by size
6    storageSpec:
7      volumeClaimTemplate:
8        spec:
9          storageClassName: gp3          # AWS EBS gp3, or your storage class
10          accessModes: ["ReadWriteOnce"]
11          resources:
12            requests:
13              storage: 50Gi

bash

1helm install prometheus prometheus-community/kube-prometheus-stack \
2  --namespace monitoring \
3  --create-namespace \
4  -f values.yaml

Verify the PVC

bash

1kubectl get pvc -n monitoring
2# NAME                                                        STATUS   VOLUME    CAPACITY
3# prometheus-prometheus-kube-prometheus-prometheus-db-...      Bound    pvc-...   50Gi
4
5kubectl get pv
6# Shows the persistent volume bound to the PVC

Configuring Storage via Custom Resource

If you manage the Prometheus Operator directly (not via Helm):

yaml

1apiVersion: monitoring.coreos.com/v1
2kind: Prometheus
3metadata:
4  name: prometheus
5  namespace: monitoring
6spec:
7  replicas: 2
8  retention: 30d
9  retentionSize: 50GB
10  storage:
11    volumeClaimTemplate:
12      spec:
13        storageClassName: gp3
14        accessModes: ["ReadWriteOnce"]
15        resources:
16          requests:
17            storage: 50Gi
18  resources:
19    requests:
20      memory: 2Gi
21      cpu: 500m
22    limits:
23      memory: 4Gi
24      cpu: 2000m

bash

kubectl apply -f prometheus.yaml

Storage Classes by Cloud Provider

yaml

1# AWS EBS (gp3)
2storageClassName: gp3
3
4# AWS EBS (gp2)
5storageClassName: gp2
6
7# GCP Persistent Disk (SSD)
8storageClassName: premium-rw
9
10# Azure Managed Disk
11storageClassName: managed-premium
12
13# Local storage (bare metal)
14storageClassName: local-storage
15
16# Default (uses cluster default storage class)
17storageClassName: ""  # Or omit storageClassName entirely

bash

# List available storage classes
kubectl get storageclass

Sizing the Volume

Prometheus TSDB storage requirements depend on:

Ingestion rate: Number of time series × scrape interval
Retention period: How long you keep data
Bytes per sample: ~1-2 bytes per sample after compression

1Rough formula:
2Storage = ingestion_rate × bytes_per_sample × retention_seconds
3
4Example:
5100,000 active time series × 15s scrape interval
6= 6,666 samples/second
7= ~10 KB/s = ~0.86 GB/day
830 days retention = ~26 GB
9
10Add 20% overhead for WAL and compaction = ~32 GB
11Request 50Gi for safety margin

Expanding an Existing Volume

If you run out of space, expand the PVC (requires the storage class to support volume expansion):

bash

1# Check if storage class allows expansion
2kubectl get storageclass gp3 -o jsonpath='{.allowVolumeExpansion}'
3# true
4
5# Edit the PVC to increase size
6kubectl edit pvc prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 -n monitoring
7# Change storage: 50Gi → storage: 100Gi

Or update the Helm values and upgrade:

yaml

1prometheus:
2  prometheusSpec:
3    storageSpec:
4      volumeClaimTemplate:
5        spec:
6          resources:
7            requests:
8              storage: 100Gi  # Increased from 50Gi

bash

helm upgrade prometheus prometheus-community/kube-prometheus-stack -f values.yaml -n monitoring

Prometheus WAL and Compaction

yaml

1# Prometheus TSDB directory structure on the volume:
2# /prometheus/
3# ├── wal/           # Write-Ahead Log (buffered, not yet compacted)
4# ├── chunks_head/   # Recent data in memory-mapped chunks
5# ├── 01ABCDEF.../   # Compacted blocks (2-hour intervals)
6# └── lock           # TSDB lock file

yaml

1# Configure WAL compression to save disk space
2prometheus:
3  prometheusSpec:
4    walCompression: true  # Reduces WAL size by ~50%

High Availability with Persistent Storage

yaml

1prometheus:
2  prometheusSpec:
3    replicas: 2                          # Two Prometheus instances
4    storageSpec:
5      volumeClaimTemplate:
6        spec:
7          storageClassName: gp3
8          accessModes: ["ReadWriteOnce"]  # Each replica gets its own PVC
9          resources:
10            requests:
11              storage: 50Gi

Each replica gets its own PVC. With ReadWriteOnce, each volume is attached to one pod. Both replicas scrape the same targets independently, providing redundancy.

Thanos/Cortex for Long-Term Storage

For retention beyond what local disk can hold, use Thanos or Cortex to ship data to object storage:

yaml

1prometheus:
2  prometheusSpec:
3    thanos:
4      objectStorageConfig:
5        secret:
6          type: S3
7          config:
8            bucket: prometheus-thanos
9            endpoint: s3.amazonaws.com
10            region: us-east-1
11    retention: 2h        # Short local retention
12    storageSpec:
13      volumeClaimTemplate:
14        spec:
15          storage: 10Gi  # Small local disk — Thanos handles long-term

Common Pitfalls

Forgetting storageSpec entirely: Without storageSpec, the Prometheus Operator uses emptyDir. Every pod restart, rollout, or node drain deletes all metrics. Always configure persistent storage for production.
Storage class does not support ReadWriteOnce: Prometheus requires ReadWriteOnce access mode. Some NFS-based storage classes only support ReadWriteMany. Verify your storage class supports RWO before deploying.
Volume too small for retention period: If the volume fills up, Prometheus stops ingesting new data and may crash-loop. Monitor disk usage with prometheus_tsdb_storage_limit_bytes and set retentionSize as a safety cap.
Not enabling WAL compression: WAL files can consume significant disk space during high ingestion rates. Enable walCompression: true to reduce WAL size by approximately 50%.
PVC not deleted on Helm uninstall: Helm does not delete PVCs when uninstalling a release (by design, to prevent data loss). Manually delete PVCs with kubectl delete pvc -l app.kubernetes.io/name=prometheus -n monitoring when you intentionally want to remove data.

Summary

Configure storageSpec.volumeClaimTemplate in the Prometheus Operator to use persistent volumes
Size the volume based on ingestion rate, retention period, and a 20% overhead margin
Enable walCompression: true to reduce disk usage
Set both retention (time) and retentionSize (bytes) to prevent disk exhaustion
Each Prometheus replica gets its own PVC with ReadWriteOnce access mode
For long-term storage beyond local disk, use Thanos or Cortex with object storage (S3, GCS)