Practice Now
Prometheus
Kubernetes
Operators
Database
Monitoring

Use Prometheus operator with DB volume for k8s

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design
KubernetesDatabasesDevOps & Deployment

Introduction

By default, the Prometheus Operator deploys Prometheus with emptyDir storage, meaning all metrics data is lost when the pod restarts. To persist data across pod restarts and upgrades, you must configure a persistent volume (PV) through the storage spec in the Prometheus custom resource or through Helm values. This ensures Prometheus retains its time-series database (TSDB) on a durable disk, which is critical for production monitoring where you need historical data.

The Problem: Data Loss on Restart

bash
# Without persistent storage, restarting Prometheus loses all data
kubectl delete pod prometheus-prometheus-kube-prometheus-prometheus-0
# Pod recreates → all metrics gone → dashboards empty

Configuring Storage via Helm Values

kube-prometheus-stack (Most Common)

yaml
1# values.yaml for prometheus-community/kube-prometheus-stack
2prometheus:
3  prometheusSpec:
4    retention: 30d                      # Keep data for 30 days
5    retentionSize: "50GB"               # Or limit by size
6    storageSpec:
7      volumeClaimTemplate:
8        spec:
9          storageClassName: gp3          # AWS EBS gp3, or your storage class
10          accessModes: ["ReadWriteOnce"]
11          resources:
12            requests:
13              storage: 50Gi
bash
1helm install prometheus prometheus-community/kube-prometheus-stack \
2  --namespace monitoring \
3  --create-namespace \
4  -f values.yaml

Verify the PVC

bash
1kubectl get pvc -n monitoring
2# NAME                                                        STATUS   VOLUME    CAPACITY
3# prometheus-prometheus-kube-prometheus-prometheus-db-...      Bound    pvc-...   50Gi
4
5kubectl get pv
6# Shows the persistent volume bound to the PVC

Configuring Storage via Custom Resource

If you manage the Prometheus Operator directly (not via Helm):

yaml
1apiVersion: monitoring.coreos.com/v1
2kind: Prometheus
3metadata:
4  name: prometheus
5  namespace: monitoring
6spec:
7  replicas: 2
8  retention: 30d
9  retentionSize: 50GB
10  storage:
11    volumeClaimTemplate:
12      spec:
13        storageClassName: gp3
14        accessModes: ["ReadWriteOnce"]
15        resources:
16          requests:
17            storage: 50Gi
18  resources:
19    requests:
20      memory: 2Gi
21      cpu: 500m
22    limits:
23      memory: 4Gi
24      cpu: 2000m
bash
kubectl apply -f prometheus.yaml

Storage Classes by Cloud Provider

yaml
1# AWS EBS (gp3)
2storageClassName: gp3
3
4# AWS EBS (gp2)
5storageClassName: gp2
6
7# GCP Persistent Disk (SSD)
8storageClassName: premium-rw
9
10# Azure Managed Disk
11storageClassName: managed-premium
12
13# Local storage (bare metal)
14storageClassName: local-storage
15
16# Default (uses cluster default storage class)
17storageClassName: ""  # Or omit storageClassName entirely
bash
# List available storage classes
kubectl get storageclass

Sizing the Volume

Prometheus TSDB storage requirements depend on:

  • Ingestion rate: Number of time series × scrape interval
  • Retention period: How long you keep data
  • Bytes per sample: ~1-2 bytes per sample after compression
 
1Rough formula:
2Storage = ingestion_rate × bytes_per_sample × retention_seconds
3
4Example:
5100,000 active time series × 15s scrape interval
6= 6,666 samples/second
7= ~10 KB/s = ~0.86 GB/day
830 days retention = ~26 GB
9
10Add 20% overhead for WAL and compaction = ~32 GB
11Request 50Gi for safety margin

Expanding an Existing Volume

If you run out of space, expand the PVC (requires the storage class to support volume expansion):

bash
1# Check if storage class allows expansion
2kubectl get storageclass gp3 -o jsonpath='{.allowVolumeExpansion}'
3# true
4
5# Edit the PVC to increase size
6kubectl edit pvc prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 -n monitoring
7# Change storage: 50Gi → storage: 100Gi

Or update the Helm values and upgrade:

yaml
1prometheus:
2  prometheusSpec:
3    storageSpec:
4      volumeClaimTemplate:
5        spec:
6          resources:
7            requests:
8              storage: 100Gi  # Increased from 50Gi
bash
helm upgrade prometheus prometheus-community/kube-prometheus-stack -f values.yaml -n monitoring

Prometheus WAL and Compaction

yaml
1# Prometheus TSDB directory structure on the volume:
2# /prometheus/
3# ├── wal/           # Write-Ahead Log (buffered, not yet compacted)
4# ├── chunks_head/   # Recent data in memory-mapped chunks
5# ├── 01ABCDEF.../   # Compacted blocks (2-hour intervals)
6# └── lock           # TSDB lock file
yaml
1# Configure WAL compression to save disk space
2prometheus:
3  prometheusSpec:
4    walCompression: true  # Reduces WAL size by ~50%

High Availability with Persistent Storage

yaml
1prometheus:
2  prometheusSpec:
3    replicas: 2                          # Two Prometheus instances
4    storageSpec:
5      volumeClaimTemplate:
6        spec:
7          storageClassName: gp3
8          accessModes: ["ReadWriteOnce"]  # Each replica gets its own PVC
9          resources:
10            requests:
11              storage: 50Gi

Each replica gets its own PVC. With ReadWriteOnce, each volume is attached to one pod. Both replicas scrape the same targets independently, providing redundancy.

Thanos/Cortex for Long-Term Storage

For retention beyond what local disk can hold, use Thanos or Cortex to ship data to object storage:

yaml
1prometheus:
2  prometheusSpec:
3    thanos:
4      objectStorageConfig:
5        secret:
6          type: S3
7          config:
8            bucket: prometheus-thanos
9            endpoint: s3.amazonaws.com
10            region: us-east-1
11    retention: 2h        # Short local retention
12    storageSpec:
13      volumeClaimTemplate:
14        spec:
15          storage: 10Gi  # Small local disk — Thanos handles long-term

Common Pitfalls

  • Forgetting storageSpec entirely: Without storageSpec, the Prometheus Operator uses emptyDir. Every pod restart, rollout, or node drain deletes all metrics. Always configure persistent storage for production.
  • Storage class does not support ReadWriteOnce: Prometheus requires ReadWriteOnce access mode. Some NFS-based storage classes only support ReadWriteMany. Verify your storage class supports RWO before deploying.
  • Volume too small for retention period: If the volume fills up, Prometheus stops ingesting new data and may crash-loop. Monitor disk usage with prometheus_tsdb_storage_limit_bytes and set retentionSize as a safety cap.
  • Not enabling WAL compression: WAL files can consume significant disk space during high ingestion rates. Enable walCompression: true to reduce WAL size by approximately 50%.
  • PVC not deleted on Helm uninstall: Helm does not delete PVCs when uninstalling a release (by design, to prevent data loss). Manually delete PVCs with kubectl delete pvc -l app.kubernetes.io/name=prometheus -n monitoring when you intentionally want to remove data.

Summary

  • Configure storageSpec.volumeClaimTemplate in the Prometheus Operator to use persistent volumes
  • Size the volume based on ingestion rate, retention period, and a 20% overhead margin
  • Enable walCompression: true to reduce disk usage
  • Set both retention (time) and retentionSize (bytes) to prevent disk exhaustion
  • Each Prometheus replica gets its own PVC with ReadWriteOnce access mode
  • For long-term storage beyond local disk, use Thanos or Cortex with object storage (S3, GCS)
Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.