Is there a way to monitor kube cron jobs using prometheus
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
Yes, you can monitor Kubernetes CronJobs with Prometheus, and the usual foundation is kube-state-metrics. It exposes CronJob and Job state from the Kubernetes API, which lets Prometheus answer questions such as "Was this CronJob scheduled?", "Is it suspended?", "Does it currently have active runs?", and "Did the Jobs it created fail?"
What Prometheus can and cannot tell you
There are two levels of monitoring for CronJobs:
- Controller-level monitoring: did Kubernetes schedule and run the Job objects correctly?
- Workload-level monitoring: did the actual batch task succeed logically?
kube-state-metrics is great for the first category. It exposes the state of CronJobs, Jobs, and Pods. It does not know whether your backup file was valid or whether your ETL produced correct business results. For that, the job itself should emit an application-level success signal.
Start with kube-state-metrics
In most Kubernetes monitoring stacks, Prometheus already scrapes kube-state-metrics. If not, add it and make sure Prometheus discovers its /metrics endpoint.
A minimal scrape config looks like this:
Once that is being scraped, CronJob-related metrics are available for PromQL queries and alert rules.
Useful CronJob and Job metrics
Some of the most useful series are:
- '
kube_cronjob_status_last_schedule_time' - '
kube_cronjob_next_schedule_time' - '
kube_cronjob_status_active' - '
kube_cronjob_spec_suspend' - '
kube_job_status_succeeded' - '
kube_job_status_failed' - '
kube_job_owner'
The CronJob metrics tell you about the scheduler/controller state. The Job metrics tell you about the concrete Jobs created by the CronJob. kube_job_owner is particularly helpful because it lets you join Jobs back to the owning CronJob.
Example PromQL queries
Find CronJobs that are currently suspended:
Find CronJobs that still have active Jobs:
A simple overdue-schedule style alert can be approximated with:
The extra 300 seconds is a grace window. Tune it to your environment because controller latency, clock skew, and job startup time all matter.
To find failed Jobs owned by CronJobs:
That gives you failed Jobs and carries over the owning CronJob name through the owner_name label.
Example alert rule
A simple alert for CronJob-backed Job failures might look like this:
That is a good start, but production monitoring usually adds job-specific thresholds and routing rules.
For short-lived jobs, add an application success signal
CronJobs are short-lived, so relying only on pod scraping can be unreliable. A batch job may start and finish between Prometheus scrapes. That is why controller-state metrics from kube-state-metrics are useful.
But they still do not tell you whether the job's business logic succeeded. For important batch pipelines, add a metric such as job_last_success_unixtime or job_records_processed_total from the job itself and publish it somewhere durable. Some teams use Pushgateway for this pattern, but it should be used deliberately and cleaned up carefully.
Common Pitfalls
The biggest mistake is expecting Prometheus to infer business success from Kubernetes state alone. A Job can complete successfully while still producing bad output.
Another issue is alerting on schedule timing without a per-job grace window. Some CronJobs run hourly, some daily, and some have controller or startup delays, so one universal threshold creates noise.
Developers also forget to join Job metrics back to CronJobs. Without kube_job_owner, Job failures can be hard to attribute in alerts.
Finally, label names can vary slightly across kube-state-metrics versions, so validate your exact metric output in /metrics before finalizing rules.
Summary
- Yes, Prometheus can monitor CronJobs, usually through
kube-state-metrics. - Use CronJob metrics for scheduling state and Job metrics for concrete execution outcomes.
- Join Job metrics back to CronJobs with
kube_job_owner. - Add application-level success metrics for important batch jobs because Kubernetes state is not the same as business success.
- Build alerts with realistic grace windows and verify metric names against your installed kube-state-metrics version.
Related reading
- Is there a way to share a configMap in kubernetes between namespaces?
- Is there a way to share secrets across namespaces in Kubernetes?
- is there a way to solve Kubeconfig user entry is using deprecated API version client.authentication.k8s.io/v1alpha1 with pulumi
- Is there a way to two containers with same app connect to same DB on Kubernetes?
- Is there a way to run initial SQL when creating an RDS database instance using CloudFormation?
- Is there a way to suppress the messages TensorFlow prints?
- Is there a way to use Kubernetes LeaderElection across multiple clusters?
- Is there a way to view the Kubernetes image download progress during pod initialization?

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.