Calculating Container CPU Usage in Kubernetes with Prometheus

Monitoring CPU usage is crucial for ensuring efficient resource utilization and identifying performance bottlenecks in Kubernetes environments. Prometheus, an open-source monitoring solution, provides a robust framework to capture and analyze container metrics, including CPU usage. This article will guide you through understanding and calculating containers' CPU usage using Prometheus within a Kubernetes cluster.

Understanding Kubernetes and Prometheus Metrics

Kubernetes dynamically allocates resources like CPU and memory based on the needs of running containers. Each node in a Kubernetes cluster has a kubelet agent that exposes metrics through a REST API. These metrics can be consumed by monitoring systems such as Prometheus for detailed analysis.

Prometheus is designed to collect metrics from various locations, store them efficiently, and make them available for querying. It uses a powerful query language called PromQL and supports custom alerts to notify operations teams of potential issues.

Key Metrics for Calculating CPU Usage

To accurately calculate CPU usage, Prometheus needs to be configured to scrape the appropriate metrics from Kubernetes nodes. Two key metrics can be used:

1. container_cpu_usage_seconds_total: The total CPU usage time measured in seconds for each container.
2. container_cpu_cfs_quota and container_cpu_cfs_period: Metrics useful for understanding the CPU constraints applied to the container via cgroups (control groups).

Steps to Calculate CPU Usage in Prometheus

1. Install Prometheus in Kubernetes: Use Helm or any preferred method to deploy Prometheus. Ensure it's configured to scrape metrics from the kubelet endpoints.
2. Configure Metric Collection: Confirm that the necessary metrics are enabled on the kubelet. The kubelet typically exposes these metrics on port 10250.
3. Write PromQL Query: Use Prometheus Query Language (PromQL) to calculate CPU usage.

   rate(container_cpu_usage_seconds_total{namespace="your-namespace", pod="your-pod-name"}[1m])

This query computes the per-second average rate of CPU usage over the last minute for the specified pod in the namespace your-namespace.

1. Normalize CPU Usage: Sometimes, analysis might require normalization of CPU usage. You can divide the usage rate by the number of available CPU cores to get a percentage usage:

   100 * sum(rate(container_cpu_usage_seconds_total{namespace="your-namespace", pod="your-pod-name"}[1m])) 
   / sum(machine_cpu_cores{})

1. Visualize and Alert: Use the Prometheus UI, Grafana (which integrates well with Prometheus), or any other tool to visualize these metrics. Set up alerts to notify you when CPU usage crosses a predefined threshold.

Example Prometheus Configuration

Below is an example of a prometheus.yml configuration file snippet to scrape metrics from kubelets:

1scrape_configs:
2  - job_name: 'kubernetes-nodes'
3    scheme: https
4    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
5    tls_config:
6      insecure_skip_verify: true
7    kubernetes_sd_configs:
8      - role: node
9    relabel_configs:
10      - action: labelmap
11        regex: '__meta_kubernetes_node_label_(.+)'

Summary Table

Additional Considerations

• Scaling Prometheus: As your cluster grows, consider scaling Prometheus or using Thanos for long-term storage and high availability.
• Security: Secure Prometheus endpoints with HTTPS and authentication to protect sensitive metrics.
• Resource Optimization: Use insights from CPU usage analysis to optimize resource allocations by updating Kubernetes resource requests and limits for different workloads.

Final Thoughts

By leveraging Prometheus' powerful monitoring capabilities, Kubernetes administrators can gain deep insights into their containers' CPU usage. This not only helps in maintaining the performance and reliability of applications but also assists in efficient resource management and cost optimization.