Understanding GKE Cluster Node Pool Autoscaling Issues

Google Kubernetes Engine (GKE) is known for its powerful and dynamic autoscaling capabilities, which help manage workloads efficiently by scaling the number of nodes in a cluster as per demand. However, users might occasionally encounter situations where the autoscaling feature does not operate as expected. In this article, we delve into possible reasons and troubleshoot scenarios when a GKE cluster node pool suddenly stops autoscaling.

How GKE Autoscaling Works

Before diving into issues, it's critical to understand how GKE's Cluster Autoscaler operates. GKE autoscaling can adjust the size of the cluster, typically based on the following:

• Pod Resource Requests: Scales nodes based on the resource requests of pods that cannot be scheduled due to insufficient resources.
• Predefined Min/Max Nodes: Adheres to the minimum and maximum node limits defined in the node pool configuration.
• Scale Up/Down Events: Automatically adds nodes when resources are scarce and removes nodes when they are underutilized.

Possible Causes for Autoscaling Failure

1. Insufficient Resource Requests:
   • Pods that do not specify resource requests might not trigger scaling. It's crucial that CPU and memory requests are appropriately defined.
2. Imbalanced Resource Requests:
   • If requests are skewed, the scheduler might struggle to place pods efficiently even if the autoscaler adds nodes.
3. Node Pool Configuration Issues:
   • The min and max node limits might have been configured inaccurately.
4. Pod Affinity/Anti-Affinity Rules:
   • Strict rules can prevent pods from being scheduled on newly scaled nodes.
5. Resource Quotas and Limits:
   • The project or cluster could be hitting set quotas, preventing further scaling.
6. Temporary API Issues:
   • GCP might occasionally experience API service issues affecting autoscaler operations.
7. Autoscaler Stabilization:
   • Autoscaler has built-in mechanisms to avoid flapping which might delay scaling.

Troubleshooting Steps

1. Review Autoscaler Logs:
   • Use kubectl logs to review the autoscale controller manager logs to identify underlying issues.
2. Check PodResource Requests:
   • Verify that pods have appropriate CPU and memory requests set using kubectl describe pod .
3. Inspect Node Pool Configurations:
   • Confirm that min/max settings in the node pool configuration align with expected scaling behavior.
4. Evaluate Cluster Resource Utilization:
   • Monitor usage with commands like kubectl top nodes/pods to determine if resources are accurately represented.
5. Review Quotas:
   • Check the GCP Console quotas section for the project to ensure that limits are not being breached.
6. Test with a Scale Deployment:
   • Manually scale a deployment to observe if new nodes are added appropriately.
7. Consult GCP Status Dashboard:
   • Investigate potential API issues or disruptions on the GCP Status Dashboard.

Example Scenario

Consider a scenario where a GKE cluster houses a critical application requiring occasional high bursts of processing capacity. The node pool expects to scale from 3 to a maximum of 10 nodes. However, upon increased load, no scaling occurs. Here's an illustration of possible investigations:

• Resource Requests: Ensure that pods defining the microservices have accurate resource requests set:
  • name: service
• Node Pool Settings: Verify the node pool configuration:
• Quotas: Use the following command to check relevant quotas: