Can not override replicas of deployment k8s

Kubernetes

Deployment

Replicas

Troubleshooting

DevOps

Can not override replicas of deployment k8s

Master System Design with Codemia

Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.

Start Practicing Learn More

Introduction

When you cannot override the replicas field of a Kubernetes Deployment, it is almost always because a Horizontal Pod Autoscaler (HPA) is managing the replica count, or because your manifest is being overwritten by a GitOps tool (ArgoCD, Flux). The HPA continuously adjusts replicas based on metrics, overriding any manual kubectl scale or manifest change within seconds. To fix this, either modify the HPA's minReplicas/maxReplicas or remove the HPA entirely before manually scaling.

The Problem

bash

1# You try to scale manually
2kubectl scale deployment my-app --replicas=5
3
4# It appears to work
5kubectl get deployment my-app
6# NAME     READY   UP-TO-DATE   AVAILABLE   AGE
7# my-app   5/5     5            5           2d
8
9# But seconds later, it reverts
10kubectl get deployment my-app
11# NAME     READY   UP-TO-DATE   AVAILABLE   AGE
12# my-app   3/3     3            3           2d

The replica count reverts because something else is controlling it.

Cause 1: Horizontal Pod Autoscaler (HPA)

bash

1# Check if an HPA targets your deployment
2kubectl get hpa
3# NAME     REFERENCE           TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
4# my-app   Deployment/my-app   45%/80%   2         10        3          5d

The HPA overrides the replicas field based on CPU/memory utilization. When you manually scale to 5, the HPA evaluates its metrics and scales back to what it calculates is needed.

Fix: Adjust the HPA

yaml

1# Update HPA min/max to include your desired count
2apiVersion: autoscaling/v2
3kind: HorizontalPodAutoscaler
4metadata:
5  name: my-app
6spec:
7  scaleTargetRef:
8    apiVersion: apps/v1
9    kind: Deployment
10    name: my-app
11  minReplicas: 5    # Set minimum to your desired count
12  maxReplicas: 20
13  metrics:
14    - type: Resource
15      resource:
16        name: cpu
17        target:
18          type: Utilization
19          averageUtilization: 80

bash

1# Apply the updated HPA
2kubectl apply -f hpa.yaml
3
4# Or patch it directly
5kubectl patch hpa my-app -p '{"spec":{"minReplicas":5}}'

Fix: Remove the HPA Temporarily

bash

1# Delete HPA to allow manual control
2kubectl delete hpa my-app
3
4# Now manual scaling works
5kubectl scale deployment my-app --replicas=5
6
7# Recreate HPA when ready
8kubectl apply -f hpa.yaml

Cause 2: GitOps Tool Overriding

If you use ArgoCD, Flux, or a similar GitOps tool, the tool continuously reconciles the cluster state with the Git repository. Manually changing replicas in the cluster is overwritten on the next sync.

bash

1# ArgoCD — check sync status
2argocd app get my-app
3
4# The manifest in Git has replicas: 3
5# ArgoCD syncs and reverts your manual change to 3

Fix: Change Replicas in Git

yaml

1# Update the deployment manifest in your Git repository
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5  name: my-app
6spec:
7  replicas: 5  # Change here, commit, and push

bash

1git add deployment.yaml
2git commit -m "Scale my-app to 5 replicas"
3git push
4
5# ArgoCD/Flux will sync the new replica count

Fix: Ignore Replicas in ArgoCD

yaml

1# ArgoCD Application — ignore replicas field during sync
2apiVersion: argoproj.io/v1alpha1
3kind: Application
4metadata:
5  name: my-app
6spec:
7  ignoreDifferences:
8    - group: apps
9      kind: Deployment
10      jsonPointers:
11        - /spec/replicas

This tells ArgoCD not to revert manual replica changes, allowing HPA or manual scaling to work alongside GitOps.

Cause 3: Kustomize or Helm Overriding

bash

1# Kustomize may set replicas in an overlay
2# kustomization.yaml
3replicas:
4  - name: my-app
5    count: 3
6
7# Helm values.yaml may override
8replicaCount: 3

If you deploy with Kustomize or Helm, changing the Deployment manifest directly is overwritten on the next kustomize build or helm upgrade. Change the value in the Kustomize overlay or Helm values file.

Cause 4: Resource Quotas or Limits

bash

1# Check if namespace has resource quotas
2kubectl get resourcequota -n my-namespace
3# NAME        AGE   REQUEST                     LIMIT
4# my-quota    30d   pods: 5/10, cpu: 2/4        cpu: 4/8
5
6# If you hit the pod limit, new replicas stay Pending
7kubectl get pods -n my-namespace | grep Pending

bash

1# Check events for scheduling failures
2kubectl describe deployment my-app
3# Events:
4#   Warning  FailedCreate  replicaset-controller
5#   Error creating: pods "my-app-xxx" is forbidden:
6#   exceeded quota: my-quota, requested: pods=1, used: pods=10, limited: pods=10

Verifying What Controls Replicas

bash

1# Check HPA
2kubectl get hpa --all-namespaces | grep my-app
3
4# Check if a VPA (Vertical Pod Autoscaler) exists
5kubectl get vpa --all-namespaces | grep my-app
6
7# Check deployment events
8kubectl describe deployment my-app | tail -20
9
10# Check who last modified the deployment
11kubectl get deployment my-app -o jsonpath='{.metadata.managedFields[*].manager}'
12# "argocd-controller" or "kubectl" or "kube-controller-manager"

The managedFields output shows which controller last modified the replicas field.

Common Pitfalls

Forgetting about HPA: The most common reason manual scaling does not stick. Always check kubectl get hpa before debugging replica issues. The HPA controller runs every 15 seconds by default and will quickly override manual changes.
Editing the Deployment but not the HPA: Setting replicas: 5 in the Deployment manifest while the HPA has minReplicas: 2, maxReplicas: 4 means the HPA immediately scales back to at most 4. The HPA's range takes precedence.
GitOps sync overwriting changes: ArgoCD and Flux continuously reconcile. Any kubectl change to a GitOps-managed resource is temporary. Always change the source (Git), not the live cluster.
Node resources insufficient: Scaling up replicas works only if the cluster has enough CPU/memory. If nodes are full, new pods stay Pending. Check kubectl describe pod <pending-pod> for scheduling failures.
KEDA or custom controllers: Advanced autoscalers like KEDA (Kubernetes Event-Driven Autoscaling) also manage replicas. Check for ScaledObject resources: kubectl get scaledobjects.

Summary

HPA is the most common reason manual replica changes revert — it continuously adjusts replicas based on metrics
Check for HPA: kubectl get hpa and verify minReplicas/maxReplicas
GitOps tools (ArgoCD, Flux) overwrite manual changes — update replicas in the Git source instead
Use ignoreDifferences in ArgoCD to allow manual or HPA-managed scaling
Check resource quotas and node capacity if new replicas stay in Pending state
Inspect managedFields to determine which controller is managing the replicas field