Helm upgrade doesn't pull new container
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
When a helm upgrade appears successful but pods keep running the old image, the problem is usually not Helm itself. Helm updates Kubernetes objects, but Kubernetes only creates a new pod when the rendered deployment changes in a way that affects the pod template. Image tags, pull policy, and rollout triggers all matter.
Understand What Helm Actually Updates
Helm renders templates into Kubernetes manifests and submits them to the cluster. If the rendered Deployment pod template is effectively unchanged, Kubernetes may not start new pods.
Typical example:
If you push a new image with the same tag and run helm upgrade without changing the manifest meaningfully, nodes may continue using the cached image.
Prefer Immutable Image Tags
The best fix is to stop reusing mutable tags such as latest for deployable releases.
Use versioned or commit-based tags:
Then deploy with:
When the tag changes, the pod template changes, and Kubernetes performs a rollout.
Know When imagePullPolicy Matters
imagePullPolicy affects whether the kubelet tries to fetch the image when a pod starts.
Common values:
- '
IfNotPresentpulls only if image is not already cached' - '
Alwaysasks the runtime to pull on each pod start'
If you insist on mutable tags, Always reduces the chance of stale images:
But this is still weaker than immutable tagging because rollout correctness now depends on node pull behavior and registry state.
Force a Deployment Rollout
If the image tag stayed the same and you need pods recreated, you must change the pod template. One common pattern is updating an annotation with a timestamp or release value.
Chart snippet:
Because the pod template annotation changes on each upgrade, Kubernetes creates new pods even if the image tag is unchanged.
This is useful for config changes too, but it should not replace good image versioning.
Verify the Rendered Deployment
Before blaming the cluster, inspect what Helm actually rendered.
Then check the live deployment:
Compare:
- image repository and tag
- image pull policy
- pod template annotations
If the rendered pod template did not change, no rollout should be expected.
Watch the Actual Image on Running Pods
After upgrade, confirm what image the pods are using.
Look at the exact image reference and, if possible, the pulled image digest. This tells you whether the node actually refreshed the image or simply reused cache.
CI and Release Pipeline Strategy
The reliable operational pattern is:
- build image once
- tag it immutably
- push to registry
- pass that exact tag into Helm
This removes ambiguity from deployments and makes rollback straightforward. If your pipeline uses mutable tags, you are pushing complexity into runtime behavior instead of release engineering.
Common Pitfalls
- Reusing
latestand expecting Kubernetes to notice new registry content automatically. - Assuming
helm upgradealone forces new pod creation. - Using
IfNotPresentwith mutable image tags. - Not checking whether the rendered pod template actually changed.
- Debugging Helm when the real issue is image versioning policy.
Summary
- Helm updates manifests; Kubernetes only rolls pods when the pod template changes.
- Immutable image tags are the safest fix for stale image problems.
- '
imagePullPolicy: Alwayshelps with mutable tags but is not the best long-term strategy.' - A rollout annotation can force pod recreation when needed.
- Verify rendered manifests and running pod image references before troubleshooting further.
Related reading
- Helm UPGRADE FAILED cannot patch ... with kind Job, by update field image
- Helm V3 - Cannot find the official repo
- Helm Variables inside ConfigMap File
- Helm/Kube Error query failed to query with labels stream error
- hostPath as volume in kubernetes
- How are intermediate containers formed?
- Heroku deploying Deep Learning model
- Heroku tensorflow 2.2.1 too large for deployment

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.