Update kubernetes secrets doesn't update running container env vars
Master System Design with Codemia
Enhance your system design skills with over 120 practice problems, detailed solutions, and hands-on exercises.
Introduction
When a Kubernetes Secret is injected into a container as environment variables, those values are read only when the container starts. Updating the Secret object later does not mutate the already running process environment, which is why the pod keeps seeing the old value until it is restarted or replaced.
Why Secret-Based Environment Variables Stay Stale
Environment variables are part of a process startup contract. Kubernetes resolves the Secret reference during pod startup, then passes the resulting values to the container runtime. After that, the process inside the container owns its environment, and Kubernetes does not patch it in place.
A Deployment like this is valid:
If api-secret changes later, the container keeps the old API_KEY value because the shell process is already running.
Mounted Secret Volumes Behave Differently
This is where a lot of confusion comes from. Secret volumes and Secret-backed environment variables are not the same update model.
If you mount a Secret as files:
Kubernetes can update the mounted files when the Secret changes. Applications that read the file at runtime can therefore observe updated content without a full pod restart. That behavior does not apply to environment variables.
So the rule is:
- Secret as env var: fixed until container restart
- Secret as mounted file: can update on disk
Whether the application notices the new file content still depends on how the application reads configuration.
Restart the Pods to Pick Up New Env Vars
If the Secret is consumed through env or envFrom, the normal fix is a rollout restart.
That causes new pods to start, and those new pods read the updated Secret value during container startup.
You can also force a new ReplicaSet by changing a pod template annotation:
The annotation value itself is not special. It just changes the pod template hash so Kubernetes creates new pods.
Use a Controller If You Want Automatic Restarts
In many teams, Secret rotation should automatically restart workloads that depend on it. Kubernetes does not do that automatically for env vars, so people often use a controller such as a reloader operator.
The workflow becomes:
- Secret changes.
- The reloader detects the change.
- The controller patches the Deployment.
- Kubernetes rolls the pods.
That gives you predictable behavior without asking every operator to remember a manual restart.
Prefer Mounted Files for Live-Reloadable Applications
If your application can reload config at runtime, a mounted Secret file is often a better fit than env vars. For example, an app can watch a file, reread credentials periodically, or reload TLS material on signal.
That pattern avoids unnecessary pod restarts, but it only works when the application is designed to reread files rather than cache the value once.
Common Pitfalls
The biggest mistake is assuming that because Secret volumes refresh, Secret-backed environment variables must refresh too. Another common issue is updating the Secret successfully but forgetting to restart the workload, which makes it look like Kubernetes ignored the change. Teams also sometimes move sensitive config into env vars when the application would actually be better served by reading mounted files. Finally, an automated reload controller helps only if the workload really references the Secret and the rollout policy is safe for that service.
Summary
- Secret values injected as environment variables are captured only at container startup.
- Updating the Secret does not mutate the environment of already running containers.
- Secret volumes can refresh on disk, which is a different behavior from env vars.
- To pick up new env-var values, restart or roll the pods.
- If you need live reload, prefer mounted Secret files and application logic that rereads them.

