Kubernetes
deployment
initialization
single-execution
DevOps

Kubernetes - deployment initialization - how to ensure it happens only once?

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

In Kubernetes, “run this only once” is trickier than it sounds because a Deployment manages replicas, not one-time workflows. If an initialization step truly must happen a single time for a rollout or for the whole cluster, an init container is usually the wrong tool.

Why Init Containers Do Not Solve This by Themselves

Init containers run once per pod, not once per deployment. That is a useful guarantee, but it is not the guarantee most people actually need.

If your deployment has three replicas, the init container runs three times. If a pod is rescheduled, it runs again. If the deployment is rolled out later, it runs again for each new pod. That behavior is correct for per-pod setup such as waiting for a dependency, copying a config file, or preparing a writable volume.

It is not correct for global actions such as:

  • database schema migration
  • seed-data insertion
  • creating a shared bucket or topic
  • registering a singleton external resource

For those cases, you need a different Kubernetes primitive.

Use a Job for True One-Time Initialization

A Job is the normal way to model a finite task that should run to completion. You can apply the job before or alongside the deployment and make the application depend on the completed outcome.

yaml
1apiVersion: batch/v1
2kind: Job
3metadata:
4  name: app-migration
5spec:
6  backoffLimit: 3
7  template:
8    spec:
9      restartPolicy: Never
10      containers:
11        - name: migrate
12          image: myorg/myapp:1.0.0
13          command: ["python", "manage.py", "migrate"]

This gives Kubernetes a resource whose job is exactly “finish once successfully.” It is also easier to monitor because kubectl get jobs and kubectl logs job/app-migration map directly to the operational question.

Make the Initialization Idempotent Anyway

Even with a Job, you should still design the initialization to be idempotent. Kubernetes provides orchestration, not magical exactly-once execution in the distributed-systems sense.

For example, a migration command should be safe to re-run. If a job pod crashes after partially completing work, Kubernetes may start it again. If the cluster operator deletes and reapplies the job, it runs again. Idempotent design turns those situations from incidents into routine retries.

A simple database-backed lock or migration table is often more reliable than trying to encode every guarantee in YAML alone.

Coordinating the Application With the Job

There are a few common patterns for making sure the main application does not start before initialization is done.

The cleanest pattern is deployment ordering from your delivery system. Run the job, wait for success, then roll out the deployment.

If you need the deployment manifest itself to wait, an init container can poll for a migration marker, but note the separation of concerns: the init container is no longer doing the one-time action. It is only waiting for the result of a separate job.

yaml
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: web
5spec:
6  replicas: 2
7  selector:
8    matchLabels:
9      app: web
10  template:
11    metadata:
12      labels:
13        app: web
14    spec:
15      initContainers:
16        - name: wait-for-migration
17          image: bitnami/kubectl:latest
18          command:
19            - sh
20            - -c
21            - kubectl wait --for=condition=complete job/app-migration --timeout=300s
22      containers:
23        - name: web
24          image: myorg/myapp:1.0.0

Operationally, many teams prefer to avoid kubectl inside application pods and instead let CI, Helm hooks, or a GitOps controller manage the ordering. That is often simpler to audit.

When You Really Mean “One Leader Should Do It”

Sometimes the task should not happen once per rollout, but once at runtime by whichever pod becomes leader. In that case, leader election or an external lock may be the right design.

Examples include warming a shared cache or periodically syncing reference data. Those are not deployment initialization tasks; they are distributed coordination tasks. Trying to force them into init containers usually creates fragile startup behavior.

Common Pitfalls

The most common mistake is assuming init containers run once per deployment. They do not. They run once per pod lifecycle.

Another problem is putting non-idempotent database setup in a deployment startup path. If the pod restarts, the code runs again and may duplicate records or fail due to existing state.

Teams also sometimes hide one-time logic inside the main application entrypoint. That makes startup slower, harder to observe, and more dangerous during scaling events.

Finally, avoid depending on perfect ordering alone. Even if a job usually runs first, the initialization logic should still tolerate retries and existing state.

Summary

  • Init containers run once per pod, not once per deployment.
  • Use a Job for true one-time initialization such as migrations or seed tasks.
  • Keep the initialization idempotent because retries and reapplications still happen.
  • Let your deployment pipeline or a waiting init container coordinate application startup after the job completes.
  • Use leader election or external locks only for runtime coordination problems, not as a default substitute for jobs.

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.