Kubernetes
cronjob
private registry
docker image
troubleshooting

Pulling an Image from Private Registry in Kubernetes cronjob fails

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

When a Kubernetes CronJob fails to pull an image from a private container registry, the pod enters an ImagePullBackOff or ErrImagePull state. The root cause is almost always a missing or misconfigured imagePullSecrets on the CronJob's pod template. Unlike Deployments, CronJobs create new pods on each schedule tick, so every pod needs valid credentials. The fix involves creating a Docker registry secret and referencing it in the CronJob spec — or attaching it to the ServiceAccount so all pods in the namespace inherit it automatically.

The Error

When you describe the failed pod, you see:

bash
1kubectl describe pod my-cronjob-28456789-abc12
2
3# Events:
4#   Warning  Failed   pull image "registry.example.com/my-app:latest":
5#     rpc error: code = Unknown desc = failed to pull and unpack image:
6#     failed to resolve reference: pulling from host registry.example.com
7#     failed with status 401: Unauthorized

The pod status shows ImagePullBackOff, meaning Kubernetes tried to pull the image, got a 401 Unauthorized, and is now backing off before retrying.

Fix 1: Add imagePullSecrets to the CronJob

Create a Docker registry secret and reference it in the CronJob's pod template:

bash
1# Create the registry secret
2kubectl create secret docker-registry myregistrykey \
3  --docker-server=registry.example.com \
4  --docker-username=myuser \
5  --docker-password=mypassword \
6  --docker-email=[email protected] \
7  -n my-namespace
yaml
1apiVersion: batch/v1
2kind: CronJob
3metadata:
4  name: my-cronjob
5  namespace: my-namespace
6spec:
7  schedule: "0 */6 * * *"
8  jobTemplate:
9    spec:
10      template:
11        spec:
12          imagePullSecrets:
13            - name: myregistrykey
14          containers:
15            - name: my-container
16              image: registry.example.com/my-app:latest
17          restartPolicy: OnFailure

The imagePullSecrets field must be at the pod spec level (spec.jobTemplate.spec.template.spec), not at the container level.

Fix 2: Attach Secret to ServiceAccount

Instead of adding imagePullSecrets to every CronJob, attach the secret to the default ServiceAccount. Every pod in the namespace then inherits it:

bash
1# Patch the default service account
2kubectl patch serviceaccount default \
3  -n my-namespace \
4  -p '{"imagePullSecrets": [{"name": "myregistrykey"}]}'
bash
1# Verify
2kubectl get serviceaccount default -n my-namespace -o yaml
3# imagePullSecrets:
4# - name: myregistrykey

Now any CronJob in that namespace pulls private images without needing explicit imagePullSecrets in the spec.

Fix 3: ECR, GCR, and ACR-Specific Solutions

Cloud registries have their own authentication mechanisms:

yaml
1# AWS ECR — use a CronJob to refresh the ECR token every 6 hours
2# because ECR tokens expire after 12 hours
3apiVersion: batch/v1
4kind: CronJob
5metadata:
6  name: ecr-token-refresh
7spec:
8  schedule: "0 */6 * * *"
9  jobTemplate:
10    spec:
11      template:
12        spec:
13          serviceAccountName: ecr-refresher
14          containers:
15            - name: refresh
16              image: amazon/aws-cli
17              command:
18                - /bin/sh
19                - -c
20                - |
21                  TOKEN=$(aws ecr get-login-password --region us-east-1)
22                  kubectl delete secret ecr-secret --ignore-not-found
23                  kubectl create secret docker-registry ecr-secret \
24                    --docker-server=123456789.dkr.ecr.us-east-1.amazonaws.com \
25                    --docker-username=AWS \
26                    --docker-password=$TOKEN
27          restartPolicy: OnFailure

For GKE with Google Container Registry, configure Workload Identity. For AKS with Azure Container Registry, use az aks update --attach-acr.

Debugging Steps

bash
1# Check pod events
2kubectl describe pod <pod-name> -n my-namespace
3
4# Verify the secret exists
5kubectl get secret myregistrykey -n my-namespace
6
7# Decode and inspect the secret
8kubectl get secret myregistrykey -n my-namespace -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d
9
10# Test pulling the image manually
11docker login registry.example.com
12docker pull registry.example.com/my-app:latest
13
14# Check if the CronJob has imagePullSecrets
15kubectl get cronjob my-cronjob -n my-namespace -o jsonpath='{.spec.jobTemplate.spec.template.spec.imagePullSecrets}'

Common Pitfalls

  • Wrong namespace: The secret must be in the same namespace as the CronJob. Secrets are namespace-scoped and cannot be shared across namespaces without tools like Sealed Secrets or External Secrets.
  • Expired credentials: Cloud registry tokens (ECR, GCR) expire. A static secret with an expired token causes recurring failures. Use a token refresh CronJob or workload identity.
  • Typo in secret name: imagePullSecrets references the secret by name. A typo means Kubernetes silently ignores the missing secret and fails with 401.
  • imagePullSecrets at wrong level: Placing imagePullSecrets under spec.jobTemplate.spec instead of spec.jobTemplate.spec.template.spec has no effect. It must be in the pod template spec.
  • Private registry with self-signed cert: Even with correct credentials, a self-signed TLS certificate causes x509: certificate signed by unknown authority. Configure the node's container runtime to trust the CA.

Summary

  • CronJob image pull failures from private registries are caused by missing imagePullSecrets
  • Create a docker-registry secret and reference it in the CronJob's pod template spec
  • Attach the secret to the ServiceAccount to avoid repeating it in every workload
  • Cloud registries (ECR, GCR, ACR) need token refresh or workload identity because credentials expire
  • Always verify the secret exists in the correct namespace and the name matches exactly

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.