Introduction

Running a private registry inside Kubernetes is common for air-gapped clusters and controlled software supply chains, but image pull failures can come from several layers at once. DNS, TLS, authentication, and runtime trust settings all affect whether pods can download images. A structured troubleshooting flow is the fastest way to resolve these errors.

Core Sections

Typical Failure Symptoms

You usually see one of these event messages:

• 'ImagePullBackOff'
• 'ErrImagePull'
• unauthorized or authentication required
• x509 certificate error
• no such host or connection refused

Start by reading pod events first:

kubectl describe pod APP_POD -n APP_NAMESPACE
kubectl get events -n APP_NAMESPACE --sort-by=.lastTimestamp

The exact error string points to the right layer.

Verify Registry Service Reachability

If registry runs in-cluster, verify service and endpoint health:

kubectl get svc -n registry
kubectl get endpoints -n registry
kubectl get pods -n registry -o wide

From a debug pod, test DNS and HTTP response:

kubectl run net-debug -n default --image=busybox:1.36 --restart=Never -it -- sh
nslookup registry.registry.svc.cluster.local
wget -qO- http://registry.registry.svc.cluster.local:5000/v2/

A healthy unauthenticated registry endpoint returns a response associated with the v2 API path.

Configure Image Pull Secrets Correctly

For authenticated registries, create and reference a docker-registry secret:

1kubectl create secret docker-registry regcred \
2  --docker-server=registry.example.internal \
3  --docker-username=REG_USER \
4  --docker-password=REG_PASS \
5  --docker-email=[email protected] \
6  -n app

Attach to service account or pod spec:

1apiVersion: v1
2kind: Pod
3metadata:
4  name: sample
5  namespace: app
6spec:
7  containers:
8    - name: api
9      image: registry.example.internal/team/api:1.0.0
10  imagePullSecrets:
11    - name: regcred

Confirm namespace alignment. Secrets are namespace-scoped.

Handle TLS and Certificate Trust

If registry uses self-signed or private CA certificates, node runtimes must trust that CA. Adding cert only in application pod does not fix image pull, because pull happens at node runtime level.

High-level fix path:

1. install CA cert on all worker nodes
2. configure container runtime trust store for registry host
3. restart runtime if required

For managed clusters, use provider-supported node image customization or daemon configuration methods.

Avoid Insecure Registry Misconfiguration

Using plain HTTP registry without runtime configuration causes pull failures. If you intentionally run insecure registry in dev, container runtime must explicitly allow it.

In production, prefer TLS with valid certificates and avoid insecure registry flags.

Registry Deployment Basics That Matter

A minimal reliable deployment includes:

• persistent volume for registry storage
• readiness probes on registry pod
• stable service name
• optional ingress with TLS termination

Without persistent storage, restarts can make previously pushed images unavailable.

Confirm Image Name and Tag

Many failures are simple naming mistakes:

registry.example.internal/team/api:1.0.0

Check:

• correct registry host
• repository path
• exact tag
• no hidden typo in deployment manifest

Use kubectl get deploy -o yaml to inspect final applied image string.

End-to-End Verification Workflow

After changes, verify in this order:

1. push test image to registry
2. pull test image from node or debug environment
3. deploy a pod with same image reference
4. monitor events until pod reaches running state

This staged check reduces false assumptions and narrows fault domain.

Observability and Incident Prevention

To prevent repeat incidents:

• monitor registry pod health and storage usage
• alert on image pull error rates in cluster events
• document pull secret rotation process
• maintain registry certificate expiry alerts

Most recurring pull issues are operational drift, not one-time bugs.

Common Pitfalls

• Creating image pull secret in wrong namespace.
• Trusting self-signed cert only inside app containers instead of node runtime.
• Using internal service DNS name in image reference where nodes need externally resolvable host.
• Forgetting persistent storage for registry and losing images after restart.
• Debugging deployment logic before confirming raw registry connectivity and authentication.

Summary

• Start with pod events to classify image pull failures by layer.
• Validate registry reachability, authentication, and certificate trust in order.
• Configure imagePullSecrets per namespace and workload context.
• Use TLS and stable registry deployment practices for production reliability.
• Add monitoring and runbooks to prevent recurring pull incidents.