Introduction

An unhealthy ingress backend usually means traffic reaches ingress but fails at service or pod resolution. The failure may be caused by readiness probes, wrong service ports, endpoint absence, or network policy blocks. Effective troubleshooting follows the request path step by step.

Verify Backend Endpoints First

Start with service and endpoint health. If the service has no ready endpoints, ingress will always report backend issues.

1kubectl get ingress -A
2kubectl describe ingress my-ingress -n app
3
4kubectl get svc my-service -n app -o wide
5kubectl get endpoints my-service -n app -o yaml
6kubectl get pods -n app -l app=my-backend -o wide

If endpoint list is empty, inspect pod readiness and selector labels. Selector mismatch is one of the most common causes of unhealthy backend status.

Confirm Port and Path Mapping

Ingress path must map to the correct service port, and service target port must map to container port. Misalignment at any one level breaks routing.

1apiVersion: v1
2kind: Service
3metadata:
4  name: my-service
5  namespace: app
6spec:
7  selector:
8    app: my-backend
9  ports:
10    - name: http
11      port: 80
12      targetPort: 8080

1apiVersion: networking.k8s.io/v1
2kind: Ingress
3metadata:
4  name: my-ingress
5  namespace: app
6spec:
7  ingressClassName: nginx
8  rules:
9    - host: demo.example.com
10      http:
11        paths:
12          - path: /
13            pathType: Prefix
14            backend:
15              service:
16                name: my-service
17                port:
18                  number: 80

Validate that container actually listens on 8080 and readiness probe uses a path that returns success.

Check Controller Logs and Events

Ingress resources are interpreted by the ingress controller. Controller logs often reveal TLS, annotation, and upstream connectivity issues.

kubectl logs -n ingress-nginx deploy/ingress-nginx-controller --tail=200
kubectl get events -n app --sort-by=.lastTimestamp | tail -n 30

Also verify DNS and host header behavior. A healthy backend can still appear broken if request host does not match ingress rule.

Common Recovery Sequence

A practical recovery sequence is: fix selector labels, confirm endpoints, verify service port mapping, then retest from inside cluster and from outside. This sequence isolates each layer and avoids random trial-and-error edits.

Use a temporary debug pod to call service directly. If service succeeds from inside cluster, focus on ingress rule and controller configuration.

Advanced Diagnostic Checks

If basic checks pass but ingress still marks backend unhealthy, inspect network policy and TLS configuration next. Network policy may block ingress controller namespace from reaching application pods even when service endpoints exist. TLS misconfiguration can also surface as backend failures when upstream protocol expectations do not match. Validate whether backend expects plain HTTP while ingress forwards HTTPS, or the reverse. For NGINX ingress, inspect generated upstream configuration and backend health status in controller debug logs. In cloud-managed ingress solutions, review provider-specific health check path and timeout settings because defaults may not match your application startup time. Keep a runbook that maps each symptom to a layer, such as DNS, ingress rule, service, endpoint, pod, and application health route. Structured diagnosis consistently reduces recovery time.

kubectl get networkpolicy -A
kubectl -n app exec deploy/my-backend -- curl -sS localhost:8080/health
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller --tail=300 | grep -i upstream

Verification Checklist

Capture one known-good request trace from ingress to pod and keep it as a baseline. During incidents, compare failing traces against this baseline to detect differences in host, path, status code, and upstream target quickly.

Common Pitfalls

• Service selector does not match pod labels, leaving endpoint list empty.
• Readiness probe path fails even though container is running.
• Ingress backend points to wrong service port name or number.
• Request host header does not match ingress rule host.

Summary

• Validate endpoints before debugging ingress annotations.
• Align container port, service target port, and ingress backend port.
• Use controller logs and events for precise failure signals.
• Test from inside cluster to isolate service-level issues.
• Follow a layer-by-layer recovery path for faster resolution.