Kubernetes
Istio
Health Check
Service Mesh
Cloud Native

How health check of Kubernetes work with Istio?

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

Introduction

Kubernetes health checks still come from the kubelet even when Istio injects a sidecar into the pod. The complication is that Istio changes traffic flow inside the pod, so HTTP, TCP, and gRPC probes need special handling if you want liveness and readiness checks to keep reflecting application health accurately.

Start with Kubernetes Probes

Kubernetes supports liveness, readiness, and startup probes. A normal HTTP readiness probe might look like this:

yaml
1readinessProbe:
2  httpGet:
3    path: /ready
4    port: 8080
5  periodSeconds: 10
6  timeoutSeconds: 1

Without a service mesh, the kubelet sends that request directly to the container through the pod IP and target port.

Why Istio Changes Probe Behavior

Istio sidecars intercept pod traffic with iptables rules. According to the Istio docs, that creates two common problems:

  • with strict mTLS, the kubelet does not have an Istio certificate, so direct HTTP probes can fail
  • TCP probes can become misleading because the sidecar makes the port appear open even when the application behind it is not ready

To solve this, Istio can rewrite HTTP, TCP, and gRPC application probes so the kubelet talks to the sidecar agent instead. The sidecar then forwards the health check to the original application endpoint.

Understand Probe Rewrite

When sidecar injection is enabled, Istio rewrites supported probes at pod-injection time. That is why the running pod may contain probe paths such as /app-health/... even if your deployment YAML originally used /ready or /healthz.

The kubelet is still the thing performing the probe. Istio is changing the route the probe takes so it still works inside the mesh.

This is also why debugging health failures in an injected pod should include inspecting the live pod spec, not just the original deployment manifest.

Disable Rewrite Only When You Mean To

Istio documents an annotation for disabling HTTP probe rewrite on a pod:

yaml
metadata:
  annotations:
    sidecar.istio.io/rewriteAppHTTPProbers: "false"

Use that only if you are sure the direct kubelet probe can still succeed with your mesh settings. In strict mTLS setups, disabling rewrite often causes healthy workloads to fail probes.

Choose Probe Types Carefully

Command probes are often the least affected by Istio because they run inside the container. HTTP probes are usually a good fit when rewrite is enabled. TCP probes deserve more caution because "the port accepted a connection" is not a very strong health signal once a proxy sits in front of the app.

If you want application-level confidence, HTTP or command probes are usually better than plain TCP probes in a sidecar-based environment.

Common Pitfalls

The biggest mistake is assuming Istio ignores Kubernetes health checks. It does not. The kubelet still performs them, but the mesh changes how they must be routed.

Another common issue is turning off probe rewrite without understanding why it was enabled. That can break health checks immediately in mTLS-protected environments.

People also trust TCP probes too much when a sidecar is present. A listening proxy is not the same as a healthy application.

Finally, when debugging readiness failures, inspect the injected pod spec and sidecar behavior instead of only reading the original YAML definition.

Summary

  • Kubernetes health checks still come from the kubelet when Istio is present.
  • Istio rewrites supported probes so they continue to work with traffic interception and mTLS.
  • HTTP, TCP, and gRPC probes can be rewritten; command probes run inside the container.
  • The annotation sidecar.istio.io/rewriteAppHTTPProbers: "false" disables rewrite, but use it carefully.
  • In Istio environments, HTTP or command probes are usually more trustworthy than bare TCP probes.

Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design

All Rights Reserved.