Three probes exist, they do completely different things, and the same five timing fields configure all of them, which is why they are so often configured identically and wrongly.

What each probe does when it fails

The distinction is entirely in the consequence of failure.

A failing liveness probe causes the node agent to kill and restart the container. A failing readiness probe removes the pod from service endpoints, so it receives no new traffic, and leaves it running. A failing startup probe kills the container, like liveness, but it runs only until it first succeeds, after which it is never evaluated again.

So the question each answers is different: liveness asks whether a restart would help, readiness asks whether traffic should arrive now, and startup asks whether the container is still legitimately starting. Giving all three the same handler, which is the most common configuration in the wild, means all three questions get the same answer, which is right only by coincidence.

The four handler types

A probe checks something in one of four ways, and the choice has cost implications.

An exec handler runs a command inside the container and treats exit zero as success, which is the most flexible and by far the most expensive, since it forks a process every period. A request handler makes a request to a path and port and treats a response status in the two hundreds or three hundreds as success. A socket handler opens a connection to a port and treats a successful connection as success. A procedure-call handler uses the standard health checking protocol for that style of service.

The socket handler is the weakest check, since a listening socket says nothing about the application behind it, and is the one most likely to report a hung process as healthy. The exec handler on a one-second period in a dense cluster is a measurable amount of processor time doing nothing but forking.

The five timing fields

Every probe takes the same parameters and their interaction is where the arithmetic lives.

An initial delay before the first check. A period between checks. A timeout for each check. A success threshold, the consecutive successes needed to be considered passing, which must be one for liveness and startup. A failure threshold, the consecutive failures needed to be considered failing.

The defaults are an initial delay of zero, a period of ten seconds, a timeout of one second, and both thresholds at three for readiness and failure at three for liveness. The timeout default of one second is the one that causes trouble, because an application under load that takes one and a half seconds to answer a health request is marked failing by a probe that would have succeeded at two seconds.

Detection time arithmetic

One formula tells you how long a failure takes to be noticed, and it is the number that matters.

The worst case from the moment a container becomes unhealthy to the action being taken is the period times the failure threshold, plus up to one period of delay before the first failing check, plus the timeout of the failing checks:

With the defaults, a failure threshold of three and a period of ten seconds, that is around forty seconds before a dead container is restarted and around forty seconds before an unready pod stops receiving traffic. Forty seconds of errors is a long time for a service under traffic, and shortening the period is usually the right lever since lowering the threshold makes the probe twitchy.

The startup probe and what it replaces

One probe exists to fix a specific conflict and it does so cleanly.

Before startup probes, a slow-starting application needed either a liveness initial delay long enough to cover the worst start, which delayed detection of real failures by that much forever, or a high failure threshold, with the same effect. Both traded permanent detection latency for a one-time startup allowance.

A startup probe separates them: it can be generous, with a long period and a high failure threshold, bounding total startup time, and once it passes, the liveness probe takes over with tight settings. An application that may take five minutes to load its data and should be restarted within twenty seconds of hanging afterwards is expressible only this way.

Bounding startup time deliberately

The startup probe's failure threshold is a budget and it is worth computing.

Total allowed startup is the period times the failure threshold, so a period of ten seconds and a threshold of thirty allows five minutes. Exceeding it kills the container, which then restarts and tries again, accumulating crash backoff.

The value should come from the measured worst-case start plus margin, not from a round number. Too low means a container that would have started is killed repeatedly, producing a crash loop whose cause looks like the application and is actually the probe. Too high means a genuinely broken start sits for a long time before anything happens, which for a rollout means the deployment waits on it.

The probe that can set its own grace period

One field on a liveness probe is easy to miss and occasionally essential.

A liveness probe can specify a termination grace period that overrides the pod's own, used when the probe's failure causes the kill. The use is a pod with a long grace period for ordinary shutdown, perhaps two minutes for draining, which should nevertheless be killed quickly when it is hung, because a hung process will not use the drain time productively.

Without it, a liveness failure on such a pod waits the full grace period before the kill, so a hung container occupies its slot for two minutes after being detected. Setting the probe's own grace period to a few seconds makes the restart prompt while leaving a deliberate shutdown unhurried.

Where probes run from

One implementation detail resolves a recurring confusion.

Probes are executed by the node agent on the pod's own node, against the pod's address directly, not through a service. So a probe does not test service routing, does not traverse any load balancer, and does not depend on cluster name resolution unless the probe itself resolves a name.

That has two consequences. A probe succeeding tells you nothing about whether traffic can reach the pod from elsewhere, so a network policy blocking traffic leaves a perfectly ready pod receiving nothing. And a probe cannot fail because of a network problem between nodes, which makes probe failures a reliable signal about the container rather than about the cluster.

Probes on init containers

One placement is worth knowing because it makes sidecar ordering real.

Ordinary init containers may have startup probes, which gate the init sequence's progress on the probe passing rather than on the process existing. For a restartable init container acting as a sidecar, that is what makes the ordering guarantee meaningful, since otherwise the sequence proceeds as soon as the process starts, which is not the same as it being usable.

Liveness and readiness probes on restartable init containers are also supported, which lets a sidecar be restarted when it hangs and lets its readiness contribute to the pod's. Without that, a hung sidecar leaves a pod ready and broken, which is a failure mode worth closing for anything the application depends on.

Probes and the container, not the pod

One scoping detail explains a behaviour people find surprising.

Probes are defined per container, so a pod with three containers has up to nine probes, and a liveness failure restarts only the container that failed, leaving the others running. The pod is not recreated, keeps its address and its volumes, and its restart count increments for that container alone.

Readiness composes differently: the pod is ready only when every container with a readiness probe is passing, so one unready container makes the whole pod unready and removes it from endpoints. That asymmetry is deliberate and useful, since a broken sidecar should stop traffic to the pod while a broken application container should be restarted without disturbing its neighbours.