The word runtime covers at least three different programs, and almost every confusing container failure becomes clear once you know which of them owns the step that failed.

Four programs, one container

Starting a container in an orchestrated cluster involves four distinct pieces of software, each with a narrow job.

The node agent receives the desired state and decides what should be running. It talks to a container runtime over a defined interface, asking for sandboxes and containers. That runtime manages images, storage and container state, and for each container it starts a small supervisor process. That supervisor invokes a low-level runtime, which makes the actual kernel calls and then gets out of the way.

The division is deliberate. The high-level runtime is a long-lived service holding state; the low-level runtime is a short-lived program that does one thing and exits; the supervisor is what remains attached to the container when both of the others are gone.

The two specifications involved

Two separate specifications define the interfaces, and conflating them is a common source of confusion.

The runtime specification defines what a low-level runtime does: it takes a directory containing a root filesystem and a configuration document, and it provides a small set of operations over the container's lifecycle. Anything implementing it is interchangeable, which is why alternative low-level runtimes are a configuration change rather than a migration.

The container runtime interface is a different thing entirely: a remote procedure call interface between the node agent and the high-level runtime, covering images, sandboxes, containers, logs and statistics. It exists so the orchestrator does not depend on any particular runtime implementation.

The bundle is the unit the low-level runtime understands

The handoff between the high-level and low-level runtime is a directory on disk, which makes it inspectable.

A bundle is a directory containing a root filesystem and a configuration document. The configuration names the process to run, its arguments, environment and working directory, the namespaces to create, the mounts to perform, the control group path, the capability sets, the seccomp filter, the resource limits and the hooks to run at various points.

That document is the complete, declarative description of a container, and every manifest field that affects isolation ends up expressed there. Reading it for a running container is the definitive answer to what the container was actually started with, as opposed to what the manifest asked for.

The lifecycle operations

The runtime specification defines a small set of operations and a specific state machine, and the two-step start is the part worth understanding.

Creating a container sets everything up: namespaces, filesystem, control group, and a process that is blocked waiting to proceed. Starting it unblocks that process so it executes the real entrypoint. Separating the two means all the setup can be verified before anything in the container runs, and it gives hooks a defined moment to act.

The remaining operations are querying state, sending a signal, executing an additional process inside, and deleting. That is the whole surface, which is why a conformant low-level runtime can be a few thousand lines and why swapping one for another changes nothing above it.

Hooks, and where platform behaviour gets injected

The configuration lists programs to run at defined points, and a surprising amount of platform behaviour lives there.

There are hooks that run after the namespaces are created but before the entrypoint, hooks that run after start, and hooks that run before deletion. Network plugins use the early ones to create interfaces inside the new network namespace. Device plugins use them to expose hardware. Storage integrations use them to mount volumes.

So a container that fails to start with an error naming nothing you recognise is frequently a hook failing, and the error text comes from a program the platform installed rather than from the runtime. Knowing hooks exist turns that from mysterious to searchable.

Reading the chain for a running container

Everything above is observable on a node, which is the practical payoff.

1crictl ps -o json | jq '.containers[0].id'           # ask the runtime, not docker
2ctr -n k8s.io containers info <id> | jq '.Spec.linux.namespaces'
3cat /run/containerd/io.containerd.runtime.v2.task/k8s.io/<id>/config.json | jq '.process'
4ps -ef | grep -E 'containerd-shim|runc'               # one shim per container, no runc

The last check is the one that surprises people: there is no low-level runtime process running, because it exited after setting the container up. A container with a running runtime process is a container mid-operation, not a healthy one.

Namespaces within the runtime, and the one that confuses everyone

The high-level runtime has its own namespace concept, unrelated to kernel namespaces, and mixing the two wastes a lot of debugging time.

It partitions its own state into named spaces so that several consumers can share one daemon without seeing each other's containers. An orchestrated cluster uses one well-known name for everything it creates, and a developer using the runtime's own command line tool gets a different default.

This is why running the runtime's native tool on a node shows no containers at all on a cluster that is obviously running hundreds: the tool is looking in its own default space, not the orchestrator's. Passing the right namespace shows everything. The lesson generalises: a tool reporting nothing on a node is evidence about the tool's configuration before it is evidence about the node.

Plugins, snapshotters and content stores

The high-level runtime is itself modular, and knowing its internal division helps when the failure is in storage rather than execution.

It separates content, which is the immutable blobs pulled from registries, from snapshots, which are the writable filesystem views assembled for containers. The snapshotter is pluggable, which is how a node can use one union filesystem implementation or another, or a remote filesystem that lazily fetches layer contents on access.

The practical value is that errors name the component. A content store error means a problem with pulled blobs or disk. A snapshotter error means a problem assembling or mounting the filesystem. These are separate failures with separate fixes, and the error text distinguishes them before you have to guess.

What this layering costs in startup latency

Each layer adds a step, and the total is worth knowing because it bounds how fast a container can possibly start.

The sequence for a cold start is an image pull if needed, unpacking layers into snapshots, creating the sandbox including the network setup, creating the bundle, invoking the low-level runtime, and executing the entrypoint. On a warm node with the image present, everything after the pull typically totals a few hundred milliseconds, with the network plugin usually the largest single contributor.

The pull dominates when it happens, which is why image size is the main lever on start latency and why pre-pulling matters for anything latency-sensitive. Blaming the runtime for slow starts is almost always misattributed: the measurement will show the pull or the network plugin.

The node agent's side of the contract

The agent is the only client of the interface in a cluster, and what it does with the answers is worth knowing.

It maintains a desired state per pod, reconciles it against what the runtime reports, and takes the smallest action that closes the gap: create a sandbox, create a container, start it, kill it, remove it. It also owns the restart policy, the probe schedule, the log rotation and the eviction decisions, none of which the runtime knows anything about.

So a container that restarts repeatedly is an agent decision based on the exit status the supervisor collected, and a container killed for failing a probe is an agent decision too. The runtime did what it was told. Attributing restart behaviour to the runtime sends people reading the wrong logs, and the agent's log is where the reasoning actually appears.

Where the layers keep their state

Four programs mean four state locations, and knowing them shortens an investigation considerably.

The orchestrator's view lives in the cluster's own store and is what the interface reports upward. The agent keeps a local store of pod state and checkpoints. The high-level runtime keeps its content store, its snapshot metadata and its container records under its own directory. The low-level runtime keeps a small state directory per container, and the supervisor keeps its socket and log alongside.

When those four disagree, the disagreement itself is the finding. A pod the cluster believes is running with no container on the node, or a container the runtime knows about that the agent does not, each point at a specific recovery path rather than at a mystery.