A volume is a directory made available inside a container, and the only question that matters about any volume is how long its contents survive. Every storage mistake in a cluster is a lifetime mistake.

Three lifetimes, not one

One classification prevents most data loss.

A volume whose lifetime is the container survives a process restart but not a container replacement, which is what a writable container filesystem gives you. A volume whose lifetime is the pod survives container restarts within the pod and dies when the pod is deleted or rescheduled. A volume whose lifetime is a claim survives the pod entirely and is reattached to whatever pod takes over.

People reach for the second when they mean the third. A scratch volume shared between containers in a pod is the right use of a pod-lifetime volume; a database's data directory on one is a guaranteed loss the first time the pod moves, and it will work perfectly in testing.

The writable container layer

One default is both convenient and a trap.

A container can write anywhere its filesystem permits, into the writable layer above the image's read-only layers. That is what makes a container feel like a machine, and it has three problems: the data is gone when the container is replaced, writes go through the storage driver's copy-on-write path and are slower than a volume, and the space consumed counts against the node's ephemeral storage, so an unbounded log file gets the pod evicted.

The sound posture is a read-only root filesystem with explicit volumes for the few paths that need writing, which turns all three problems into configuration you can see.

Pod-lifetime scratch space

One volume type covers temporary data properly.

An empty directory created when the pod is assigned to a node and deleted when the pod leaves it, shared by all containers in the pod that mount it. It is backed by the node's disk by default, or by memory if asked, in which case it counts against the pod's memory limit rather than its storage.

The memory-backed form is the right answer for a cache that must be fast and for anything that should never touch a disk, and it is a common cause of surprise memory kills, because a process filling a memory-backed directory is consuming memory the author thinks of as disk. A size limit on the volume turns that into a write failure instead, which is almost always preferable.

Configuration as files

One mechanism delivers configuration and secrets as volumes.

A configuration map or secret mounted as a volume appears as a directory of files, one per key. The node agent refreshes the contents when the object changes, on a cache period of up to a minute or so, by replacing a symbolic link to a new directory atomically.

Two consequences follow. The update is atomic across all keys, so an application reading several files never sees a mixed state, which is a genuinely useful property. And a file mounted at a specific path with a sub-path reference is not refreshed at all, because the symbolic link trick does not apply, so a configuration file mounted that way is frozen at pod start and the application will never see a change.

Projected volumes and the token

One composite type matters more than it looks.

A projected volume combines several sources, configuration maps, secrets, downward information about the pod, and service account tokens, into one directory. The token source is the important one: it requests a token with an audience and an expiry, which the node agent refreshes before it expires.

That refresh is why an application must re-read the token file rather than reading it once at start. A client library that caches the token at start works for the token's lifetime and then fails, which produces an authentication failure hours after a deployment that appeared successful, and it is one of the more confusing failure modes in a cluster.

Node-local storage, and the honest case for it

One type trades portability for speed.

A local volume is a disk or directory on a specific node, presented as a claim with node affinity so the scheduler places the pod where the data is. It gives the node's full device performance with no network in the path.

The cost is that the data is on one node: node loss is data loss, and the pod cannot be rescheduled elsewhere, so a drain blocks or the workload goes down. It is the right choice for workloads that replicate at the application layer and want local device speed, and the wrong choice for anything relying on the storage layer for durability. Stating which of those applies is the whole decision.

Host paths, and why they are almost always wrong

One type should raise an objection in review.

A host path mounts a directory from the node into the pod. It is necessary for node-level agents that must read the node's filesystem, and for almost nothing else.

The problems are that the path may not exist or may have different contents on another node, that the pod writes outside any accounting so it can fill the node's disk unnoticed, that it is a straightforward privilege escalation path since mounting the right directory gives access to credentials or the container runtime's socket, and that nothing about it is portable. Policy should restrict it to specific paths for specific service accounts rather than leaving it available.

The ephemeral storage dimension

One resource is easy to forget until a node fills.

The writable layer, pod-lifetime empty directories backed by disk, and container logs all consume the node's ephemeral storage. Requests and limits exist for it, and a pod exceeding its limit is evicted.

Without limits, one pod writing a large file can fill the node and trigger eviction of other pods under disk pressure, which is a noisy-neighbour failure that looks like a cluster problem. Setting an ephemeral storage limit on every pod, and a size limit on every empty directory, converts a cluster-wide incident into one pod's write failure.

Mount propagation and sub-paths

Two options exist that are mostly ways to hurt yourself.

Mount propagation controls whether mounts made inside the container are visible on the node and vice versa, which is needed by storage and monitoring agents and by nothing else. Sub-path mounts expose one file or directory from a volume rather than the whole thing.

The sub-path case is the one that appears in ordinary applications, usually to place a single configuration file next to application files. It defeats the refresh mechanism, as above, and historically has been a source of escape vulnerabilities. Mounting the whole directory and having the application read from it is the configuration that keeps working.

How to reason about any volume

One question sequence settles it.

What is the lifetime of the data, container, pod or claim. Who else needs to read or write it, one container, several in a pod, or several pods. Does it need to survive the node, which decides between local and network storage. How much space, and is that bounded by anything. And what happens when the pod is rescheduled, which is the question that exposes a wrong answer to the first.

Answering those five in order produces the right volume type almost mechanically, and skipping the last is how pod-lifetime volumes end up under databases.

Read-only mounts and shared volumes

One pair of settings expresses intent precisely.

A volume may be mounted read-only in one container and read-write in another, which is the right way to let a sidecar read what the main container writes without being able to damage it. Secret and configuration volumes should always be read-only, since nothing should write to them and the mount makes that explicit.

Within a pod, several containers mounting the same volume see the same filesystem with no coordination of any kind, so an init container that prepares a directory and a main container that consumes it is the safe pattern, and two containers writing the same files concurrently is a bug the platform will not catch.