A container's root filesystem is not a copy of the image. It is a view assembled at mount time from several read-only directories and one writable one, and almost every surprising thing about container storage follows from how that view is built.

Four directories make one filesystem

The union filesystem in standard use takes four arguments, and knowing what each one is explains the rest.

The lower directories are the image's layers, ordered, read-only, and shared between every container using that image. The upper directory is this container's own writable space, empty when it starts. The work directory is scratch space the kernel needs for atomic operations. The merged directory is the resulting view, which is what the container sees as its root.

Reading a file searches the upper directory first and then each lower directory in order, taking the first match. So a file present in three layers is read from the topmost one, which is exactly what an image build expects when a later instruction replaces an earlier file.

Deletions are recorded, not performed

A read-only lower layer cannot have anything removed from it, so deletion is represented rather than executed.

Removing a file that exists only in a lower layer creates a character device node with zero major and minor numbers in the upper directory, at the same path. The union filesystem treats that marker as meaning the path does not exist, so the file disappears from the view while remaining in the layer.

Removing a whole directory sets an attribute on the upper directory's copy marking it opaque, which hides every lower entry beneath it rather than needing a marker per file. The consequence that matters for images is that deleting a file in a later build instruction does not reduce the image size at all, because the bytes stay in the earlier layer and a marker is added on top.

Writing triggers a copy

The first write to a file that lives in a lower layer copies the whole file upward before the write happens, and that single behaviour is the source of most container storage performance problems.

The sequence is: resolve the file in a lower layer, create the parent directories in the upper layer if needed, copy the file's entire contents and metadata up, then apply the write to the copy. Subsequent writes go straight to the upper copy with no further cost.

So appending one byte to a one gigabyte file in the image copies a gigabyte first. Opening a large file for writing and immediately truncating it still copies it, because the copy happens on open rather than on the data write in most implementations. Workloads that modify large files from the image should either write to a volume or have those files generated into the writable layer at startup.

What the container sees versus what is on disk

Two numbers that people conflate are the apparent size of the filesystem and the space the container is actually using.

Inside the container, the root filesystem reports the size of whatever device backs the upper directory, which is usually the node's disk, so a container appears to have hundreds of gigabytes available regardless of its configuration. The space it is actually consuming is only what is in its upper directory, because the lower layers are shared with every other container using the same image.

This is why ten containers from one two gigabyte image occupy two gigabytes plus their individual writes rather than twenty gigabytes, and it is also why a container's apparent free space is not a limit. The limit, where there is one, is the ephemeral storage limit the orchestrator enforces by measuring the upper directory and the logs.

The layer order comes from the build

The ordering of lower layers is decided by the image and is not negotiable at run time.

Each build instruction that changes the filesystem produces one layer containing the differences that instruction made, and the layers stack in instruction order. Instructions that change only metadata, such as setting an environment variable or a label, produce no layer and only alter the configuration document.

So an image's layer list is a readable history of its build, and inspecting it is the fastest way to find why an image is larger than expected. A single instruction contributing hundreds of megabytes is usually a package installation whose cache was not cleaned in the same instruction, which is the most common image size mistake and the one most clearly explained by the layer model.

Trying it without a container

The union filesystem is mountable by hand, which is the fastest way to make the behaviour concrete.

Creating three directories, putting different files in two of them, and mounting the union produces a merged view you can experiment with: read a file present in both lowers and see which wins, delete a lower file and find the whiteout node, write to a lower file and watch the copy appear in the upper directory.

1mkdir -p lower1 lower2 upper work merged
2echo one > lower1/shared.txt; echo two > lower2/shared.txt
3mount -t overlay overlay -o lowerdir=lower2:lower1,upperdir=upper,workdir=work merged
4cat merged/shared.txt          # two: the first lowerdir listed wins
5rm merged/shared.txt; ls -l upper/shared.txt   # a 0,0 character device

Ten minutes of this removes the mystery from copy-up, whiteouts and layer precedence, and those three ideas cover most of what goes wrong with container storage in production.

The lower layer limit, and deep images

The number of lower directories a single mount can take is bounded, and the bound is low enough to meet in practice.

Older kernels accepted a few dozen, which an image built with one instruction per dependency can exceed, and the failure is a mount error at container start rather than anything at build time. Newer kernels raised the limit substantially, so the problem is now mostly historical, and the symptom is distinctive enough to recognise: an image that builds and pushes fine and refuses to run on a particular node.

Beyond the hard limit there is a soft cost, because every path lookup that misses walks the whole stack. An image with eighty layers pays eighty lookups for a path that does not exist, which makes directory-heavy workloads slower in a way that profiling attributes to the filesystem rather than to the image. Both arguments point the same way: fewer, more meaningful layers are better than one per command.

Mounting volumes over image content

A volume mounted at a path that the image already populated hides what was there, and the behaviour differs enough between cases to be worth knowing.

An empty volume mounted over a populated directory makes that directory appear empty, because the mount replaces the view rather than merging with it. Some runtimes copy the image's contents into a new empty volume on first use, which makes the behaviour look merged, and others do not, so the same configuration behaves differently in two environments.

The practical rule is to choose paths for volumes that the image does not populate, and to generate anything that must coexist with mounted data at startup rather than shipping it in the image at that path. A configuration directory that is half image content and half mounted secrets is the arrangement that produces the most confusion.

Why the model suits services and not storage

It is worth being explicit about which workloads the design serves well, because the answer is most of them.

A typical service reads its binary, its libraries and its static assets from the image and writes almost nothing, so it pays no copy-up, shares all its layers with every replica, and starts as fast as the process itself. The union filesystem is close to free for that pattern, which is why it became the default rather than the block-level alternatives.

Storage workloads invert every one of those properties: they write continuously, modify large files in place, depend on precise rename and flush semantics, and need their data to outlive the container. Running them on the writable layer fights the design rather than using it. The useful framing is that the image filesystem is a delivery mechanism for code, not a disk, and workloads whose job is holding data need a real one attached.