0%
AWS in Practice: Services, Limits, and Failure Modes
Account Foundations
Compute
Databases
Networking
Delivery and Visibility
Limits, Cost and Recovery
EFS and Shared Filesystems
EBS gives one instance a disk. S3 gives any number of clients an object store over HTTP. Between them sits a need that neither meets well: many machines reading and writing the same files, through ordinary file system calls, with the semantics programs written for a local disk expect. Content management systems, shared home directories, machine learning training data, legacy applications that write reports to a directory another process watches, and containers that need persistent state no matter which host they land on all fall into this gap. AWS fills it with Elastic File System and the FSx family, and choosing well among them, or deciding that S3 would serve better, is the subject of this lesson.
What a file system promises that S3 does not
A POSIX file system offers operations S3 does not. A program can open a file, seek to the middle, and overwrite four bytes. It can append to a log line by line. It can rename a directory of ten thousand files atomically. It can take a lock that other processes respect. It can list a directory and see changes made a moment ago by another client. S3 offers whole object reads and writes, prefix listings, and no partial updates, renames or locks. Software written against files assumes these operations, and rewriting it to use objects is often more work than the storage is worth, which is the main reason shared file systems remain in demand.
NFS in one paragraph
EFS speaks NFS version 4.0 and 4.1, the Network File System protocol that Unix systems have used to share directories for decades. A client mounts a remote export at a local path, and the kernel's NFS client turns file system calls into network requests to the server. The client caches attributes and data to avoid a round trip for every call, and uses a consistency model called close to open: changes a client writes become visible to other clients when the writing client closes the file and the reading client opens it afterwards. Understanding that model explains most surprises people meet with shared file systems, including stale reads and lost updates when two writers do not coordinate.
The latency cost of sharing
Every operation that must be checked with the server costs a network round trip, which on EFS is roughly a millisecond for reads and a few milliseconds for writes, compared with microseconds for a local disk and the page cache. Operations that touch many small files, such as git status on a large repository, npm install, or compiling a codebase, issue thousands of metadata operations and feel dramatically slower on NFS than on local disk. Large sequential reads and writes, in contrast, run at high throughput because each request moves a lot of data. The shape of a workload's I/O, many small metadata operations or few large data operations, predicts how it will feel on a shared file system far better than its total size does.
The options on AWS
Elastic File System is the managed NFS service for Linux, elastic in capacity with no provisioning, regional by default with data stored across multiple zones, and billed per gigabyte stored. FSx for Lustre is a high performance parallel file system for compute heavy workloads, often linked to an S3 bucket. FSx for Windows File Server provides SMB shares with Active Directory integration for Windows workloads. FSx for NetApp ONTAP offers NFS, SMB and iSCSI with NetApp's snapshots, cloning and replication, often chosen for migrations from on premises NetApp. FSx for OpenZFS provides low latency NFS with ZFS features. Mountpoint for Amazon S3 and S3 Files offer file access to S3 buckets with restricted semantics. Each fits a different workload, and the most common mistake is choosing EFS for a workload that needs one of the others, or a file system at all for a workload that should use S3.
When not to use a shared file system
Shared file systems add latency, cost per gigabyte several times that of S3, and a consistency model that is easy to misuse. Many workloads that reach for one would be better served otherwise. User uploads in a web application usually belong in S3, served through CloudFront, with the application writing objects through the SDK. Logs belong in a log pipeline, not a shared directory every server appends to. Coordination between processes belongs in a queue or database, not lock files on NFS. Container images belong in a registry. A shared file system is the right answer when software genuinely needs file semantics across machines and cannot reasonably be changed, or when its performance profile, such as Lustre for parallel training reads, is exactly what is needed.
A short history of the pattern
Before managed shared file systems, teams on AWS ran their own NFS server on an EC2 instance with EBS volumes, which created a single point of failure that everyone mounted. When that instance failed, every client hung, because NFS clients by default retry forever on a hard mount. EFS removed the server from the team's responsibility and spread data across zones, which is why it replaced most self managed NFS servers. Some teams still run their own for cost or control, and they should know they are taking on the failure modes EFS was built to remove.
Mount behaviour when the server is unreachable
A hard NFS mount, the default, makes every process touching the mount block until the server responds, which keeps data safe but can hang an entire host when the file system is unreachable, including processes as simple as ls or a shell's prompt that checks the current directory. A soft mount returns errors after a timeout, which keeps the host responsive but can surface as data corruption in applications that do not check write errors. AWS recommends hard mounts with sensible timeouts for EFS, and designs that keep anything critical to the host's health, such as its own logs and binaries, off the shared mount so that an EFS problem degrades the application rather than freezing the machine.
Metadata is the hidden workload
People size file systems by bytes, but shared file systems are often limited by metadata operations: opening, closing, checking attributes, listing directories and creating files. A web application that checks whether a template file has changed on every request issues a stat call each time, and across fifty servers and thousands of requests per second that becomes a large metadata load even though almost no data moves. Directories with hundreds of thousands of entries make listings slow and are best split into a hashed hierarchy of subdirectories. Thinking about how many metadata operations a workload performs per second, and not only how many bytes it reads, predicts shared file system performance much better.
Atomic rename as the safe write pattern
Because readers may open a file while a writer is part way through it, the standard pattern for publishing a file on a shared file system is to write it under a temporary name in the same directory, flush and close it, and then rename it to its final name. Rename within a directory is atomic on the server, so readers see either the old complete file or the new complete one, never a partial one. Configuration files, generated reports and model artefacts shared across machines should all be written this way, and tools that watch a directory should ignore temporary names.
What the shared file system replaces
When a team adopts a shared file system, it is worth writing down which coordination it is standing in for. If it is replacing a database, a queue or a cache, the team has usually chosen a weaker tool for that job. If it is giving unmodified software the file semantics it was built for, it is doing exactly what it is meant to do. That one line in a design document prevents years of slow growth in which more and more services treat a shared directory as their integration point.
A shared file system gives many machines POSIX semantics over the network at the cost of latency per operation and close to open consistency. Use one when software genuinely needs file semantics across hosts, and prefer S3, a database or a queue when it does not.