Design A Distributed File System - System Design

Requirements

Functional Requirements:

CRUD - create / read / update / delete files
Concurrent file access
Distributed locking
Replication
Metadata management

Non-Functional Requirements:

Hight availability
Consistency
Scalability
Reliability

Capacity Estimation

API Design

1.Create:

Request:
{
  path: "/home/mordyk/readme.txt"
  size: "8mb"
}
Response:
{
file_id: "id1",
chunk_size: "4mb",
chunks: [
{
chunk_id: "id1",
servers: ["server1", "server2", "server3"]
},
{
chunk_id: "id2",
servers: ["server1", "server2", "server3"]
}
]
}

2.Read:

Request:
{
path: "/home/mordyk/readme.txt"
}
Response:
{
file_id: "id1",
chunks: [
{
chunk_id: "id1",
servers: ["server1", "server2", "server3"]
},
{
chunk_id: "id2",
servers: ["server1", "server2", "server3"]
}
]
}

3.Delete:

Request:
{
path: "/home/mordyk/readme.txt"
}
Response:
{
status: "deleted"
}

4.List directory:

Request:
{
path: "/home/mordyk/"
}
Response:
{
files: [
"file1",
"file2",
]
}

5.Create/Delete directory:

Request:
{
path: "/home/mordyk/"
}
Response:
{
status: "done"
}

6.Lock file:

Request:
{
path: "/home/mordyk/file.txt",
lockType: "write",
}
Response:
{
lock: "lockid",
status: "aquired",
}

7.Unlock file:

Request:
{
lock: "lockid",
}
Response:
{
status: "done"
}

7.Get Metadata

Request:
{
path: "/home/mordyk/file.txt",
}
Response:
{
file_id: "id1",
size: "8mb",
created_at: timestampt,
version: "ver_1",
owner: "user1",
}

High-Level Design

1. Client Layer

Client sends read/write requests.
Requests first go through a Load Balancer (LB).
LB distributes traffic across multiple API Gateways.

2. API Gateway Layer

Provides a single entry point.
Handles:
- Authentication/authorization
- Request routing
- Rate limiting
- Forwarding requests to backend services

3. Metadata Service Layer

Responsible for managing file/object metadata.

Main components:

Namespace Manager → manages directory/file hierarchy.
Chunk Manager → tracks data chunks and their locations.
Lease Manager → handles ownership/locking for writes.
Permission Manager → controls access rights.
Replica Manager → maintains replication information.
Health Manager → monitors service and node health.

4. Metadata Storage

Metadata service maintains state in:
- In-memory cache for fast access.
- Persistent database for durability.
Uses a leader/follower model:
- Metadata leader handles writes.
- Followers replicate data for availability.

5. Storage Layer

Stores the actual user data.

Components:

Multiple Storage Nodes (Node 1...N).
Each storage node contains:
- Chunk Manager → manages data chunks.
- Read/Write Manager → handles IO operations.
- Checksum Manager → verifies data integrity.
- Local Storage Manager → manages physical disk storage.

Database Design

files - stores files.

files (
    file_id: "id",
    directory_id: "id",
    filename: "name",
    owner: "name",
    size: "4mb",
    version: "1",
    created_at: "TIMESTAMP",
    updated_at" "TIMESTAMP",
);

Directories - stores directories.

directories (
    directory_id: "id",
    parent_directory_id: "id",
    name: "name",
);

Chunk - each file is divided into chunks.

chunks (
    chunk_id: "id",
    file_id: "id",
    chunk_index: "offset",
    checksum: "hash",
    version: "1",
    size: "64mb",
);

Storage_nodes - represents every storage server.

storage_nodes (
node_id: "id",
hostname: "name",
rack: "rack",
status: "status",
free_space: "12600 mb"
);

chunk_replicas - a chunk can exist on multiple nodes.

chunk_replicas (
chunk_id: "id",
node_id: "id",
);

Lease - tracks who currently coordinates writes.

leases (
chunk_id: "id",
primary_node: "node",
lease_expiry: "TIMESTAMP",
);

Detailed Component Design

API Gateway (API GW) is the single entry point for all client requests into a distributed system:

Request Routing
Authentication/Authorization
Rate Limiting
Load Balancing

Metadata service level - stores only metadata:

Namespace Manager - responsible for the file system hierarchy.
Chunk Manager - maps a file to its chunks.
Replica Manager - knows where every chunk is stored.
Lease Manager - coordinates writes.

A locking mechanism is needed to control concurrent access to the same data by multiple users or applications. Without locks, multiple clients could modify the same file at the same time, causing data corruption, inconsistency, or lost updates.

Permission Manager - stores ACLs or RBAC information.
Health Manager - receives heartbeats from storage nodes.

We need raft to ensure all Metadata Servers have the same data, even if servers crash or the network is unreliable.

Storage Nodes are the workers of a distributed file system. They store the actual file data, while metadata is maintained by the Metadata Server (Master).

A Storage Node is responsible for:

Storing chunks of files
Read/Write chunks
Verifying data integrity using checksums
Sending health information to the Metadata Server
Recovering corrupted data
Serving multiple clients concurrently

Each Storage Node has:

Chunk manager - manages all data chunks stored on the node.
Read/Write manager - clients download/upload data directly from Storage Nodes.
Checksum manager - check chunks hash sum.
Local Storage Manager - this component interacts with the operating system and disks.

Allocate disk space
Organize chunk files
Delete obsolete chunks
Handle SSD/HDD differences
Perform background cleanup
Idempotent chunk writes
Re-replication on disk failure

Optimizations for read operation in storage node:

Read-ahead
Buffering
SSD caching
Zero-copy I/O
Memory-mapped files

To scale metadata storage, the Metadata Service uses sharding/partitioning across multiple metadata servers. Files/directories are distributed using strategies like hash-based partitioning or namespace-based partitioning. Each metadata shard is replicated using Raft/Paxos for high availability and consistency. A metadata router directs client requests to the correct shard.

Sharding strategies:

Hash-Based Partitioning - hash(filename) % number_of_metadata_servers
Namespace-Based Partitioning - split by directory