System requirements

Functional:

• File operation(Read/Write/Append/etc.)
• Directory operation.
• Security.

Non-Functional:

• Availability
• Consistency
• Fault tolerance
• Reliability

Capacity estimation

Estimate the scale of the system you are going to design...

API design

# REST Examples

GET /file/path/to/file.txt

PUT /file/path/to/file.txt

POST /file/path/to/file.txt?append=true

DELETE /file/path/to/file.txt

Database design

File Representation

• Files are split into fixed-size chunks (e.g., 64MB - 256MB).
• Each chunk has a unique ID (UUID).
• Files and directories stored in a hierarchical namespace maintained by the Metadata Server.

Chunk Storage

• Each chunk is replicated across R storage servers (e.g., default R=3).
• Stored using local file systems (ext4, xfs).

nodes Table – Files and Directories

TABLE nodes (

id,

name ,

parent_id,

is_directory,

size,

created_at,

modified_at,

permissions,

owner_id UUID,

);

chunks Table – Logical Data Blocks

TABLE chunks (

id,

node_id,

chunk_index,

checksum,

size,

);

locks Table – Distributed Locking

TABLE locks (

node_id ,

client_id,

lock_type,

acquired_at,

expires_at

);

chunk_replicas Table – Physical Replication Mapping

TABLE chunk_replicas (

chunk_id,

server_id,

replica_state,

last_heartbeat,

);

storage_servers Table – Data Node Info

TABLE storage_servers (

id

hostname ,

ip_address ,

last_heartbeat ,

disk_usage ,

is_active

);

High-level design

• API gateway: Interface through which users/applications interact with the system.
• Metadata Server: Maintains metadata like file hierarchy, chunk mappings, and permissions.
• Chunk Servers: Store actual file data in fixed-size chunks.
• Distributed Locking: Ensures consistency and coordinates concurrent access.
• Replication: Each chunk is stored on multiple chunk servers for fault tolerance.
• Heartbeats: Chunk servers send periodic status updates to the Metadata Server.

Request flows

Client wants to upload /user/video/video.mp4 (384 MB)

Client ──► Metadata Server: Request to create file

Metadata Server ──►Lock /user/video

Metadata Server ──► Checks parent directory, permissions, and uniqueness

Metadata Server ──► Allocates 3 chunk IDs (for 3 chunks of 128MB each)

──► Selects 3 storage nodes per chunk for replication

──► Returns chunk plan to Client

Client ──► Sends each chunk to its assigned Storage Servers

Storage Servers ──► Acknowledge receipt and replication success

Client ──► Metadata Server: Finalize write (commit metadata)

Metadata Server ──► Updates file entry, chunk locations, sizes

Metadata Server ──►Unlock

Client reads /user/alice/video.mp4

Client ──► Metadata Server: Request file metadata (including chunk list)

Metadata Server ──► Returns chunk IDs + list of replica locations

Client ──► Directly contacts closest or fastest Storage Server per chunk

Storage Servers ──► Stream chunks to Client

Detailed component design

1. API GW

Responsibilities:

• User/API interface for uploading/downloading files
• Talks to Metadata Server to get file metadata and chunk layout
• Streams chunks directly to/from Storage Servers
• Handles retries, caching, checksum validation

Design Notes:

• Thin client — stateless, logic-heavy on server side
• Can run as SDK, CLI, or web interface

2. Metadata Server

Responsibilities:

• Manages the namespace (directories, file structure)
• Tracks chunk metadata (IDs, replica locations, sizes)
• Validates permissions, quotas
• Coordinates file creation, deletion, and finalization

Subcomponents:

• Namespace Manager: Hierarchical file/directory structure
• Chunk Manager: Allocates chunk IDs, tracks replica placement
• Lock Manager Interface: Coordinates locking per file/path
• Database Interface: Reads/writes metadata in DB

Performance Design:

• Write-ahead log for metadata durability
• In-memory caching of hot metadata

3. Chunk Servers

Responsibilities:

• Store raw data chunks (e.g., 128MB blocks)
• Receive chunk write/read requests from clients
• Perform local replication, validation, cleanup
• Report health, storage, and chunk status to Metadata Server

Storage Layout:

• Chunks stored as flat files with metadata (e.g., checksums)
• Local disk cache + fast access index

Design Notes:

• Stateless from global view; recoverable via re-registration

4. Lock Manager

Responsibilities:

• Ensures consistency during concurrent access (create, write, rename)
• Manages fine-grained locks: file-level, chunk-level, or path-level
• Supports lease-based or advisory locks

Implementation Options:

• Database-based (e.g., SELECT FOR UPDATE)
• Distributed (e.g., ZooKeeper, etcd)
• In-memory with TTL fallback

5. Replication Manager

Responsibilities:

• Monitors chunk replication factor
• Automatically re-replicates lost or under-replicated chunks
• Balances replicas across Storage Servers
• Manages versioning and chunk aging (for overwrite, deletion)

Design Notes:

• Periodically scans metadata and storage health reports
• Can throttle or prioritize replication traffic

6. Monitoring

Responsibilities:

• Heartbeats from Storage Servers
• Tracks disk usage, failures, latency, throughput
• Marks failed or slow nodes
• Alerts for manual intervention or automated failover

Design Notes:

• Store time-series metrics
• Optional UI for dashboards

7. Metadata Database

Responsibilities:

• Stores all persistent metadata
  • File nodes, directory structure
  • Chunk IDs and mappings
  • Replication state
• Supports transactions for locking and consistency

Design Schema:

• nodes table: file/directory entries with hierarchical structure
• chunks table: chunk mappings per file
• replicas table: physical chunk replicas and their node locations
• locks table (optional): record ongoing file locks

DB Choices:

• SQL (PostgreSQL, MySQL): ACID compliance
• NoSQL (Cassandra, etcd): scale-out, but requires custom locking

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?