System requirements

Functional:

• File operation(Read/Write/Append/etc.)
• Directory operation.
• Security.

Non-Functional:

• Availability
• Consistency
• Fault tolerance
• Reliability

Capacity estimation

Estimate the scale of the system you are going to design...

API design

# REST Examples

GET /file/path/to/file.txt

PUT /file/path/to/file.txt

POST /file/path/to/file.txt?append=true

DELETE /file/path/to/file.txt

Database design

File Representation

• Files are split into fixed-size chunks (e.g., 64MB - 256MB).
• Each chunk has a unique ID (UUID).
• Files and directories stored in a hierarchical namespace maintained by the Metadata Server.

Chunk Storage

• Each chunk is replicated across R storage servers (e.g., default R=3).
• Stored using local file systems (ext4, xfs).

nodes Table – Files and Directories

TABLE nodes (

id,

name ,

parent_id,

is_directory,

size,

created_at,

modified_at,

permissions,

owner_id UUID,

);

chunks Table – Logical Data Blocks

TABLE chunks (

id,

node_id,

chunk_index,

checksum,

size,

);

locks Table – Distributed Locking

TABLE locks (

node_id ,

client_id,

lock_type,

acquired_at,

expires_at

);

chunk_replicas Table – Physical Replication Mapping

TABLE chunk_replicas (

chunk_id,

server_id,

replica_state,

last_heartbeat,

);

storage_servers Table – Data Node Info

TABLE storage_servers (

id

hostname ,

ip_address ,

last_heartbeat ,

disk_usage ,

is_active

);

High-level design

• API gateway: Interface through which users/applications interact with the system.
• Metadata Server: Maintains metadata like file hierarchy, chunk mappings, and permissions.
• Chunk Servers: Store actual file data in fixed-size chunks.
• Distributed Locking: Ensures consistency and coordinates concurrent access.
• Replication: Each chunk is stored on multiple chunk servers for fault tolerance.
• Heartbeats: Chunk servers send periodic status updates to the Metadata Server.

Request flows

Client wants to upload /user/video/video.mp4 (384 MB)

Client ──► Metadata Server: Request to create file

Metadata Server ──►Lock /user/video

Metadata Server ──► Checks parent directory, permissions, and uniqueness

Metadata Server ──► Allocates 3 chunk IDs (for 3 chunks of 128MB each)

──► Selects 3 storage nodes per chunk for replication

──► Returns chunk plan to Client

Client ──► Sends each chunk to its assigned Storage Servers

Storage Servers ──► Acknowledge receipt and replication success

Client ──► Metadata Server: Finalize write (commit metadata)

Metadata Server ──► Updates file entry, chunk locations, sizes

Metadata Server ──►Unlock

Client reads /user/alice/video.mp4

Client ──► Metadata Server: Request file metadata (including chunk list)

Metadata Server ──► Returns chunk IDs + list of replica locations

Client ──► Directly contacts closest or fastest Storage Server per chunk

Storage Servers ──► Stream chunks to Client

Detailed component design

1. API GW

Responsibilities:

• User/API interface for uploading/downloading files
• Talks to Metadata Server to get file metadata and chunk layout
• Streams chunks directly to/from Storage Servers
• Handles retries, caching, checksum validation

Design Notes:

• Thin client — stateless, logic-heavy on server side
• Can run as SDK, CLI, or web interface

2. Metadata Server

Responsibilities:

• Manages the namespace (directories, file structure)
• Tracks chunk metadata (IDs, replica locations, sizes)
• Validates permissions, quotas
• Coordinates file creation, deletion, and finalization

Subcomponents:

• Namespace Manager: Hierarchical file/directory structure
• Chunk Manager: Allocates chunk IDs, tracks replica placement
• Lock Manager Interface: Coordinates locking per file/path
• Database Interface: Reads/writes metadata in DB

Performance Design:

• Write-ahead log for metadata durability
• In-memory caching of hot metadata

3. Chunk Servers

Responsibilities:

• Store raw data chunks (e.g., 128MB blocks)
• Receive chunk write/read requests from clients
• Perform local replication, validation, cleanup
• Report health, storage, and chunk status to Metadata Server

Storage Layout:

• Chunks stored as flat files with metadata (e.g., checksums)
• Local disk cache + fast access index

Design Notes:

• Stateless from global view; recoverable via re-registration

4. Lock Manager

Responsibilities:

• Ensures consistency during concurrent access (create, write, rename)
• Manages fine-grained locks: file-level, chunk-level, or path-level
• Supports lease-based or advisory locks

Implementation Options:

• Database-based (e.g., SELECT FOR UPDATE)
• Distributed (e.g., ZooKeeper, etcd)
• In-memory with TTL fallback

5. Replication Manager

Responsibilities:

• Monitors chunk replication factor
• Automatically re-replicates lost or under-replicated chunks
• Balances replicas across Storage Servers
• Manages versioning and chunk aging (for overwrite, deletion)

Design Notes:

• Periodically scans metadata and storage health reports
• Can throttle or prioritize replication traffic

6. Monitoring

Responsibilities:

• Heartbeats from Storage Servers
• Tracks disk usage, failures, latency, throughput
• Marks failed or slow nodes
• Alerts for manual intervention or automated failover

Design Notes:

• Store time-series metrics
• Optional UI for dashboards

7. Metadata Database

Responsibilities:

• Stores all persistent metadata
  • File nodes, directory structure
  • Chunk IDs and mappings
  • Replication state
• Supports transactions for locking and consistency

Design Schema:

• nodes table: file/directory entries with hierarchical structure
• chunks table: chunk mappings per file
• replicas table: physical chunk replicas and their node locations
• locks table (optional): record ongoing file locks

DB Choices:

• SQL (PostgreSQL, MySQL): ACID compliance
• NoSQL (Cassandra, etcd): scale-out, but requires custom locking

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

1. Metadata Server Failure (Single Point of Failure)

Impact:

• File creation, deletion, and path resolution fail.
• No client can read or write new files (though ongoing chunk operations may continue).

Mitigation:

• Use primary-backup architecture (active-passive)
• Implement metadata server replication or distributed metadata
• Regular metadata checkpoints and write-ahead logs for recovery

2. Storage Node Failure

Impact:

• Some chunks become temporarily unavailable
• Risk of data loss if enough replicas are lost

Mitigation:

• Replication factor ≥ 3
• Monitor nodes via heartbeat
• Use replication manager to re-replicate missing chunks from healthy peers

3. Network Partition or Latency

Impact:

• Inconsistent view of cluster state
• Clients may time out or hang
• Metadata or replication messages may fail

Mitigation:

• Timeout + retry logic in clients
• Gossip protocols or quorum-based updates for consistency
• Geo-redundancy and network zoning

4. Bottleneck: Metadata Server Scalability

Impact:

• High metadata request volume causes latency spikes
• Hot directories or files get contention

Mitigation:

• Horizontal scaling (shard by directory prefix or user)
• Metadata caching on client side
• Use lock striping or fine-grained locks for hot files

5. Bottleneck: Small Files Problem

Impact:

• Too many small files → metadata bloat
• Poor disk efficiency and high memory usage on Metadata Server

Mitigation:

• Small file packing (combine small files into larger container files)
• Cache hot small files
• Store metadata in optimized formats (LSM, RocksDB)

6. Slow or Failed Replication

Impact:

• Under-replicated chunks = risk of data loss
• Read latency increases if only distant replica available

Mitigation:

• Asynchronous background replication
• Track replication lag
• Throttle new writes if replication queue grows too large

7. Locking Deadlocks or Starvation

Impact:

• Clients hang waiting for locks
• Can block writes or even reads in some systems

Mitigation:

• Use timeout-based locks and automatic retries
• Maintain a lock queue with fairness
• Avoid locking entire directories (favor per-path or per-chunk locks)

8. Corrupt Chunks or Disk Failures

Impact:

• Read or checksum verification fails
• Data becomes unavailable or invalid

Mitigation:

• Use checksums per chunk and periodic verification
• Self-healing with re-replication
• Use ECC and RAID on storage nodes

9. Hotspot Files or Directories

Impact:

• Some files (e.g., logs or large videos) get massive traffic
• Metadata server and chunk servers for those files get overloaded

Mitigation:

• Load balance reads across replicas
• Allow read-only replicas or near caching
• For metadata: shard popular directories

10. Insufficient Storage Space

Impact:

• New chunk allocations fail
• Replication fails
• System becomes read-only

Mitigation:

• Monitor disk usage thresholds (e.g., 85% max capacity alert)
• Auto-scale storage nodes
• Rebalance chunks to underutilized nodes

11. Inconsistent Metadata (Split-Brain)

Impact:

• Metadata corruption or conflict
• Data loss or logical inconsistency

Mitigation:

• Use consensus protocols (e.g., Paxos/Raft)
• Ensure single writer for each file (or append-only)
• Strong versioning and logs for metadata 12. Security Failures

Impact:

• Unauthorized access to files
• Data leakage or destruction

Mitigation:

• Enforce authentication + access control
• Use TLS encryption in transit, and encryption at rest
• Audit trails and monitoring

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?