Design a distributed File Storage System

Question

Design a distributed file storage system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **File Upload/Download**: Users can upload and download files of various formats (images, documents, etc.).
2. **File Metadata**: Store and retrieve metadata associated with files (e.g., file size, type, upload date).
3. **Access Control**: Implement user authentication and authorization for file access.
4. **File Versioning**: Support version control for files, allowing users to access previous versions.
5. **Search**: Enable searching by file metadata and content.

### Non-Functional Requirements
1. **Scalability**: The system should handle millions of requests and scale horizontally.
2. **Availability**: Aim for 99.99% uptime, ensuring files are accessible at all times.
3. **Durability**: Guarantee data durability, ensuring files are not lost or corrupted.
4. **Latency**: Strive for low latency in file retrieval (less than 200ms for 95% of requests).
5. **Consistency**: Prioritize eventual consistency for metadata updates to balance performance.

Assuming Square/Block has around 10 million active users, with an average of 10 file interactions (uploads/downloads) per user per month:

- **Total Requests per Month**: 10M users * 10 requests/user = 100M requests/month.
- **Average File Size**: Assuming 1 MB per file, total storage required = 100M MB (or ~100 TB).
- **Daily Requests**: Approximately 3.3M requests/day.
- **Peak Load**: Considering peaks, we could assume 4x normal activity during busy hours, leading to ~500 requests/sec.
- **Storage Growth**: If we estimate a 20% growth year-over-year, we would need to accommodate up to 120 TB by the next year.

### Component Diagram
1. **Client**: Web and mobile applications.
2. **API Gateway**: Routes requests to appropriate services (built with **AWS API Gateway**).
3. **File Storage Service**: Handles file uploads and downloads (using **Amazon S3** for storage).
4. **Metadata Service**: Stores file metadata in a **NoSQL database** (e.g., **DynamoDB**) for scalability and fast access.
5. **User Authentication Service**: Manages user sessions and permissions (using **OAuth2** with **Auth0**).
6. **Search Service**: Enables searching across files (using **Elasticsearch**).
7. **Version Control Service**: Manages file versions (could use **AWS Lambda** for microservices handling).
8. **Monitoring and Logging**: **Prometheus** and **Grafana** for real-time metrics and logging.

### Schema Design
1. **File Metadata Table**:  
   - **FileID** (Primary Key): Unique identifier for each file.
   - **UserID**: ID of the user who uploaded the file.
   - **FileName**: Name of the file.
   - **FileSize**: Size of the file in bytes.
   - **UploadDate**: Timestamp of file upload.
   - **FileVersion**: Version number.
   - **AccessControl**: User permissions (e.g., read/write).

2. **File Versions Table**:  
   - **VersionID** (Primary Key): Unique identifier for each version.
   - **FileID**: ID of the associated file.
   - **VersionData**: Pointer to the stored version (in S3).
   - **VersionDate**: Timestamp of version creation.

1. **Consistency vs. Availability**: Choosing eventual consistency for metadata updates allows for better performance and availability, but may lead to temporary discrepancies in metadata.
2. **Latency vs. Durability**: Using S3 provides high durability but can introduce latency in file retrieval. We can mitigate this by caching frequently accessed files in **Redis**.
3. **Complexity vs. Performance**: Implementing complex features like A/B testing and feedback loops for model retraining increases system complexity. We may choose to simplify by limiting the scope of experimentation to critical features.
4. **Real-time vs. Batch Processing**: For file search and retrieval, we prioritize real-time access but may batch process less frequently accessed files to optimize costs.

Design a distributed File Storage System

Square/Block

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Square/Block Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search