Design a high-throughput File Storage System

Question

Design a high-throughput file storage system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements:**  
  1. Store and retrieve files of various types (images, documents, etc.) with high throughput.  
  2. Support versioning and metadata storage for each file.  
  3. Provide a RESTful API for file upload, download, and management.  
  4. Implement user authentication and authorization for file access.  
  5. Enable tagging and searching of files based on metadata.

- **Non-Functional Requirements:**  
  1. **Scalability:** Must handle millions of requests per second as Redfin scales its user base.  
  2. **Availability:** Aim for 99.99% uptime, especially during peak usage times.  
  3. **Performance:** Average response time for file operations should be under 100ms.  
  4. **Security:** Ensure data is encrypted in transit and at rest, with strong authentication methods.  
  5. **Monitoring and Logging:** Real-time monitoring of request metrics, error rates, and user activity logs.

- Assuming Redfin has around 10 million users, and on average, each user makes 5 requests per day for file operations.  
  - **Total Daily Requests:**  
    - 10 million users * 5 requests/user = 50 million requests/day.  
  - **Peak QPS Calculation:**  
    - If we estimate peak traffic to be 10% of daily, we have:  
    - 50 million requests/day * 10% = 5 million requests during peak hours.  
    - Dividing by 3600 seconds/hour gives approximately 1390 QPS.  
  - **Storage Requirements:**  
    - Assume an average file size of 1 MB.  
    - If each user stores an average of 5 files, then total storage = 10 million users * 5 files * 1 MB = 50 TB.

The architecture consists of the following components:

1. **API Gateway:**  
   - Handles incoming requests and routes them to appropriate services (e.g., AWS API Gateway).

2. **File Storage Service:**  
   - Uses Amazon S3 for file storage, providing durability and scalability.

3. **Metadata Service:**  
   - A microservice (built with Node.js and Express) managing file metadata, storing it in a NoSQL database like Amazon DynamoDB for quick access.

4. **Caching Layer:**  
   - Implements Redis for caching frequently accessed metadata and file references to reduce database load and improve performance.

5. **Content Delivery Network (CDN):**  
   - CloudFront to serve files quickly to users globally, reducing latency.

6. **Authentication Service:**  
   - Utilizes OAuth 2.0 with JSON Web Tokens (JWT) for user authentication and authorization.

The data model consists of the following schemas:

1. **File Metadata Table (DynamoDB):**  
   - **Attributes:**  
     - `file_id` (Primary Key)  
     - `user_id`  
     - `file_name`  
     - `upload_timestamp`  
     - `file_size`  
     - `tags` (list of strings)  
     - `version` (integer)

2. **User Table (DynamoDB):**  
   - **Attributes:**  
     - `user_id` (Primary Key)  
     - `email`  
     - `password_hash`  
     - `created_at`  
   
3. **Access Logs Table (DynamoDB):**  
   - **Attributes:**  
     - `log_id` (Primary Key)  
     - `user_id`  
     - `file_id`  
     - `action_type` (upload/download)  
     - `timestamp`

- **Consistency vs Availability:**  
  - Choosing S3 and DynamoDB offers eventual consistency for file metadata but high availability. This trade-off fits Redfin’s use case where low-latency access to metadata is prioritized over strict consistency.

- **Caching Strategy:**  
  - Using Redis for caching enhances performance but introduces potential staleness in metadata. Regular cache invalidation strategies, such as TTL and cache-busting updates on file changes, must be implemented.

- **Database Choice:**  
  - NoSQL (DynamoDB) is chosen for its scalability and flexibility in handling diverse metadata structures, trading off relational integrity for performance and scalability.

- **Monitoring and Alerting:**  
  - Implementing AWS CloudWatch for monitoring request metrics, error rates, and system health provides operational visibility but adds complexity in managing alerts and responses.

Design a high-throughput File Storage System

Redfin

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Redfin Product Manager Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search