Design File Storage Infrastructure for mobile apps

Question

Design a multi-tenant file storage system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements:**  
  1. **Multi-Tenant Support:** Enable file storage for multiple mobile app clients with isolation.  
  2. **File Operations:** Support create, read, update, delete (CRUD) operations for files.  
  3. **Shared Access:** Allow users to share files with others, requiring collaboration features.  
  4. **Versioning:** Include version control for files to track changes over time.  
  5. **Search Functionality:** Implement a search feature to find files based on metadata.

- **Non-Functional Requirements:**  
  1. **Scalability:** Handle millions of concurrent requests efficiently.  
  2. **Availability:** Ensure 99.99% uptime, with a focus on global availability.  
  3. **Consistency:** Provide eventual consistency with options for strong consistency when required.  
  4. **Performance:** Response time for file operations should be under 100 milliseconds.  
  5. **Security:** Implement strong authentication and authorization mechanisms to protect file access.

- Assume Adobe has 1 million active users across mobile apps, each generating an average of 5 file interactions per day.  
  - **Daily Requests:**  
    - 1,000,000 users * 5 requests = 5,000,000 requests per day.  
  - **Peak Load Estimation:**  
    - Assuming peak usage of 10% of users at the same time (10% of 1M users = 100,000 users).  
    - If these users make 3 requests per minute, that results in:  
    - 100,000 users * 3 requests = 300,000 requests per minute.  
    - 300,000 requests/minute translates to 5,000 requests/second on average.  
  - **Storage Requirements:**  
    - Assuming each user stores an average of 10 files of 1MB each, total storage required would be approximately 10TB initially.

The architecture will consist of the following components:  
1. **Client Layer:** Mobile apps interact with the API Gateway.  
2. **API Gateway:** Handles routing, authentication, and rate limiting.  
3. **File Storage Service:**  
   - **Object Storage:** Use Amazon S3 for scalable file storage.  
   - **Database:** PostgreSQL for metadata storage with a multi-tenant schema.  
4. **Caching Layer:** Redis for caching frequently accessed file metadata.  
5. **Load Balancer:** Distribute incoming requests across multiple instances of the API Gateway.  
6. **Message Queue:** Use Kafka for asynchronous processing of file uploads and shared file notifications.  
7. **CDN:** Utilize a Content Delivery Network for faster file delivery across geographies.

The schema design will include the following tables:  
1. **Users Table:**  
   - `id (UUID, PK)`, `name (VARCHAR)`, `email (VARCHAR, UNIQUE)`, `tenant_id (UUID)`.  
2. **Files Table:**  
   - `id (UUID, PK)`, `tenant_id (UUID)`, `user_id (UUID, FK)`, `file_name (VARCHAR)`, `file_size (INT)`, `created_at (TIMESTAMP)`, `updated_at (TIMESTAMP)`, `version (INT)`.  
3. **File Sharings Table:**  
   - `id (UUID, PK)`, `file_id (UUID, FK)`, `shared_with (UUID)`, `permission (ENUM: READ, WRITE)`, `shared_at (TIMESTAMP)`.  
4. **File Metadata Cache:**  
   - Caching frequently accessed file metadata to reduce load on the database.

- **Consistency vs. Availability:**  
  - Chose eventual consistency for file operations to optimize availability and performance, especially during high traffic.  
  - Implement strong consistency options for critical file operations, such as during collaborative editing.

- **Performance vs. Complexity:**  
  - Utilizing Redis for caching improves read performance but adds complexity in cache invalidation strategies.  
  - Designing for multi-tenancy increases complexity but is essential for resource isolation.

- **Asynchronous Processing vs. User Experience:**  
  - Using Kafka for file processing ensures that uploads are handled asynchronously, improving user experience.  
  - However, it may result in delays in file availability immediately after upload.

- **Global Scale Deployment:**  
  - Using AWS regions for increased availability and lower latency, but requires careful management of data replication and consistency across regions.

Design File Storage Infrastructure for mobile apps

Adobe

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Adobe Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search