Design a File Storage for Twilio

Question

Design a real-time file storage system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **User Authentication**: Users must authenticate to upload or download files securely.
2. **File Upload**: Users should be able to upload files of varying sizes (up to 5 GB).
3. **File Download**: Users should be able to download files quickly, with support for resumable downloads.
4. **Real-time Access**: Users need real-time access to their uploaded files, with eventual consistency in updates.
5. **File Metadata**: Store metadata for each file (e.g., user ID, upload timestamp, file type).

### Non-Functional Requirements
1. **Scalability**: The system should handle millions of requests per day without degradation in performance.
2. **Availability**: Aim for 99.99% uptime, ensuring files are accessible at all times.
3. **Durability**: Files must be stored reliably with redundancy to prevent data loss.
4. **Latency**: Upload and download operations should be completed within a few seconds.
5. **Security**: Ensure data is encrypted in transit and at rest.

Assuming Twilio serves around 1 million users, with each user uploading an average of 2 files per week, we can calculate:
- **Total Uploads per Week**: 1,000,000 users * 2 files = 2,000,000 files/week 
- **Total Uploads per Day**: 2,000,000 / 7 ≈ 285,714 files/day 
- **Peak Hourly Uploads**: Assuming peak traffic occurs during business hours, we estimate around 50% of daily uploads occur in 8 hours, giving us approximately 35,714 uploads/hour or about 10 uploads/second.
  
If we account for file sizes averaging 10 MB, the total data stored weekly would be approximately:
- **Weekly Data**: 2,000,000 files * 10 MB = 20 TB/week 
- **Monthly Data**: 80 TB/month

Thus, the system should be designed to handle at least 80 TB of storage per month.

### Component Diagram
The architecture consists of the following components:
1. **Load Balancer**: Distributes incoming requests across multiple application servers.
2. **Application Server**: Handles user requests for uploading and downloading files, performing authentication, and interacting with storage services.
3. **File Storage**: Uses object storage (e.g., AWS S3) for file storage, ensuring durability and scalability.
4. **Metadata Database**: A NoSQL database (e.g., DynamoDB) for storing file metadata, optimized for quick access and scalability.
5. **CDN (Content Delivery Network)**: For caching and serving static files quickly, reducing latency.
6. **Authentication Service**: Manages user authentication and authorization tokens.

### Technology Choices
- **Load Balancer**: AWS Elastic Load Balancing
- **Application Server**: Node.js / Express
- **File Storage**: Amazon S3
- **Metadata Database**: DynamoDB
- **CDN**: AWS CloudFront

### Schema Design for Metadata Database
**File Metadata Table**: (DynamoDB)
- **Partition Key**: `user_id` (string)
- **Sort Key**: `file_id` (string)
- **Attributes**:
  - `file_size` (number)
  - `upload_timestamp` (string)
  - `file_type` (string)
  - `file_url` (string)
  - `status` (string, e.g., 'uploaded', 'processing')

### Access Patterns
- Query files by user ID to list all files uploaded by a user.
- Retrieve file metadata using `user_id` and `file_id` for download requests.

### Architectural Decisions
1. **Consistency vs. Availability**: Opting for eventual consistency in file uploads to prioritize availability and performance, allowing users to access files immediately after upload, even if the metadata update is not yet consistent.
2. **Storage Cost vs. Performance**: Using Amazon S3 for file storage due to lower costs and built-in redundancy, accepting slightly higher latency in file retrieval compared to faster, more expensive block storage options.
3. **Caching Strategy**: Implementing a CDN for frequently accessed files to reduce load on the storage service and improve download speeds, while caching metadata in-memory for quick access during high traffic.
4. **Horizontal Scaling**: Designing the application to scale horizontally by adding more application servers as user load increases, ensuring that we can handle spikes in traffic effectively.

Design a File Storage for Twilio

Twilio

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Twilio Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search