Design File Storage Infrastructure for real-time analytics

Question

Design a low-latency file storage system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **File Upload**: Users can upload files of various formats (e.g., JSON, CSV, images) to the system.
2. **File Retrieval**: Users can retrieve files quickly, with low latency for real-time analytics.
3. **File Indexing**: The system should index files for quick search and retrieval based on metadata (e.g., user ID, timestamp).
4. **Real-Time Analytics**: The system should support real-time updates and analytics on uploaded files.
5. **Access Control**: Implement user authentication and authorization to control access to files.

### Non-Functional Requirements
1. **Low Latency**: The system should achieve file retrieval times of less than 50ms under normal load.
2. **Scalability**: Must handle millions of concurrent requests, scaling horizontally to accommodate growth.
3. **High Availability**: The system should provide 99.99% uptime, with redundancy to handle failures.
4. **Durability**: Files must be stored reliably with mechanisms for data recovery and backup.
5. **Monitoring and Alerting**: Implement logging and monitoring for system performance and error rates.

### Back-of-Envelope Calculations
Assuming Meta has an estimated user base of 100 million users, with an average of 10 file requests per user per day:
- **Total Daily Requests**: 100 million users * 10 requests = 1 billion requests/day.
- **Peak Traffic**: Assuming peak usage occurs during specific hours, if 10% of requests happen in peak hours:
  - Peak Requests: 1 billion requests/day * 10% = 100 million requests/hour = ~27,800 requests/second.
- **Storage Requirements**: Assuming an average file size of 1MB:
  - Total Storage: 100 million files * 1MB = 100TB of storage needed.
Taking into account growth, we plan for a system that can handle 200TB of storage and around 40,000 requests/second during peak.

### Component Diagram
- **User Client**: Web or mobile application for users to interact with the file storage.
- **Load Balancer**: Distributes incoming requests across multiple web servers.
- **Web Servers**: Handle file upload, retrieval, and indexing requests. Use **Node.js** for low-latency processing.
- **Caching Layer**: Implement **Redis** for caching popular files and metadata to reduce database load.
- **File Storage**: Use **Amazon S3** for durable and scalable file storage, with lifecycle policies for archiving.
- **Database**: Use **PostgreSQL** for metadata storage, providing strong consistency and supporting complex queries.
- **Analytics Engine**: Use a real-time processing framework like **Apache Kafka** for handling event streams and analytics.
- **Monitoring**: Implement **Prometheus** and **Grafana** for monitoring system health and performance.

### Schema Design
1. **Files Table**:
   - `file_id` (UUID, Primary Key)
   - `user_id` (UUID, Foreign Key)
   - `file_name` (VARCHAR)
   - `file_size` (INTEGER)
   - `upload_timestamp` (TIMESTAMP)
   - `metadata` (JSON)
   - `file_url` (VARCHAR)

2. **Users Table**:
   - `user_id` (UUID, Primary Key)
   - `username` (VARCHAR)
   - `password_hash` (VARCHAR)
   - `created_at` (TIMESTAMP)

### Access Patterns
- Frequent reads of file metadata for analytics, hence the need for indexing on `user_id` and `upload_timestamp`.

### Architectural Decisions
1. **Consistency vs. Availability**: Opted for eventual consistency in the caching layer to improve read performance, while ensuring strong consistency in the primary database for file metadata.
2. **Caching Strategy**: Utilizing Redis for caching introduces complexity in cache invalidation but significantly reduces latency for frequently accessed files.
3. **Load Balancing**: Using a centralized load balancer can introduce a single point of failure; we will implement multiple load balancers in an active-active configuration to enhance fault tolerance.
4. **Data Storage Choices**: Using Amazon S3 provides excellent durability and scalability, but the potential latency for file retrieval is a trade-off compared to a more traditional database approach. We mitigate this with caching.
5. **Monitoring Complexity**: Implementing comprehensive monitoring and alerting increases operational overhead but is necessary for maintaining system performance and reliability.

Design File Storage Infrastructure for real-time analytics

Meta

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Back-of-Envelope Calculations

Submit Your Answer

Meta Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search