Design File Storage Infrastructure for real-time analytics

Question

Design a high-throughput file storage system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **File Upload and Retrieval**: Users should be able to upload files and retrieve them with low latency.
2. **Real-Time Analytics**: The system should provide real-time analytics on file usage, such as upload/download frequencies and user interactions.
3. **Versioning**: Each file must support versioning to allow users to access previous versions.
4. **Access Control**: Implement user authentication and authorization to ensure that only authorized users can access certain files.

### Non-Functional Requirements
1. **Scalability**: The system must handle millions of requests per minute, scaling horizontally as needed.
2. **Availability**: Aim for 99.99% uptime to ensure users can access their files at any time.
3. **Consistency**: Ensure files are consistently available across replicas, using a model that fits the use case (e.g., eventual consistency for analytics).
4. **Performance**: Average file upload and retrieval times should be under 200ms.

Assuming Notion has approximately 10 million active users, with an average of 3 file interactions (upload/download) per user per day:

- **Daily Requests**: 10M users * 3 interactions = 30M requests/day
- **Requests per second**: 30M requests/day / 86400 seconds/day ≈ 347 requests/second
- To account for peak traffic, assume a 10x increase during peak hours, leading to ~3,470 requests/second.
- **Storage Needs**: If each user uploads an average of 500MB of files:
  - Total storage = 10M users * 500MB = 5,000TB (or 5PB).
- **Analytics Data**: Assuming 1% of interactions generate analytics data, that adds an additional 3.5M records per day.

### Component Diagram
- **Client Application**: Web and mobile clients.
- **Load Balancer**: Distribute incoming requests across multiple application servers.
- **Application Servers**: Handle business logic, file uploads/downloads, and analytics processing. Technology choice: Node.js or Go for performance.
- **File Storage**: Use a distributed file storage system like Amazon S3 for durability and scalability.
- **Database**: Use PostgreSQL for metadata storage (file info, access logs) and ClickHouse for analytics due to its performance in handling large datasets.
- **Caching Layer**: Redis for caching frequently accessed file metadata and analytics results.
- **Monitoring**: Use Prometheus and Grafana for real-time monitoring and alerting on system performance.

### Schema Design
1. **Files Table**: 
   - `file_id`: UUID (Primary Key)
   - `user_id`: UUID (Foreign Key)
   - `file_name`: VARCHAR
   - `file_size`: BIGINT
   - `created_at`: TIMESTAMP
   - `updated_at`: TIMESTAMP
   - `version`: INT

2. **User Access Table**: 
   - `user_id`: UUID (Foreign Key)
   - `file_id`: UUID (Foreign Key)
   - `access_level`: ENUM('read', 'write', 'admin')
   - `access_time`: TIMESTAMP

3. **Analytics Table**: 
   - `analytics_id`: UUID (Primary Key)
   - `file_id`: UUID (Foreign Key)
   - `action`: ENUM('upload', 'download')
   - `timestamp`: TIMESTAMP
   - `user_id`: UUID (Foreign Key)

This design supports efficient querying of file access and analytics data.

1. **Consistency vs. Availability**: Choosing eventual consistency for analytics can improve performance and availability but may lead to temporary discrepancies in analytics data. This is acceptable given the use case.
2. **Caching Strategy**: Using Redis for caching metadata improves read performance but requires cache invalidation strategies to ensure data consistency.
3. **Storage Choice**: Using S3 for file storage provides durability and scalability but introduces some latency for file retrieval compared to local storage options.
4. **Load Balancing**: Implementing an active-active architecture across multiple data centers can enhance availability but increases complexity in consistency management across regions.

Design File Storage Infrastructure for real-time analytics

Notion

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Notion Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search