Design File Storage Infrastructure for global users

Question

Design a real-time file storage system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements:**
  - Store and retrieve files (images, videos, documents) for millions of global users.
  - Support real-time access and updates to files.
  - Implement versioning for file updates.
  - Enable A/B testing capabilities on file-related features.
  - Provide a feature store for machine learning models, allowing for quick access to engineered features relevant to user interactions.

- **Non-functional Requirements:**
  - **Scalability:** Handle millions of concurrent requests with low latency.
  - **Availability:** 99.99% uptime to ensure users can access files at all times.
  - **Consistency:** Ensure eventual consistency for file updates across distributed data centers.
  - **Performance:** Response time under 100ms for file retrieval and under 500ms for uploads.

Assuming Reddit has around 500 million active users:
- Let's estimate that each user uploads an average of 2 files per month, leading to:
  - **Total uploads per month:** 500M users * 2 files/user = 1 billion files/month.
- Breaking this down further:
  - **Daily uploads:** 1 billion files / 30 days ≈ 33.3 million files/day.
- If we assume an average file size of 5 MB:
  - **Daily data ingestion:** 33.3M files * 5 MB = 166.5 TB/day.
- For storage needs, considering a retention period of 1 year:
  - **Total storage requirement:** 166.5 TB/day * 365 days = 60.8 PB/year.

The architecture will consist of the following components:
1. **Client Layer:**
   - Frontend applications (web/mobile) for users to upload and retrieve files.
2. **API Gateway:**
   - AWS API Gateway to route requests to the appropriate microservices.
3. **File Storage Service:**
   - Amazon S3 for durable file storage with versioning enabled.
4. **Database Layer:**
   - DynamoDB for metadata storage (user info, file links, versioning).
5. **Feature Engineering Pipeline:**
   - Apache Kafka for real-time data streaming and processing.
   - Apache Spark for batch processing and feature extraction.
6. **Model Serving Infrastructure:**
   - TensorFlow Serving for real-time model serving with low latency.
7. **Monitoring and Logging:**
   - Prometheus and Grafana for monitoring performance and health of the system.

The schema design for the metadata storage in DynamoDB:
- **Users Table:**
  - `UserID (PK)`
  - `Username`
  - `Email`
  - `CreatedAt`

- **Files Table:**
  - `FileID (PK)`
  - `UserID (FK)`
  - `FileName`
  - `FileSize`
  - `FileURL`
  - `CreatedAt`
  - `UpdatedAt`
  - `Version`

- **Feature Store Table:**
  - `FeatureID (PK)`
  - `UserID (FK)`
  - `FeatureValues (JSON)`
  - `Timestamp`

This design allows for efficient queries on user files and supports quick retrieval of features for model serving.

- **Consistency vs Availability:**
  - Choosing eventual consistency for file updates allows the system to be more available, especially during high load, but might result in users experiencing stale data temporarily.

- **Complexity vs Latency:**
  - Using a microservices architecture increases complexity in deployment and maintenance, but provides better scalability and separation of concerns, which is critical for handling high request volumes.

- **Batch vs Real-time Processing:**
  - Real-time processing using Apache Kafka enables immediate feedback loops for machine learning, but incurs higher costs and management overhead compared to batch processing, which is simpler and more cost-effective.

- **Storage Costs:**
  - Choosing Amazon S3 for file storage is cost-effective but may introduce latency in file retrieval compared to a faster but more expensive solution like a distributed file system.

Design File Storage Infrastructure for global users

Reddit

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Reddit Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search