Design File Storage Infrastructure for IoT devices

Question

Design a fault-tolerant file storage system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **File Upload**: IoT devices must be able to upload files (like logs, images, sensor data) to the storage system.
2. **File Retrieval**: Users or other services must be able to retrieve files stored by IoT devices.
3. **File Metadata Management**: Store and manage metadata (e.g., timestamps, device IDs, file types).
4. **File Deletion**: Provide functionality to delete files when they are no longer needed.
5. **Versioning**: Support versioning of files to track changes over time.

### Non-Functional Requirements
1. **Scalability**: The system must handle millions of requests per second (QPS) and scale horizontally.
2. **Fault Tolerance**: Ensure system remains operational even during hardware or network failures.
3. **Low Latency**: File uploads and retrievals should occur with minimal delay.
4. **Data Durability**: Files must be stored reliably with redundancy.
5. **Security**: Implement authentication and encryption for data at rest and in transit.

Assuming Grubhub has around 1 million active IoT devices, each generating an average of 10 file uploads per day (each being 2 MB) and 5 file retrievals:

- **Daily Uploads**: 1,000,000 devices * 10 uploads = 10,000,000 uploads/day.
- **Daily Retrievals**: 1,000,000 devices * 5 retrievals = 5,000,000 retrievals/day.
- **Total Requests per Second (QPS)**: 
  - 10,000,000 uploads/day = ~115 uploads/second.
  - 5,000,000 retrievals/day = ~58 retrievals/second.
  - Total = 173 QPS peak.

### Storage Calculation
- **Total File Size**: 10,000,000 uploads * 2 MB = 20 TB/day.
- **Monthly Storage Requirement**: 20 TB/day * 30 = 600 TB.
- **Replication Factor**: Assuming a replication factor of 3 for fault tolerance, total storage = 600 TB * 3 = 1.8 PB.

### Component Diagram
- **IoT Devices**: Send files via HTTP/HTTPS.
- **API Gateway**: Handles incoming requests, routes them to appropriate services.
- **File Storage Service**: Manages file uploads, retrievals, and metadata storage (using AWS S3 or Google Cloud Storage).
- **Metadata Service**: Stores metadata in a NoSQL database (like DynamoDB) for quick access.
- **Message Queue**: (e.g., Kafka) for asynchronous processing of file uploads and retrievals.
- **Load Balancer**: Distributes incoming requests across multiple instances of services.
- **Monitoring and Logging**: Tools like Prometheus and ELK stack for monitoring system health and logs.

### Technology Choices
1. **Storage**: AWS S3 for durable file storage with high availability.
2. **Database**: DynamoDB for metadata storage due to its scalability and low latency.
3. **Message Queue**: Apache Kafka for handling asynchronous uploads.

### Schema Design
1. **Files Table (in DynamoDB)**:
   - **FileID** (Primary Key): Unique identifier for the file.
   - **DeviceID**: Identifier for the IoT device that uploaded the file.
   - **Timestamp**: When the file was uploaded.
   - **FileSize**: Size of the file in bytes.
   - **Version**: Version number for file management.
   - **Metadata**: Additional key-value pairs for file attributes.

### Access Patterns
- Files will be accessed primarily by `DeviceID` and `Timestamp` for retrieval and deletion operations, necessitating efficient indexing.

### Architectural Decisions
1. **Consistency vs. Availability**: Opting for eventual consistency in file storage allows for higher availability and fault tolerance, which is crucial given the scale and nature of IoT devices.
2. **Replication vs. Latency**: Choosing a replication factor of 3 ensures durability but introduces some latency during writes. This trade-off is acceptable given the need for data durability.
3. **Synchronous vs. Asynchronous Processing**: Using a message queue (Kafka) allows for asynchronous processing of file uploads, improving responsiveness. However, it adds complexity in maintaining message integrity.
4. **Cost vs. Performance**: Utilizing cloud storage (like AWS S3) optimizes costs while providing scalable performance; however, reliance on third-party services may introduce vendor lock-in.

Design File Storage Infrastructure for IoT devices

Grubhub

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Grubhub Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search