Build a distributed Video Streaming Pipeline

Question

Design a distributed video streaming system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
- **User Management**: Users should be able to create accounts, log in, and manage their profiles.
- **Video Upload**: Users can upload videos, which will be processed and stored for streaming.
- **Video Streaming**: Users can stream videos in real-time with adaptive bitrate streaming.
- **Search and Recommendations**: Implement search functionality and personalized video recommendations based on user behavior.
- **Analytics**: Provide analytics about video views, user engagement, and performance metrics.

### Non-Functional Requirements
- **Scalability**: The system should handle millions of concurrent streams without degradation.
- **Availability**: The system must be highly available with a target of 99.99% uptime.
- **Security**: End-to-end encryption for video data and secure authentication for users.
- **Latency**: The system should aim for a video startup latency of less than 2 seconds.

To estimate capacity:
- **Concurrent Users**: Assume 10 million registered users with an average of 10% active at peak times.
- **Requests Per Second (QPS)**: If each active user generates 2 requests per second for streaming, we can estimate:
  - Peak QPS = 10 million * 10% * 2 = 2 million QPS.
- **Storage**: Each uploaded video is approximately 500 MB. If we expect 100,000 new video uploads daily:
  - Daily Storage Requirement = 100,000 * 500 MB = 50 TB/day.
- **Bandwidth**: For adaptive streaming, assume average bitrate of 3 Mbps per stream. For 2 million concurrent streams:
  - Bandwidth Requirement = 2 million * 3 Mbps = 6 Tbps.

### Components
1. **Load Balancer**: Distributes incoming traffic to multiple application servers.
2. **Application Servers**: Handle user requests, video upload, and streaming logic (using Node.js).
3. **Video Processing Pipeline**: Utilizes FFmpeg for transcoding and prepares videos for adaptive streaming.
4. **Storage**: Amazon S3 for storing video files and DynamoDB for metadata.
5. **CDN**: Cloudflare or AWS CloudFront to cache and serve video content closer to users.
6. **Messaging Queue**: Apache Kafka for handling asynchronous processing of video uploads and notifications.

### Diagram
```
[Load Balancer] --> [Application Servers] --> [Messaging Queue] --> [Video Processing] --> [Storage]
                       |--> [Analytics Service] --> [DynamoDB]
```

### Schema Design
- **User Table**: `user_id (PK)`, `username`, `password_hash`, `email`, `created_at`
- **Video Table**: `video_id (PK)`, `user_id (FK)`, `title`, `description`, `file_url`, `created_at`, `view_count`
- **View Table**: `view_id (PK)`, `video_id (FK)`, `user_id (FK)`, `view_timestamp`

### Access Patterns
- Frequent reads for streaming and analytics (video views).
- Write-heavy for video uploads and user interactions (likes, comments).

### Consistency vs Availability
- **Eventual Consistency**: For video metadata (DynamoDB), we can afford eventual consistency, allowing for fast reads but possibly stale data.
- **Strong Consistency**: For user authentication and transactions, we prioritize strong consistency to prevent unauthorized access.

### Other Trade-offs
- **Caching**: Use Redis to cache frequently accessed video metadata and user sessions, trading off memory for speed.
- **Replication**: Choose multi-region replication for S3 storage, improving availability but increasing costs and potential latency.
- **Cost vs Performance**: Using AWS services provides high scalability but incurs higher costs compared to self-hosting, which may reduce operational complexity but increase management overhead.

Build a distributed Video Streaming Pipeline

Mastercard

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Mastercard Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search