System requirements

Functional:

1. The system must allow the users the Ability to upload videos
2. The system must allow users to stream videos across different kinds of clients like mobiles, TV, laptop etc.
3. The system must allow the user to download the video.

Non-Functional:

1. The system must be highly available
2. The system should be fairly scalable
3. The system should allow smooth video streaming - fairly reliable.
4. Support Large Videos

Capacity estimation

1. 10 M daily Active users
2. 10% of Users upload videos
3. Average video size = 500 MB
4. Total space required for the videos = 10% * 10M * 500 MB ~ 500 TB of data to be stored
5. CDN cost - We definitely need a CDN to be able to render videos closer to an user.

API design

1)Upload a Video-Post/api/v2/video

2) Stream a Video - Get/api/v2/Video/{id}

Database design

1. Video metadata - This will contain information like the video name, size, uploaded by user details, and S3 urls for the different formats of the video. To store this, we can choose a database like Cassandra, which will provide a high availability.
2. Actual Video - For the actual video, we will use a S3 storage. User will be able to upload the video directly onto the S3 storage using a pre-signed URL.

High-level design

1. Video Metadata Service - This service will store information in the Video Metadata Database.
2. Video Transcoders - This will convert the video into different formats so that they can be stored and dowloaded by clients in different network conditions
3. Worker and Task Queues - There will be several worker and task queues in order to perform the various video processing tasks in parallel
4. CDN - CDN renders the video for the client to stream. The video will be rendered from the CDN closest to the client.
5. Audio processing Service - Process audio for the video uploaded
6. Transcript Processing Service - Generate Transcript for the video
7. Thumbnailing Service - Add thumbnails to the video
8. Watermarking Service - Watermark the video if required
9. Metadata DB
10. Load balancers and Rate limiters

Request flows

Video Uploading Workflow

1. The client posts a request using the presigned url for S3 to upload the video directly to S3.
2. Parallely the video metadata service stores the video metadata information in the Metadata DB
3. When the video is uploaded to S3, it triggers a S3 notification to the Video Processing service to start the Video transcoding process.
4. Video Splitter splits the video into GOP(Group of pictures alignment) these are individual playable segments.
5. The segments are transcoded into different formats , and they are also thumbnailed and watermarked.
6. Similarly the audio is also encoded, and these different formatted segments are stored in S3 , also put in a CDN along with their respective metadata stored in the metadata cache.

Video Streaming Workflow

1. For Video streaming, the videos are rendered from the CDN closest to the user.
2. An adaptive bitrate streaming workflow is followed for the streaming purpose. which is where the different formats of the video are rendered based on the network latencies and conditions

Detailed component design

Video Transcoding and Directed Acyclic Graph Structure

• Video Upload:
• Initial step where the raw video file is uploaded to the streaming platform. 
• Video Pre-processing:
  • Encoding: Conversion of the video into a streaming-friendly format like H.264 or VP9, with different bitrate profiles for adaptive streaming. 
  • Transcoding: Creating multiple versions of the video at different resolutions to cater to varying network bandwidths. 
  • Metadata Generation: Creating descriptive information about the video (title, description, thumbnails). 
• Content Storage:
  • Storage on CDN: Distributing encoded video segments across a Content Delivery Network for fast access based on user location. 
  • Manifest File Creation: Generating a file that details the available video segments and bitrates for adaptive streaming. 
• Delivery to Client:
  • Client Request: User initiates a playback request specifying desired quality. 
  • Dynamic Bitrate Adaptation (DAB): Based on network conditions, the client selects the optimal bitrate segment from the manifest file. 

Video Playback: The video segments are streamed to the client for playback.

Trade offs/Tech choices

Transcoding is a very CPU intensive process and following a DAG further instensifies that.

Which is why there will be several task queues for workers and tasks to effectively manage the process and help the completion of the several tasks.

All of these will be done during the video upload process.

Failure scenarios/bottlenecks

1. For all services, there will be load balacners.
2. Database uses replication
3. Cassandra as a Metadata DB provides eventual consistency.

Future improvements

1. Video Feed recommendation
2. Ability to save and comment on videos.