Loading...
List functional requirements for the system (Ask the chat bot for hints if stuck.)...
List non-functional requirements for the system...
Estimate the scale of the system you are going to design...
Number of Daily Active Users = 100M
~80 % users watch 3 videos per day. Num videos watched per day = 80M * 3 = 250M approx
~20% users upload 2 videos per day. Num videos uploaded per day = 20M * 2 = 40M
Total number of requests per day = 250M + 40M = 300M approx.
Number of queries per second(QPS) = 300M/10^5 = 3000 queries/second. Most are read queries.
Approx size of video = 50 MB.
Size of all videos uploaded per day = 50 MB * 40 M = 2000TB
Size of all videos uploaded in a year = 2000TB * 365 = 720PB approx
Define what APIs are expected from the system...
Upload video to S3
================
PUT https://
Response: 201 created
Stream a video
=============
GET https://
Response over a streaming protocol like HLS(HTTP Live Streaming)
Search
=======
GET https://
Returns and array of JSON responses
{
videoId:
Name:
Length:
Created By:
Thumbnail:
},
....
Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...
Use flat object store for storing raw and transcoded videos.
MySQL DB for storing user data.
User DB
========
UserId Name Location MemberSince Password(encrypted SHA)
User Columnar DB for Video Metadata and Stats to support analytical Queries
=========================================================
BigQuery or Amazon Redshift can be used for this.
Video Metadata
=============
VideoId Name URL Timestamp Format UploadedBy Channel Tags Length
Video Stats Metadata
===================
VideoId NumLikes NumDislikes NumViews NumComments
Use Elastic Search for Searching Videos
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Video Upload
===========
S3 - Amazon S3 - object storage for storing raw and transcoded videos.
Video Viewing/Streaming
=====================
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
Video Upload
===========
S3 - Amazon S3 - object storage for storing raw and transcoded videos.
Video Viewing/Streaming
=====================
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
The video streaming service follows a microservices architecture so that all components are independently scalable.
Upload Service
===========
The upload service uses a DAG (Directed Acyclic Graph) model for the 4 stages of Preprocess, Transcode, Watermark, Thumbnail. Also, multiple workers execute these steps in parallel for different videos for high scalability.
As transcoding is an expensive process, it can be done lazily on demand for certain formats as the video is requested.
Download Service
==============
Video is streamed to the clients browser or mobile device, decoded and played by the internal video player. The most popular streaming protocol is HTTP Live Streaming (HLS).
Use adaptive streaming to serve the video format according to a user's network quality.
Most modern client browsers and devices support buffering. A small portion of the video is buffered in advance so that the user does not experience interruption due to a network blip.
Explain any trade offs you have made and why you made certain tech choices...
Signed URL: allows user to directly upload the raw video without involving the API server.
Concurrent pipelines for the DAG model across videos: Introduce a queue and workers to process the DAG pipeline so that multiple videos can be processed in parallel.
The video streaming service follows a microservices architecture so that all components are independently scalable.
Try to discuss as many failure scenarios/bottlenecks as possible.
Interruption When User is Uploading a Video - this can be addressed by using MultiPart upload where the video is uploaded in small chunks so that upload can be resumed where it left.
Also require a mechanism to clean incomplete uploads on S3 due to failures.
Many duplicate videos - introduce a de-duplication and compression job avoid load on the video store
Explosion in the number of uploaded videos - as the service becomes more popular, the number of videos can increase exponentially. To maintain performance, use hierarchical storage to store and retrieve effeciently at such scale.
Network Blip during Streaming: use Adaptive Streaming. The bitrate of the video changes dynamically based on network condition.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?
Interruption When User is Uploading a Video - this can be addressed by using MultiPart upload where the video is uploaded in small chunks so that upload can be resumed where it left.
Also require a mechanism to clean incomplete uploads on S3 due to failures.
Many duplicate videos - introduce a de-duplication and compression job avoid load on the video store
Explosion in the number of uploaded videos - as the service becomes more popular, the number of videos can increase exponentially. To maintain performance, use hierarchical storage to store and retrieve effeciently at such scale.
Network Blip during Streaming: use Adaptive Streaming. The bitrate of the video changes dynamically based on network condition.
Sharding of videos as a single site or server cannot support high scale.
Use Recommendation Engine and algorithms to suggest videos. Pro-actively cache videos a user is likely to view soon.