Loading...
Traffic Estimation
Bandwidth estimation
Storage estimation (for videos only here)
load_video(video_id) -> loads a video with this video ID
upload_video(video_file, title, description, thumbnail_photo) -> uploads the video onto our system
search(query) -> search videos based on the keywords given in the query
like(video_id) -> likes videos
comment(video_id) -> comment on videos
We will need a number of Databases
Database 1: Videos DB
We will use a cloud storage like Google cloud to store our videos
Database 2: Video meta data, like and comments
The meta data will include title, description, and thumbnail URL. In a separate table we will keep track of the likes and comments. these two tables are relational, hence we might want to use a SQL database to keep track of these data. However the issue with SQL DB is that it is not very scalable, our system might have a lof of people liking and commenting on many different videos, hence we might want to use a NoSQL database like MongoDB instead
Service 1: Downloading videos
Service 2: Uploading videos
Service 3: Searching videos
Service 4: Likes and comments
Database 1: Video meta data, likes and comments
Database 2: Cloud storage for video chunks
As you can see there are 4 different types of services and 2 different types of databases that are needed to support a system like YouTube
scenario 1: Loading video
The server will fetch the meta data of the video from DB, then it will load up the first chunk of the video from our cloud storage to return to the user, as the user continues to watch the video, our system will continue to load up more and more chunks from the cloud storage to return to our user
scenario 2: uploading video
the system will break up the video into multiple chunks and uplaod them onto our cloud storage, our system will save meta data like title, description and thumbnails onto our DB
scenario 3: searching
The system will take in the query and break the query into multiple keywords or key phrases, with these key words they will perform a search into our metadata DB to retrieve 10 videos to return to our user, as the user requests to load more suggestions, our system will continue to return more
scenario 4: liking and commenting
The system will enter the relevant information into our likes and commetns table
Detail 1 : Loading videos
When loading videos into memory, the videos should be loaded up in chunks. If you load up a video in full before returning it to the user, it will take up way to long, especially for videos above 10 minutes. Hence when a user clicks on the video, the first chunk can be loaded up very quickly and returned, as the user continues to watch he video the next 3-5 chunks are being loaded up into memory, the chunks will be returned to the client in order. We can do this by maintaining a linked list of chunks in memory, where each linked list is tied to a userID of the user that is currently viewing the video. Every time the front node has been loaded into memory, it will be sent to the client and the linked list moves on to the next chunk. This prevents the system from returning the chunks to the client out of order.
Detail 2: uploading
When uplaoding the video, the system saves the title, description and thumbnail into memory. But the video file itself will be broken up into different chunks and stored into the cloud storage, the chunks will be about 30 seconds to 1 minute long each and each chunk will have a metadata tied to it. They will be stored in various locations of the cloud storage and they will be linked by having a 'next_chunk' URL, which will allow the current chunk to be able to find the next chunk of the video when the video is being loaded up
Detail 3: Searching
the query will be broken into keywords and key phrases using an algorithm that we will write ourselves, words like 'the' or 'is' or 'a' dont provide meaning itself, but when put together with other words like 'the cat' or 'is a amazing stunt' the phrase will be more useful to be used as key word searches. We will devise an algorithm to break down the query like this. We then go into the DB to load up 10 videos with some of the keywords that are inside, as the user scrolls down, more suggestions will be loaded
Detail 4: Likes and comments
The system simply stores them in the DB and loads them up when requested
Detail 5: Data sharding
How can we divide the chunks between our cloud storages? We could lump all chunks in a video into one of the cloud storage, but the problem would potentially be hot videos, if there is 1 or a few popular videos that reside in a single cloud storage, this storage instance will have a lot of load requests, leading to a slower instance or even crash. hence we will divide the chunks by their (video_id, chunk_id), this ensures that the chunks are evenly distributed across the storages, preventing a single storage from being overloaded.
Detail 6: Caching
We will want to cache popular videos for each day. However, what we dont want to do is to cache the whole video into the cache, this minimizes the number of videos we can store in a cache. Instead we should only store the first 2-3 minutes of each popular video into memory. This gives enough time for our system to load up the remaining chunks of the video and return it to our user. Assuming these 2-3 minutes are 2MB each, we could potentially store up to 100,000 videos in a 200GB cache.
Detail 7: Clearing up expired videos
We will want to clear videos that are over 10 years old. We can do that by using a separate server that will clear these videos.
The bad thing about this system is that the user can potentially watch videos half way and then not have a chunk to show (due to slow internet for example) then the video would have to 'buffer' before resuming. This is a trade off we have to make because if we potentially try to load up a full video or huge chunks into memory before we play the video for the user, then it would take too long for a video to play, the user will have to wait for an extremely long time before it does play.
We want back up storages for each cloud storage, in case one of them crashes
When searching for videos, we will want to return highly suggested videos with the key words instead of just videos with the keywords. This can be done by creating a sophisticated algoirithm that takes in the user's tastes and the current popular videos
In addition, using a similar algorithm, we could have a suggestions tab that displays videos that a user might probably like to see.