Loading...
User could share videos
User could search videos
User could get snapshot of videos
User could previous videos
The system should have low latency.
System space should be efficient
Estimate the scale of the system you are going to design...
upload_video(local_path)
User table
user_name
user_email
avatar
VideoTable
video_id
upload_time
user_id
title
status: Pending Upload/Created
duration
metadata: {
videos: [
{
resolution:
chunks: []
snapshot_ids: []
}
]
}
Chunk table
chunk_id
chunk_server
video_id
path
duration
start_time
end_time
size
Snapshot table
video_id
thumb_id
path
size
moment
You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...
Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...
During video upload, the client side will cut the video into smaller chunks. Each chunk will be uploaded one after another.
The client side will first go to server to get the chunk server/encoding server address. Then it will start to transfer each chunk. The chunks will be cut in local, so for each chunk, it will have a sum. We can use checksum to ensure data integrity. If any failure occurs, the data upload will reupload that specific chunk so that we do not waste time on uploading data that has already uploaded.
Once the data is uploaded, it will be navigated to encoding server. The encoding server will transform the original video into uniform format. The encoded video will be stored in the distributed filesystem.
For different resolutions, it could be an opportunity to transform into different resolutions. During this process, the database will also be updated with different resolutions.
During encoding, it can also create snapshots and put into snapshot database. Snapshot should have its own distributed FS.
An indexing service could also be added to Encoding service so that users could search the videos. Indexing service includes tokenization, inverted index creation. This we can build a indexing on the videos.
When watching videos, it will fist go to the server and find the right chunks of videos. It will start downloading the chunks one by one. Note, it can preload the next chunk to accelerate data loading.
For distributed FS, recommend to use three replicas. The replicas do not need to be handled by encoding server. There could another service which slowly syncs the data in the backend.
During upload, the server should also check if the video has already been uploaded. If the video is uploaded already, it will directly show the upload is successful.
The snapshot will have much higher read rate. So, we can use master-slave method to store cache.
To scale, we can add a cache layer. For popular videos, they could be stored in cache. We can also leverage CDN to deliver the results globally with low latency.
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?