Loading...
Users should be able to watch videos
Users should be able to upload/manage their videos
Users should be able search for videos from keywords in description
Users should be able to comment on videos
System should be highly available with low downtime
We will have eventual consistency, a few seconds after a video has been uploaded, changed, etc
Streaming should be fast and optimized for different device types
All videos should be available globally
Estimate the scale of the system you are going to design...
GET /video/:id -> Video
POST /video ->
header: Session Token
{
"name":
"description"
}
binary video payload
PUT/DELETE /video/:id
header: Session Token
GET /video/search?query={}&filters={}&sort={} -> Video[]
POST /comment/:video_id
header: Session Token
{
"comment":
}
GET /comment/:video_id -> Comment[]
User:
id
name
Video:
id
name
description
streams
duration
Comment:
user_id
video_id
comment
We use a Load Balancer and an API gateway to enable us to separately scale our specialized services and to facilitate horizontal scaling if needed.
The video service is the heart of the application where when users make a GET call they get a response with the videos metadata and link to the video's stream that the client can playback on their device. Videos can be uploaded on the platform using a POST request where the caller sends the video binary as a payload to the service which then uploads it to external blob storage like Amazon S3. After the video is uploaded a link to the video in blobstorage is stored in the Video database. We can use a relational database here to store the relationship between videos and users.
The comment service is a simple CRUD service backed by a NoSQL database which is keyed by video_id quickly and easily retrieve all the comments for a video. We use a NoSQL database here because of its ability to scale out of the box and because our data doesn't require strong ACID properties and is weakly relational and is well represented as a document with a key.
The Search service just queries the database with direct query to the database to look for names or descriptions similar to the query string. We can improve this by introducing an elastic search index that is hydrated by a change data capture service to keep the index populated. It can be used to create a "reverse" index to make searching for keywords fast!
The POST request to the Video service starts an upload to the blob sotrage and then creates entries in the video database and also queues a job into the video transcoding queue to produce multiple copies of the video at different resolutions and bitrates to optimize steaming for different mediums. We can also compress the original video to store it more efficiently
On the GET request to the video service we get a list of stream urls which can be used by the client to stream an appropriate format for the user's device and bandwidth.
The comments DB will have high read and write loads so it should be sharded using a key such as video ID so that reads and writes for different videos are sent to different DB deployments.
Video Uploads can be slow and fail in the middle causing a bad user experience. We should make this better by uploading videos in chunks and checking for chunks that have already been uploaded by using a hash like MD5. This is allow users to resume uploads if left off.
Our services are stateless and can scale horizontally to provide redundancy for our services and we can create replicas for our databases to prevent data loss. For our Video DB we spread out records by using a sharding strategy to help with this also.
We can introduce a cache for the video service and comments service to store results for the most popualar entities on the platform. This would reduce strain on the database and provide better response times to the user.
Videos can be distributed to the clients using CDNs to have close-to-optimal download and streaming speeds. CDNs will distribute the videos across the globe to provide low latency and high availability for assets that are used often according to different access patterns across the globe.
Comments for popular videos can be queried and written to multiple times during peak times, so we can introduce a write through cache to prevent the DB from being overloaded on both reads and writes.