Requirements

Functional Requirements:

for such a video source website, I believe we should definitely start with the basic functions first. I can list a few functions, we can then discuss and let me know what you think on these items:

1. Uploading video for YouTuber
2. User to watch a pre-saved video
3. User to watch a streaming video
4. User to talk with other users, like a comment session
5. prossible search bar is required

Non-Functional Requirements:

We will definitely need some low-latency play-back mechanism, and since this is YouTube, I assume the global users would be huge, so high-availability is a must, as well as scalability.

Capacity Estimation

Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...

API Design

We will need the following RESTful control services:

1. POST: uploading a new video, fetch the assigned uploading URL
2. POST: posting a signal that put an end of the video
3. GET: playback segment API for user to retrieve the correct data from the Data Plane
4. GET: metadata for a video, like title, description, others
5. GET: search for video/uploader/etc...
6. POST: probably need a heartbeat signal

Additionally we need a Data Plane services:

1. Raw source Object Bucket, segment/manifest bucket
2. CDN in front of the segment bucket

High-Level Design

So basically like what I have drawn in the Diagram section: a client is talking to two basic data plane. One is the control plane, the other is the object bucket (some said blob storage).

Firstly the control plane is holding metadata services, providing all APIs that I listed above. But before the Metadata Service, I believe we should have an API Gateway, where our Load Balancer is integrated in it to ensure high availability and high scalability, we probably can have rate limiter protecting our services for free if we choose any modern cloud platform. We will then have a metadata DB which could be SQL based, as the metadata format is very likely to be structural, but as a SQL based DB we need a cache layer to protect it, because we are talking about YouTube/Netflix that has huge huge query-per-second (assuming 100k/s, roughly).

Then for the Data Plane, we need an Object Bucket, like my diagram shown, but once the raw data or let's say the origin copy is stored, we start processing with the transcoding services, which handles by a transcoding message queue, and then a distributor (depending on how many different encoding version we need to do) then writes to the Object Bucket again.

Database Design

Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...

Detailed Component Design

Let's discuss about the Metadata Control Plane:

1. We need to have multiple server available for read, but maybe we can have much less write server, as the read/write ratio is probably 1000:1, or even higher. Also, for write operation, we don't even need a strong consistency, as long as it is eventually persisted, so we really just need to scale for read as that's our main service for.
2. For the Database, a SQL like storage is for video title, uploader, and like/repost etc... those are quite structural data, we can take SQL relational Database, but considering what we mentioned for the read heavy service, probably cache is not sufficient, we may still suffer from a huge amount of read operation frequently, so my idea is to do Database sharding in this case, there could be many strategies but I am not going to dive too deep here, one simple way is sharding by some key as our data is structural. Perhaps we can continue polishing the design in practice like using some other database type (cassandra etc..)