System Requirements

Functional:

1. We want to implement an authentication feature that lets users log in using Google, Facebook etc.
2. Users should be able to upload videos of indefinite size any number of times.
3. Users should be able to watch videos without downloading them, that is, stream videos.
4. Users should be able to search for specific videos.
5. Users should be able to store videos, encode them, process them in blocks, store them in different resolutions, distribute them all over the world.

Non-Functional:

1. Availability - Service should be highly available, we should never be able to not watch or upload videos.
2. Consistency - We are fine with not having strong consistency but would like eventual consistency. New videos uploaded by a channel can show up a few minutes late to the subscribers of the channel.
3. Durability - Users can lose uploaded videos. Even deleted videos are kept in storage.
4. Backup - Videos should be replicated across different data centers to ensure durability.
5. Latency - Streaming should be smooth, in case the user has a poor internet connection, the stream should automatically switch to a lower resolution video. Ideally, we want to send the user the first 1 minute of the video within half a second and then keep downloading video blocks in the background.

Capacity Estimation

1. We assume that each video we have is 1 GB on average. This seems like a large number but with 4K videos getting prevalent along with HDR and Dolby Vision, this seems like a good estimate. We also assume that we store 1 Billion videos on our service, meaning we need a storage of 10^9 * 10^9 bytes of data = 10^18 bytes of data = 1 Exabyte.
2. We would also need to replicate our videos across different data stores for backup, leading to at least 5 Exabytes of needed storage.
3. We assume 1 million active concurrent users, leading to the creation of roughly 500k requests per second.
4. We assume that roughly 5% of our total videos are considered popular at any given time, all of which would need to be stored in CDNs.
5. Amount of data needed to be stored in CDNs = 5% of 1 Exabyte = 5 * 10^13 = 50 TB of data should be stored in CDNs around the globe.

API Design

1. Watch Video
   1. User makes a request to watch a certain video.
   2. This request is forwarded to a an API server via the load balancer.
   3. The API server retrieves the video metadata from the metadata database.
   4. If it is present in the cache (CDN), the user streams the video from the CDN.
   5. If not, the user is connected to the appropriate object store and gets the video chunk from there.
2. Video Upload
   1. The client starts uploading a video over a stateful server.
   2. In case of a disconnection, the user is reconnected to the same server and they can resume their upload from where they left it.
   3. The S3 sends the video to SQS for further processing.
   4. The encoding worker retrieves the video chunks from the object store and begins processing the video and then storing them in a different object store which will our main store that all has the encoded videos.
   5. The Object Store the user uploads to serves as the original video store only.
   6. The SQS server also extracts metadata from the video and uploads in the metadata database.
   7. This architecture depicts a distributed system designed for scalability and efficiency, leveraging services like load balancers, caches, distributed storage (S3), and a coordination service (Zookeeper) for managing distributed nodes. It specifically addresses the needs of video processing and storage, including the initial upload, processing, and retrieval of video data.

Database Design

We have two primary stores, one fr storing videos and the other for storing video metadata. The data stores are detailed below.

1. Video Store
   1. Stores the original video uploaded by the user, these are never deleted.
   2. Stores the processed videos. The processed videos are videos that are stored as small chunks, which are typically 1 to 4 MB in size. The small chunks allow us to seamlessly switch video resolutions based on network quality.
   3. The processed videos are copies of the original video but stores in different resolutions and formats, ensuring we support a wide range of devices, specially older ones that do not support larger resolutions and newer audio and video codecs.
2. Video Metadata
   1. We can use a relational database like PostgreSQL to store video metadata, like its title, duration, number of likes, dislikes, views etc.
   2. We can also store this information in a non-relational database like MongoDB which will allow for better scalability.
3. User Metadata
   1. We can use a relational database to store user data like their name, login credentials etc.

High-level design

Initially, a client requests to read data by connecting to a load balancer, which then forwards this request to a Server. The Server interacts with a Metadata database to access necessary metadata. For reading operations, the Server first attempts to retrieve the s3 url from a CDN; if the required data isn't available there, it proceeds to fetch the data from the metadata to get the S3 chunks.

For write operations, a client sends a write request to the LoadBalancer, which is then directed to a server. This server asks Zookeeper for a write node, receiving a node ID in response. With this ID, the server performs a stateful write operation and confirms the action back to the client.

When a client begins sending a video, the data is routed to a Stateful Node. This node is responsible for storing the raw video data in an S3 store and then sends the raw S3 URL for processing to an AWS SQS. An Encoding Worker retrieves this task from SQS, processes the video into S3 chunks, and updates the Metadata database accordingly.

The read flow updates the 'hotness' of the video and makes the adjustment for the CDN and how long the video should live in the CDN.

flowchart LR B[client] -- Read video/id --> C[LoadBalancer] C[LoadBalancer] -- credentials --> Q[Auth Service] Q[Auth Service] -- signed auth token --> B[Client] C --> D[Server] D[Server] --> Y[Metadata database] D[Server] -- READ | Queries metadata --> F[CDN] D[Server] -- READ | Requests data from S3 --> M[S3 Chunks] D[Server] -- video id not found --> B[client] A[client] -- write video data --> Z[LoadBalancer] --> I[Server] -- request write node --> K[Zookeeper] K[Zookeeper] -- provide node ID for writing --> I[Server] I[Server] -- write data to storage node --> A[client] A[client] <--> Q[Auth Service] A[client] -- stream video content --> U[Stateful Node] U[Stateful Node] -- upload raw data to S3 --> H[Raw S3 Store] U[Stateful Node] -- provide S3 URL for processing --> T[SQS] J[Encoding Worker] <--- T[SQS] J[Encoding Worker] --> M[S3 chucnks] J[Encoding Worker] --> Y[Metadata database]

Request flows

In a system involving multiple components for handling read and write operations, as well as video processing, the sequence of interactions begins with a client initiating a read request. This request is first received by a LoadBalancer, which then forwards the request to a Server. The Server performs multiple tasks based on this request: it accesses a Metadata Database to retrieve necessary metadata, checks with a CDN (Content Delivery Network) for cached content ("READ | first here"), and accesses S3 Chunks for data not found in the CDN ("READ | go to s3"). It's noted that reads are managed through either the CDN or directly from S3 Chunks, depending on where the requested data resides.

For write operations, the Client sends a write request to the LoadBalancer, which again forwards this request to a Server. This time, the Server requests a write node from Zookeeper, a service that manages the distribution of write operations across nodes. Zookeeper responds with the node ID that should take the write operation. The Server then communicates back to the Client, indicating it can perform a stateful write.

When the Client begins sending a video, it directly interacts with a Stateful Node. This node is responsible for putting the raw video data into an S3 store designated for raw content. It also sends the raw S3 URL for processing to SQS (Simple Queue Service), which queues the video for processing. An Encoding Worker picks up this task from SQS, processes the video into S3 Chunks, and updates the Metadata Database with the new metadata related to the processed video. This series of interactions ensures that read and write operations, along with video processing, are efficiently managed across different components of the system.

sequenceDiagram participant Client as Client participant LoadBalancer as LoadBalancer participant Server as Server participant MetadataDatabase as Metadata Database participant CDN as CDN participant S3Chunks as S3 Chunks participant Zookeeper as Zookeeper participant StatefulNode as Stateful Node participant RawS3Store as Raw S3 Store participant SQS as SQS participant EncodingWorker as Encoding Worker Client->>+LoadBalancer: Read video LoadBalancer->>+Server: Forward read request Server->>+MetadataDatabase: Access Metadata Server->>+CDN: READ | Initial content load Server->>+S3Chunks: READ | Retrieve content from S3 Client->>+LoadBalancer: Write video LoadBalancer->>+Server: Forward write request Server->>+Zookeeper: Request write node assignment Zookeeper-->>+Server: Provide node ID for writing Server->>+Client: Confirm successful data write Client->>+StatefulNode: Begin video transfer StatefulNode->>+RawS3Store: Store video raw data in S3 StatefulNode->>+SQS: Dispatch S3 URL for further processing SQS->>+EncodingWorker: Notify of new processing task EncodingWorker->>+S3Chunks: Save encoded data chunks EncodingWorker->>+MetadataDatabase: Update video metadata

Detailed component design

The first clarification is to ensure that whenever we are only polling from the SQS queue when the enqueuing service is ready to take the load. With that in mind, it's pretty obvious to ensure that the encoding service is able to read from the S3 bucket and process the video doing the following steps:

1. decoding the video from whichever type it is (.imec, mp4, etc.) into a standard video format of our service
2. decoding video into many resolutions from (140p to 4k HD).
   1. Note how encoding may take a longer time for a higher resolution, so we'll make sure that the video can be published sooner at a lower resolution.
   2. Another important thing to see is that uploading the chunks to S3 should be idempotent, we don't care if the chunk is uploaded more than one time as long as it's uploaded at least once. That allows to utilization an unbounded workflow engine like AWS Step Functions to make this encoding process easier to implement
   3. We also want to ensure that the video that is being uploaded doesn't contain restricted material, the encoding service would handle these checks asynchronously with the encoding steps.
   4. Encoding is very computationally heavy and it is the bottleneck of our system

The second thing we would to focus on is our CDN design. To ensure the users have a good experience, we'd like to ensure that we can provide these core functionalities in our service:

1. The chunks are small enough that we can buffer them in the client's memory to allow for smoother video play
2. We can dynamically upgrade/downgrade from the resolution on the fly by maintaining a stateful read flow and checking if the user's internet connection is able to receive chunks fast enough
3. The user is able to preload/download the entire video to allow for offline play in any resolution
4. Our CDN stores link to the S3 URL of the chunk (not the other video file) to perform fast sequential-like reads

Trade offs/Tech choices

The key idea is that we want to trade off write latency/consistency (writes can be really slow) for availability (our system relies on reading from highly available S3 service), latency (most of the 'hot movie chunks' are stored in our store in our CloudFront CDN), and durability - we persist the write in the SQS to make sure that the message is read at least once.

We do have eventual consistency in the system because we don't want to wait until MongoDB finishes replicating the metadata across all nodes before the video is considered available.

Additionally, we trade-off waiting until high resolution is available by allowing to see 140p encoded video which completes faster

Failure scenarios/bottlenecks

1. We have quite a bit of single point of failure - which is our S3 bucket. If S3 is down, we pretty much can't do any reads or writes. The good part is that AWS S3 is highly available and that is not a major concern, but if it is down, it would bring the entire system down.

2. We can be also concerned that in case the authentication service is down, we would completely lose access to writes, but we don't have to completely lose access to reads. We can consider letting users watch the videos, even if they can't sign in.

3. Another point of failure/bottleneck is the availability and deployment of CDN, we would want to ensure CDN is deployed across the entire globe (as well as our database nodes are distributed) to minimize latency worldwide.

Future Improvements

1. The system can definitely be improved by adding AWS Step Functions as the workflow engine for processing videos, we would need to ensure that the upload function is idempotent and we can upload something at least once to make sure that it works. That would be an ideal example of a workflow since we have a sequence of tasks that need to be performed at least once.
2. The second portion to figure out would be the CDN approach. I'd love to get more research done on the video owner/video author to determine if this video needs to be preloaded in Cache to allow for faster reads.
3. It would be great to dive deep into the moderation service. Moderation service would take the entire video and use Ml/AI to detect if it infringes copyrighted material (either in audio or video). If the copyrighted material is only in audio, we can remove the voice audio from the encoded video, notify the user, and upload the video. If the video is copyrighted, we can notify the user of the certain frames that we deleted and ask the author to explain/remove the content. If the video is outright malicious/inappropriate (excessive violence, adult content) - the video can be outright rejected and the author warned and or struck.
4. The auth service can definitely be improved to allow either sign-ups with the password/email or token from SSO services like Google, Facebook, etc. User data can be stored in MySQL database.