Loading...
POST /videos/upload
GET /videos/{video_id}
video_id (UUID, Primary Key): Unique identifier for the video.title (String): Title of the video.description (Text): Description provided by the uploader.upload_time (Timestamp): When the video was uploaded.uploader_id (Foreign Key): References the uploader's user ID.duration (Integer): Length of the video in seconds.thumbnail_url (String): URL of the video's thumbnail.storage_url (String): Location of the video in the storage system.status (Enum): Status of the video (e.g., processing, available, deleted).Database Choice
For a platform with a high volume of video uploads and views, Cassandra is a strong candidate due to its horizontal scalability, high availability, and ability to handle high read and write throughput. Cassandra’s architecture, based on a leaderless replication model, ensures that data is always available, even in the event of node failures, making it well-suited for a global platform.
Metadata, including video titles, descriptions, tags, and storage URLs, is typically stored in Cassandra. By sharding the data using the video ID as the partition key, queries targeting specific videos can be executed efficiently as the data is localized to the responsible node.
Hot Partitions
However, while sharding by video ID ensures uniform data distribution under normal circumstances, it introduces challenges when a video becomes viral. Viral videos result in concentrated read and write traffic, leading to hot partition. To mitigate this, consistent hashing with virtual nodes can be employed.
Virtual nodes divide each physical node into multiple logical partitions, enabling better distribution of the data and reducing the impact of any single hot partition.
Batch Updates
High-frequency metadata updates, such as view counts or comments, present another challenge. For instance, view counts on viral videos might see millions of updates within a short period. Directly writing each update to the database can lead to contention and latency.
Instead, batching updates and applying them periodically reduces the write load. For example, view counts can be accumulated in memory and written to the database in intervals. This approach balances consistency with performance, ensuring updates remain near real-time without overwhelming the system.
Caching
Caching provides a vital layer of optimization for metadata management. Frequently accessed metadata, such as information for popular videos, can be stored in an in-memory cache like Redis.
Tiered caching enhances this strategy further. Metadata is cached at multiple levels: a primary in-memory layer for ultra-low latency, and a secondary cache on slower storage like SSDs for less frequently accessed items. This layered approach ensures efficient use of resources while maintaining high performance for both popular and less frequently accessed data.
Client
Users interact with the platform through web browsers, mobile apps, or smart TVs. The client uploads videos, fetches metadata, and streams content.
Load Balancer
Distributes incoming requests across multiple API servers to ensure even resource utilization and high availability.
API Servers
Handle user requests like video uploads, fetching video metadata, and retrieving playlists. These servers are stateless and horizontally scalable.
Video Metadata Service
Stores and retrieves metadata such as titles, descriptions, and upload timestamps. This service uses a database like Cassandra for scalability and high availability.
Blob Storage
Stores original video files and transcoded video segments. Services like Amazon S3 or Google Cloud Storage are used.
Transcoding Pipeline
Processes uploaded videos into multiple resolutions and formats using a Directed Acyclic Graph (DAG) scheduler. Outputs are stored in transcoded storage.
Content Delivery Network
Distributes video segments and thumbnails globally to minimize latency for users and reduce the load on central servers.
Caching Layer
Caches frequently accessed metadata (e.g., video descriptions, view counts) using an in-memory store like Redis.
Analytics Service
Tracks metrics such as view counts and user interactions. Handles batch updates to reduce database contention.
Recommendation System
Suggests videos to users based on viewing history and preferences.
Parallel Uploads
In a traditional upload process, a video file is transmitted as a single continuous stream. While straightforward, this approach can become a bottleneck when uploading large video files, particularly over unreliable or slow networks. Parallel uploads address this limitation by dividing the video into smaller, independently manageable chunks that can be uploaded simultaneously.
The process begins by splitting the video into GOP-aligned (Group of Pictures) chunks. GOP alignment ensures that each chunk contains a self-contained sequence of frames, including keyframes, that can be processed or decoded independently.
Once the video is chunked, the client uploads each chunk in parallel to the designated storage, such as Amazon S3 or a similar blob storage system. Parallelism enables multiple chunks to be transmitted simultaneously over different network connections or threads. This significantly reduces the total upload time.
In addition to improving speed, parallel uploads enhance reliability. If the upload of a specific chunk fails due to a network issue, only that chunk needs to be retransmitted, rather than restarting the entire upload. This fault isolation reduces the impact of intermittent network problems.
Furthermore, parallel uploads support resumable uploads, allowing users to pause and resume their progress. Metadata associated with each chunk tracks its upload status (e.g., completed or pending). When resuming, the client skips already-uploaded chunks and focuses on incomplete ones.
Regional Upload Centers
For platforms with a global user base, the geographical location of storage servers significantly impacts upload latency and throughput. A user in Asia uploading to a storage server located in North America may experience high latency, reduced upload speeds, and increased chances of network interruptions. Regional upload centers solve this problem by positioning storage endpoints closer to users.
Regional upload centers leverage CDNs (Content Delivery Networks) or geographically distributed blob storage to act as localized upload endpoints. For instance, users in Europe might upload their videos to a storage endpoint in a European data center, while users in South America use a South American endpoint. This proximity reduces the round-trip time (RTT) for each network packet, resulting in faster and more reliable uploads.
The workflow for regional uploads involves the following steps:
By placing upload centers in high-demand regions, the platform can cater to users in diverse locations while reducing the strain on central servers. This setup also provides fault tolerance, as uploads can be redirected to alternate centers if a specific region faces issues.
Parallel Uploads and Regional Centers Combined
Parallel uploads ensure that the client can fully utilize its local resources, such as available bandwidth and CPU, to upload chunks simultaneously. Regional upload centers reduce the latency for each chunk, further enhancing overall performance.
For instance, a user in Asia uploading a large video might connect to a regional upload center in Singapore. The video is split into chunks, and each chunk is uploaded in parallel to the nearby center. This setup minimizes the time required for the user to complete their upload, even over variable network conditions. Once all chunks are uploaded, the regional center can either process the video locally or replicate it to a global storage hub for further processing.
Video Splitting
When a user uploads a video, the first step is to split it into smaller segments. This is typically done using GOP alignment as mentioned.
Splitting the video has several advantages additional advantage:
The original video file is fetched from blob storage, and the splitting operation generates several smaller chunks. These chunks are stored in temporary storage, such as a high-speed disk or blob storage, to serve as the input for the next stage of the pipeline. Temporary storage ensures reliability, allowing the pipeline to retry operations in case of failures without reprocessing the entire video.
Segment Encoding
Once the video is split into segments, each segment undergoes transcoding, a process where the video is converted into multiple formats and resolutions. Transcoding typically involves applying codecs (such as H.264 or VP9) to compress the video efficiently while maintaining visual quality. Each format is tailored to specific devices, browsers, and network conditions.
In parallel with the video encoding, the system processes the audio stream and generates additional assets, such as subtitles or closed captions. These operations are independent of the video transcoding and can be executed concurrently to maximize efficiency. Subtitles may involve transcription, translation, or formatting tasks, depending on the requirements.
The output of this stage includes several encoded video files for each segment, each representing a unique combination of format and resolution. These files are stored back into blob storage or a similar system for further processing.
Manifest File Creation
After all segments are transcoded, the system generates manifest files that serve as indices for the video. These files describe the available formats and resolutions of the video, as well as the location of each segment in blob storage or the CDN. A primary manifest file lists all versions of the video and links to individual media manifest files, which provide detailed information about specific formats and resolutions.
Manifest files are essential for adaptive bitrate streaming, where the client dynamically selects the most appropriate video segment to download based on current network conditions or user settings. These files ensure seamless playback, allowing the client to switch between resolutions without interrupting the viewing experience.
DAG Scheduling
The entire transcoding pipeline is orchestrated using a Directed Acyclic Graph model. A DAG represents the dependencies between various tasks in the pipeline, ensuring that tasks are executed in the correct order while enabling parallelism wherever possible. For example, video splitting must precede transcoding, but different segments can be processed simultaneously.
A DAG scheduler (such as Temporal, Apache Airflow, or a custom solution) manages the execution of tasks across a distributed set of worker nodes. The scheduler ensures that each worker node receives a task it can handle efficiently, taking into account resource availability and task dependencies. It also monitors task progress and handles retries in case of failures.
For example, once the video is split into segments, the DAG scheduler might assign the transcoding of Segment 1 to Worker Node A and Segment 2 to Worker Node B. Simultaneously, another worker node might process audio or subtitles. Once all segments are transcoded, the scheduler triggers the creation of manifest files and marks the pipeline as complete.
The transcoding pipeline produces the following outputs:
These outputs are uploaded to the storage system and/or CDNs, ready for streaming. The pipeline's parallel and distributed design ensures scalability, allowing the system to handle thousands of videos simultaneously while minimizing processing time.
Adaptive Bitrate Streaming is a method designed to ensure smooth and uninterrupted video playback, regardless of the user's network conditions. This approach allows clients to dynamically adjust the quality of the video they are streaming, switching between resolutions and bitrates in real-time based on bandwidth availability. The technique enhances the user experience by minimizing buffering and providing the best possible video quality for the current connection.
Segment Selection
When a user initiates video playback, the client begins by downloading a manifest file from the server or CDN. This file contains metadata about the video, including the available resolutions, bitrates, and the locations of individual video segments stored in a CDN or blob storage. The client analyzes the network's current state, such as bandwidth, latency, and stability, to determine the most suitable resolution and bitrate for the initial segment.
As the video plays, the client continuously monitors network conditions. If the bandwidth improves, the client may switch to downloading higher-resolution segments to enhance video quality. Conversely, if the network degrades, the client can reduce the resolution and bitrate, fetching smaller-sized segments to prevent buffering or playback interruptions. These transitions occur seamlessly between segments, ensuring that users experience consistent playback.
The client achieves this adaptability by relying on small, independently decodable video segments (e.g., 2–10 seconds in length). Each segment is pre-encoded at multiple resolutions and stored in blob storage or a CDN. This segmentation enables real-time quality adjustments without requiring the user to reload or restart the video.
Caching Strategy
Caching plays a crucial role in adaptive bitrate streaming, particularly in reducing latency and ensuring high-quality playback for popular content. By leveraging Content Delivery Networks, the system brings video segments closer to the end users. CDNs consist of geographically distributed edge servers that store copies of frequently accessed video data, significantly reducing the time required to fetch segments.
For videos with high demand, such as trending or viral content, the system caches the most popular segments across multiple resolutions on the edge servers. This ensures that users accessing these videos experience minimal latency, as the data is served from a nearby CDN node rather than the central storage.
The caching mechanism is supported by metadata caching strategies. Since the metadata (such as the manifest file or video details) is accessed frequently but requires minimal storage, it is often cached in an in-memory data store like Redis. An LRU (Least Recently Used) caching strategy is typically employed for metadata caching.
This strategy ensures that the most frequently accessed metadata remains in the cache, while less-used data is evicted when the cache reaches capacity. For example, if a user plays a video, the manifest file and metadata related to that video are cached. If another user requests the same video, the metadata is served instantly from the cache without querying the database, significantly reducing latency.
By batching updates for view counts, likes and etc, the system avoids the need to make immediate and frequent writes to the database, which can create contention and strain resources during periods of high traffic, such as when a video goes viral.
This approach allows the system to focus resources on optimizing other parts of the platform, such as video playback or metadata lookups, which are more critical to the user experience. However, the trade-off is that users may see slightly outdated data, like delayed view counts or interactions, which might not be ideal for content creators tracking their video’s real-time performance.
Another trade-off we might want to consider is whether we want to introduce tiered storage.
With tiered storage, we would store less frequently accessed or unpopular videos in less expensive storage tiers, such as cold storage solutions like AWS Glacier, while keeping high-demand videos in faster, more expensive storage.
This strategy would result in significant cost savings, especially given the large volume of videos uploaded daily. However, the trade-off would be higher latency when retrieving these videos from lower-cost storage. This could lead to a degraded user experience for viewers attempting to access videos stored in these tiers, as they might experience delays before playback begins.
Chunk Tracking
When a user initiates an upload, the video file is divided into smaller chunks, typically ranging from 5 to 10 MB in size. Each chunk is uniquely identified using a fingerprint, often generated as a hash of the chunk's data. This fingerprint serves as a reliable identifier to track the status of each chunk independently. The client begins by sending a request to the backend, registering the file for upload and providing its metadata. The metadata includes essential details such as the total number of chunks, their fingerprints, and the initial status of each chunk, which defaults to "NotUploaded."
The client then uploads each chunk directly to the blob storage, such as Amazon S3, using a multipart upload mechanism. After each chunk is successfully uploaded, the storage system acknowledges the completion of that chunk. The client updates the metadata with the status of the uploaded chunk, marking it as "Uploaded." This metadata is crucial because it allows the system to maintain a record of which chunks have been successfully stored and which are still pending, even if the upload process is interrupted.
If an upload is paused or interrupted, the client can query the backend to retrieve the current metadata for the file. This metadata provides a comprehensive view of the upload status, enabling the client to identify chunks that have already been uploaded and skip them during the resumption. This capability significantly reduces redundant data transfer and ensures efficient use of network resources.
Event Notifications
The storage backend plays an active role in keeping the metadata accurate and synchronized. For instance, in systems like Amazon S3, event notifications are triggered whenever a chunk is successfully uploaded. These notifications are sent to a pre-configured listener, such as an AWS Lambda function or a backend service, which processes the event and updates the corresponding metadata.
The event notification includes details such as the file identifier, chunk fingerprint, and upload status. Upon receiving this notification, the backend updates the metadata associated with the file, marking the specific chunk as "Uploaded." This mechanism ensures that the metadata is always consistent with the state of the storage system, even if the client disconnects before explicitly updating the status.
Event notifications also enable additional workflows, such as logging upload progress, triggering retries for failed chunks, or initiating downstream processing once all chunks are uploaded. For example, when the final chunk is uploaded and marked as complete, the backend can transition the file to the next stage, such as video transcoding or indexing.
The Upload-Resume Process
When a user resumes an interrupted upload, the client retrieves the latest metadata from the backend. This metadata acts as a roadmap, showing which chunks are already in storage and which remain incomplete. The client then resumes the upload process, skipping over completed chunks and focusing solely on the ones marked as "NotUploaded." Each newly uploaded chunk is acknowledged by the storage system, and its status is updated in real time through event notifications.
This process is robust enough to handle network failures, client crashes, or even server-side issues. The reliance on metadata and event notifications ensures that the upload state is preserved and can be resumed without unnecessary duplication of work. Additionally, the use of small chunks minimizes the impact of any individual failure, as only the specific chunk needs to be re-uploaded rather than the entire file.
Stateless API Servers
All API servers are stateless, the system can scale them horizontally. A load balancer distributes incoming requests among these servers, ensuring even utilization and preventing overloading of individual instances.
Elastic Video Processing
Elastic video processing further enhances scalability by addressing the resource-intensive nature of transcoding. Transcoding pipelines are designed to adapt to the number of videos being uploaded. When upload rates increase, such as during peak hours, the system provisions additional transcoding worker nodes or serverless compute resources to handle the extra workload.
Conversely, during off-peak periods, these resources can be scaled down to reduce costs. This elasticity is achieved using tools like Kubernetes for container orchestration or serverless platforms like AWS Lambda for on-demand scaling.
Handling Burst
Queuing systems act as buffers between upload requests and processing pipelines. When the system experiences a sudden influx of uploads, incoming requests are placed in a queue rather than being processed immediately.
This prevents resource overloading and ensures that every request is eventually handled. For example, if the transcoding pipeline is operating at capacity, additional videos are queued, allowing the pipeline to process them sequentially as resources become available. Queuing systems not only absorb bursts but also provide visibility into workload patterns, enabling predictive scaling and efficient prioritization of tasks.
Dynamic Load Balancing
Dynamic load balancing ensures efficient utilization of system resources under varying traffic patterns. Instead of static allocation, load balancers dynamically distribute requests based on factors such as server health, current utilization, and latency.
For example, if one API server instance becomes overloaded or experiences high latency, the load balancer redirects new requests to less-burdened servers. Advanced load balancers can also incorporate geographic routing, directing users to the nearest server or data center to minimize latency and improve response times. This adaptability ensures optimal performance even during traffic surges.
Upload Failures
For upload failures, retry mechanisms are essential. Uploads often fail due to transient network issues, particularly for large files or in regions with unstable internet connectivity. The system breaks video uploads into smaller chunks, and if a specific chunk fails to upload, the client retries only that chunk instead of restarting the entire upload.
This targeted retry approach reduces bandwidth usage and ensures faster recovery. The use of resumable uploads, combined with metadata tracking for chunk statuses, ensures that uploads can resume seamlessly from the point of failure.
Transcoding Errors
Transcoding errors require a different strategy due to the computational complexity involved. Transcoding tasks are often orchestrated using a Directed Acyclic Graph (DAG), where each node represents a specific task, such as encoding, audio processing, or thumbnail generation. If a node in the DAG fails, the system regenerates the workflow, starting from the failed task.
For example, if the encoding of a particular video segment fails, the DAG scheduler retries the encoding task while preserving the progress of other completed tasks. This approach avoids redundant work and ensures that errors in one part of the pipeline don’t propagate to the entire workflow.
Streaming Errors
Streaming errors, such as playback interruptions, often arise due to CDN issues, network fluctuations, or client-side limitations. Graceful degradation plays a key role in mitigating the impact of such errors. When high-resolution segments fail to load due to bandwidth constraints, the client can request and play lower-quality segments that are more likely to load successfully.
This ensures that users experience continuous playback, even if it’s at a reduced quality. Similarly, if a segment is unavailable in the CDN, fallback mechanisms direct the client to fetch the segment from the primary blob storage, albeit with slightly higher latency.
API Failures
System-wide resilience is bolstered by automatic failover mechanisms. In the event of an API server failure, the system’s load balancer detects the downtime and redirects requests to other available servers. Because API servers are stateless, this transition is seamless, with no data loss or user impact.
Similarly, in the metadata layer, if a database node becomes unavailable, the system promotes a replica node to serve as the new primary. This ensures that metadata queries and updates continue without interruption. For caching layers, redundant cache replicas ensure that data remains accessible even if one replica fails. These failover strategies ensure high availability and maintain the system’s operational integrity under failure conditions.
Redundancy
Redundancy complements error-handling strategies by providing additional capacity to absorb failures. In the transcoding pipeline, redundant worker nodes ensure that tasks can be reassigned if a node crashes.
For metadata caching, multiple replicas ensure that the cache remains responsive, even during high-demand scenarios or partial outages. This redundancy reduces single points of failure and enhances fault tolerance across the system.
Dynamic caching introduces the possibility of using machine learning models to predict video popularity and proactively cache likely-to-be-accessed videos, further enhancing user experience by reducing retrieval latency.
We can also leverage edge computing for pre-fetching popular video metadata and thumbnails can also reduce latency by bringing key resources closer to the user.
Adaptive replication strategies could dynamically increase the replication factor for frequently accessed metadata, distributing the load across more nodes to maintain system responsiveness under heavy demand.