Loading...
Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...
Assuming 1 billion DAU, and each user on average streams 10 videos per day. We also have on average 1 upload per 10 DAU per day. So we would get 100 million uploads per day.
We would have:
Peak read QPS = 1 billion users / 3600 / 24 * 10 videos per day * 2 (for peak hours) = 230K
Peak write QPS = 100 million uploads per day / 3600 / 24 * 2 (for peak hours) = 2300
On each day, we have 100 million video uploads. Assuming 500MB per video and 1Mb per video metadata, on each day, our storage increases by:
100 million * 500MB = 50PB. We also transcode video into ~5 renditions, so every day our storage increases by 250PB.
Metadata wise, each day the storage increases by 100 million * 1MB = 100TB.
1. Annual Storage & Lifecycle Management
At 250 PB/day, annual storage would hit ~91 EB — clearly unsustainable to keep everything in hot storage. I'd propose a tiered lifecycle:
For costs: assume hot = ~0.02/GB/mo,warm= 0.02/GB/mo,warm= 0.005/GB/mo, cold = ~$0.001/GB/mo. With ~60% of views on top 1% of content, you can aggressively tier without affecting UX.
2. Ingest Bandwidth Spike Handling
At 100M uploads/day, peak ingest bandwidth hits ~5.8 Tbps. Handled via:
3. CDN Cache Sizing
10B daily views → ~115K QPS sustained. Average video bitrate ~5 Mbps (for a 720p stream), so edge bandwidth is ~575 Gbps at peak.
For cache sizing: Assume Pareto principle — top 10% of videos get ~90% of views. If videos average 500MB (original) but transcoded renditions average ~200MB each:
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
Here are the APIs:
1. Upload Initiation (returns upload_id + presigned URL)
POST /v1/upload_video Request:{ "title": "My Cat Video", "description": "...", "file_size": 524288000, "content_type": "video/mp4"} Response:{ "upload_id": "upl_a1b2c3d4", "presigned_url": "https://upload.youtube.com/upl_a1b2c3d4?token=...", "expires_in": 3600}
The client then PUTs/POSTs the raw bytes directly to the presigned URL. This avoids funneling traffic through the app server. If the upload fails, the client resumes by sending a range header with the byte offset to the same presigned URL.
2. Complete/Trigger Processing
POST /v1/upload_video/{upload_id}/complete Response:{ "video_id": "vid_x9y8z7w6", "status": "processing"}
Triggers the transcoding pipeline. Without this, the system doesn't know when to start processing — the client might upload bytes in chunks over hours.
3. Video Metadata / Status Polling
GET /v1/videos/{video_id}Response:{ "video_id": "vid_x9y8z7w6", "status": "processing", // or "ready", "failed" "title": "...", "duration": 342, "thumbnails": { "small": "...", "large": "..." }, "manifest_url": "https://cdn.youtube.com/hls/vid_x9y8z7w6.m3u8", "created_at": "..."}
Client polls this endpoint until status = "ready", then loads the manifest URL.
4. Adaptive Bitrate Delivery (HLS/DASH)
GET /v1/stream_video/{video_id}.m3u8 Response: HLS master playlist (text/manifest) #EXTM3U #EXT-X-STREAM-INF:BANDWIDTH=800000,RESOLUTION=640x360 360p.m3u8 #EXT-X-STREAM-INF:BANDWIDTH=2500000,RESOLUTION=1280x720 720p.m3u8
No raw byte ranges. The client requests the manifest, then fetches individual .ts segments (e.g., GET /cdn/vid_x/720p/seg_001.ts) directly from the CDN. The app server only generates the manifest — CDN serves video segments.
For users to stream videos:
GET v1/stream_video {
video_id: UUID,
video_offset: Int,
video_resolution: String
}
For users to view video counts:
GET v1/view_video_count {
video_id: UUID
}
For users to search for a video:
GET v1/search_videos {
search_term: String
}
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
There are several paths in this design. For all requests, we first go through API gateway, which does authentication and rate limiting. This prevents unauthorized users, as well as any single user from sending too many requests that overwhelm the servers.
We then do load balancing, using consistent hashing of user_ids to distribute users to different web servers of a service.
When users upload a video, several things happen. We trigger the APIs to break video into blocks, and upload them to an S3. After the upload completes (finalize), we enqueue a transcode job. Workers pick it up, produce renditions, store them in S3, and emit a completion event (via Kafka) so the metadata service can mark the video as ready and update the manifest. Once this finishes, we do 2 things:
There are 2 design decisions here. For the redis layer, we use as a write-through cache, which updates the database synchronously whenever a write comes. This ensures data consistency between the cache and the database, as well as data durability. In case of cache miss, or when cache cluster is down, we can query the database directly and still get the accurate and updated data. The tradeoff is higher write latency, which is acceptable in our use case.
For the ElasticSearch index, instead of directly writing to it, we first trigger an event on Kafka, which then gets consumed by video search service and then written to the ElasticSearch index. In this way, we reduce the write throughput ElasticSearch needs to handle and improve its latency and availability. The downside is search data can be a bit stale when there are a lot of events on the queue. This is an acceptable tradeoff since search is never guaranteed to be exact, but more of an estimate.
When users try to stream a video, when a request is sent, the video streaming service processes it, and tries to fetch block of video from the CDN. In case of CDN miss, we fetch the video in the right resolution from S3, and cache it in the CDN. We also query the redis cache to read the video metadata. In cases of misses, we query the underlying relational database. If a popular key within the cache cluster expires, to prevent thundering herd, we use stale-while-revalidate and TTL jittering to protect the database.
When users finish watching or exit watching the video, we trigger events on Kafka queue, to be processed by the analytic service. Analytics service writes the detailed watch events to the cassandra cluster for storage, as well as some OLAP database like snowflake for analytics queries. For either analytics or log storage, we don't need real-time data, and eventual consistency is acceptable, so we put Kafka queue in between to reduce the write throughput needed for these databases.
For video count of a video, we also store it in the redis cluster and underlying database. Here, we use distributed counts, so we can aggregate counts from different clusters. Also, the application layer can buffer and batch the counts before they are written to redis, avoiding overloading the redis cluster with writes. In redis, we do atomic increments with sequential execution, so there is no race condition. In this use case, we can flush data from redis to the database async. While there is durability concerns in case of cache cluster crash, it is an acceptable tradeoff since consistency and durability for video count is not super critical. In this case we favor lower latency.
Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...
For the database, we want to store videos, video metadata and video counts.
For the video contents, we use distributed object storage like s3, and store videos in chucks in different resolutions. We also store the videos in CDNs to serve geologically close users.
For video metadata, we store them in 2 databases. 1 is a distributed relational SQL database, another is search index like ElasticSearch.
For the SQL database, we use it for querying video metadata in real time. It is chosen over noSQL like cassandra or mongoDb because:
The tradeoff is we won't be able to support the massive write throughput like cassandra, or have the schema flexibility of mongoDB, but both are acceptable tradeoffs in this case.
Here's a sample table schema:
table video_metadata {
video_id: UUID，
video_uploader_id: UUID,
duration: Int,
status: String,
storage_urls: List
video_title: String,
video_description: String,
uploaded_at: Timestamp,
updated_at: Timestamp,
video_category: String,
video_resolutions: String,
video_archived: Boolean,
video_search_filters: List
}
We also store the same table in ElasticSearch, which allows users to search for videos using keywords.
For both the relational database and ElasticSearch, we shard the data by user_id, using consistent hashing. This way, we distribute videos evenly to each shard, and reduce the read/write throughput per shard. To handle the huge read throughput, we also create several read replicas for each write replica. Every time a write happens, it gets propagated to the read replicas asynchronously. It is acceptable to have temporary inconsistencies as tradeoff for lower write latency. This way, we have scaled both read and write.
For hot keys, we can also break down a single hot key to several, and distribute them across the shards. We can do periodic batching and aggregation for these sharded keys.
For video counts, we store 2 things, one is the video count mapped to a video, and another is the video watch events.
For the video count, we store it in redis cluster and an underlying relational database.
For the video watch events used for analytics purpose, we write it to a cassandra database, optimized for high write throughput. Eventual consistency is acceptable in case of analytics.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
For the database, we want to store videos, video metadata and video counts.
For the video contents, we use distributed object storage like s3, and store videos in chucks in different resolutions. We also store the videos in CDNs to serve geologically close users.
For video metadata, we store them in 2 databases. 1 is a distributed relational SQL database, another is search index like ElasticSearch.
For the SQL database, we use it for querying video metadata in real time. It is chosen over noSQL like cassandra or mongoDb because:
The tradeoff is we won't be able to support the massive write throughput like cassandra, or have the schema flexibility of mongoDB, but both are acceptable tradeoffs in this case.
Here's a sample table schema:
table video_metadata {
video_id: UUID，
video_uploader_id: UUID,
duration: Int,
status: String,
storage_urls: List
video_title: String,
video_description: String,
uploaded_at: Timestamp,
updated_at: Timestamp,
video_category: String,
video_resolutions: String,
video_archived: Boolean,
video_search_filters: List
}
We also store the same table in ElasticSearch, which allows users to search for videos using keywords.
For both the relational database and ElasticSearch, we shard the data by user_id, using consistent hashing. This way, we distribute videos evenly to each shard, and reduce the read/write throughput per shard. To handle the huge read throughput, we also create several read replicas for each write replica. Every time a write happens, it gets propagated to the read replicas asynchronously. It is acceptable to have temporary inconsistencies as tradeoff for lower write latency. This way, we have scaled both read and write.
For hot keys, we can also break down a single hot key to several, and distribute them across the shards. We can do periodic batching and aggregation for these sharded keys.
For video counts, we store 2 things, one is the video count mapped to a video, and another is the video watch events.
For the video count, we store it in redis cluster and an underlying relational database.
For the video watch events used for analytics purpose, we write it to a cassandra database, optimized for high write throughput. Eventual consistency is acceptable in case of analytics.
There are several paths in this design. For all requests, we first go through API gateway, which does authentication and rate limiting. This prevents unauthorized users, as well as any single user from sending too many requests that overwhelm the servers.
We then do load balancing, using consistent hashing of user_ids to distribute users to different web servers of a service.
When users upload a video, several things happen. We trigger the APIs to break video into blocks, and upload them to an S3. After the upload completes (finalize), we enqueue a transcode job. Workers pick it up, produce renditions, store them in S3, and emit a completion event (via Kafka) so the metadata service can mark the video as ready and update the manifest. Once this finishes, we do 2 things:
There are 2 design decisions here. For the redis layer, we use as a write-through cache, which updates the database synchronously whenever a write comes. This ensures data consistency between the cache and the database, as well as data durability. In case of cache miss, or when cache cluster is down, we can query the database directly and still get the accurate and updated data. The tradeoff is higher write latency, which is acceptable in our use case.
For the ElasticSearch index, instead of directly writing to it, we first trigger an event on Kafka, which then gets consumed by video search service and then written to the ElasticSearch index. In this way, we reduce the write throughput ElasticSearch needs to handle and improve its latency and availability. The downside is search data can be a bit stale when there are a lot of events on the queue. This is an acceptable tradeoff since search is never guaranteed to be exact, but more of an estimate.
When users try to stream a video, when a request is sent, the video streaming service processes it, and tries to fetch block of video from the CDN. In case of CDN miss, we fetch the video in the right resolution from S3, and cache it in the CDN. We also query the redis cache to read the video metadata. In cases of misses, we query the underlying relational database. If a popular key within the cache cluster expires, to prevent thundering herd, we use stale-while-revalidate and TTL jittering to protect the database.
When users finish watching or exit watching the video, we trigger events on Kafka queue, to be processed by the analytic service. Analytics service writes the detailed watch events to the cassandra cluster for storage, as well as some OLAP database like snowflake for analytics queries. For either analytics or log storage, we don't need real-time data, and eventual consistency is acceptable, so we put Kafka queue in between to reduce the write throughput needed for these databases.
For video count of a video, we also store it in the redis cluster and underlying database. Here, we use distributed counts, so we can aggregate counts from different clusters. Also, the application layer can buffer and batch the counts before they are written to redis, avoiding overloading the redis cluster with writes. In redis, we do atomic increments with sequential execution, so there is no race condition. In this use case, we can flush data from redis to the database async. While there is durability concerns in case of cache cluster crash, it is an acceptable tradeoff since consistency and durability for video count is not super critical. In this case we favor lower latency.
1. Retryable Chunks & Regional Fallback
For chunked uploads, each chunk is uploaded as a separate multipart upload part to S3. The client computes an MD5/SHA256 of the chunk and includes it as an ETag in the request header. S3 returns the ETag on success. On failure (network blip, 5xx), the client retries the same chunk with exponential backoff (up to 3 retries). Since chunks are independent, a failure doesn't require re-uploading previous chunks.For regional fallback: the upload initiation endpoint returns a list of presigned URLs across multiple S3 regions (e.g., us-east-1, us-west-2, eu-west-1). The client tries the primary region first. If it detects a regional outage (repeated failures within timeout), it falls back to the next region using the pre-signed URL for that region. Each region's URL is pre-signed for the same upload ID, so the finalize call can merge chunks from multiple regions.
2. GOP-Aligned Split for Parallel Transcode
Before transcoding, the source video is analyzed to detect GOP (Group of Pictures) boundaries. A GOP starts with a keyframe (I-frame) and contains only dependent frames (P/B-frames) until the next I-frame — making each GOP a self-contained decodable unit.
The transcode job splits the source into segments at GOP boundaries (typically 2-10 seconds per segment). Each segment is assigned to an independent worker. Workers run FFmpeg (or similar) to produce renditions (360p, 720p, 1080p) of their assigned segment in parallel.
Once all segments for a rendition are complete, a merge step concatenates them into the final rendition file and updates the HLS/DASH manifest with the correct byte ranges and segment durations. If any segment fails, only that segment is retried — no re-transcoding of the entire video.