Design Youtube or Netflix

Video streaming is defined by one extraordinary asymmetry, for every video uploaded, it gets watched roughly 1,000 times. This 1000:1 view-to-upload ratio shapes every architecture decision. Upload is rare, expensive, and compute-intensive. Playback is frequent, must be fast, and bandwidth-intensive. The system must handle both extremes simultaneously, with fundamentally different optimization strategies for each path.

Functional

Upload video: Users upload video files ranging from a few megabytes to tens of gigabytes. The system must accept the raw file, transcode it into multiple resolutions and formats, generate thumbnails, and make the video available for streaming. This is an asynchronous pipeline that can take minutes to complete.

Streaming bandwidth dwarfs every other infrastructure metric. While storage grows steadily and compute spikes during transcoding, the real cost driver is the sheer volume of video bytes flowing from CDN to viewers every second of every day. The capacity math reveals exactly where your architecture budget should go.

Traffic

Start with uploads. 100,000 new videos per day means approximately 1.2 uploads per second. Each upload triggers a transcoding pipeline that runs for several minutes, but the ingest rate itself is modest.

The most important API design decision is that video upload bypasses the API servers entirely. The largest data transfers in the system (multi-gigabyte video files) flow directly from the client to object storage via presigned URLs. The API server only handles the small metadata request that initiates the upload.

Upload Initiation

POST /api/v1/videos/upload

The data in a video streaming system falls into three fundamentally different categories, each requiring its own storage technology. Trying to store video files, structured metadata, and search indexes in a single database would be like using a hammer for screws, technically possible, but wrong for every use case.

Why Blob Storage for Video Files

Video files are binary blobs ranging from megabytes to tens of gigabytes. No database is designed to store and serve these efficiently. Object storage (S3, GCS) is purpose-built for this workload, it handles multi-terabyte objects, provides built-in redundancy across availability zones, integrates directly with CDN for global delivery, and offers tiered pricing for hot versus cold content. Trying to store video files in a relational or NoSQL database would be dramatically more expensive, slower for streaming, and operationally painful at petabyte scale.

The architecture splits into two fundamentally different paths, each optimized for its unique requirements. The upload path is write-heavy, compute-intensive, and can tolerate minutes of latency. The streaming path is read-heavy, bandwidth-intensive, and must respond in milliseconds. Combining them into a single pipeline would force uncomfortable compromises, you cannot optimize for both throughput-intensive batch processing and latency-sensitive real-time delivery in the same code path.

Upload path: Client sends POST to API Server, receives presigned URL. Client uploads directly to S3. S3 event notification triggers message queue. Transcoding pipeline processes the video into multiple resolutions and formats. Transcoded segments are written back to S3. Metadata store is updated with manifest URL and status "ready." CDN caches segments on first viewer request.

Streaming path: Client requests video metadata from API Server (served from Redis cache on hit). Response includes manifest URL on CDN. Player fetches manifest and video segments directly from CDN edge locations. For popular videos, 90%+ of requests never reach the origin.

The upload flow touches every backend component and takes minutes to complete. The streaming flow mostly never reaches the backend at all, the CDN serves the vast majority of requests from edge cache. Understanding both flows reveals why the architecture is split the way it is.

Upload Flow

1. Client sends POST /api/v1/videos/upload with video title, description, and tags to the API Gateway.
2. API Gateway authenticates the request, checks rate limits, and routes to an API Server.
3. API Server creates a video metadata record in Cassandra/DynamoDB with status "uploading" and generates a presigned S3 upload URL.
4. API Server returns the videoId, presigned upload URL, and URL expiration time to the client.
5. Client uploads the video file directly to S3 using the presigned URL. For large files, this uses S3 multipart upload, the file is split into chunks (5-25 MB each), and chunks are uploaded in parallel. If a chunk fails, only that chunk is retried.
6. S3 completes the multipart upload assembly and publishes an event notification to the message queue (Kafka/SQS).
7. A transcoding worker picks up the event from the queue.
8. The transcoding worker downloads the original video from S3, splits it into segments at GOP (Group of Pictures) boundaries, and encodes each segment into multiple resolutions (1080p, 720p, 480p, 360p) and codecs.
9. Transcoded segments are uploaded back to S3 in the output bucket.
10. The transcoding worker generates HLS/DASH manifest files listing all available quality levels and segment URLs, and uploads the manifests to S3.
11. The transcoding worker updates the video metadata in Cassandra/DynamoDB with status "ready", manifest URL, thumbnail URL, available resolutions, and duration. CDN caches the segments on first viewer request.

Three engineering challenges define the core complexity of a video streaming platform: the transcoding pipeline that converts raw uploads into streamable formats, the adaptive bitrate mechanism that adjusts quality during playback, and the resumable upload system that handles multi-gigabyte file transfers reliably.

Transcoding Pipeline

Transcoding converts a raw uploaded video into multiple formats and resolutions optimized for streaming. This is the most compute-intensive operation in the entire system, but it runs only once per video, every subsequent view benefits from the pre-computed output.

Every technology choice in a video streaming platform has measurable impact on the three metrics that matter most: bandwidth cost, playback quality, and engineering effort. The right choice depends on your scale, team capabilities, and cost constraints.

HLS vs DASH

HLS (HTTP Live Streaming): Developed by Apple, universally supported on iOS and Safari. Uses .m3u8 manifest files and .ts segment files. The de facto standard for mobile streaming due to Apple's device market share.

Video streaming systems store two fundamentally different types of data with different failure characteristics. Video segments are immutable once transcoded, they can always be re-derived from the original upload. Metadata is mutable and stateful, losing it means losing the mapping between videoIds and their content. Recovery strategies differ accordingly.

Transcoding Pipeline Failure

A segment encoding task can fail due to a corrupted frame, GPU memory error, or worker crash. Because the pipeline is modeled as a DAG with independent segment tasks, only the failed segment needs to be retried. The orchestrator (Temporal/Airflow) automatically retries the failed task on a different worker with a fresh GPU. If the retry budget is exhausted (e.g., 3 attempts), the segment is marked as failed and an alert is raised. The video can still be published with the failed segment excluded from certain quality levels, or held until the issue is manually resolved.

Each future improvement introduces new engineering challenges that extend the core architecture in different directions. Live streaming changes the processing model from batch to real-time. Content moderation adds a filtering layer. DRM adds encryption. Recommendations add machine learning infrastructure. Multi-region deployment adds geographic distribution.

Live Streaming

Live streaming fundamentally transforms the architecture. Video-on-demand (VOD) transcodes once and serves cached segments indefinitely. Live streaming must transcode in real-time, every frame must be encoded, segmented, and distributed to CDN edge locations within 2-5 seconds of capture.