Requirements

Functional Requirements:

• Allow users to upload videos.
• Enable users to stream videos.
• Support analytics for view counts.
• Support searching for videos

Non-Functional Requirements:

• Scalability: the system should be able to quickly scale out/horizontally its service numbers and db instances on high demand
• availability: the system should have an uptime of 99.99% and users should always be able to see or upload videos, if their internet connection is poor, we can lower the video quality. staleness is acceptable and thus we can cache the service responses to fasten their delivery
• low latency: videos should be downloaded in streams, gradually so the users wont take too long to download an entire video right at the begining. videos stream fetching should be smooth and the calls should take less than 200ms
• cost efficiency - we wont store all videos in a cdn because it will be very costly, so we must store only popular and local videos (theres no need to store videos with a small number of views or videos not too watched in certain regions)
• resumable uploads - if a user's upload failed, we should be able to resume it

API Design

POST /upload/init body:{fileSize:string, mimeType:string, userId:string, channelId:string, videoDetails:Json} - used for initiating the upload of a video, videoDetails contains the title, description, and any additional details. channel id is for users that have multiple channels to decide on which one to post the video to. Returns { uploadId, presignedUrl } or { uploadId, chunkSize }

PUT /upload/{uploadId}/chunk/{chunkNumber} body: binary chunk data

POST /upload/{uploadId}/complete → Triggers transcoding

GET /upload/{uploadId}/status → Returns { uploadedChunks: [1,2,3], missingChunks: [4,5], complete: false }

PUT /upload/{uploadId}/chunk/{chunkNumber} → Resume uploading missing chunks

POST /stream/start

body: { userId, channelId, streamDetails }

→ Returns { streamId, ingestUrl }

Then streamer uses RTMP/WebRTC to ingestUrl directly, not HTTP POST for each chunk

GET /video/{videoId}

→ Returns { title, description, duration, manifestUrl, thumbnailUrl }

GET /video/{videoId}/manifest

→ Returns HLS/DASH playlist

GET /analytics?start_date=X&end_date=Y&channel_id=Z&video_id=W

→ Returns { views, watchTime, uniqueViewers, ... } - used for fetching anayltics for channels or videos, it returns the number of views for all the video corresponding to the channel or to the video, timespan is between start and end date

GET /search?q={term}&page=1&limit=20&sort=relevance|date|views enables searching for videos by their name or description

GET /video/{videoId}/manifest

→ Returns HLS (.m3u8) or DASH (.mpd) manifest

→ Contains signed CDN URLs for segments

→ Validates JWT for access control

GET /video/{videoId}/manifest/{quality}

→ Returns segment playlist for specific quality (360p, 480p, 720p, 1080p)

High-Level Design

clients are mobile, web or smart TVs

flow for watching a video:

1. client makes the api call to request the video to start
2. it reaches firstly api gateway which handles auth and rate limiting
3. reaches load balancer
4. forwards to api servers, we need multiple instance in order to scale out in case of high demand
5. it first reaches metadata cache and searches for the requested video
6. if not found it looks in metadata db

when watching a streaming video, it checks directly the cdn

uploading a video flow

1. client makes request
2. video is saved in original blob storage
3. then it reaches transcoding servers which perform encodings and transformations
4. in transcoded storage the transcoded data is saved
5. in cdn we save the most trendy videos
6. in parallel, when transcoding is done, we send a message to completion queue
7. completion handler consumes from the queue and updates both metadata cache and db
8. api servers inform the client that the video is uploaded

metadata is the source of truth for the entire videos, from there users are served with different qualities based on their internet speed

When the Completion Handler updates Metadata DB, it also sends the video to the Search Indexer. The indexer extracts title, description, tags, and channel info, then adds the video to an Elasticsearch index.

Under normal load, videos become searchable within 30 seconds of upload completion. Under high load, we use a priority queue — videos from high-subscriber channels index first. Maximum indexing delay is 5 minutes even during peak times.

View events are sent to a sqs queue in real-time. Two consumers process this data:

Real-time path: Aggregates counts in Redis every 5 seconds. Powers live view counters on videos. Approximate but fast.

Batch path: Every 15 minutes, a Spark job reads from sqs, deduplicates views, and writes to Analytics DB

CDN hit (99% of requests): Edge server returns cached segment immediately. Latency ~20ms.

Cache warming for popular content:

• When video trends, proactively push to all regional CDN nodes
• New uploads from popular channels pre-warmed to edges before first view

Detailed Component Design

videos are sent to original storage. then the original video is split into 3: video, audio and metadata. video is then processed by multiple tasks: inspection, video transcoding, thumbnail, watermarks, etc. audio is processed into audio encoding and then video and audio are assembled together whilst metadata is stored separately

video transcoding process:

1. preprocessor -responsabiities: video splitting - video is split into smaller GOP (group of picures). a gop is a chunk of frames arranged in a specific order, each unit is a fragment of a few seconds; generates the DAG (directed acyclic graph); cache data - preprocessor is a cache for GOP, if video encoding fails, it can retry using the temporary stored data from preprocessor
2. DAG scheduler - splits the dag graph into stages of tasks and puts them in the task queue in resource manager
3. resource manager - manages efficiently resource allocation
4. tak workers - run the tasks defined in the dag eg watermark, encoder, tuhmbnail, etc
5. temporary usage -
6. encoded video - the final result