Loading...
POST /upload/init body:{fileSize:string, mimeType:string, userId:string, channelId:string, videoDetails:Json} - used for initiating the upload of a video, videoDetails contains the title, description, and any additional details. channel id is for users that have multiple channels to decide on which one to post the video to. Returns { uploadId, presignedUrl } or { uploadId, chunkSize }
PUT /upload/{uploadId}/chunk/{chunkNumber} body: binary chunk data
POST /upload/{uploadId}/complete → Triggers transcoding
GET /upload/{uploadId}/status → Returns { uploadedChunks: [1,2,3], missingChunks: [4,5], complete: false }
PUT /upload/{uploadId}/chunk/{chunkNumber} → Resume uploading missing chunks
POST /stream/start
body: { userId, channelId, streamDetails }
→ Returns { streamId, ingestUrl }
Then streamer uses RTMP/WebRTC to ingestUrl directly, not HTTP POST for each chunk
GET /video/{videoId}
→ Returns { title, description, duration, manifestUrl, thumbnailUrl }
GET /video/{videoId}/manifest
→ Returns HLS/DASH playlist
GET /analytics?start_date=X&end_date=Y&channel_id=Z&video_id=W
→ Returns { views, watchTime, uniqueViewers, ... } - used for fetching anayltics for channels or videos, it returns the number of views for all the video corresponding to the channel or to the video, timespan is between start and end date
GET /search?q={term}&page=1&limit=20&sort=relevance|date|views enables searching for videos by their name or description
GET /video/{videoId}/manifest
→ Returns HLS (.m3u8) or DASH (.mpd) manifest
→ Contains signed CDN URLs for segments
→ Validates JWT for access control
GET /video/{videoId}/manifest/{quality}
→ Returns segment playlist for specific quality (360p, 480p, 720p, 1080p)
clients are mobile, web or smart TVs
flow for watching a video:
when watching a streaming video, it checks directly the cdn
uploading a video flow
metadata is the source of truth for the entire videos, from there users are served with different qualities based on their internet speed
When the Completion Handler updates Metadata DB, it also sends the video to the Search Indexer. The indexer extracts title, description, tags, and channel info, then adds the video to an Elasticsearch index.
Under normal load, videos become searchable within 30 seconds of upload completion. Under high load, we use a priority queue — videos from high-subscriber channels index first. Maximum indexing delay is 5 minutes even during peak times.
View events are sent to a sqs queue in real-time. Two consumers process this data:
Real-time path: Aggregates counts in Redis every 5 seconds. Powers live view counters on videos. Approximate but fast.
Batch path: Every 15 minutes, a Spark job reads from sqs, deduplicates views, and writes to Analytics DB
CDN hit (99% of requests): Edge server returns cached segment immediately. Latency ~20ms.
Cache warming for popular content:
videos are sent to original storage. then the original video is split into 3: video, audio and metadata. video is then processed by multiple tasks: inspection, video transcoding, thumbnail, watermarks, etc. audio is processed into audio encoding and then video and audio are assembled together whilst metadata is stored separately
video transcoding process:
Each upload gets a unique uploadId. Chunks are numbered and tracked in Redis. Client can query GET /upload/{uploadId}/status to see which chunks are missing and resume from there.
Finalization is idempotent: calling POST /upload/{uploadId}/complete multiple times has no effect after first success. We use a state machine (uploading → finalizing → complete) with optimistic locking. If finalization fails mid-way, client retries and system picks up where it left off.
Failed chunks retry 3 times with exponential backoff. Client tracks failed attempts per chunk.
If a region consistently fails (>50% chunk failures), client requests new upload URLs from a different region via POST /upload/{uploadId}/switch-region. Existing successful chunks are replicated to new region asynchronously. Upload continues without restarting.
Timeout per chunk: 30 seconds. Network errors trigger immediate retry; 5xx errors wait 2 seconds before retry.
Transcoding servers produce multiple resolutions in parallel: 360p, 480p, 720p, 1080p, 4K (if source supports).
Each resolution is segmented into 4-second chunks. All segments are uploaded to Transcoded Storage with naming convention: {videoId}/{resolution}/segment_{n}.ts
Task queue monitored for depth. Auto-scaling rules:
On cache miss, CDN fetches from origin using tiered pullthrough:
Cache-Control: max-age=86400, stale-while-revalidate=3600Cache refresh happens passively on miss or actively for trending content. No viewer request ever hits origin directly—regional nodes act as origin shield, collapsing duplicate origin requests into one.
Indexing is near-real-time, not batch:
Batch is only used for reindexing (schema changes, full rebuild). Day-to-day indexing is streaming/event-driven for freshness.