Requirements

Functional Requirements:

1. Count views — every legitimate view of a video gets counted (accurately, none lost)
   1. If a legitimate views a video more than once
2. Display counts — viewers and creators see near-real-time numbers
   1. We need to know if someone is a viewer or a creator
3. Filter fraud — bot/fake views must not be counted (watch-time thresholds, rate limiting, IP/device analysis)
4. Low latency read
5. De-duplication of vidoes
6. Analytic integration

Non-Functional Requirements:

Throughput: Assume YouTube-scale (we always design for millions of users). Let's do the math:

• ~1 billion views/day
• Average: 1B / 86,400 sec ≈ 12K views/sec
• Daily active users
• Peak (viral spikes): 5–10× → 50K–100K+ views/sec at peak
• Reads (displaying counts) far outnumber writes — roughly 10:1 or higher, so peak reads can hit ~1M/sec
• the ingestion pipeline must absorb 100K writes/sec, and caching handles the read-heavy side.

Multi region support since users are US and international

Latency of loading analytics

Consistency in db - handling counting a view exactly once

heavy spikes - Super Bowl view level videos

Scalability during peak hours

Eventual consistency - a viewers count doesn't need to to instantly be reflected in all global caches

Capacity Estimation

DAU - 100 Million

views per user per day = 10 per user = 1B views/day

Average write QPS 1B/24hours =

24 Hours = 86400 second = 11K views/second

Peak write = 2x avg = 22K views/second

Peak Read QPS = 10:1 ratio of read to writes = 220K views/second

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

POST /v1/videos/{video_id}/views --> Write that a viewer viewed a video - payload: user id, video id, timestamp, session_id, device_fingerprint, watch_duration

PUT /v1/views/{view_id} --> updates the existing view, how long of the video did they watch

GET /v1/views/ --> takes in a user id, based on whether they are creator or viewer, it returns the correct counts for views. a creator will see more details than a viewer.

GET /v1/views/analytics --> return a summary of views

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

CLient --> API Gateway/load balancer --> ingestion service --> sns topic #1 --> sqs #1 counter queue --> view counter worker --> batch updates to db for views to raw tables

sns topic #1 --> sqs #2 fraud queue --> fraud detection service -> s3 -- 2 topics in s3 bucket (valid and fraud).

valid s3 topic --> sns #2 --> sqs #3 --> analytics service --> PG analytics tables

fraud s3 topic --> sns#3 --> sqs#4 --> analytics fraud service --> PG fraud tables

deduplication strategy - store user_id : video_id in ingestion service

deduplication key = user id + video id + device fingerprint + timestamp + user_id

Database Design

SQL DB

.Table 1 - views - view_id, user_id, video_id, video_start_time, video_end_time, IP_address, device

Table 2 users - user_id, status(viewer/creator),

Table 3 IPs - view_id, IP_address, geo_lcoation, country, continent

Table 4 analytics - video_id, total_views_ever, total_view_last_30_days, total_geo_lcoations,

Table 5 detected_fraud - Ip,

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

fraud detection service will read the views from the database, and it will process them based on whether they are fraud. It can do some level of aggregation (like if a IP viewed the same video 1M times in 1 min, it is likely a bot, it can check user from the same IP, ...etc).

analytics service will hit the database and it will aggregate statistics hourly or daily to have it rolled up, if the user wants to know last 30 days of traffic, average length of video watched...etc. it will store the aggregate statistics in new tables in the database, then can be hit first before the un-aggregated (raw) db tables.

the ingestion service receives all the views, and it is the first layer in the system. it writes to the db in the raw db tables. it ensures consistency. it makes sure a view is not counted twice. it validates consistency by checking viewer, video id, if it is already been counted in the last 5 seconds. it allows for tracking views from the same viewer in different time periods ( a user can view the video 10 min ago, a user can restart the video, a user can view the video last year or last month). it receives the view event. a view event will contain (user id, video id, starting point in video in second (did a user start video at 0s or 120 sec...etc).

preventing fraud = synchronous filtering rate limit by IP and user ID; asynchronous filtering - anti fraud pipeline - ML model detection against DB

handling viral videos - hot keys - millions of reads and thousands of writes hitting the same key simultaneously

• separate replicas for DB for reads adn writes

QPS = queries per seconds

Use kafka because it creates decoupling between api servers and consumers (ML detection service, ingestion, and db) . Kafka acts as a buffer against unexpected traffic spikes. multiple consumers can consume from the same event without extra load on the api servers.

ingestion process gets events from kafka every 10 seconds and persist the data in postgressql db.