Requirements

Functional Requirements:

• Count the views of the video accurately.
• Display real-time or near real time video view count
• Prevent against bot views and fraudulent views

Non-Functional Requirements:

• Availability > Conistency - System should be highly available and should not fail even if it means that the view of video is synced a little late.
• Low latency - System should be able to display video views very fast and efficiently
• Highly scalable - system should be scabale enough to handle burst or spike traffic load
• High write and read throughput - System should be able to handle writes and reads efficiently.

Capacity Estimation

• ~100M DAU
• Each views ~10 videos daily i.e. 1B writes and read daily, as whenver a write happens, a read will happen first i.e. User opens a video, system displays the view count and then system writes the user view to the video's total view count.

API Design

Data entities

Video

Id

userId

url

VideoMetadata

videoId

views

//Update video view

POST /video/views/{id}

request - {

newViews

}

response - 200

GET /video/views/{id}

response - view count

High-Level Design

API Gateway

Acts as the entry point for all client requests.

Responsibilities:

• Authentication (if required)
• Rate limiting - Applied using token bucket per IP to prevent fraudulent or bot views
• Request routing

Load Balancer

Distributes traffic across multiple instances of the Read and Write services to improve scalability and availability.

View Serivce

Responsibilities

• Recieves a view event only after user has viewed >= 30 sec of video
• Validates the view event for deduplication -> if user already viewed this video is last 24 hour dont count the view -> achieved using redis wirh 24 hr TTL
• after valdiation sends the view evnet to kafka

Kafka

Responsibilities

• Acts a durable video event log
• Once view event is written, acknowledged anc onsumed it is perisited, so we can recover from downstream failures and replay events if needed. eg: When redis crashes and we lose the view count, we can use kafka to rebuild redis for video view count

View aggregator Service

Responsiblities

• Acts as a consumer, consuming view events from kakfa
• Accumulates video counts in memory using hashmap for a short period of 10 sec
• Flushes aggregated counts to Redis
• Commits Kafka offsets after successful flush (ensures no data loss)

Read Service

Responsibilties:

• Returns total view count of a particular video
• Uses redis for faster returns
• Checks redis first to see if video requested is there
• If redis hit -> return response
• If redis miss -> Go to DB fetch data
• Store data in Redis -> return respons

Redis

Responsilbities

• Acts a caching layer enabling system with low latency for writes and reads
• Stores 2 key data info:
  • videoId → total views (serving layer)
  • user-video dedupe keys (TTL = 24h)
• In case of redis crash or failure, we build the critical video view storage using Kafka persisted events plus DB snapshot
• As for user view storage when redis crash happen, i think it is a tradeoff system accepts because if that happens then it means every view in last 24 hrs could be duplicated only once,

Background worker

Responsiblities:

• Runs every 2 mins, fetches all the video views from redis and uses that to update the total video count in our DB NoSQL

NoSQL

• Acts as a persisted storage to store video counts
• We use NOSQL here because we care mroe about scalability than consistency

Used for:

• • Recovery snapshot
  • Cold reads (rare fallback, not primary serving path)

Database Design

Define the data model. Identify the main entities, their attributes, and relationships. Consider the choice of database type (SQL vs NoSQL) and justify your decision based on access patterns...

Detailed Component Design

Now we will deep deep dive into the following:

• How to prevent system against bot views and fradulent views:
  • Problem: a bot or a fradulent user tries to view videos and refresh constantly
  • Solution:
    • Only send a view event to the server after user watches >= 30 sec, this already filters 99% of bot/fradulent activities,
    • Furthermore apply IP based rate limiting, so in case request with same IP is keeps coming, the rate limiter will block/reject the request
    • Addtionally, we also use deduplication views using userId + videoId,which is stored in redis with TTL. so same user's view cant be counted twice for the same video
    • Optionally we can also add device fingerprinting i.e. track userid, ip, devideId, and if same device more than 1000 views with different accounts, flag that as suspicious or even reject requests from the device
• How do we ensure data is not lost when downstream services
  • Problem: Redis have data that is not yet synved to DB, and redis crashes. how to prevent data loss?
  • Solution:
    • Use kafka durable event log
    • Once an view event from kafka is consumed it is peristed for 7 days,
    • So when redis crashes, we will follow this flow:
      • A read request comes in for a video
      • Check redis, no data in redis
      • Fetch total view count from DB
      • Store in redis
      • Replay kafka events for that video -> this updates redis and redis is now rebuilt with correct view count
      • Return result
  • Tradeoff: the first read for the video after downstream services crahses will take some time to return response.

Redis Cache Invalidation - Here, we dont have to worry to much about cache invalidations because, we use write-back policy, meanign, each view write happens to redis first, so redis always have fresh data, the DB is synced later via Background worker.

• Hot video scalability:A video goets viral -> gets 500k views/sec, then this results in
  • Problem: one kafka partition becomes hot - suppose the video id is video123, and partitioin hashing for it always produced same key, hence one partiition is overloaded
  • Solution:
    • Use key salting:
      • to spread traffic, instead of sending id as video123, send video123_0, video123_1, video123_2 etc...
      • Then apply hash on this idea, this ensures the 500k views are distributed accorss many partitions