Loading...
Peak View = 100K/sec
Average Views = 33K/sec (assuming peak is 3x)
Read to Write Ratio = 20:1
Average reads = 660K/sec
Peak reads = 2M/sec
Storage (views data)
We will have (user_id, video_id, metadata) per view ~32bytes/view
Storage in a day = 33K * 32 * 10^5 bytes/day ~ 10^12 bytes/day = 1TB / day
Storage in a year = 365 TB
Storage in 5 years = 1.8PB
Counter store:
It will have 1 row per video.
Assuming 30M videos and each row at 200 bytes (ID, counter, timestamp) ~ 30M * 200 bytes = 6000M bytes = 6 GB (fits in a single machine's RAM)
POST /views
Request:
{
user_id: uuid,
video_id: uuid,
idempotency_key: uuid,
}
Response:
{
success: True,
current_view: string
}
GET /views/{video_id}
Response:
{
success: True,
current_view: string
}
Clients call Load Balancer, which calls API Gateways.
We have 2 services: Fraud Detection and Views Services.
Write path -> When a new view arrives, it goes through Fraud Detection Service, which includes checking if sufficient time was spent on the video, if there is sufficient time between the current view and the last registered view for the key - (video_id, user_id).
If Fraud Detection passes, it emits an event which is stored into Kafka to update the Redis with key = (video_id, user_id), and value = timestamp.
The request is then forwarded to Views Service, which persists the data into a Cassandra store and also emits an event to Kafka, which updates the counter on Redis. Key = video_id, Value = int
There is a slight delay between registering a view and it getting updated in the Redis, which is acceptable.
Read path -> Views service reads the data from Redis, with fallback to Cassandra DB. If they key doesn't exist, it updates the Redis (Read-aside cache)
To handle large burst of views on hot keys, we will shard the data onto Redis (shard by user_id % N) and then an aggregator service will combine aggregate them and update the final key. This is to prevent the update on the single video from becoming bottle neck.
In case of failed events, the events go to a DLQ instead of being lost
Counting and analytics are decoupled via separate Kafka topics; under extreme load, analytics consumers are throttled or sampled, while counting continues with a strict SLA.
The Write throughput required is large at 100K/sec, and the query pattern is also predictable either by video_id, or by (video_id, user_id) in case of analytics. Cassandra fits our use case as primary source.
For caching purposes, we will use Redis.
Cassandra:
video_id, user_id, timestamp -> indexed by video_id
Redis: Key: video_id, Value -> counter
We will deep dive into the following components.
Views Counting Methods:
We will go with the 2nd approach here, as a slight delay is acceptable trade off, for faster reads and writes.
Fraud Detection Service:
We store the (video_id, user_id) with the last view timestamp in a Redis for faster lookups. We will only let the request pass, if the stored_timestamp + configured_grace_window < current timestamp.
We are persisting the writes asynchrounously through Kafka, so there is a window where fraudulent writes could go through. This is a trade off for faster write throughput
The 30sec watch time and 5 min dedup window will be saved as config values, so product can tune without redeploying
High Read And Write Throughput:
We are using Redis counters to read for ms latency. Apart from that, the views will be cached at 3 levels: CDN (~20ms), Redis (~1ms), Cassandra (~20ms) latency
Writes: We divide the write path for hot videos vs relatively cold videos.
For normal videos, the counter is updated directly in Redis. While for hot videos, we will shard the data into N counters and then aggregate them and update the main counter.
Failure scenarios: