System requirements

Functional:

• Users: Viewers, Content Creators
• A counter is updated every time a video is viewed
• A counter is displayed in "real time"
• Fraudulent views are detected and not counted towards official count value

Non-Functional:

• reliable
• scalable
• responsive - let "real time" be within ~ 5 minutes

Capacity estimation

• 1,000,000 requests per second
• 10,000,000 videos
• Assume we need ~256 bytes to store a video Id + counter. This can be stored in RDB. ~2.56GB.
• Should employ a cache service for quick reads, i.e. Redis.

API design

• getViewCount(requestContext, videoId)

We can lookup the videoId from the cache service and retrieve the view count.

• incrementViewCount(requestContext, videoId)

The requestContext can be investigated to determine if the calling client is on a list of blocked users to determine a fraudulent view.

If valid, we can lookup the videoId from the cache service and increment the view count. The view count should be persisted to the DB and the cache should be updated.

Database design

Leverage RDB, i.e. Postgres, to persist view counts long term. Estimate ~2.56GB for 10M videos. Leverage a highly available cluster such as AWS RDS for redundancy failover and perform backups for durability.

Leverage a Redis cache cluster to improve read performance on retrieving view counts. Ensure use of a consistent hashing algorithm for the key (videoId) and leverage data partitioning to avoid cache hot spots.

High-level design

• To increment a view count:

incrementViewCount would first ensure the view is valid, i.e use the requestContext to determine if the requesting agent is on a blocked list of users. If yes, do not increment the view count. Otherwise, submit a job for processing to incrememnt the view count.

• To retrieve a view count:

getViewCount could request the view count from the Redis cache.

The API servers should sit behind an API gateway and rate limiter to protect against bursty traffic.

Request flows

Detailed component design

Consider use of websockets with browser clients to improve performance of obtaining view count and incrementing view counts in "real time".

To increment the view count, we could submit the job to a queue (i.e. SQS) to be picked up by a worker thread who can be responsible for the update. A queue system has the ability to scale its workers faster than scaling up more web servers. The worker thread can then be responsible for updating the cache. Multiple worker threads may want to update the counter for the same videoId simultaneously. This could result in a race condition when reading/writing the counter value. To avoid this, we would want to ensure using atomic operations available in Redis.

Trade offs/Tech choices

The API service could retrieve and update the view count in the database directly, however the db reads and writes are likely to be slow. Instead, leveraging a cache like Redis, that is optimised for high throughput of reads/writes, will be more performant.

Failure scenarios/bottlenecks

Service unavailable:

• if API server fails, we risk losing updating a view count for a video. Ensure we have multiple API servers running and ensure an orchastration tool such as zookeeper keeps list of healthy available servers updated for the load balancer.
• if rate limiter fails, we risk dropping requests to increment the view count as well. Ensure redundant RL nodes in case of failure scenario.
• Ensure we have read replicas in place for Redis cache in case of leader failure.
• Ensure backup strategy for postgresql for data durability.
• If under heavy load, writing to postgresql may become a bottleneck.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?