Requirements

Functional Requirements:

• Count views for each video accurately, ensuring every legitimate view is counted, even under high traffic.
• Eventual consistency is allowed, so slight delays in view count accuracy is fine.
• Display real-time or near-real-time view counts to users and content creators.
• Prevent fraudulent or bot views using techniques like watch duration thresholds, rate limiting, IP analysis, CAPTCHA, or device fingerprinting.
• Provide content creators with view count trends broken down by time period (daily, weekly, monthly) for analytics and insights.

Non-Functional Requirements:

• Needs to be accurate (View count needs to be up to date with correct information but slight delays of about 10-15 minutes in view count accuracy is allowed)
• Needs to have low latency (Needs to provide the most up to date view count at around 10-50ms)
• Needs to have high availability (Needs to have 99.99% uptime)
• Needs to be scalable (Needs to handle billions of viewers and more during peak season)

Storage Estimation

• 1,000,000 viewers a day on average * (60 seconds * 60 minutes * 24 hours * 365 days) = 31.5 trillion views a year
• Viewer metadata has around 10 KB * 31.5 trillion views a year = 300 TB

API Design

1. CreateVideo
   1. path: /createVideo
   2. request: { uploaderId: string, videoTitle: string, videoLength: integer }
   3. response: { videoId: string }
2. AddView
   1. path: /addView
   2. request: { videoId: string, viewerId: string }
   3. response: None
3. GetViews
   1. path: /getView/{videoId}
   2. request: None
   3. response: { viewCount: integer }
4. GetViewTrends
   1. path: /getViewTrends/{videoId}
   2. request: None
   3. response: { viewCounts: List[{ viewCount: integer, date: DateTime }] }

High-Level Design

1. The client will first connect to a rate limiter so that if they are trying to watch a video, they cannot keep spamming refresh to constantly add views to the video. We can have a cap on the number of views a user can add such as 1 view every 8 hours (they can still watch the video, it will just not count towards the views).
2. After that, the rate limiter will go through a load balancer to distribute the load evenly across servers.
3. The load balancer will connect to an API Gateway which will trigger the Video Handling Service.
4. The Video Handling Service will go through a Cache for all of the APIs except AddView and it will also modify the Database.
5. The AddView API will be the only API that does not directly touch the Cache and Database directly. Each call to the AddView API will send a message to an SQS to trigger a handler which will then modify the cache/database to add a view to the respective video. Any failed messages will go to the DLQ for manual review and redrive.

Detailed Component Design

1. Video Handling Service
   1. Functionality: The Video Handling Service will handle all of the APIs that we have defined. When the user creates a video, we will create a new entry in the cache and database for this video which includes a view count. When a user watches the video, it will trigger the AddView API to send a message to the SQS. The payload will include the video ID and the user ID so that we can add a view to the correct video and also make sure that the user cannot spam watch the same video to bloat the view count. When a user clicks on the video, we will also call GetViews to get the current amount of views. GetViews will first check the cache to see the current amount of views (it's okay for it to be a little stale), and if there is a cache miss, we will check the database. The cache can have a TTL that scales to the number of views, for example, a video with less than 100 views will have a small TTL of about 10 seconds, while a video with 1000 views will have a TTL of about 5 minutes, and a video with a million views can have a TTL of about 30 minutes. Finally, when the user requests to see their video trends, we can check all view count entries in the database for this video and return it to the user in a list that also includes the date that the views were added. All of these APIs can have a retry 3 times strategy with exponential backoff to handle transient errors.
   2. In order to handle race conditions, we can use DynamoDB's UpdateItem to atomically increment the view count, avoiding race conditions.
   3. We can handle idempotency by including an idempotency key in the AddView request. If the request includes the same idempotency key as the one in the database, we will not count the view.
   4. We can have a periodic reconcillation between the view counts in the cache and the database. For example, we can have a Reconsilation Service that periodically checks the cache and database every few minutes to ensure that the results are not too widely different.

Tech Choices and Trade Offs

1. We will be using DynamoDB, Redis, Amazon SQS, and API Gateway as our choice of tech.
2. We went with a cache instead of a write through cache because we want performance but we are fine with stale data because we want eventual consistency. It's fine if view counts are slightly off for a few seconds.
3. We went with a non relational database instead of a relational database because we are fine with eventual consistency and we want to be able to handle large volumes of read/write (for example, millions of AddView modifications per second). We also want to be able to scale horizontally when videos start reaching trillions of views or more.