Requirements

Functional Requirements:

• Count views for each video accurately, ensuring every legitimate view is counted, even under high traffic.
• Display real-time or near-real-time view counts to users and content creators.
• Prevent fraudulent or bot views using techniques like watch duration thresholds, rate limiting, IP analysis, CAPTCHA, or device fingerprinting.
• Provide content creators with view count trends broken down by time period (daily, weekly, monthly) for analytics and insights.

Non-Functional Requirements:

• Horizontal Scalability: We will go with simple scaling that is when the cpu utilization / ram utilization reaches 80% we will add more servers.
• High Availability by maintaining an up time of 99.99%
• Eventual Consistency: We will have a delay anywhere from 100ms to 5 seconds.
• Reliability
• Low latency: The read and write ratio for this usecase is 50:50 meaning for every view you write to database. For read we can achieve low latency by using in memory db like redis and CDN cache which will handle most of the read and keeping the latency low around 100 to 200 ms

API Design

POST /api/v1/{contentID}/view

Request Body:

contentID: Id of the content the view is for

createdAt: ISO 8601 timestamp for when the pasteContent was created

Response (201 Created):

contentID: Id of the content the view is for

createdAt: ISO 8601 timestamp for when the pasteContent was created

views: Integer

Response (400 Bad Request) If a request fails

Response (429 Too may requests) To rate limit the creation and prevent abuse and spam.

GET /api/v1/{contentID}/view

Request Body:

contentID: Id of the content the view is for

Response (200 Found)

contentID: Id of the content the view is for

createdAt: ISO 8601 timestamp for when the pasteContent was created

views: Integer

Response(404 NOT FOUND) If the pasteContent is not present.

Response (401 Unauthorized) If the user is not logged in.

Response (403 Forbidden) If the user do not have permissions

GET /api/v1/{contentID}/analytics

Request Body:

contentID: Id of the content the view is for

Response (200 Found)

Response(404 NOT FOUND) If the pasteContent is not present.

Response (401 Unauthorized) If the user is not logged in.

Response (403 Forbidden) If the user do not have permissions

High-Level Design

Database Design

I think we can go with blob storage like s3 for storing the content (no schema) and for storing the metadata we can use InfluxDB which can also be used for analytics making it 2 birds with one stone.

Details Table Schema

• contentID(Primary Key)
• CreatedAt
• views
• contentLink

Read Path

Client Click -> CDN -> API Gateway -> Load Balancer -> Getter view service -> Redis Cache -> InfluxDB

Write (Paste) Path

Client Click -> CDN -> API Gateway -> Load Balancer -> write view service-> redis AOF -> Message Queue -> Queue Worker -> InfluxDB

The loadbalancer is integrated into the api gateway api paths through a vpc endpoint. Making it possible for communications.

When ever a request comes to an api it will go through the load balancer from there it will go to the respective service. The view count is first counted in redis and periodically flushed to InfluxDB making the view count more fast.

If a request fails at message queue for 3 times then we will send it to DLQ for manual analysis and rectification. We cant send any messages to user as it does not make sense.

Since CDN also caches most of the reads will be taken care of by the CDN itself. In case of any cache hit misses it will come inside but there as well we have Redis cache. Only after missing both of these will it hit the DB directly.

For fradulent view detection we can do a few things like ip based ratelimiting. We can have Web application firewall with bot detection rules. We can even use deduplication window in redis where we count a view from a user for x seconds. Doing this we can take care of fradulent views.

Detailed Component Design

Caching Strategy

CDN (Hot Tier): Most of the read requests will be taken care by the CDN it self. There is TTL for ensuring cache is fresh.

Redis AOF: This is to count the view. Using this makes the system faster. We will be flushing the data to the influx db periodically

TradeOff:

Even though we may have faster system there is a change of losing some views data which is acceptable in this use case since we want fast writes.

Redis (Warm Tier): If a cache miss happens at CDN we still have redis ready. The data is loaded only when some clicks on the content and we can use LRU eviction for keeping the cache fresh.

When a pasteContent is deleted we have to remove it from cache i.e CDN and redis. For redis we can just delete the key and for CDN we can invalidate it.

InfluxDB:

We are using influxDB to store the metadata. Since the data is stored in a specific frequency of time we are getting timeseries data which can be used for analytics.

Key Generation:

To avoid collisions we can go with universally unique UUIDs which even if we generate millions of id per second for 100 years the chaces of collision is only 50%, Since we are using id for the content and not the views this works out since the amount of content generated will be normal compared to the views they generate.

Trade Off:

Even though it is great at avoiding collisions the Key generated is not very readable. It generates a 128 bit sized key for which we have considered in the design itself.

It works for this use case.

Rate Limiting:

Since we are using API Gateway it can help us in Ratelimiting as well. We can set it to throttle based on the rate or the burst. If we can use api keys per client then we can rate limit per client as well.

Trade off:

We cannot rate limit based on IP which maybe useful in some cases. If we want to ratelimt based on IP then we may have to use Web Application Firewall.