Requirements

Functional Requirements:

• Count views for each video accurately, ensuring every legitimate view is counted, even under high traffic.
• Display real-time or near-real-time view counts to users and content creators.
• Prevent fraudulent or bot views using techniques like watch duration thresholds, rate limiting, IP analysis, CAPTCHA, or device fingerprinting.
• Provide content creators with view count trends broken down by time period (daily, weekly, monthly) for analytics and insights.

Non-Functional Requirements:

• The system should support high view traffic (~100K/s)
• The system should support spikes during high view times.
• The system should be reliable and show accurate view counts.
• System should be resilient in face of abuse.
• System should show counts with minimal latency.

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

Video view -

POST v1/view {videoID: <>, loc: , , , ... , other metadata in headers} -> success

Video view count -

GET v1/counts// -> count

Video Analytics

GET v1/analytics// -> Analytics

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

View hit -

1. The client will send the information on the view to an edge server.
2. The edge server will then send this information to the abuse detection service.
3. The abuse detection service will run the request parameters through comprehensive checks using YARA rules to detect abuse. The set of YARA rules will have some wide ranged and a lot of fine-tuned rules developed over time using abuse patterns to weed abuse. Previous actor history is persisted to the abuse DB for better signal enrichment.
4. If abuse detected, the edge server will respond back to the client for mitigation/verification measures like captcha, rate limiting etc.
5. If no abuse detected, the edge server will pass on the view information to Kafka. A flink instance will pick this streamed information and aggregate them. It will push the aggregation to a time series DB periodically. Meanwhile it will do aggregation on a minute basis and flush that information to a distributed cache for fine-grained view counts.

View counts -

1. The request is routed to a count service, which uses the the time series DB to get the number of views in the past and add to it the granular view counts from the distributed cache. The count service will be able to query for weekly, monthly and yearly data from the time series DB with ease.

Analytics -

1. A background worker queries the DB and submits the results to the analytics service.
2. The service will then generate analytics and persist them into an analytics DB.
3. The request is routed to the analytics service which returns the response from the analytics DB.

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

1. Edge servers - they reduce the load on the main servers and group counts based on videoID and other analytics events. They cover a geography and thus are the first points of aggregation and abuse detection. They will smooth over region specific popular content and act as spike absorbers. They can be scaled horizontally as they do not evaluate any state. They also publish information to Kafka topics on views, isp, IP and other analytical events. If we remove the edge servers, the main aggregation engine - flink would be drowned in lots and lots of data and will have to run the abuse detection which will become a bottleneck. The con is that there will be some delay in the view event being sent due to abuse checks, but it can be mitigated by adding a timestamp field which will be honored by Flink downstream.
2. Abuse check - we have discussed this above. The abuse DB will have historical actor data that can be used to run YARA rules in conjunction with request data. Since we are placing the abuse detection service at the edge server level, the time for mitigation for regional specific anomalies will be reduced and more targeted. This can also be scaled horizontally. If placed somewhere downstream, abuse detection will become a bottleneck due to the volume of events.
3. Kafka - will have topics on views, and other analytical heads. Has persistence so data is durable and is not lost in case of consumer failure. Can be used to ensure proper delivery semantics like exactly-once delivery in case of multiple flink consumers. Kafka also supports high volume high throughput which is perfect for our case.
4. Flink - aggregate the counts and other analytics. Useful for current count as time period can be made granular. Can reprocess messages in case of failures due to checkpointing. Supports a variety of aggregation techniques. If a flink server goes down, another server will spawn due to auto-scaling and resume the operations because of the checkpointing feature.
5. Time series DB - makes it easier to store and parse time related data. Queries like how many counts in the last 10 days will be easier to run as compared to traditional SQL/NoSQL databases. The DB would be replicated to ensure no single point of failure.
6. Analytics service - A background worker will pull events from the TSDB and submit them to the analytics service. Since analytics require a collection of events, there will be a periodicity in which a particular video's counts and metadata are queried. The service will generate the analytics and store them into an analytics DB. The service will be stateless so it will be horizontally scalable. Analytics will be stored in a day/week/month format for easier fetch.

All the services are stateless, except the flink server which also stores state periodically using checkpoints. Thus all services can be horizontally scaled with auto-scaling groups.

Optionally, we can ease the burden from the TSDB and the analytics DB, and store past information which is not regularly used into a blob storage. Partitions should be made so it is easier to fetch data - we should not go inside a partition too much. Hour level partitions seem like a good place to start - we can decide more authoritatively with metrics on how data is fetched.