Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

Edge servers - they reduce the load on the main servers and group counts based on videoID and other analytics events. They cover a geography and thus are the first points of aggregation and abuse detection. They will smooth over region specific popular content and act as spike absorbers. They can be scaled horizontally as they do not evaluate any state. They also publish information to Kafka topics on views, isp, IP and other analytical events. If we remove the edge servers, the main aggregation engine - flink would be drowned in lots and lots of data and will have to run the abuse detection which will become a bottleneck. The con is that there will be some delay in the view event being sent due to abuse checks, but it can be mitigated by adding a timestamp field which will be honored by Flink downstream.
Abuse check - we have discussed this above. The abuse DB will have historical actor data that can be used to run YARA rules in conjunction with request data. Since we are placing the abuse detection service at the edge server level, the time for mitigation for regional specific anomalies will be reduced and more targeted. This can also be scaled horizontally. If placed somewhere downstream, abuse detection will become a bottleneck due to the volume of events.
Kafka - will have topics on views, and other analytical heads. Has persistence so data is durable and is not lost in case of consumer failure. Can be used to ensure proper delivery semantics like exactly-once delivery in case of multiple flink consumers. Kafka also supports high volume high throughput which is perfect for our case.
Flink - aggregate the counts and other analytics. Useful for current count as time period can be made granular. Can reprocess messages in case of failures due to checkpointing. Supports a variety of aggregation techniques. If a flink server goes down, another server will spawn due to auto-scaling and resume the operations because of the checkpointing feature.
Time series DB - makes it easier to store and parse time related data. Queries like how many counts in the last 10 days will be easier to run as compared to traditional SQL/NoSQL databases. The DB would be replicated to ensure no single point of failure.
Analytics service - A background worker will pull events from the TSDB and submit them to the analytics service. Since analytics require a collection of events, there will be a periodicity in which a particular video's counts and metadata are queried. The service will generate the analytics and store them into an analytics DB. The service will be stateless so it will be horizontally scalable. Analytics will be stored in a day/week/month format for easier fetch.

All the services are stateless, except the flink server which also stores state periodically using checkpoints. Thus all services can be horizontally scaled with auto-scaling groups.

Optionally, we can ease the burden from the TSDB and the analytics DB, and store past information which is not regularly used into a blob storage. Partitions should be made so it is easier to fetch data - we should not go inside a partition too much. Hour level partitions seem like a good place to start - we can decide more authoritatively with metrics on how data is fetched.

During high volume spikes, a lot of the spike would be handled at the edge layer. Using metrics on the number of events being generated, we can increase the number of kafka partitions and flink servers to handle the load better.

Since Kafka has in-built message persistence, we would be able to refetch any data as required. Thus in case of a flink server going down, the new spawned server will be able to fetch the messages that were fetched by the crashed instance and lost. Thus the system would ensure reliability and no messages being lost.

The cache design would be read through. The DB would be queried only on cache misses. The cache will hold the popular videos and the eviction policy can be either LFU or LRU.

Submissions

Displaying Solutions with Scores Ranging from 1 to 10