Count views for each video accurately, ensuring every legitimate view is counted, even under high traffic.
Display real-time or near-real-time view counts to users and content creators.
Prevent fraudulent or bot views using techniques like watch duration thresholds, rate limiting, IP analysis, CAPTCHA, or device fingerprinting.
Provide content creators with view count trends broken down by time period (daily, weekly, monthly) for analytics and insights.
Non-Functional Requirements:
High Availability: The system should be highly available : 99.999% (around 4 s of downtime per year), so that the users never feel left out, and always use the system.
Scalability: The system should handle high read traffic from users, for their engagement, analysis over forecast data or daily data. DAU: 1M
Large number of reads compared to writes: The system should be handle large number of reads: Read: Write:: 100: 1
Reliability: The system should be highly reliable. It should be protected against fraudulent data like bot actions.
Security: The system should be secure while moving requests, and also storing data. Encryption at rest using encryption and SSL/TLS while at transit is preferable.
Malicious IPs must be tracked and prevented from entering the system.
Durability: The system should be durable. The data stored in the system should not be erased by network attacks or hardware issues.
Eventual Consistency: Since the system should be highly available, eventual consistency can be employed on read replicas of database using consistent hashing.
API Design
POST /api/v1/video/count payload: {videoId: videoId}
GET /api/v1/video/count?videoId={videoId}
GET /api/v1/video/count/trends?videoId={videoId}&trend={trend} RBAC based auth (only for content creators not for users) trend can be: daily, week, month
High-Level Design
Client: Can be website/ ios/ android
CDN: To cache the counts of the popular videos inorder to provide low latency at edge locations and prevent the call to system as much as possible within a TTL. Eviction Policy: LRU.
WAF : Firewall to whitelist IPs and blacklist IPs to prevent requests from Bots and malicious IP traffic. AWS Shield can also be integrated with WAF to prevent network based attacks such TCP Sync attacks and UDP flooding. Captcha can be integrated with this for additional security to prevent bot attacks.
Load Balancer: Distributes the traffic among different instances of the backend server based on a rule. For optimum use of servers, round-robin based static application load balancer is used, there by ensuring equal distribution of traffic. API Gateway can be integrated with this to implement RBAC based authentication.
Rate Limiter: Ensure that the server doesnot get overwhelming traffic at a short duration.Leaky bucket algorithm to be employed to maintain a constant flow of traffic to the server, and return 429 status code in case there are too many requests.
Video Count Service: The main backend service that handles fetching and increasing the count of a particular video. Is connected with DB for metadata about the video and the S3 bucket for the actual video content.
Database: For high availability, No-SQL basd DB such as Cassandra can be used, since it is highly scalable, durable, reliable and available. Due to its leaderless system for replicas, replication will be faster and can handle high number of reads at large traffic. S3 can be used to store the actual blob file of the video.
Detailed Component Design
User: Can be a website/ ios/ android app
CDN: Content - Delivery network that caches the counts for popular / most viewed videos to provide low latency and near-real time updates at edge locations.
Captcha: A security middleware that helps in bot detection by introducing detections for bots. The request is rejected if bot is detected.
WAF: Firewall for whitelisting/ blacklisting IPs. AWS Shield can be used for protection against network attacks such TCP sync attacks and UDP flooding.
Load Balancer: Static Load balancer with round-robin algorithm for equal traffic distribution. To prevent single point of failure, we can have 3 stand by load balancers in different availability zones, where traffic is redirected if primary load balancer is unhealthy.
Video Count Service: The main server orchestration of managing video counts. The server based on the video id will create a job and push it to a message broker like redis queue, where the count value is incremented and saved in redis. Saving in DB is handled asynchronously, thus making our APIs fast.
Redis: Redis queue is used as a message broker for high scalability. Redis will be a distributed cache, and failover, there will be a standby cluster/ instance so that recovery will be possible from primary to replica. Master-Slave to be followed among redis clusters.
Worker: Queue worker polls redis queue and asynchronously writes to DB. This gives true decoupling.
DB Proxy: Proxy should be used for maintaining and re using connections to DB, there by not allowing DB to handle very high traffic, and handle constant traffic from proxy.
Database: A Key-Value database to be used for high availability instead of RDBMS. Dynamo DB can be used for high scalability and high availability, in combination with DAX. sample entry: { videoId: videoId, s3_url: s3_url, count: count}. Cassandra can also be considered for a columnar based DB, giving high availability and scalability.
Things to consider:
Encryption: For securing data at transit, SSL/TLS should be used, with ACM attached to Load balancer for certificated for DNS. For rest encryption, KMS should be used for key management.
Single Point of failure: Load balancer should not be exposed as a single point of failure. Stand by Load balancers should be used in multi-AZ and multi region for high availability and disaster recovery for less RPO and RTO.
Caching Strategies: Write-Back used so that the updates would be faster for the user. DB write can be handled asynchronously. This makes the api faster, and eventually the updates are replicated to other clusters which act as replicas. Eviction Policy: LRU for removing keys for keeping only the hot videos in the cache.
Firewall: The WAF should be able to blacklist/ whitelist IPs for prevention of malicious users into the system.Captcha should be used for bot detection and prevent bots from entering the system.
Eventual Consistency: Cassandra can be used since it has leaderless architecture, thus keeping all the replicas in sync at all times.