System requirements

Functional:

Accurate View Counting: Count a view when a user watches a video for a specified duration (e.g., at least 10 seconds).

Real-time Updates: Provide real-time or near-real-time updates of the view count to both content creators and viewers.

Fraud Prevention: Implement mechanisms to prevent counting fraudulent views, such as bots or repeated views within a short time.

High Traffic Handling: The system should be able to handle millions of concurrent views during peak times.

Historical View Data: Store and allow retrieval of historical view data for analytics purposes.

Public and Private Views: Differentiate between views by all users and views by specific groups (e.g., subscribers).

Non-Functional:

Scalability: The system must scale horizontally to accommodate growing numbers of videos and users.

Reliability: Ensure high availability with minimal downtime.

Consistency: Ensure eventual consistency for view counts across distributed components.

Performance: Provide low-latency responses, especially for counting views and updating the view count.

Security: Protect against unauthorized access and ensure data integrity.

Fault Tolerance: The system should be resilient to failures and recover quickly.

Capacity estimation

Given the updated capacity planning considerations, here’s a more detailed breakdown:

1. Data Volume Estimation

• Number of Videos: 1 million videos in the system.
• Number of Users: 10 million daily active users.

Daily Data Volume:

• Views per Video per Day: On average, assume each video gets 100 views per day.
• Total Views per Day = 1 million videos * 100 views = 100 million views per day.
• Storage for View Records:
• Assume each view record (including VideoID, UserID, SessionToken, Timestamp, and metadata) requires approximately 100 bytes.
• Daily Storage Requirement = 100 million views * 100 bytes = 10 GB per day.

5-Year Data Volume:

• Total Views over 5 Years:
• Total Views = 100 million views/day * 365 days/year * 5 years = 182.5 billion views.
• Storage Requirement over 5 Years:
• Total Storage = 182.5 billion views * 100 bytes = ~18.25 TB over 5 years.

2. Transactions per Second (TPS) Estimation

• Avg TPS:
• Average TPS = Total Views per Day / Seconds per Day
• Avg TPS = 100 million views / 86,400 seconds ≈ 1,157 TPS.
• Peak TPS:
• Assume peak traffic is 10 times the average.
• Peak TPS = 1,157 TPS * 10 ≈ 11,570 TPS.

3. Resource Estimation

To handle this load, the system will need to be equipped with:

• Database Resources:
• NoSQL Database (DynamoDB):
• Provisioned throughput to handle at least the peak TPS (11,570 write operations per second). Additional capacity may be needed to handle occasional spikes beyond this.
• Capacity to store and manage 18.25 TB of view data over 5 years.
• SQL Database (MySQL/PostgreSQL):
• Adequate storage to handle historical data queries efficiently, with indexing on important fields like VideoID and Timestamp.
• Read replicas to distribute the load and handle high read queries for historical data.
• Cache (Redis):
• Sufficient memory to store view counts for frequently accessed videos. Given the large scale, plan for at least several GBs of in-memory storage to minimize read latency.
• Compute Resources:
• API Servers: Multiple instances of stateless API servers to handle an average load of 1,157 TPS and scale to handle peak loads of up to 11,570 TPS.
• Load Balancer: To distribute incoming requests evenly across API servers.
• Real-Time Update Service: Horizontally scalable WebSocket servers to push real-time updates to users.
• Networking:
• High bandwidth network infrastructure to support real-time updates and efficient data transfers between components, especially during peak traffic periods.

API design

POST /video/{videoId}/view: Register a view for a video. Input: VideoID, SessionToken, UserID (optional). Output: Status (success/failure).

GET /video/{videoId}/view-count: Retrieve the current view count for a video. Input: VideoID. Output: ViewCount.

GET /video/{videoId}/view-history: Retrieve historical view data for a video. Input: VideoID, TimeRange. Output: Historical View Data.

POST /video/{videoId}/fraud-detection: Manually flag a view or set of views for fraud detection. Input: VideoID, UserID, IPAddress. Output: Status.

Database design

Video: Stores video metadata (VideoID, Title, UploaderID, etc.).

ViewCount: Stores current view count (VideoID, TotalCount, LastUpdated).

ViewHistory: Stores historical view data (VideoID, UserID, ViewTimestamp).

FraudDetection: Stores flagged views for fraud detection (ViewID, VideoID, UserID, IPAddress, Reason).

Sharding:

• Video Sharding: Based on VideoID, ensuring that each shard contains data for a distinct set of videos, which helps distribute the load evenly across shards.
• User-Based Sharding: Secondary sharding strategy based on UserID for tables where user-specific data is important, such as ViewHistory.

Partitioning:

• Time-Based Partitioning: Used for ViewHistory to split data by time intervals (e.g., daily or monthly partitions) to optimize query performance and data management.
• Geographical Partitioning: Partition data by region for tables that might benefit from regional queries, enhancing performance in geographically distributed systems.

Replication Strategy:

• Multi-Region Replication: Replicate data across multiple geographic regions to ensure high availability and disaster recovery.
• Synchronous Replication: Critical data like ViewCount and Video tables use synchronous replication to ensure data consistency across primary nodes.
• Asynchronous Replication: Less critical data, such as logs and historical data (ViewHistory), use asynchronous replication for eventual consistency, reducing latency during write operations.

High-level design

YAPI Gateway: Manages incoming requests and routes them to the appropriate service.

View Service: Handles logic for counting views, including deduplication and fraud detection.

Real-Time Update Service: Pushes updates to clients about view counts in real-time using WebSockets or similar technologies.

Fraud Detection Service: Analyzes view patterns to detect and flag fraudulent views.

Data Storage:

• NoSQL Database (e.g., DynamoDB): For real-time view counts, ensuring fast read and write operations.
• SQL Database (e.g., MySQL/PostgreSQL): For storing historical view data and supporting complex queries.

Cache (e.g., Redis): Stores frequently accessed view counts to reduce database load.

Request flows

View Count Increment:

1. User Action: A user watches a video, triggering a POST /video/{videoId}/view request.
2. API Gateway: Routes the request to the View Service.
3. Deduplication: The View Service checks for duplicate views using session tokens and IP addresses.
4. Count Update: If the view is valid, the View Service increments the view count in the Cache and NoSQL Database.
5. Fraud Check: The view is also passed to the Fraud Detection Service for analysis.
6. Real-Time Update: The Real-Time Update Service pushes the new view count to subscribers or viewers.

Retrieve View Count:

1. User Request: A user requests the current view count via GET /video/{videoId}/view-count.
2. Cache Check: The View Service checks the Cache for the view count.
3. Database Fallback: If the count is not in the cache, it retrieves it from the NoSQL Database and updates the cache.

Detailed component design

View Service:

• Scalability: Stateless, allowing horizontal scaling. Uses in-memory cache (Redis) for fast access to view counts.
• Deduplication Logic: Implements deduplication using session tokens and IP addresses. Multiple views from the same user within a short window are not counted.
• Database Sync: Asynchronously updates the NoSQL Database to ensure the view count is stored durably.

Token Bucket Rate Limiting: Controls the rate of incoming requests to prevent overloading the system, allowing for short bursts while maintaining an average rate.

• Behavioral Analysis: Tracks user interaction patterns to differentiate between bots and legitimate users, using features like mouse movement, click patterns, and video engagement.

Fraud Detection Service:

• Machine Learning: Uses supervised and unsupervised learning models to detect fraudulent views (e.g., rapid views from the same IP).
• Bloom Filters: Employs Bloom filters to efficiently track and detect repeated fraudulent views.

Real-Time Update Service:

• WebSockets: Pushes real-time updates to clients, allowing viewers to see the updated view count without refreshing.
• Horizontal Scaling: Scales across multiple instances to handle large numbers of simultaneous connections.

Automatic Reconciliation:

• Consistency Checks: Compare view counts across replicas and logs periodically or after suspected anomalies.
• Conflict Resolution: Use last-write-wins (LWW) or majority consensus to reconcile any detected inconsistencies.

Trade offs/Tech choices

NoSQL vs. SQL: NoSQL (DynamoDB) is chosen for real-time view counting due to its scalability and performance. SQL (MySQL/PostgreSQL) is used for storing historical data where complex queries might be needed.

Caching: Redis is used for caching view counts to reduce load on the primary database, with a trade-off of potential stale data if cache invalidation is not handled properly.

Eventual Consistency: The system is designed to be eventually consistent, ensuring that while immediate view counts may slightly differ across regions, they will converge over time.

Failure scenarios/bottlenecks

Cache Invalidation: If the cache is not properly invalidated, stale view counts might be served. Mitigation includes implementing a TTL (Time to Live) for cache entries and ensuring periodic synchronization with the database.

Database Overload: During peak traffic, the database might become a bottleneck. Mitigation includes sharding the database, using read replicas, and offloading some queries to the cache.

Fraud Detection Failure: If the fraud detection model fails, fraudulent views might inflate the count. Implementing a fallback to simpler rule-based detection can mitigate this risk.

Network Latency: High network latency between services can slow down view count updates. Mitigation includes deploying services in multiple regions and optimizing network paths.

Bot Traffic and Artificial Inflation:

• Advanced Behavioral Analysis: Use sophisticated models to track detailed user interactions, identifying and filtering out bot traffic.
• Rate Limiting: Apply strict rate limits to known suspicious IPs or patterns.

Ensuring Consistency During High Traffic Spikes:

• Dynamic Throttling: Prioritize critical operations like view count updates while deferring non-essential tasks, controlled by the token bucket algorithm.
• Backpressure Management: Implement backpressure in the message queue to prevent downstream components like the database from being overwhelmed.

Cache Failure:

• Durable Write-Ahead Logs (WAL): Before updating the cache, log every operation to a persistent store, ensuring that data can be recovered in case of cache failure.
• Cache Replication: Use replicated caches to ensure that data isn’t lost if one cache node fails.

Redundancy and Consistency:

• Automatic Reconciliation: Compare data from primary databases, replicas, and logs regularly to identify and correct any discrepancies, ensuring consistency across all nodes.

Future improvements

Advanced Fraud Detection: Continuously improve machine learning models for fraud detection, incorporating new patterns and behaviors.

Global Distribution: Deploy the system across multiple geographical regions to reduce latency and improve fault tolerance.

Streaming Architecture: Move to a streaming architecture using tools like Apache Kafka for handling real-time processing of view events at scale.

Personalized Analytics: Provide more detailed analytics to content creators, such as demographic breakdowns of views or engagement metrics.