Requirements

Functional Requirements:

List the key functional requirements for the system (Ask the AI for hints if stuck)...

• Users should be able to see live view count/run-time changes

Non-Functional Requirements:

List the key non-functional requirements (performance, scalability, reliability, etc.)...

• System should be highly scalable(High read/write throughput)
• ViewCount should not be incremented because of bots refreshing the page
• Count should not include fraudulent views
• System should be highly available
• System should be realistically reliable(not payment system level accuracy expected)
• System should be eventually consistent

Estimations

• total 100 M users, 10% users visit website daily
• DAU : 10 M users
• Average length of content watched = 2 hours
• attention span = 30s to 2 min and they will watch the complete video if they like it
• so total no of videos watched by 1 person = 50
• So 500 M daily count
• we will later scale it to multi continent, but for now we can assume multiple cities across the country.(so still a multi region)
• Assumptions:
  • The scope of the problem is to show the count only and not the user-ids list who has watched the video.
  • The authZ/authN has already been taken care of

API Design

Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...

update count: POST count (accepts positive integer only)

view count: GET count

• POST /api/videos/{videoID}/count
  • Description: Register a new video count.
  • Request Body: {"userID": "acbde" }
  • This will also use other meta info while deciding eligibility like client IP address, location etc.
  • Response: { "message": "View count updated." }

• GET /api/videos/{videoID}/count
  • Description: Get the number of views for a video.
  • Response: { "videoID": "12345", "totalViews": 15000, "uniqueViews": 12000 }

• GET /api/videos/{videoID}/analytics
  • Description: Get a detailed breakdown of views for a video, per month.
  • Response: { [{ "date": int, "totalViews": 15000, "uniqueViews": 12000 } , ...] }

High-Level Design

Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.

• clients hit server to update count
• Servers will maintain uniqueness by using some time based cache: multiple bloom filters to find duplicates in last n time units.(1 bloom filter for 1 time unit and it will be refreshed after its greater than n time units old)
  • Other than the technique mentioned above, there is one more option for de-duplication which is using Redis like in memory cache for storing recent views.
• Event stream for not losing data
• Even stream is separately and asynchronously read and processed by another server and it updates DB by processing one chunk at a time
• For reading the count value, we will read it from cache if its absent there we will read it from DB.
• Eventual consistency is fine here.

For DB we will use simple key value store, as its good for high write and read throughput. Consistency is not a big dealbreaker here.

This architecture is designed for high scalability and allows you to maintain eventual consistency, as specified in your requirements.

Detailed Component Design

Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.

How we will scale this:

• Kafka or event stream can easily handle quarter million requests per second per node; so its not a bottleneck for our design
• we can divide processing servers based on read and write traffic
• We can divide event stream, processing server and DB based on video id hash; so that all the load will not come on single node and we don't have to lock row at all; only one node will access one type of row and that way sharding will take care of contingency of resources and avoid concurrent writes

Database Design

• Both SQL or NoSQL can be used here; but NoSQL will be preferred for high write throughput.
• Table/Collection: View, User, Video

Sharding:

• Video Sharding: Based on VideoID, ensuring that each shard contains data for a distinct set of videos, which helps distribute the load evenly across shards.
• User-Based Sharding: Secondary sharding strategy based on UserID for tables where user-specific data is important, such as ViewHistory.

Partitioning:

• Time-Based Partitioning: Used for ViewHistory to split data by time intervals (e.g., daily or monthly partitions) to optimize query performance and data management.
• Geographical Partitioning: Partition data by region for tables that might benefit from regional queries, enhancing performance in geographically distributed systems.

Replication Strategy:

• Multi-Region Replication: Replicate data across multiple geographic regions to ensure high availability and disaster recovery.
• Synchronous Replication: Critical data like ViewCount and Video tables use synchronous replication to ensure data consistency across primary nodes.
• Asynchronous Replication: Less critical data, such as logs and historical data (ViewHistory), use asynchronous replication for eventual consistency, reducing latency during write operations.