System requirements

Functional:

• The system should be able to count views for a particular video effectively.
• The system should display the view count correctly.
• The system should be able to filter out fraudulent views.
• The system should provide real-time updates to users.

Non-Functional:

• System should be robust enough to handle large amount of traffic.
• The system can be eventually consistent, view updates should not impact video quality or viewing experience.
• Out of CAP theorem we can probably compromise on consistency.

API Design

We separate the heavy ingestion of data (writes) from the user-facing display of data (reads).

1. Record a View Event (Fire and Forget) POST /v1/videos/{video_id}/events/view JSON

JSON

(Note: The API Gateway returns a 202 Accepted immediately upon dropping this payload into the message queue, rather than waiting for database confirmation).

2. Fetch Video Statistics GET /v1/videos/{video_id}/stats JSON

JSON

High-Level Design

• Client: Video player (web or mobile) emitting telemetry events at specific watch intervals.
• API Gateway / Load Balancer: Receives the massive influx of view events and handles rate limiting.
• Message Broker (Apache Kafka): The critical shock absorber. It ingests millions of events per second and stores them in partitioned topics, decoupling the fast ingestion from the slower processing.
• Stream Processing Engine (Apache Flink / Spark): Consumes the Kafka streams in real-time. It handles the complex logic: filtering out fraud, aggregating valid views, and creating micro-batches.
• Primary Database (Apache Cassandra): A NoSQL, wide-column store optimized for extremely high write throughput, acting as the permanent source of truth for the aggregated counts.
• Read Cache (Redis): Stores the frequently accessed view counts to serve the GET API requests with sub-millisecond latency.

Gemini said

This is exactly how you want to present your thoughts during a high-stakes technical whiteboard session. Structuring your answer this way demonstrates to an engineering panel that you can take a massive, open-ended problem and systematically break it down into scalable components.

Here is the complete system design document for the Video View Count System, swapping out the fragile synchronous approach for the robust, event-driven architecture we discussed.

System Requirements

Functional Requirements:

• The system must record view events when a user watches a video.
• The system must accurately aggregate and display the total view count for any given video.
• The system must identify and discard fraudulent or spam views (e.g., bot nets, rapid page refreshing).

Non-Functional Requirements:

• Extreme Write Scalability: The system must handle millions of concurrent view events without dropping requests or crashing databases (e.g., when a highly anticipated trailer drops).
• High Availability: The view counting mechanism must not bring down the main video playback service if it experiences lag or failure.
• Eventual Consistency: It is perfectly acceptable for the displayed view count to lag behind the actual real-time count by a few seconds or minutes.
• Low Latency Reads: Fetching the current view count to display on the client application must be lightning fast (< 50ms).

API Design

We separate the heavy ingestion of data (writes) from the user-facing display of data (reads).

1. Record a View Event (Fire and Forget) POST /v1/videos/{video_id}/events/view JSON

JSON

(Note: The API Gateway returns a 202 Accepted immediately upon dropping this payload into the message queue, rather than waiting for database confirmation).

2. Fetch Video Statistics GET /v1/videos/{video_id}/stats JSON

JSON

High-Level Design

• Client: Video player (web or mobile) emitting telemetry events at specific watch intervals.
• API Gateway / Load Balancer: Receives the massive influx of view events and handles rate limiting.
• Message Broker (Apache Kafka): The critical shock absorber. It ingests millions of events per second and stores them in partitioned topics, decoupling the fast ingestion from the slower processing.
• Stream Processing Engine (Apache Flink / Spark): Consumes the Kafka streams in real-time. It handles the complex logic: filtering out fraud, aggregating valid views, and creating micro-batches.
• Primary Database (Apache Cassandra): A NoSQL, wide-column store optimized for extremely high write throughput, acting as the permanent source of truth for the aggregated counts.
• Read Cache (Redis): Stores the frequently accessed view counts to serve the GET API requests with sub-millisecond latency.

Detailed Component Design

Let's dig into the core mechanics that make this architecture capable of handling YouTube-level scale.

1. Asynchronous Ingestion (The Shock Absorber) When a video goes viral, traffic spikes unpredictably. If our API tried to write directly to a database or even a Redis cluster, the connections would max out. By routing all POST requests directly to Kafka, the API Gateway simply appends a message to a log. Kafka can easily handle millions of these appends per second, acting as a buffer so the rest of the system can process the backlog at its own maximum safe speed.

2. Real-Time Fraud Detection (Time-Windowing) The Apache Flink workers pull batches of events from Kafka. Before counting a view, Flink evaluates it against sliding time windows.

• Rule Engine: If Flink sees 50 view events from the same ipAddress or userId within a 1-minute window, it flags them as bot activity and drops the events from the pipeline entirely.
• Validation: It can also cross-reference the watchDurationSeconds to ensure the user actually watched enough of the video to constitute a "view" (e.g., > 10 seconds).

3. Micro-Batching and Aggregation Instead of the "Write-Through" approach where 10,000 views result in 10,000 database updates, Flink aggregates the valid views in memory over a short interval (e.g., 10 seconds).

• If "Video A" gets 5,000 valid views in that 10-second window, Flink issues a single update command to the database: Add 5,000 to Video A's total. This drastically reduces the write load on the primary database.

4. Cache Invalidation Strategy A separate background worker monitors the database for updates. When a video's count is updated in Cassandra, the worker pushes the new total to the Redis cluster. The client application always reads from Redis, ensuring the database is never overwhelmed by read queries.

Database Design

For a system where the primary operation is incrementing counters at a massive scale, an AP (Available and Partition-tolerant) NoSQL database like Cassandra is the industry standard due to its masterless architecture and specific data types.

Table: Video_View_Counts | Column Name | Data Type | Constraints / Indexes | Description | | :--- | :--- | :--- | :--- | | videoId | UUID / String | Partition Key | Uniquely identifies the video. | | totalViews | Counter | | A special Cassandra data type optimized for concurrent increments without read-before-write locking. |

Table: Fraud_Logs (Optional, for Analytics) | Column Name | Data Type | Constraints / Indexes | Description | | :--- | :--- | :--- | :--- | | ipAddress | String | Partition Key | The offending IP. | | flaggedAt | Timestamp | Clustering Key | When the fraud was detected. | | reason | String | | e.g., "RATE_LIMIT_EXCEEDED". |