Loading...
99.99% availability:. Unlike most systems where a brief outage just delays operations, a view counting outage permanently loses data because views happen whether the system is ready or not.
Horizontal scalability for viral spikes: A single viral video can generate 100K+ views per second. The system must absorb these spikes without dropping events or degrading the counting pipeline for other videos.
Low-latency reads: Displaying a video's view count should return in under 50ms. This is the read path; it must be fast regardless of write load.
Eventual Consistency: View counts do need strict real time accuracy, but few sec delay can work.
Traffic:
1 Billion views/day-> 10^9/10^5->10^4 -> 10,000 views/sec
Peak Throughput:100K+ views/sec
Read/Write Ratio: 10:1 (Every video page load fetches the view count (read), but only a fraction of page loads generate a view event (write, after minimum watch duration))
Storage:
Each counter is video_id(8 bytes)+ counter(8bytes) + Timestamp(8 bytes)->24 bytes/video
With 500M videos->12 GB /day
for Year: 365*12->4380*10^9/10^12 -> 4.3 TB /year
Kafka event retention: Each view event is roughly 200 bytes (video_id, timestamp, user context, event ID). At 1B events/day: 200GB/day. With 7-day retention: 1.4TB. This is the buffer that absorbs spikes and enables replay.
POST /v1/views
Headers: X-Request-ID (idempotency key)
Body: {
video_id: string,
watch_duration_ms: number,
client_timestamp: ISO-8601,
device_fingerprint: string (optional)
}
Response: 202 Accepted { event_id: string }
GET /v1/videos/{video_id}/views
Response: 200 OK {
video_id: string,
total_views: number,
last_updated: ISO-8601
}
GET /v1/videos/{video_id}/analytics?period=daily&start=2026-01-01&end=2026-01-31
Response: 200 OK {
video_id: string,
period: "daily",
data: [{ date: "2026-01-01", views: 45230 }, ...]
}
1.DynamoDB/ Cassandra Store
video_view_counts
video_id STRING Partition Key
total_views NUMBER Atomic counter
last_updated NUMBER Unix timestamp (ms)
daily_view_stats
video_id STRING Partition Key
date STRING Sort Key (YYYY-MM-DD)
views NUMBER
3.Redis Cache
Key: views:{video_id}
Value: {total_views, last_updated}
TTL: 300 seconds (5 minutes)
The architecture separates the write path (counting views) from the read path (displaying counts).
View Ingestion Service: Validates the request format and publishes the event to Kafka
Fraud Detection: A filtering layer that examines watch duration, IP patterns, device fingerprints, and rate limits.
Kafka: The durable buffer that decouples ingestion from counting. Events are partitioned by video_id so all views for the same video land on the same partition.
Aggregation Consumer: Reads batches of events from Kafka, groups them by video_id, counts occurrences, and flushes atomic increments to DynamoDB.
Dead-Letter Queue: Events that repeatedly fail processing (malformed data, schema violations) are routed to a DLQ rather than blocking the pipeline.
Client to Redis Cache: When a user loads a video page, the view count service checks Redis first. Cache hits (the common case for popular videos) return in under 1ms.
Cache Miss to DynamoDB: For less popular videos not in cache, the service reads from DynamoDB, returns the count, and populates Redis with a 5-minute TTL. Subsequent reads for that video hit cache.
The aggregation consumer is the heart of the system. It reads a configurable batch of events from Kafka (typically every 5 seconds or 5,000 events, whichever comes first), groups them by video_id, and produces a count per video. For a batch of 5,000 events spanning 200 unique videos, it issues 200 atomic ADD operations to DynamoDB, a 25x reduction in write volume compared to per-event processing.
The system handles this through multiple mechanisms:
Consumer auto-scaling:Monitoring tracks consumer lag (the gap between the latest produced offset and the last consumed offset). When lag exceeds a threshold (such as 10,000 unconsumed events), the auto-scaler launches additional consumer instances.
Partition count headroom: The system uses more partitions than consumers (such as 30 partitions with 10 consumers). During a spike, new consumers can absorb partitions immediately without requiring a partition increase
Backpressure and graceful degradation:Under extreme load, non-critical consumers (analytics aggregation, notification triggers) are throttled or paused while the primary counting pipeline continues uninterrupted.
Fraud detection operates as a filter before Kafka, using multiple signals in combination:
Watch duration: Views under 30 seconds are flagged.
IP rate limiting: More than 100 views per hour from a single IP triggers temporary blocking.
Device fingerprinting: Browser and device characteristics (screen resolution, installed fonts, WebGL renderer) create a fingerprint.
Behavioral analysis: Legitimate viewers exhibit natural patterns: variable watch durations, organic arrival times, diverse referral sources.
Event IDs: Each view event carries a unique event_id (UUID generated by the client). The aggregation consumer maintains a sliding window deduplication set in Redis: before counting an event, it checks if the event_id exists.
Offset tracking: The consumer commits offsets only after successful DynamoDB writes. On restart, it resumes from the last committed offset.