Loading...
Define the APIs expected from the system. This is your chance to analyze and define the read and write paths so that you can come up with the high-level design...
API will be post/v1/views
Headers: X-Request-ID (idempotency key)
Body: {
video_id: string,
watch_duration_ms: number,
client_timestamp: ISO-8601,
device_fingerprint: string (optional)
}
GET/v1/video_id/views:
REsponse:
200 OK {
video_id:string,
total views:number
last_update:timestamp
}
Analytics:
GET /v1/video_id/analytics?period =daily&start=2026-01-01&end=2026-01-31
Response: 200 OK {
video_id: string,
period: "daily",
data: [{ date: "2026-01-01", views: 45230 }, ...]
}
Describe the overall system architecture. Identify the main components needed to solve the problem end-to-end. Use the diagramming tool to create a block diagram.
Client to API Gateway: The video player client sends a view event after minimum watch duration (30 seconds). The API gateway load-balances across ingestion service instances.
View Ingestion Service: Validates the request format and publishes the event to Kafka. This service is stateless and horizontally scalable. It does not make counting decisions; it just gets events into the pipeline reliably.
Fraud Detection: A filtering layer that examines watch duration, IP patterns, device fingerprints, and rate limits. Invalid views are discarded with a metric logged (for monitoring false positive rates). Valid views proceed to Kafka.
Kafka: The durable buffer that decouples ingestion from counting. Events are partitioned by video_id so all views for the same video land on the same partition. This ensures a single consumer handles all events for a given video, preventing write conflicts.
Interview Tip
Kafka serves double duty: it is both a buffer that absorbs traffic spikes and a replay log that enables recovery. If the aggregation consumer has a bug that miscounts views, you fix the bug and replay events from Kafka to recalculate correct counts. No other component in the pipeline provides this combination.
Aggregation Consumer: Reads batches of events from Kafka, groups them by video_id, counts occurrences, and flushes atomic increments to DynamoDB. After a successful flush, it updates Redis and commits the Kafka offset. This ordering (write to DB, update cache, commit offset) ensures at-least-once processing.
Client to Redis Cache: When a user loads a video page, the view count service checks Redis first. Cache hits (the common case for popular videos) return in under 1ms.
Cache Miss to DynamoDB: For less popular videos not in cache, the service reads from DynamoDB, returns the count, and populates Redis with a 5-minute TTL. Subsequent reads for that video hit cache.
This architecture handles viral spikes gracefully: Kafka absorbs the burst, consumers process at their own pace, and the read path is completely isolated from write load.
Deep dive into 2-3 key components. Explain how they work, how they scale, discuss tradeoffs, capacity, and any relevant algorithms or data structures.
DATABASE DESIGN:
video_view_counts
video_id STRING Partition Key
total_views NUMBER Atomic counter
last_updated NUMBER Unix timestamp (ms)
video_id STRING KEY
total view number
last_update timestamp
A single table with video_id as the partition key. DynamoDB's atomic ADD operation increments total_views without read-modify-write cycles, eliminating race conditions during concurrent batch flushes.
daily_view_stats
video_id STRING Partition Key
date STRING Sort Key (YYYY-MM-DD)
views NUMBER
Composite key (video_id + date) enables efficient range queries: "give me daily views for video X from January to March." The aggregation consumer updates both tables in the same batch flush.
redis cache:
Key: video_id
Value{total_view, last_update}
TTL: 300 seconds
The system's reliability comes from its ability to handle failures without losing view counts. Each failure scenario maps to a specific architectural decision made earlier.
Consumer crash recovery: Kafka retains events from last committed offset. New consumer replays and deduplicates.
Scenario: The aggregation consumer crashes after reading a batch of 5,000 events but before committing the Kafka offset.
What happens: Kafka retains all events from the last committed offset. A new consumer instance (launched by the container orchestrator) starts reading from that offset. The 5,000 events are replayed. If the crash happened after DynamoDB writes but before offset commit, the idempotent processing layer (event ID dedup) ensures replayed events are not counted twice.
Why this works: The commit-after-write ordering guarantees at-least-once delivery. Idempotency converts at-least-once into effectively exactly-once. No views are lost, and no views are double-counted.
Key Insight
Dead-letter queues prevent silent data loss: events that fail processing repeatedly (schema errors, encoding bugs) are routed to a DLQ rather than being discarded or blocking the pipeline. Without a DLQ, a single malformed event could block all processing behind it, or worse, be silently dropped. The operations team inspects the DLQ, fixes the root cause, and replays the events.
Scenario: DynamoDB becomes temporarily unavailable during a batch flush (network partition, service degradation).
What happens: The consumer's DynamoDB write fails. The consumer does not commit the Kafka offset (because the write did not succeed). It retries with exponential backoff: 1 second, 2 seconds, 4 seconds, up to a maximum of 60 seconds. Meanwhile, view events continue flowing into Kafka, which buffers them durably. When DynamoDB recovers, the consumer processes the accumulated backlog at full speed.
Why this works: Kafka's retention (7 days) means events are safe for far longer than any realistic database outage. The consumer's backoff prevents overwhelming a recovering database with retry storms.
Scenario: Redis becomes unavailable or a cache node fails.
What happens: Read requests that would normally hit Redis fall back to DynamoDB directly. Read latency increases from sub-1ms (Redis) to 5-10ms (DynamoDB), noticeable but not catastrophic. The aggregation consumer's Redis update step fails silently (it is best-effort). When Redis recovers, cache entries repopulate organically through cache-aside on read misses.
Why this works: Redis is a performance optimization, not a correctness requirement. The system functions correctly (just slower) without it.
Scenario: A global event (New Year's, Super Bowl halftime) causes all traffic to spike simultaneously, overwhelming the aggregation consumers.
What happens: Consumer lag increases rapidly. The auto-scaler launches additional consumer instances. Non-critical consumers (analytics aggregation, milestone notifications) are paused to free Kafka consumer group capacity for the primary counting pipeline. If lag continues growing, the system activates graceful degradation: cache TTLs are extended (serving slightly staler counts) and analytics writes are deferred.
Why this works: Prioritizing core counting over secondary features ensures the primary guarantee (accurate view counts) is maintained even under extreme load.