Requirements

System Requirements

Functional Requirements:

• Track product views in real-time across millions of active users.
• Calculate and serve "Frequently Viewed" products over multiple sliding time windows (24h, 7d, 30d).
• Calculate "Co-Viewed" products (Users who viewed X also viewed Y) for cross-selling.
• Allow users to filter trending products dynamically by category.

Non-Functional Requirements:

• Massive Write Scalability: Must absorb extreme clickstream data (e.g., thousands of views per second) without database lockups.
• Ultra-Low Latency Reads: The Product Page must fetch the trending and co-viewed lists in < 20ms to prevent slowing down page rendering.
• High Cardinality Tolerance: The system must survive the combinatorial explosion of millions of unique users viewing millions of unique products.

API Design

1. Ingest a Product View POST /v1/views

JSON

Returns: 202 Accepted. This is a fire-and-forget endpoint. The client does not wait for the aggregation to complete.

2. Get Trending Products (Filtered) GET /v1/trending?category=cat-elec&window=24h&limit=10 Returns: 200 OKwith an ordered array of product IDs and their view counts.

3. Get Co-Viewed Products GET /v1/products/{product_id}/co-viewed Returns: 200 OK with an array of related product IDs.

High-Level Design

• View Ingestion Service: A lightweight REST/gRPC service that simply validates the incoming view payload and publishes it directly to a message broker.
• Message Broker (Kafka): The highly durable buffer. It acts as a shock absorber, preventing traffic spikes from overwhelming the databases.
• Stream Processing Engine (Apache Flink): The core computational brain. It reads raw events from Kafka, holds state in memory, and continuously calculates sliding window aggregates and session-based co-view occurrences.
• Cold Storage / Data Lake (S3 / GCS): Raw views are dumped here directly from Kafka for compliance and batch machine learning. We explicitly avoid a Time Series Database (TSDB) here to prevent High Cardinality memory crashes.
• Hot Read Store (Redis): Stores the continuously updated aggregated leaderboards and co-viewed matrices. We explicitly avoid an RDBMS here to prevent complex ORDER BY bottlenecks on the read path.

.

Detailed Component Design

Detailed Component Design

Here is the mechanics of the Flink to Redis pipeline.

1. Stream Aggregation (Apache Flink) Flink consumes the raw_product_views Kafka topic and splits the stream into two parallel jobs:

• Job A (The Sliding Windows): Flink maintains sliding windows (e.g., a 24-hour window that updates every 1 minute). It counts views per product_id. When the window slides, Flink emits an upsert event: (product-9876, 24h_views: 1500, category: cat-elec).
• Job B (The Sessionizer): Flink groups events by user_id using a Session Window (closing after 30 minutes of inactivity). When the session closes, Flink looks at all products viewed, generates pairs (e.g., [prod-A, prod-B]), and increments a co-view counter for that specific pair.

2. The Read/Sort Bottleneck Resolution If we wrote Job A's output to an RDBMS (PostgreSQL), the API would have to execute: SELECT product_id FROM views WHERE category = 'cat-elec' ORDER BY views_24h DESC LIMIT 10. At thousands of queries per second, this dynamic sorting is highly CPU-intensive and slow.

• Solution: Flink writes the aggregates directly to Redis Sorted Sets (ZSETs). Redis maintains the exact sorted order in memory at all times. Fetching the top 10 items is an O(log(N) + M) operation that takes less than a millisecond