System requirements

Functional:

• Generate product recommendations personalized by user behavior.
• Input signals include: user clicks, product views, cart additions, purchases.
• Serve recommendations on homepage, product page, and cart checkout.
• APIs to get/store user activity, retrieve recommendations.
• Support feedback loop for training (likes, skips, etc).

Non-Functional:

• Low latency (<200ms) for recommendation fetch.
• High availability (99.99%).
• Horizontal scalability to handle global traffic.
• Real-time and batch inference support.
• Privacy-compliant user data storage (GDPR, CCPA).

Capacity estimation

• DAU: 10 million
• QPS for homepage recommendations: 50k/s peak
• Product catalog: 1M+ items
• Data ingestion: 200k events/s (clicks, views, etc)
• Model refresh frequency: every 6 hours (batch) + online updates

API design

1. POST /api/v1/user-events
   • Payload: {user_id, event_type, product_id, timestamp}
2. GET /api/v1/recommendations/home?user_id=...
   • Returns: list of product_ids
3. POST /api/v1/feedback
   • For recording positive/negative feedback to improve ranking

Database design

• User Activity Store: append-only (e.g., BigQuery, S3)
• Feature Store: key-value store for ML features (e.g., Redis, DynamoDB)
• Recommendation Store: precomputed results (e.g., Cassandra, Redis)
• Product Metadata: relational (e.g., PostgreSQL)

High-level design

[Client/App] ──▶ [API Gateway]

│

▼

[Kafka / PubSub]

│

┌──────────┼────────────┐

▼ ▼

[ClickStream Processor] [Batch Processor]

(Flink/Spark) (Spark)

│ │

▼ ▼

[Feature Store] [Offline Model Training]

│ │

▼ ▼

[Online Ranker Service] ◀──── [Trained Models]

│

▼

[Recommendation API] ──▶ [Client]

Request flows

1. User event ingestion:
   • Client logs viewed_product, add_to_cart events
   • Sent to Kafka, Flink does ETL & writes to Feature Store
2. Offline Training:
   • Spark job on historical data
   • Trains collaborative/content-based model
3. Online serving:
   • Ranker Service queries Feature Store + model inference
   • API returns top-N items in <200ms

Detailed component design

Flink Processor

• Consumes Kafka events in real-time
• Joins with product metadata from Redis
• Writes user embeddings & product co-visitation matrix
• Fault tolerance via Flink checkpointing

Feature Store

• Stores dense/sparse features
• Use Redis (low-latency)
• TTL policies for stale features

Model Inference (Online)

• Preload model in memory (TensorFlow/ONNX)
• Use ANN search (Faiss) to find similar items
• Latency budget: <150ms

Trade offs/Tech choices

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?