Design a Recommendation Service

Question

Design a fault-tolerant recommendation system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - Provide personalized recommendations based on user interactions, preferences, and historical data.
  - Support real-time and batch processing of recommendations.
  - Allow users to provide feedback on recommendations to improve accuracy.
  - Integrate with existing ServiceNow applications for seamless user experience.

- **Non-Functional Requirements**:
  - **Scalability**: Handle up to 10 million requests per day, with peak loads reaching 5,000 QPS.
  - **Availability**: 99.99% uptime, ensuring the service is available at all times.
  - **Latency**: Response time of less than 200 ms for 95% of requests.
  - **Fault Tolerance**: System must gracefully handle failures and ensure data consistency.

- Assuming an average of 5 recommendations per request, with 10 million requests per day:
  - Total recommendations per day = 10 million requests * 5 = 50 million recommendations.
  - For a 30-day month, we would need to handle 1.5 billion recommendations.
  - Estimated storage for user interactions could be around 100 TB, considering user profiles and interaction history.
  - Bandwidth estimation: If each recommendation requires 1 KB of data, then:
    - 50 million recommendations/day * 1 KB = 50 GB/day, leading to ~1.5 TB/month.

- **Component Diagram**:
  1. **User Service**: Handles user data and authentication (e.g., OAuth for secure logins).
  2. **Recommendation Engine**: Core logic for generating recommendations (using Apache Spark for batch processing and real-time recommendation via Redis).
  3. **Data Store**: NoSQL database (e.g., MongoDB) for storing user interactions and preferences.
  4. **Caching Layer**: Redis for caching frequently accessed recommendations to reduce latency.
  5. **Feedback Service**: Collects user feedback to adjust recommendation algorithms.
  6. **Monitoring and Alerting**: Prometheus for monitoring metrics and Grafana for alert dashboards.

- **User Collection**:
  ```json
  {
    "userId": "string",
    "preferences": ["string"],
    "interactionHistory": [{
      "itemId": "string",
      "timestamp": "datetime",
      "feedback": "int"
    }]
  }
  ```

- **Recommendation Collection**:
  ```json
  {
    "recommendationId": "string",
    "userId": "string",
    "recommendedItems": ["string"],
    "timestamp": "datetime"
  }
  ```
- **Access Patterns**: Fast lookups for user interactions and recommendations, ensuring minimal read/write latency.

- **Consistency vs Availability**: Opting for eventual consistency to ensure high availability of recommendations. This allows the system to remain responsive during high load.

- **Database vs Caching**: Choosing MongoDB for persistence due to its schema flexibility and scalability, while Redis serves as a powerful cache to meet latency requirements.

- **Batch Processing vs Real-time**: Implementing a hybrid approach, where the recommendation engine uses batch processing for initial recommendations and real-time processing for updates based on user interactions.

- **Failure Handling**: Implementing retries and fallbacks for critical components, ensuring that service remains operational even if one part fails. Use of circuit breakers to prevent cascading failures.

Design a Recommendation Service

ServiceNow

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

ServiceNow Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search