Design Recommendation Infrastructure for real-time analytics

Question

Design a low-latency recommendation system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - Provide real-time recommendations based on user activity, preferences, and historical data.
  - Handle millions of concurrent requests with low latency (sub-100ms).
  - Support A/B testing for recommendation algorithms.
  - Allow for user feedback on recommendations to improve the model.

- **Non-Functional Requirements**:
  - High availability: 99.99% uptime.
  - Low latency: Responses must be returned in under 100 milliseconds.
  - Scalability: Must handle spikes in traffic, especially during peak Slack usage times.
  - Consistency: Recommendations should be consistent across different devices and sessions.
  - Security: Ensure user data privacy and compliance with GDPR.

- Assume Slack has 12 million daily active users (DAUs) with an average of 100 interactions per user per day.
- This results in approximately $ 12M 	imes 100 / 86400 \approx 1,385 $ requests per second (QPS).
- For peak hours, assume a 5x increase due to usage spikes, leading to a peak QPS of approximately 6,925.
- Each recommendation request may involve processing user preferences and historical data, requiring approximately 1 KB of data per request. 
- Estimated storage for user preferences and interaction history for 12 million users over a year: $ 12M 	imes 1KB 	imes 365 \approx 4.38 TB $.

The architecture consists of several components:

- **API Gateway**: Handles incoming requests, load balancing across multiple services.
- **Recommendation Service**: Core service that generates recommendations, built using Python with Flask.
- **User Profile Database**: Stores user preferences in a NoSQL database (e.g., MongoDB) for flexible schema and quick reads.
- **Cache Layer**: Redis for caching frequent queries and recent recommendations to minimize database hits.
- **Analytics Engine**: A separate service that processes user interactions and feedback to improve recommendation algorithms, using Apache Kafka for event streaming.
- **Monitoring**: Prometheus for metrics collection and Grafana for visualization, along with alerting via PagerDuty for downtime or latency spikes.

The user profile schema in MongoDB could look like:

```json
{
  "userId": "string",
  "preferences": {
    "channels": ["channel_id_1", "channel_id_2"],
    "keywords": ["keyword1", "keyword2"]
  },
  "interactionHistory": [
    {
      "timestamp": "ISODate",
      "actionType": "string",
      "itemId": "string"
    }
  ]
}
```

- **Access Patterns**:
  - Frequent reads for recommendations based on user preferences.
  - Writes for updating interaction history and preferences as users interact with Slack.
  - Index on `userId` for quick lookups and potentially on `preferences` for complex queries.

- **Consistency vs Availability**: Opt for eventual consistency in the recommendation service to ensure low latency. Use a cache to serve quick responses, accepting the risk of slightly outdated recommendations.
- **Database Choice**: NoSQL (MongoDB) offers flexibility and scalability, but may not support complex queries as efficiently as SQL databases. Prioritize speed and scale over complex querying capabilities.
- **Caching Strategy**: Use Redis for caching recommendations to reduce database load, but this introduces complexity in cache invalidation.
- **Failure Handling**: Implement circuit breakers for the recommendation service to handle failures gracefully without impacting user experience. Introduce retries and fallbacks for downstream services.

Design Recommendation Infrastructure for real-time analytics

Slack

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Slack Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search