Design a Recommendation for Redfin

Question

Design a event-driven recommendation system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **User Interaction**: Enable users to receive personalized property recommendations based on their search history, preferences, and location.
2. **Real-time Updates**: Provide real-time recommendations as users interact with the platform, such as viewing or saving properties.
3. **Feedback Loop**: Allow users to provide feedback on recommendations to improve the system.
4. **Diversity of Recommendations**: Ensure that the recommendations include a variety of properties (e.g., price range, location, type).

### Non-Functional Requirements
1. **Scalability**: The system should handle millions of requests per day, especially during peak times like weekends.
2. **Low Latency**: Aim for a response time of less than 200ms for recommendation queries to enhance user experience.
3. **Availability**: System uptime should be greater than 99.9% to ensure users can access recommendations anytime.
4. **Data Freshness**: Recommendations should reflect the latest property listings and user interactions in near real-time.

To estimate capacity, let's assume:
- **Daily Active Users (DAU)**: 1 million users
- **Average Requests per User per Day**: 10 requests
- **Total Daily Requests**: 1M users * 10 requests = 10 million requests
- **Peak Load**: Assuming 20% of requests occur in a 2-hour window during peak times, that results in 2 million requests/hour or ~555 requests/second.

To account for growth, we should design for a 3x increase over the next few years, leading to a target capacity of around 1,665 requests/second.

### Component Diagram
- **Frontend**: React-based web app for user interaction.
- **API Gateway**: AWS API Gateway to manage incoming requests and route them to appropriate microservices.
- **Recommendation Service**: A microservice built using Python (Flask) to handle business logic for generating recommendations.
- **Event Bus**: AWS SNS/SQS for managing events and communication between services.
- **Feature Store**: A NoSQL database (DynamoDB) to store user profiles, property features, and interaction histories.
- **Batch Processing**: AWS Lambda functions to update recommendations at regular intervals based on new data.
- **Model Serving**: Use AWS SageMaker for hosting the ML model, ensuring low-latency access.
- **Monitoring**: AWS CloudWatch for logging and monitoring system performance.

### Schema Design
- **User Table**:  
  - `user_id` (PK)  
  - `location`  
  - `preferences` (JSON)  
  - `interaction_history` (JSON)

- **Property Table**:  
  - `property_id` (PK)  
  - `price`  
  - `location`  
  - `features` (JSON)

- **Recommendations Table**:  
  - `user_id` (FK)  
  - `property_id` (FK)  
  - `score` (float)  
  - `timestamp`

This structure allows for efficient retrieval of user preferences and interaction history while enabling simple joins to fetch relevant property features.

### Architectural Decisions
1. **Consistency vs. Availability**: Choosing eventual consistency for the recommendation updates allows for better availability and user experience, without delays in returning results.
2. **Real-time vs. Batch Processing**: While real-time recommendations are ideal, using a hybrid approach where batch processing updates the recommendations periodically (e.g., every few minutes) can reduce load on the system and still keep data relatively fresh.
3. **Model Complexity vs. Latency**: Opting for a simpler model (e.g., collaborative filtering) may yield lower latency for serving recommendations, while more complex models (e.g., deep learning) could improve accuracy but introduce latency. A/B testing can be used to find the right balance.
4. **Feedback Loop Implementation**: Implementing a feedback loop to refine recommendations can introduce complexity in data handling, but is crucial for improving model accuracy and user satisfaction.

Design a Recommendation for Redfin

Redfin

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Redfin Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search