Design Recommendation Infrastructure for mobile apps

Question

Design a geo-distributed recommendation system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - **User Personalization**: Provide real-time product recommendations based on user preferences, previous orders, and location.
  - **Location Awareness**: Recommendations should be sensitive to the user's geographic location, highlighting nearby restaurants and delivery options.
  - **Scalability**: Handle millions of concurrent requests during peak times (e.g., lunch and dinner).

- **Non-Functional Requirements**:
  - **Latency**: The system must respond to recommendations within 200 milliseconds.
  - **Availability**: The system must maintain 99.9% uptime, even during regional outages.
  - **Consistency**: Users should see consistent recommendations across sessions, with eventual consistency for less critical data.
  - **Security**: Ensure user data is encrypted both at rest and in transit.

Assuming Postmates has approximately 10 million active users, with a peak of 1 million concurrent users during busy hours.

- **Request Volume**:
  - Each user generates approximately 10 recommendation requests per hour.
  - Peak request rate = 1 million users * 10 requests/hour = 2,778 requests/second.

- **Data Storage**:
  - Assuming 100 bytes per recommendation entry (including metadata), we expect:
    - 2,778 requests/second * 100 bytes/request * 3600 seconds/hour = ~1 GB/hour.
  - For a week, that translates to ~168 GB of data, necessitating a scalable storage system.

- **Recommendation Updates**:
  - Assume 1 million updates per day for user preferences and restaurant data, leading to ~12 updates/second.

The architecture consists of the following components:

- **Client-side (Mobile Apps)**: Interfaces for users to view recommendations.
- **API Gateway**: Handles incoming requests and routes them to appropriate services, implemented using **AWS API Gateway**.
- **Recommendation Service**: Core logic for generating recommendations, using **Apache Spark** for batch processing of historical data and **TensorFlow** for machine learning models.
- **User Profile Service**: Stores user preferences and behavior, using **Amazon DynamoDB** for high availability and low latency.
- **Geolocation Service**: Determines user location, leveraging **Google Maps API**.
- **Data Store**: **Amazon Aurora (PostgreSQL)** for relational data and **Amazon S3** for unstructured data (like user interaction logs).
- **Cache Layer**: **Redis** to store frequently accessed recommendations for faster retrieval.
- **Message Queue**: **Apache Kafka** for asynchronous processing of user events and recommendations.

The data model will consist of the following key entities:

1. **Users**:
   - `UserID`: String (PK)
   - `Preferences`: JSON (e.g., favorite cuisines, dietary restrictions)
   - `Location`: Geolocation coordinates
   - `OrderHistory`: List of past orders

2. **Restaurants**:
   - `RestaurantID`: String (PK)
   - `Name`: String
   - `Location`: Geolocation coordinates
   - `Menu`: List of items
   - `Ratings`: Average rating

3. **Recommendations**:
   - `RecommendationID`: String (PK)
   - `UserID`: String (FK)
   - `RestaurantID`: String (FK)
   - `Score`: Float (based on user preferences and history)
   - `Timestamp`: Datetime

This schema allows efficient querying for user-specific recommendations, while the denormalized structure supports fast read operations.

1. **Consistency vs. Availability**: Following the CAP theorem, we opt for eventual consistency for recommendations, allowing users to see slightly outdated data while ensuring high availability across geo-distributed nodes.

2. **Latency vs. Freshness**: We maintain a caching layer for frequently accessed recommendations to reduce latency, but this may serve stale data. To mitigate this, we implement a cache invalidation strategy based on user actions.

3. **Complexity vs. Scalability**: Using a microservices architecture increases operational complexity but improves scalability. We will use container orchestration (e.g., **Kubernetes**) to manage the deployment and scaling of services.

4. **Operational Excellence**: We will set SLOs for latency (<200ms) and availability (99.9%), implementing monitoring with **Prometheus** and using **Grafana** for visualizing metrics. We will also adopt chaos engineering practices to test the resilience of the system under failure conditions.

Design Recommendation Infrastructure for mobile apps

Postmates

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Postmates Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search