Architect a fault-tolerant Feed Generation Engine

Question

Design a fault-tolerant feed generation system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Feed Generation**: The system must generate personalized feeds for millions of users based on their interactions and preferences.
2. **Real-Time Updates**: The feed should be updated in real-time as users interact with the platform.
3. **API Access**: The system must expose a RESTful API for clients to request feeds.
4. **Rate Limiting**: Implement rate limiting to prevent abuse and ensure fair usage among users.

### Non-Functional Requirements:
1. **Fault Tolerance**: The system must continue to operate under various failure scenarios, ensuring users can always access their feeds.
2. **Scalability**: Handle millions of requests with low latency (target < 100ms).
3. **Consistency vs. Availability**: The system should aim for eventual consistency while prioritizing availability.
4. **Performance**: Optimize for high throughput, targeting hundreds of thousands of requests per second.

Assuming Zoom has approximately 300 million users:
- **Daily Active Users (DAU)**: Approximately 10% of users are active daily, totaling 30 million users.
- **Feed Requests**: If each user requests their feed 10 times a day, that results in 300 million feed requests daily or about 3.5k requests per second (RPS).
- **Peak Load**: During peak times (e.g., meetings), this could spike to 10,000 RPS.
- **Data Operations**: Assuming each feed generation requires 10 read operations to gather content, the system must handle 30,000 read operations per second at peak times.

### Components:
1. **Load Balancer**: Distributes incoming API requests across multiple instances of the feed generation service.
2. **Feed Generation Service**: Stateless microservices that generate feeds using data from the database and cache.
3. **Message Queue (e.g., Kafka)**: Handles asynchronous processing of feed updates and events, ensuring messages are processed even under load.
4. **Database (e.g., Cassandra)**: A distributed NoSQL database for storing user interactions and feed data.
5. **Cache (e.g., Redis)**: A distributed cache to store frequently accessed feeds for quick retrieval.
6. **Event Processor**: Handles real-time processing of user interactions to update feeds accordingly.

### Technology Choices:
- **Load Balancer**: AWS Elastic Load Balancer
- **Message Queue**: Apache Kafka
- **Database**: Apache Cassandra for horizontal scalability
- **Cache**: Redis for low-latency access

### Schema Design:
1. **Users Table**:
   - `user_id (UUID)`
   - `preferences (JSON)`
   - `last_updated (TIMESTAMP)`

2. **Interactions Table**:
   - `interaction_id (UUID)`
   - `user_id (UUID)`
   - `content_id (UUID)`
   - `timestamp (TIMESTAMP)`

3. **Feeds Table**:
   - `user_id (UUID)`
   - `feed_data (JSON)`
   - `last_generated (TIMESTAMP)`

### Access Patterns:
- Query feeds for a user: `SELECT feed_data FROM Feeds WHERE user_id = ?`
- Update interactions: `INSERT INTO Interactions (user_id, content_id, timestamp) VALUES (?, ?, ?)`

1. **Consistency vs. Availability**: By opting for eventual consistency with Cassandra, we ensure high availability and scalability, but some users might see stale data.
2. **Caching Strategy**: Using Redis for caching feeds balances the load on the database but introduces potential cache invalidation issues, requiring careful management of TTL and cache updates.
3. **Asynchronous Processing**: Using Kafka allows for decoupling of feed updates from requests but can introduce latency in feed visibility, impacting real-time expectations.
4. **Database Choice**: Choosing Cassandra over a relational database allows for horizontal scaling but sacrifices complex querying capabilities, requiring denormalization and careful schema design.

Architect a fault-tolerant Feed Generation Engine

Zoom

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Submit Your Answer

Zoom Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search