Build a high-throughput Feed Generation Pipeline

Question

Design a high-throughput feed generation system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **High Throughput**: Support a minimum of 10 million requests per second for generating market feeds.
2. **Real-Time Processing**: Ensure that the feed generation pipeline delivers data with a latency of less than 100 milliseconds.
3. **Data Accuracy**: Guarantee that generated feeds are accurate, reflecting the latest market data.
4. **Multi-Region Support**: Enable deployment across multiple regions to ensure low latency for users globally.
5. **Scalable API**: Provide a RESTful API for external access to the generated feeds.

### Non-Functional Requirements
1. **Availability**: Achieve 99.99% uptime to ensure that users can access feeds at all times.
2. **Scalability**: The system should easily scale horizontally to handle increased loads during market surges.
3. **Security**: Implement robust authentication and authorization mechanisms to secure the data.
4. **Monitoring and Logging**: Provide detailed logs and monitoring metrics for operational insights.

To estimate the capacity needed for the feed generation pipeline, we can use the following assumptions:
1. **Users**: Assume 100,000 concurrent users, each generating 100 requests per second.
2. **Total Requests**: This results in 100,000 * 100 = 10 billion requests per second at peak load.
3. **Data Size**: Each feed update averages 1 KB in size.
4. **Throughput Calculation**: 
   - **Total Data**: 10 billion requests * 1 KB = 10 TB of data per second.
   - **Monthly Data**: 10 TB * 60 seconds * 60 minutes * 24 hours * 30 days = 25 PB per month.
5. **Message Queue**: A message queue system like Kafka should be capable of handling this load with sharding and partitioning to balance the throughput.

### Component Diagram
- **Load Balancer**: Distributes incoming requests across multiple instances of the feed generation service.
- **Feed Generation Service (Microservice)**: Implements the core logic for processing incoming requests and generating feeds. Technologies: Node.js or Go for performance.
- **Message Queue (Kafka)**: Handles asynchronous processing and decouples the feed generation from the request handling.
- **Data Store (Cassandra)**: A NoSQL database optimized for write-heavy workloads to store feed data with high availability.
- **Cache (Redis)**: Caches frequently accessed feed data to reduce latency.
- **Monitoring (Prometheus + Grafana)**: For tracking system performance and health.
- **Authentication Service (OAuth2)**: Manages user authentication and token issuance.

### Schema Design
- **Feed Table**:
  - **feed_id** (UUID, Primary Key)
  - **market_symbol** (String, Indexed)
  - **price** (Decimal)
  - **volume** (Integer)
  - **timestamp** (Timestamp)
- **User Requests Table**:
  - **request_id** (UUID, Primary Key)
  - **user_id** (UUID, Indexed)
  - **feed_id** (UUID, Foreign Key)
  - **status** (String, e.g., 'PENDING', 'COMPLETED')

### Access Patterns
- High write throughput for feeds during market updates.
- Frequent reads for the latest market data, hence the use of caching.

### Architectural Decisions
1. **Consistency vs. Availability (CAP Theorem)**: Opting for eventual consistency using Cassandra allows for higher availability and partition tolerance, which is crucial for a trading system where feeds must be available at all times.
2. **Choice of Message Queue**: Kafka is chosen for its durability and ability to handle high throughput, with trade-offs in terms of complexity for operational maintenance.
3. **Caching Strategy**: Using Redis for caching improves read performance but introduces the challenge of cache invalidation and potential stale data.
4. **Deployment Strategy**: Adopting a phased rollout with canary releases allows for incremental testing and risk mitigation, ensuring that the system can handle real-world loads before full deployment.

Build a high-throughput Feed Generation Pipeline

Optiver

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Optiver Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search