Build a high-throughput Monitoring Pipeline

Question

Design a high-throughput monitoring system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Real-time Monitoring**: The system should provide real-time data on API request metrics, such as latency, throughput, and error rates.
2. **Data Ingestion**: Ability to receive and process millions of monitoring events per second from various services within Twitter/X.
3. **Data Storage**: Store detailed logs and metrics for at least 30 days.
4. **Alerting Mechanism**: Trigger alerts based on defined thresholds (e.g., high latency, increased error rates).
5. **Dashboard**: Provide a user interface to visualize metrics and trends over time.

### Non-Functional Requirements
1. **Scalability**: Must handle spikes in traffic, especially during major events or outages.
2. **Availability**: 99.9% uptime to ensure continuous monitoring.
3. **Latency**: End-to-end processing latency should not exceed 1 second.
4. **Consistency**: Event data should be eventually consistent but must guarantee that all events are processed.
5. **Security**: Ensure data is transmitted securely and access to monitoring data is restricted.

Assuming Twitter/X processes approximately 1 billion API requests per day, and we want to monitor 100 metrics per request:
- **Total Requests per Second (RPS)**: 1 billion / 86400 seconds = ~11,574 RPS.
- **Total Monitoring Events**: 11,574 RPS * 100 metrics = ~1.157 million monitoring events per second.
- **Data Storage**: If each monitoring event is approximately 256 bytes, total storage per day would be 1.157 million * 256 bytes * 86400 seconds = ~25.2 TB.
- **Storage Retention**: For a 30-day retention policy, total storage required would be 25.2 TB * 30 = ~756 TB.

### Component Diagram
- **API Gateway**: Handles incoming requests and applies rate limiting using **AWS API Gateway** or **Kong**.
- **Message Queue**: Use **Apache Kafka** for buffering incoming monitoring events to handle spikes in traffic and decouple the ingestion from processing.
- **Processing Layer**: A cluster of **Apache Flink** or **Apache Spark Streaming** nodes for real-time processing and aggregation of metrics.
- **Data Storage**: Use **Amazon S3** for raw logs and **Amazon DynamoDB** or **InfluxDB** for time-series metrics storage.
- **Dashboard**: A front-end built with **React** fetching data from a backend service that queries the metrics database.
- **Alerting System**: Implement **Prometheus** for monitoring and alerting based on metrics.

### Schema Design
1. **MonitoringEvent** Table
   - `event_id` (UUID, Primary Key)
   - `timestamp` (DATETIME)
   - `service_name` (STRING)
   - `metric_name` (STRING)
   - `value` (FLOAT)
   - `status` (STRING, e.g., 'success', 'error')

2. **Alerts** Table
   - `alert_id` (UUID, Primary Key)
   - `metric_name` (STRING)
   - `threshold` (FLOAT)
   - `alert_type` (STRING, e.g., 'latency', 'error_rate')
   - `last_triggered` (DATETIME)

### Access Patterns
- Frequent writes for monitoring events and reads for alert checks and dashboard visualizations.

1. **Consistency vs. Availability**: Opting for eventual consistency allows for higher availability. Given the nature of monitoring, some data can be processed out of order.
2. **Latency vs. Throughput**: Kafka provides high throughput but introduces some latency. Ingestion may have slight delays but ensures that spikes are handled gracefully.
3. **Cost vs. Performance**: Using S3 for long-term storage is cost-effective but may have higher retrieval times compared to using a dedicated database like DynamoDB. For real-time needs, a hybrid approach can be taken.
4. **Complexity vs. Features**: Introducing a complex processing layer like Flink may provide more features but also increases operational complexity. This needs to be balanced with the team's expertise and operational overhead.

Build a high-throughput Monitoring Pipeline

Twitter/X

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Twitter/X Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search