Design a Monitoring Service

Question

Design a geo-distributed monitoring system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements:
1. **Real-time Monitoring**: The system should monitor metrics such as API response times, error rates, and system resource utilization in real-time.
2. **Alerting**: Users must receive alerts for predefined thresholds (e.g., CPU usage exceeding 80%).
3. **Dashboards**: Provide a web interface for users to visualize metrics over time and drill down into specific instances or services.
4. **Data Retention**: Store historical data for at least 90 days for analysis.

### Non-Functional Requirements:
1. **Scalability**: The system must handle millions of requests efficiently by scaling horizontally.
2. **Availability**: Ensure a 99.99% uptime, with geo-distributed deployment to minimize downtime.
3. **Performance**: Ensure metrics can be queried with a latency of less than 100ms.
4. **Consistency**: Eventual consistency is acceptable for monitoring data due to the nature of metrics.
5. **Security**: Implement authentication and authorization for accessing monitoring data.

Assuming Robinhood has approximately 10 million users, with each user generating an average of 1 request per minute:

- **Total Requests**: 10 million requests/minute = 600 million requests/hour.
- **Data points**: If each request generates 10 data points (e.g., CPU, memory, disk I/O, etc.), we’ll have:
  - 600 million requests/hour * 10 data points = 6 billion data points/hour.
- **Storage Requirement**: Assuming each data point is 100 bytes, we will need:
  - 6 billion data points/hour * 100 bytes = 600 GB/hour.
- **Historical Data**: For 90 days retention: 600 GB/hour * 24 hours * 90 days = 1.296 PB of data storage.

This necessitates a robust storage solution capable of handling high write throughput and efficient querying.

### Components:
1. **Load Balancer**: Use AWS Elastic Load Balancer to distribute incoming requests across multiple instances.
2. **API Gateway**: Implement AWS API Gateway to manage API requests and rate limiting.
3. **Monitoring Service**: A microservice architecture using Node.js or Go to handle incoming monitoring requests.
4. **Message Queue**: Utilize Apache Kafka for buffering incoming metrics before processing.
5. **Data Storage**: Use a combination of:
   - **Time-Series Database**: InfluxDB for storing monitoring metrics.
   - **NoSQL Database**: DynamoDB for user and alert configurations.
6. **Caching Layer**: Implement Redis as an in-memory cache for frequently accessed metrics.
7. **Frontend Dashboard**: A React-based web application for users to visualize data.

### Diagram:
*Components are deployed across multiple AWS regions for geo-distribution and resilience.*

### Schema Design:
1. **Metrics Table (InfluxDB)**:
   - `measurement` (e.g., `cpu_usage`): string
   - `tags`: (e.g., `instance_id`, `region`): string
   - `fields`: (e.g., `value`): float
   - `timestamp`: timestamp

2. **Users Table (DynamoDB)**:
   - `user_id`: string (Primary Key)
   - `email`: string
   - `alert_thresholds`: JSON (stores user-defined thresholds)

3. **Alerts Table (DynamoDB)**:
   - `alert_id`: string (Primary Key)
   - `user_id`: string (Foreign Key)
   - `metric`: string
   - `threshold`: float
   - `is_active`: boolean

### Access Patterns:
- Frequent writes to the Metrics Table for real-time data ingestion.
- Read patterns for querying metrics and alerts, which will involve both time-series queries and user-specific configurations.

1. **Consistency vs. Availability**: Opting for eventual consistency allows the system to be more available under high loads, which is critical for monitoring services where immediate consistency is not as crucial.
2. **Caching**: Implementing Redis as a caching layer helps in reducing read latencies but introduces complexity in cache invalidation strategies.
3. **Rate Limiting**: Using AWS API Gateway for rate limiting simplifies implementation but may lead to throttling under peak loads. A balance needs to be struck between user experience and backend load.
4. **Geo-Distribution**: While deploying across multiple regions increases availability and redundancy, it also introduces challenges like data synchronization and network latency.
5. **Database Selection**: Choosing InfluxDB for time-series data is optimal for write-heavy workloads but requires careful handling of data retention policies to manage storage costs.

Design a Monitoring Service

Robinhood

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements:

Capacity Estimation

Submit Your Answer

Robinhood Machine Learning Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search