Design a Monitoring Service

Question

Design a real-time monitoring system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Real-time Monitoring**: Capture and display metrics such as request latency, error rates, and system health in real-time.
2. **Alerting System**: Send alerts based on predefined thresholds for various metrics (e.g., high error rates).
3. **Dashboard**: Provide a user-friendly dashboard for users to visualize metrics and alerts.
4. **Data Retention**: Store historical metrics data for at least 30 days for insights and trend analysis.
5. **API Rate Limiting**: Implement rate limiting to prevent abuse of monitoring endpoints.

### Non-Functional Requirements
1. **Scalability**: The system should handle millions of requests per second (QPS).
2. **Availability**: 99.99% uptime requirement to ensure monitoring is always available.
3. **Performance**: Latency for retrieving metrics should be under 100ms.
4. **Consistency**: Eventual consistency is acceptable for metrics data but strong consistency for alert states.
5. **Security**: Ensure secure access to monitoring APIs and dashboards.

### Back-of-Envelope Calculations
1. **Traffic Estimation**: Assume Notion has 1 million active users, and each user generates an average of 10 requests per minute for monitoring data. This results in:
   - Total Requests = 1,000,000 users * 10 requests/user/min = 10,000,000 requests/min.
   - QPS = 10,000,000 / 60 = **166,667 requests/second**.

2. **Data Storage Needs**: Assuming each metric entry takes up 1 KB and we expect to store 1 million metrics per minute:
   - Data per minute = 1,000,000 metrics/min * 1 KB = 1 GB/min.
   - For 30 days, total storage = 1 GB/min * 60 min/hour * 24 hours/day * 30 days = **43.2 TB**.

3. **Bandwidth**: If each request/response pair is approximately 1 KB, the bandwidth requirement would be:
   - Bandwidth = 166,667 requests/sec * 1 KB = **161.5 MB/sec**.

### Component Diagram
- **API Gateway**: Handles incoming requests, performs rate limiting, and routes requests to the appropriate service. (e.g., AWS API Gateway)
- **Load Balancer**: Distributes traffic across multiple instances of the monitoring service. (e.g., NGINX)
- **Monitoring Service**: Core service responsible for collecting metrics and processing alerts. (e.g., Node.js with Express)
- **Data Store**: Time-series database (e.g., InfluxDB) for storing metrics, with a separate relational database (e.g., PostgreSQL) for alert configurations and states.
- **Caching Layer**: Redis for caching recently accessed metrics to improve response times.
- **Alerting Service**: Triggers alerts based on metrics processed, integrated with notification services (e.g., PagerDuty).
- **Dashboard**: Frontend application (e.g., React) for displaying metrics and alerts.

### Schema Design
1. **Metrics Table** (Time-series data):
   - `metric_id` (UUID, Primary Key)
   - `timestamp` (DATETIME)
   - `value` (FLOAT)
   - `source` (VARCHAR)
   - `type` (VARCHAR)

2. **Alerts Table** (Relational data):
   - `alert_id` (UUID, Primary Key)
   - `metric_id` (UUID, Foreign Key)
   - `threshold` (FLOAT)
   - `condition` (VARCHAR)
   - `status` (ENUM: 'active', 'resolved')
   - `created_at` (DATETIME)

### Access Patterns
- Frequent reads for fetching metrics data for dashboard visualization.
- Periodic writes for incoming metrics data from various sources.

### Architectural Decisions
1. **Consistency vs Availability**: Opted for eventual consistency for metrics data to allow for high availability and performance. Strong consistency is enforced only for alert states, which are critical for real-time responses.
2. **Caching Strategy**: Using Redis for caching metrics can lead to stale data; however, it significantly improves read performance. A TTL (time-to-live) policy can mitigate staleness.
3. **Database Choice**: Selected InfluxDB for time-series data due to its optimized performance for metrics storage and retrieval, but it may introduce complexity in data migration.
4. **Rate Limiting**: Implementing rate limiting at the API Gateway prevents abuse but may restrict legitimate users during peak times. Consideration for a dynamic rate-limiting approach could help balance this.
5. **Monitoring and Alerting**: Implementing comprehensive logging and monitoring for the monitoring service itself is crucial. Tools like Prometheus can be integrated for self-monitoring, but they add overhead.

Design a Monitoring Service

Notion

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Back-of-Envelope Calculations

Submit Your Answer

Notion Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search