Design a large-scale Monitoring Platform

Question

Design a low-latency monitoring system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - Real-time monitoring of system metrics (CPU, memory, disk usage, request latency, etc.)
  - Support alerts and notifications for anomalous behavior (threshold-based, anomaly detection)
  - Dashboard for visualizing metrics over time with customizable views
  - API for clients to submit custom metrics and retrieve monitoring data

- **Non-Functional Requirements**:
  - Low latency (<100ms) for metric ingestion and retrieval
  - High availability (99.9% uptime) with minimal downtime for maintenance
  - Scalability to handle millions of requests per second
  - Multi-region deployment for resilience and reduced latency in global access
  - Data retention policies for compliance and storage efficiency (e.g., 1 year of data for metrics)

- **Assumptions**:
  - Assume Brex has 10 million monthly active users.
  - Each user generates an average of 5 metrics per minute.
  - Each metric is a data point with a timestamp, user ID, and metric value.

- **Calculations**:
  - Total metrics per minute = 10M users * 5 metrics/user = 50M metrics/minute.
  - Total metrics per second = 50M / 60 ≈ 833,333 metrics/second.
  - To handle peak loads, assume a 3x buffer, targeting 2.5M metrics/second.
  - Data retention for 1 year with an average of 60 bytes per metric yields:
    - Total storage = 2.5M metrics/second * 60 seconds/minute * 60 minutes/hour * 24 hours/day * 365 days/year * 60 bytes ≈ 315TB/year.

- **Component Diagram**:
  - **Data Ingestion Layer**:
    - Use **Apache Kafka** for high-throughput message ingestion.
    - Producers to send metrics to Kafka topics.
  - **Processing Layer**:
    - **Apache Flink** for real-time processing and anomaly detection.
    - Data stored temporarily in **Redis** for quick access during processing.
  - **Storage Layer**:
    - Use **Cassandra** for time-series data storage, ensuring scalability and high write availability.
    - **Elasticsearch** for indexing metrics for fast search and retrieval.
  - **API Layer**:
    - Node.js application serving RESTful APIs for custom metrics submission and data retrieval.
  - **Dashboard**:
    - React-based frontend for visualizing metrics and alerts.

- **Deployment**:
  - Deploy components in multiple AWS regions for low latency and high availability.

- **Schema Design**:
  - **Metrics Table (Cassandra)**:
    - `user_id (UUID)`: Primary key to partition data.
    - `metric_name (TEXT)`: Name of the metric (e.g., CPU, memory).
    - `timestamp (TIMESTAMP)`: Time of metric collection (clustering column).
    - `value (FLOAT)`: Value of the metric.
    - **Access Patterns**:
      - Query for a user’s metrics over time.
      - Query for the latest metric value for alerts.
  - **Alerts Table (Cassandra)**:
    - `alert_id (UUID)`: Primary key.
    - `user_id (UUID)`: Foreign key related to metrics.
    - `threshold (FLOAT)`: Alert threshold.
    - `status (TEXT)`: Current status of the alert (active, resolved).

- **Consistency vs. Availability**:
  - Using **Cassandra** allows for tunable consistency, favoring availability and partition tolerance. This is crucial for a monitoring system where losing some metric data is acceptable if it ensures continuous operation.

- **Eventual Consistency**:
  - With an eventual consistency model, metrics might not be immediately visible across regions. However, this is acceptable in non-critical monitoring contexts where real-time updates are not required for all metrics.

- **Operational Excellence**:
  - Implement chaos engineering practices to test system resilience (e.g., introducing failures in Kafka or Cassandra). 
  - SLOs for the monitoring system should target <100ms latency for API responses and <1% error rate for metric ingestion.

- **Phased Rollout Plan**:
  - Start with an MVP focusing on core metrics ingestion and basic alerting, then gradually add features like anomaly detection and advanced visualization based on user feedback.

Design a large-scale Monitoring Platform

Brex

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Brex Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search