Requirements

Functional Requirements:

• Data Collection
  • Gathers a variety of metrics (CPU use, RAM, disk) across many servers.
  • Nodes have a heartbeat to ensure metrics sampling is reliable.
• Data Storage
  • Persists metrics for a year (flexible policy).
  • Uses InfluxDB.
  • Support realtime and historical queries.
• Analytics
  • Provide insightful summary statistics on metrics.
• Visualization
  • Display recent history on dashboards.
  • Support customization.
  • Uses Grafana for dashboards.
• Alerting
  • Sends alerts for low resources.
  • Detects anomalies by comparing for irregularities in seasonality-adjusted measures, for example, average CPU usage compared to the same hour over the past 7 days.
  • Supports multiple channels: SMS, email, slack.
• APIs
  • Provides APIs for data ingestion, querying, management.
  • Secure access.

Non-Functional Requirements:

• Performance
  • Insert and query performance should be fast on the database.
• Scalability
  • System can flexibly adapt to rapid growth in size of problem.
• Reliability
  • Consistently provides reliable metrics on system performance.
• Maintainability
  • Easy to maintain over routine turnover in hardware and versions.
• Compliance
  • Provides truthful data that enables compliance monitoring and enforcement in the system.

API Design

Endpoint: /api/v1/metrics

Method: POST

Description: Writes metrics to database.

Request:

{

"source_id": "host_1234",

"timestamp": "2025-12-05T12:12:12Z",

"metrics": {

"cpu_utilization": "71.876435",

"ram_usage": "324234",

"disk_space": "234323"

}

}

Response:

{

"status": "success",

"message": "Data written to db successfully"

}

Endpoint: /api/v1/metrics/query

Method: GET

Description: Queries metrics from database.

Request:

{

"source_id": "host_1234",

"metric_name": "cpu_utilization",

"start_time": "2025-12-01T00:00:00Z",

"end_time": "2025-12-02T00:00:00Z"

}

Response:

{

"source_id": "host_1234",

"metric_name": "cpu_utilization",

"data_points": {

{"timestamp": "2025-12-01T01:02:12Z", "value": "0.02"},

{"timestamp": "2025-12-01T01:04:12Z", "value": "1.056"}

}

}

High-Level Design

• Time Series Database
  • We use InfluxDB as a time series database choice.
  • This allows efficient storage of timestamps in addition to fast queries of data for use in visualization and analytics tasks.
• Data Collection
  • We use a "push" model where nodes measure metrics in a period fashion and then push their values to the time series database via API.
  • A daemon runs on each node collecting relevant system metrics at a pre-configured frequency.
  • The process also has a heartbeat that tracks the overall health of the metric collections.
• Alerting System
  • The alerting service checks for unusual behavior in system metrics on a continuous basis.
  • Queries the time series database via API and calculates seasonally-adjusted measures of utilization across the main metrics covered.
  • Behavior that is an outlier, in terms of both magnitude and duration, fires an alert that is communicated to the system admins via email, SMS, or slack.
• Visualization
  • A Grafana dashboard set up to display the evolution of the system metrics over time.

Detailed Component Design

• Data Collection Daemon
  • The daemon starts at boot time for each node and runs continuously while the machine is operational.
  • At a predetermined frequency, metrics are collected and pushed to the time series database via the POST endpoint.
• Alerting System