Design a Metrics Monitoring and Alerting System

A metrics monitoring system looks like a simple pipeline. Collect data, store it, show dashboards. But with 10K servers each emitting 100 metrics every 10 seconds, the system ingests 100K data points per second. The real challenge is not collection or storage. It is evaluating thousands of alert rules against this firehose in near-real-time without drowning on-call engineers in false positives.

Functional

Collect infrastructure metrics: Agents on servers collect CPU usage, memory consumption, disk I/O, and network throughput. Services expose application-level metrics (request latency, error rates, queue depths) via HTTP endpoints. The system supports both push-based agents and pull-based endpoint scraping.

The numbers drive every architecture decision. 100K data points per second means a single TSDB node cannot handle all writes. You need sharding. 860 GB per day of raw data means you need downsampling for anything beyond 7 days.

Traffic

Start with the scale: 10K servers, each reporting 100 metrics, one sample every 10 seconds.

Four API groups cover the system. The ingestion API handles 100K points per second, so batch writes are essential, sending one HTTP request per data point would mean 100K requests per second of overhead. The query API supports aggregation functions and time-range filtering. The alert API encodes the core business logic. What condition to watch, how long it must persist, and who to notify.

Metric Ingestion

1POST /api/v1/metrics
2Content-Type: application/json
3
4{
5"batch": [
6{
7"metric": "cpu_usage",
8"tags": {"host": "web-01", "region": "us-east"},
9"value": 78.5,
10"timestamp": 1709654400
11}
12]
13}
14
15Response: 202 Accepted

Time-series databases exist because general-purpose databases fail at this workload. At 100K writes per second of monotonically timestamped data, a B-tree index fragments instantly. Every insert goes to the rightmost leaf, causing constant page splits and write amplification. TSDBs use append-only compressed blocks, achieving 10-15x compression and 300K writes per second on a single node.

Data Model

Every metric data point has four components:

The architecture splits into a write path and a read path that diverge at Kafka. Writes flow from collectors through Kafka to the TSDB. A unidirectional pipeline optimized for throughput. Reads fan out from dashboards and alert evaluators to the TSDB through a cache layer. Optimized for latency. This separation means you can scale ingestion and query capacity independently.

Write Path

Two core flows define the system: getting metrics in and getting alerts out. The ingestion flow is optimized for throughput. Batch, buffer, write. The alert triggering flow is optimized for correctness. Evaluate, deduplicate, notify.

Metric Ingestion Flow

The alert rule evaluation engine is the GATE. The core challenge that separates a toy monitoring system from a production-grade one. With 5,000 alert rules and 100K data points per second, the naive approach of checking every rule against every data point cannot keep up. The solution is partitioning rules across evaluator instances by metric name, where each evaluator maintains windowed state for duration-based thresholds.

Alert Rule Evaluation Engine

Three design decisions define the system's character: how metrics are collected, when data is aggregated, and which TSDB engine runs underneath. Each has a clear trade-off that interviewers expect you to articulate.

Pull vs Push Collection

Pull model (Prometheus-style scraping). Collectors periodically scrape HTTP endpoints on each server. The collector controls the schedule, and a failed scrape is itself a health signal (the target is down). The limitation: short-lived containers that start and stop between scrape intervals are never observed. Services behind network firewalls that cannot accept incoming connections are unreachable.

A monitoring system that fails during an outage is worse than no monitoring at all. It creates false confidence that everything is fine while problems compound undetected. Each failure mode requires a specific mitigation.

TSDB Unavailable

Four extensions would elevate the system from reactive monitoring to proactive infrastructure intelligence.

ML-Based Anomaly Detection

Static thresholds miss gradual degradation. CPU creeping from 40% to 75% over two weeks never breaches a 90% threshold, but the trend signals impending failure. Machine learning models trained on each metric's historical patterns learn normal baselines and flag deviations, detecting the slow drift that static rules ignore.