System requirements

Functional:

• users can send metrics to the system
• users can set alerts based on metrics they sent
• users can view their metrics in some dashboard UI

Non-Functional:

• The availability is most critical for this system, we probably need to achieve like 99.999%
• The durability of metrics data is also important

Capacity estimation

Let's say we have 1M machines which sends 10 metrics per minute. It would be 200k QPS

If each metric is 100B in average, it would be 20MB Per Sec, it should be 630 TB per year.

API design

The API design isn't key for this system.

Database design

We will relies on time-series databsae like influxDB which is designed specifically for metrics. It has good optimization in terms of metric data storage, query optimization and query functionality support

High-level design

In the high-level, we break the whole system into 4 parts:

• the monitoring ingestion part, including the client, message queue, metric collector and timesDB
• the query parting, including the query server and in memory db, which is relied on by the alerting and the visualizing
• the alerting part and the visualizing part.

Request flows

Metric ingestion flow:

• metric client will set up a agent in locally to push to the message queue
• message queue is to buffer requests to deal with traffic spike
• metric collector pull metrics from message queue and write to times DB

Alert setup flow:

• clients configure alerts through the alert manager
• Alert manager will combine alerts and store them in the alert DBs

Alert flows:

• alrter manger periodically poll from metric query server
• query server check in-mem cache and timesDB and returns the results to alert manager
• alert managers fire alert targerts, which could be emails, text messages, phone calls and oncall apps
• In order to achieve at-least one delivery, we can add callback links in those notifications and requires the receivers to ack. The information of that is stored in alert DB so that alert managers can do retry if the receivers hasn't acked.

Visualizing

• when users open the link of some monitoring dashboard, it can call visualizer server to create the dashboard.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

The alert manager:

• combine alert queries to minimize the query costs
• it can support multiple alert targets so that we can make sure the receivers/oncalls can receive the alerts
• we needs to support callback links in those notifications/event to track if the receivers has ACKed

TimesDB:

• storage optimization:
  • time can be recorded as in delta format
  • labels can be predined

Trade offs/Tech choices

client side push or pull mode:

In fact they both have pros and cons. We choose push here for lower system complexity

Push pros: 1) lower system complexity as it only needs to know the address of collector where pull has to maintain a service discovery for alive workload to pull 2) feasible to send metrics for short-live jobs which pull cannot cons: 1) cannot do locally debugging while pull support that by pulling /endpoint 2) it cannot detect if server is down or it is network issue on the monitoriong side but pull can do that

metric batching/aggregation: there are multiple places that we can do.

• on the client side, we can batch requests by firing by size or time
• on the client side, we can aggregate the metrics
• on the timesDB, we can aggregate metrics for old metrics, like ones older than 2 weeks, we compress them.
• on the alert manager side, we aggregate the queries

metric precompute: for the alerting, we can also precomputing the metrics for alerting so it can give better performance on the alerting.

Failure scenarios/bottlenecks

The metric collector can be bottlenecks on the traffic spike so we add a message queue to solve that.

The metric collector/query server alert manager/visuallizer server are all stateless machine and they can be easily horizontally scaled up.

Future improvements

meta monitoring: we need to monitor the health of our system. In theory, we cannot monitor ourselves so we have to introduce another monitoring system to monitor ourselves and it shouldn't depend any of these components at all.