System requirements

Functional:

List functional requirements for the system (Ask the chat bot for hints if stuck.)...

• collect metrics from a set of servers and services
• show these metrics using a visualization mechanism
• generate alerts based on pre-determined criteria
• aggregate metrics as follows: for 1 week, metrics available at 1 min granularity. For next 30 days, roll up to 10 mins. For later, roll upto 1 hour.

Non-Functional:

List non-functional requirements for the system...

• low latency (metrics visualized and alerts generated in near real-time)
• scalable, fault tolerant
• easy to integrate and use

Capacity estimation

Estimate the scale of the system you are going to design...

1000 servers, each generate 10 metrics per second.

So total metrics data generated = 10K metrics per second

Given the high volume of data and queries required for visualization, a TimeSeries DB like Prometheus or InfluxDB must be used.

10K writes are made per second.Read load spikes every time the metrics are queried by the Monitoring Service.

API design

Define what APIs are expected from the system...

• Data sources push data to Collection System. This can be through TCP or UDP.

POST https:///metrics

{

metric type:

labels: {}

timestamp:

value:

}

• Query a given metric type or within a time window or for a region, etc. These APIs are exposed by the visualization system.

Database design

Defining the system data model early on will clarify how data will flow among different components of the system. Also you could draw an ER diagram using the diagramming tool to enhance your design...

As both read and write load are high, SQL DB is not suitable. The amount of data stored is also above the capacity handled by a traditional SQL Database.

Many proven TimeSeries databases like Prometheus, InfluxDB and Gorilla are available in the market for metrics data. They are suitable for this system.

Metrics data consists of a metric type, value, timestamp and labels (key-value pairs). This is called line definition. For example, number of requests (metric type) at 10:59PM UTC(timestamp) on server A(label) with location us-west(label) was 500 (value).

High-level design

You should identify enough components that are needed to solve the actual problem from end to end. Also remember to draw a block diagram using the diagramming tool to augment your design. If you are unfamiliar with the tool, you can simply describe your design to the chat bot and ask it to generate a starter diagram for you to modify...

Data Sources: Servers which produce the metrics data. This data is fed into the Metrics Monitoring and Alerting System

Data Collection: This layer is responsible for collecting data from the sources. It can follow a push or pull model. It can also perform aggregation of metrics data before sending them to the storage layer.

Storage: Stores the collected metrics data. Data is stored in a TimeSeries DB like Prometheus or InfluxDB as they are specialized in this use-case and support efficient querying of metrics data compared to traditional SQL.

TimeSeries DB: As explained above, a Timeseries DB is optimized for effecient storage and retrieval of metrics data.

Query Engine: Reponsible for querying the DB periodically to get metrics data for monid

Visualization: Visialization tools like Grafana are used to visualize and monitor metrics in real time.

Alerting: This system sends an alert for any anomalies. Alerting channels are email, PagerDuty and web-hooks.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Data Sources: Servers which produce the metrics data. This data is fed into the Metrics Monitoring and Alerting System

Data Collection: This layer is responsible for collecting data from the sources. It can follow a push or pull model. It can also perform aggregation of metrics data before sending them to the storage layer.

Storage: Stores the collected metrics data. Data is stored in a TimeSeries DB like Prometheus or InfluxDB as they are specialized in this use-case and support efficient querying of metrics data compared to traditional SQL.

TimeSeries DB: As explained above, a Timeseries DB is optimized for effecient storage and retrieval of metrics data.

Query Engine: Reponsible for querying the DB periodically to get metrics data for monid

Visualization: Visialization tools like Grafana are used to visualize and monitor metrics in real time.

Alerting: This system sends an alert for any anomalies. Alerting channels are email, PagerDuty and web-hooks.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

A data collection process collects all data from the data sources. Aggregation can be done at this layer. This can be done via a pull or push model. Push model is adopted in this design so that data sources(servers) do not have to open ports for the Collection engine to fetch data. Using a push model also avoids the use of Service Discovery layer to find the available data sources.

Metrics data consists of a metric type, value, timestamp and labels (key-value pairs). This is called line definition. For example, number of requests (metric type) at 10:59PM UTC(timestamp) on server A(label) with location us-west(label) was 500 (value).

If the storage layer is slow or unavailable, an intermediate data pipeline with Kafka can be added between the Collection and Storage layers. But maintaining such a pipeline requires a complex architecture and maintenance. Modern TimeSeries storage layers like Facebook Gorilla provide in-memory data storage if the disks are inaccessible.

Query performance of metrics data is not effecient on a traditional SQL database. This is because the data volume is very high and requires many indexes. Hence, off-the-shelf TimeSeries DB like Prometheus or InfluxDB is used. This database also offers caching for low latency.

Delta Encoding is a common technique used for efficient storage of metrics data. Besides this, compression techniques can also be used to compress metrics data stored in the DB. Historical data is moved to tape backup.

Query engine uses the query language provided by the DB to make refresh the metrics data periodically. This is a very thin layer and light-weight service.

Queried data is fed to the visualization system which displays it in a dashboard. Well-known dashboard visualization tools like Grafana are used instead of building a custom tool.

Depending on pre-configured alerting rules, alerts are generated and sent to respective channels as shown in the diagram.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Push vs Pull Model

• TimeSeries DB instead of custom built DB or SQL

• Data aggregation is performed at the Data Collection layer. This avoids storing extra data on the DB and avoids aggregation latency at query time.

• Timeseries DB like Facebook's Gorilla is used to avoid building an maintaining a message pipeline between Collection and Storage layers.

• Compression, delta encoding, down-sampling for efficient storage

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

• As the number of data sources increase, the collection and storage pipelines must be scaled
• Failures in the alerting must be retried or reported

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?

• Use ML techniques to predict future failures or anomalies
• Analysis tool to analyse historical failures or patterns