Requirements

Functional Requirements:

1. The user would be able to monitor CPU utilization, memory usage, disk IO and throughput.
2. The user should be able to set notifications or alerts within the system wherein if the metrics exceed or fall outside any limits or expected numbers that the user sets, they get notified automatically.
3. The user should also be able to create visualizations of the metrics like histograms for latency distributions.
4. Any new service or apps would automatically be onboarded onto the metrics monitoring system.

Non-Functional Requirements:

1. For the nonfunctional requirements. As you mentioned. We expect roughly 1 sample every 10 seconds, so. And we have roughly 100 metrics and across 10,000 servers that roughly amounts to 100,000 data points per second. So our system should be able to scale up to right at least 100,000 data points per second.
2. I am going to prioritize. For different parts of the system, different. Uh, I the priority would change for consistency and availability in the sense that for writing our metrics into the system, our system should always be available. But it's all right. Actually, I'll, I'll correct myself. I I'm going to prioritize availability over consistency in the sense that it's OK if we if the data that we monitor in dashboards. Are the metrics that get aggregated in dashboards if they are? Eventually consistent, but I should be able to monitor them at least to the point of the preceding 5 seconds of matrix or preceding 10 seconds of metrics.
3. The metric should be. There should be a clear storage hierarchy for all the metrics that we collect. This for example, any immediate, any metrics computed or collected over the preceding. Day or two days should reside in or the preceding day should reside in hot storage. Up to a week back should reside in warm storage, and anything beyond that can reside in cold storage. We can potentially use something like. S3 storage object for these purposes.

Capacity Estimation

We are expecting 100,000 metrics per second, 100,000 data points per second. Let me assume that roughly each data point is a byte of data. So that would make it 100,000 bytes per second, that is. 100,000 bytes per second would roughly be 100 kilobytes per second. Hundred. We have 3600 seconds in a minute. That makes it roughly 3 point. 3600 * 100 kilobytes. That is like 36 GB per hour Scaling up to over a day. at roughly 1 TB. So we would have to use sharding/partitioning or systems that scale horizontally.

API Design

Let's use post post style for our API design.

1. The first API API method would be a GET method to monitor our CPU to monitor all the dashboard metrics like CPU utilization, memory usage, disk IO. And throughput. The. The method would take a particular deployment or a service as an input along as a part of the API itself as a query parameter and return. CPU utilization. Which would be in gigabytes or megabytes, etc. Memory usage, again in gigabytes or megabytes, the CPU percentage. And throughput. Return object would be a JSON style return object for our first. For the get method.
2. The second request would be the for the actual ingestion of this data into our databases, so thus all the services would pass along with the service name. CPU utilization, memory usage, disk IO and throughput every 10 seconds along with the time timestamp. Into our database and this will get written into our database. Would also contain the service name obviously.
3. service would be a. Post request again this would create. Any new rules or notifications? It would be essentially. Post/create rule. Capturing the alert ID, What the rule ID is? Which metric this rule is set for? And any potential e-mail addresses to which the notification should be sent to.

High-Level Design

Let's assume that there are multiple services. There are, For example, there are Deployment 1, Deployment 2, Deployment 3, Each has it's own service name, et cetera. Our ingestion service keeps monitoring each of these deployments and using a GET method that we discussed in the API design. It it fetches. Umm, the CPU metrics, memory utilization, disk network utilization, etcetera. If there is any GPU, any GPU utilization and ingest that data into our write heavy write optimized database, which would be primarily something like Cassandra which we had discussed. Our database design. In this dotted line we intend to capture using some sort of a change data CDC mechanism. We intend to capture any most recent changes into. Write database into a something like a read database, something like Dynamo or DB or something that again as previously explained, the reason to choose something like Dynamo DB is it provides us with a flexibility of using multiple indexes. So if we were going to query across. Timestamps across CPU utilizations or across services. We could set up timestamp to be our primary index and use a global secondary index like service names or CPU utilization or any other metric basically. So from the Dynamo DB we using our gets using a get service or we can populate our dashboards with all these metrics. Now the third requirement, third functional requirement which we had discussed was. About alerting users potentially when there are, when the when the metrics like exceed some thresholds or something like that. Again, using in within the dashboard, we can set up an alert mechanism, we will, we can. Using the Create Alert service and this Create Alert service would update the alert rules along with many notification emails or phone numbers that we intend to notify into our Cassandra database. The these are our primary 3. Requirements fulfilled within. We'll have to next create a separate job which monitors. Are metrics in the Dynamodb and whenever it identifies that and. After monitoring the metrics, whenever it identifies that somehow any of the metric has failed a particular rule set up for an alert, we will trigger an alert notification to the appropriate. E-mail or something.

Database Design

1. deployments - deploymentId, deploymentName, serviceName
2. CPU metrics - metricId, deploymentName, cpuUtilization, memory%, disk, network, timestamp, temperature
3. network metrics - metricId, requestLatency, errorstates, timestamp
4. database metrics - metricId, queryType (read/write), query, timestamp
5. alert - alertId, rule, notificationType, notificationEmail, notificationPhone, timestamp

The database - especially tables like CPU metrics would become huge in a short time span because of having 100K data points per second, we will have to use a partition key by date. If we were to partition by serviceName or deploymentName, and if there are services that have high traffic vs services with not so high traffic there could still be hotspots. Since we are okay with eventual consistency, we could choose something like cassandra which is optimal for high writes to be chosen. Alternately if our search is going to be across multiple different potential primary keys, then we could use something like Dynamo DB which provides features like GSI (global secondary index) - to optimize for reads.

Detailed Component Design

Since we expect more than 100,000 data points per second, around 100,000 data points per second umm, having that messaging service in between between the ingestion service and the right dat right database is optimal. We can partition the messaging service by. Uh, we can partition the messaging service by service type or by deployment name or deployment ID or even by timestamp so that. The my uh, the mess messages stick to one particular partition, essentially and. The ingestion service could for for every data point or metric that we extract, we could retain item potency. We we could add an item potentially key so that we do not have duplicate metrics for the same timestamp. I know it might not be a huge effort to deduplicate our database at a later point, but we do not want that. So having an idempotent item potency key within. Ingestion service that gets translated through the Kafka messaging service to ensure that there are no duplicates in our rights in our metrics would be helpful. And as I may, as we discussed earlier, Kafka innately provides storage for at least by default for at least seven days. So immediate warm storage, immediate storage necessities are taken care of. Within the Kafka messaging service.

The alert notification uh uh, the alert evaluation engine will use up in this design at least. The alert notification job will use a polling mechanism wherein it keeps polling our Dynamo read database. A very, uh, very frequently. Say a minute or umm, based upon the latency needed, even lower. To ensure that to see if there are any metrics for any deployments that are out of. Umm, best to ensure that the deployments metrics fall within the rules that are set up now. There is I see a potential issue in here. Let's say there are 200 or $300 every time. Umm, not 200 or 300. When we are running a huge cluster or something like that, we may not have just 10 or 20 alerts. We could have 2003 thousand, maybe thousands of alerts then. If this job parses through each alert one at a time, identifies any metric based upon that alert that's gone wrong. Umm and then comes back with a feedback we could that that could be a potential delay in sending out the alert. So this mechanism has to be optimized.

The evaluation job which could potentially be horizontally scaled reads from Dynamo db. Though it does not need transactional latency it still needs to be run frequently at streaming speed. So flink instead of spark could be used for the evaluation job and alert notifications. This takes care of grouping, deduping and send an actionable notification. Alternatively, we capture a state that processes the data in fixed time windows - for example anything in the preceding one hour alone is evaluated and before that the assumption is evaluation is already done. In which case spark could do a reasonable job.