Loading...
The purpose of this system is to gather and visualize health of computer systems in a data center.
[Mid-level deep dive topic.
It is important to discuss and agree on what kind of metrics we are trying to gather, at an early stage of the interview. In a way, this is a potential deep dive topic. But, you do not want to spend more than a couple of minutes at this stage. Try to balance depth and speed here.]
There are different types of data we are interested:
(1) Structured data: Information from computing nodes (e.g. servers) about their health. Liveliness, CPU, memory, disk, etc.
(2) Unstructured data: e.g. application logs. These are free formatted text.
(3) Semi-structured data: e.g. web server request logs. They are stored as a text, but has a structure, e.g., timestamp, error code, response time.
In this solution, we will focus on (1) because that's what the prompt calls for.
(2) and (3) would require a different design, e.g., a database that is more suitable for unstructured data (e.g., Cassandra), stream processing to extract structure out of unstructured data, text search capability (e.g. Elastic Search), etc.
- 10K servers, each server reports every second
- Per day, 10000 * 60 * 60 * 24 ~ 1B requests
- Each req - 256B (0.25KB)
- Per day, 256 * 1 billion = 256GB / day
- Per year, 256 * 365 = 93TB
- Server sends report_data(node_id, timestamp, data) every second
data is JSON format of perf data and service response time data
- Client sends get_data(node_id, start_time, end_time, type)
-> returns JSON output
- configure_alerts(user_id, metrics_to_alert, threshold, email_addresses)
Data Model:
The data each node reports would include:
We observe:
(a) Timestamp plays a critical role. We expect queries would be time-based, e.g., draw a graph of CPU usage between 10AM and 11AM.
(b) The data model is well structured.
(c) There would be many small writes.
These criteria lead us to think Time Series DB would be a good choice to store these Events. TSDBs are optimized for small writes of well structured data. It supports time based range query.
To improve write and read throughput, we should partition data. Tenant ID would be a good partition key because users are typically only interested in (and allowed to) access data from servers in their own tenant.
It is helpful to split read path from write path in this system.
Write Path receives many small writes. It needs to do two tasks: (1) storing the events in the database, and (2) processing the events into the format Read Path needs (e.g. aggregated events), and stores them in cache for fast access. We will introduce Message Queue as a way to trigger these two tasks.
Read Path reads the processed data from cache, or the raw data from Events DB, for visualization and alerting.
Nodes report metrics via report_data() API to Events Service.
Events Service writes the events to Message Queue.
Events in Message Queue by read by two different services:
[Mid-level deep dive topic]
This system's read path, i.e., visualization and alerting, require real-time processing of data. For example, it is more likely an alert is set for a 5 minute average of CPU or memory capacity, rather than an individual event (which is reported every second and is highly variable).
The processing needs to be real-time because both monitoring and alerting need to react to the most recent event: in seconds or in minutes.
Streaming Service, built with a streaming framework like Apache Flink, would be a good fit. It supports several windowing algorithms (e.g. hopping, tumbling, sliding) to efficiently aggregating raw events.
[Junior-level deep dive topic]
If we drastically scale up the system, which components would become the bottleneck?
Events Service, Read Service, and Alert Service are all stateless. We can run multiple nodes of these services. We can use weighted round-robin to distribute work between the nodes.
Message Queue receives the raw events. We can use horizontally scalable message broker such as Kafka. We can partition the queue by node_id. We can expect the traffic pattern to be mostly stable (all nodes report the same amount of data every second). Kafka natively supports scalability by having multiple nodes, while each node serving multiple partitions.
We should also use Time Sequence DB's native scalability support for Events DB.
Let's look at some of the common scenarios components of this system can fail, and how we can mitigate the challenges.
[Senior-level deep dive topic.]
If Events DB is lost, for example by a disk crash or corruption, or natural disaster, that would do irreversible harm, as we would lose important past data. As such, we need to replicate data. As the first line of defense, we should use Time Series DB's native replication mechanisms, such as leader-follower replication. Furthermore, data should be backed up in a less expensive storage (e.g. Amazon S3) in the same data center, as well as in another data center (for disaster recovery).
DB Worker and Streaming Service may fail, or encounter a performance problem and cannot catch up to the stream of incoming events. Message Queue with a permanent storage (e.g. Kafka) absorbs the difference of speed between Producer (nodes) and Consumer (DB Worker and Streaming Service). Message Queue itself can use Kafka's native fault tolerance mechanisms, such as leader-follower replication and automatic leader election.
Nodes are likely to fail to upload logs. Crash, slowness, or network partition can cause this. In this case, Events Service wouldn't receive events from these nodes. Even if something terrible is happening on the node, the system (and the users) wouldn’t know.
We have to have a special category of metrics that shows which nodes are reporting events in a healthy manner. If Events Service does not receive events for more than N minutes, we should flag them as unhealthy. Effectively, this is a heartbeat mechanism. This metrics itself can be shown visually in a graph. Alerts can be set on this metrics. E.g., notify the admin if M nodes fail to upload events for more than N minutes.
As we discussed in the Requirements, the big next step is to support unstructured and semi-structured data.