Design A Log Collection And Analysis System - System Design

Requirements

Functional Requirements:

Users should be able to monitor the logs across any of their deployments.
Users should be able to filter for specific logs based upon timestamps, log types, service types, servers, etc.
Users should be able to get automated notifications from the logs when a service is say down etc.
Users should be able to run traces to capture any failed system or to monitor latencies of services.

Non-Functional Requirements:

Should be able to scale to up to a billion logs per hour - also to millions of users.
Prioritize availability over consistency - It is okay if the logs are a split second late to update on the platform but the system should be available always to capture the logs.
Since the logging might capture potentially sensitive PII data, the logs should be encrypted/hashed and secured both in rest and at transit.
Logs should reside in hot storage for a day, warm storage for 30 days and beyond that in cold storage to be served for any analytics.

Capacity Estimation

Estimate the scale of the system. Consider daily active users, read/write ratio, storage requirements, bandwidth, and any relevant QPS calculations...

API Design

GET - /getLogs?cursorId=1
1. returns {logs}
GET - /getLogs?startTimeStamp=""&endTimeStamp=""
1. returns {logs}
GET - /getService?serviceName=""
1. returns {service}
POST - /createAlertRule
1. body: {alertId, alertSeverity, alertCondition, notificationEmail}
2. returns alertStatus -> boolean
GET - /getTrace?api=""
1. returns: {callingApi, apiResponseCode, responseTimestamp}
POST - /writeLogs
1. body: {serviceName, logType, logText, logTimeStamp, deploymentName}
2. returns: status -> boolean

High-Level Design

Our services will reside behind an API Gateway which can. Takes care of the authorization and any rate limiting the API gateway. The API Gateway interacts as the uh. A step between the client and. All our services, the get trace service will ping our database. I'll separate the read path from the read path. The get trace, get logs, get service. All these three services are read heavy and they can read directly from a postgres database to retrieve any traces, any logs with any filter criteria like timestamps or any livelocks. Et cetera. And the get service would would be one of the APIs that fetches. All the services, all the deployments that are currently present. Umm, along with on the fields that we aligned on in our database design. Coming to the right path, the right log service takes the logs from the individual services and. Using a Kafka event stream. Uh, which enables us to capture the. Uh, capture any late coming events, et cetera. We can write it into a write optimized database like Cassandra so that we don't. Overload the read path since the system is likely to be something. Were you write a log once but you read it multiple times so the write can. The read has to be completely separated from the right path, but at the same time the right also has to be fast, which is why choosing a database like Cassandra makes sense. Because it's a white column database, it almost provides a document structure Nosql style database and where even deletes or updates to data or records. Happened as happened. Appended logs basically from Kasan from the Kafka stream. We'll also use a Flink job to. Data into any analytics databases from Cassandra based upon the Heartworm and. And cold storage. UH rules that we have identified earlier. Data gets archived into an S3 storage in parquet format and there is a change data capture from Cassandra to the Postgres database. Umm, which will have to run periodically so that all the reads get are taken from the Postgres database. Any precomputed views like across services, the number of errors or number of warnings? Any, sorry, any statistics like number of errors, number of warnings, et cetera can be kept as precomputed views from the Postgres database, which can be served as statistics to the user. Also. The PostgreSQL database, if it is indexed correctly based upon the service or the service ID or the timestamp, would enable us to. Provide faster search instead of being able to search over billions and billions of rows.

Database Design

service - serviceId, serviceName, serviceName, serviceType,
logs - logId, serviceId, serviceName, logText, logTimestamp, logType
deployment - deploymentId,serviceId, environmentType, deploymentName
alert_rule - alertId, alertSeverity, alertCondition, notificationEmail
session - sessionId, callingServiceId, callingServiceTimeStamp, responseCode, responseTimeStamp

The logs in cold storage would reside in S3 storage in parquet format for size optimization.

Detailed Component Design

The Postgres database needs to be indexed correctly based upon the search. Based upon the search. That needs to be performed if the search is going to be something structured like based upon timestamps based upon the log types like warnings or the health. Or maybe. Errors then post uh, creating indexes based upon. Serve creating composite indexes based upon the service. The error and the error type or the warning type? Warning or actual log info basically would help speed up the search especially. From, instead of having to search over 2 billion rows, it could be brought down to millions or so rows, or even lower. That is 1 optimization that can be done on the read path.

And usage of Kafka stream even stream decouples our writes from the actual. Uh, logging portion of it, meaning that. Once we write our events into the Kafka event stream, we have durability lasting up to even a week and. The logging system. The service that is calling the logging. Umm, logger basically need not wait until the write is successful, and need not wait until it gets an acknowledgement from. In our database write database stating that the write has been successful. That can be it's own process from the Kafka event stream. And to avoid duplicate logging entries each logger event will have a unique ID which is transmitted all the way downstream.