Loading...
Main endpoints:
GET /logs
Arguments: { timeframe, resourceName (optional), listOfTags (optional) }
POST /logs
Arguments: { resourceName, [ { timestamp, severity, message }, ... ] }
The design is split between data collecting and data visualizing/analyzing
A queue is used to process logs asynchronously so we can handle peaks
Metrics should be collected at a regular interval therefore they don't necessarily need a queue mechanism
We use a timeseries oriented database to store logs and metrics
The main bottleneck is the logs processing because we need to process large volumes of data that can vary greatly during time.
A queue system is used to absorb spikes of logs data traffic. We can have multiple instances of the queue component.
The most simple solution would be to distribute data across the different queue instances using a round-robin algorithm. However this can lead to situation where a log session is not consistent (for instance if 2 logs of the same session are processed by two different consumers but one of them is stuck).
To overcome this problem we instead use a key-based distribution mechanism. The key is a combinaison of data from the logs: part of timestamp + resourceid.
The queue is configured to act like a priority queue so important logs like errors are processed first.
In case a log can't be processed repeatedly, it is inserted in a dead letter queue for later analysis
Logs processing workers are depoyed using a K8S cluster so they can scale dynamically based on the workload.
This queue design is able to handle large volume of data but it will lead to some delay due to the asynchronous processing.
The time series database is also sharded using a composite key: (day, hash(resource_id) % N)
Some resources can produce much more data than other.
Data integrity and availability is ensured by having multiple database replicas. Each shard is composed of a leader and a follower.
We don't keep data older than 1 year. A purge service regularly deletes (or archive with compression?) old data.
The load balancer is a SPOF so we deploy an active-passive configuration.