Loading...
Capacity estimation
We could either have a push model or pull model. API for push model:
// Server send metrics
reportData(serverId, timestamp, data)
// Read data
getData(query) // query is a custom query language
// Configure alerts.
addAlert(predicate) // predicate is a custom language based on query
Server sends metrics data to EventService, which gets queued on EventQueue for processing.
PersistenceService takes the data from EventQueue, and persist in DB.
AggregationServices is a real-time processing service (e.g. Flink or Spark Streaming) that computes predefined aggregation.
On the read path, the QueryService can query the DB, accelerated by a in-memory cache in front.
The alert service periodically runs queries (or look at predefined queries) and puts alerts into NotificationQueue, which gets picked up by NotificationService and processed.
Database
We choose a time series DB for its efficient write and query of time series data. The database can be sharded by server id or time ranges. Either way, having sharded database complicates query that spans across shards.
The query language:
If this is an internal facing service, then we can expose a custom language (SQL like language is okay) that hides TSDB specific query language, since we might want the same language to work across TSDB and maybe a no-sql DB in the future.
Most services are stateless and can be horizontally scaled. The queues can be partitioned by server ID.