Loading...
Say we process 1 billion events per day.
QPS = 11.6k
Assume peak is 2x. QPS = 23.2k
Each event may be 500 bytes.
1 billion * 500 * 365 = 183 TB / yr
Not all events are worth storing for long time. Say we store 3 yrs, and account for some annual increase, this may be anywhere between 200 to 600 TB for up to 3 yrs.
Write an event:
POST /metrics/events, payload is event details, including service id, event id, event_type, timestamp, and details
Write/Aggregate a type of events for a service for a time window:
void aggregate(service_id, event_type, agg_method, time_window)
This will write a record to aggregated events DB.
Read or visualize an event:
GET /metrics/events/
Read or visualize an aggregation
GET /metrics/events?params
Params are filtering params, e.g., service_id, event_type, agg_method, time_window.
We don't choose RDB because the write speed may not be ideal and amount of data is huge. Plus, event schema may vary a lot.
Some options may be MongoDB or Cassandra.
If event schema do not change very frequently, we choose Cassandra due to very fast write, and fast read on recent data. We can partition by event types.
If event schema might change a lot, we choose MongoDB because of its flexible schema design.
See high level diagram
We need to keep some most recent data, say 1-3 days, in memory for very fast read.
We may keep aggregated stats in memory, at a longer horizon, say 7-30 days, or use cache to store them, for very fast read.
We use queue between server where accepts initial client events and the service which writes data to DB, so that the interface server doesn't block client while event flow is high.
For where we use memory to store recent data, we need replicas so that if the leader is dead, followers can be brought up in very short period of time. It's ok to have a minimal discrepancy should this happen.
We also need replicas for DB in case of hardware failure, etc., so that we don't lose data. Some data may be required for auditing, etc.
Client may provide timestamp, but server shall timestamp every event being received as source of truth for timing. For some events, a global strict increasing sequence number may be required to establish strict ordering within an event type.
We use different queues for different event topics to achieve very high throughput at client input side.
We can shard DBs by event type, because different event types do not overlap in terms of their data. Even if we do comprehensive analysis, we still can have aggregation service to query multiple DB instances for different event topics, and put them together at service level.
Future evolvement of event types, data scale may change our DB design and how we process events.