Loading...
Initial 10000 RPS
one metric is 100 bytes we need to store metrics for 30 days thus 10000 metrics * 100 * 60 * 60 * 24 * 30 = 25.,9 TB of storage
If one server holds 1 TB we need 26 servers
If one server can handle 1000 requests we need 10000 / 1000 = 10 servers to handle requests
post /event/push&event_id
post /event/pull&event_id
post /subscribe/notification¬ification_type
post /notification/rules¬ification_id
get /notification/rules¬ification_id
get /event&event_id
post /ratelimit/adjust
post /connect/sink
delete /connect/sink&sink_id
Event : +int EventId PK INDEX
Event : +String Source
Event : +String Labels
Event : + DateTime timestamp INDEX
Notification : +int EventIdFK
Notification : +enum Destination (sms, phone, browser)
Notification : + DateTime timestamp INDEX
Notification : + enum type (warning, info, debug)
Notification : + enum hazard (severe, danger, light)
Notification : + enum status (sent, failed, lost)
the system consists of the following:
1) Client agent that collects events and performs simple aggregations like: collect events for 1 minute
]2) Rate limiter - imits the number of pushes of events to the system
]3) Event collector - collects events from multiple soruces and performs more complex aggregations like a sliding windwon aggregate
4) Queueue service - a service to queue db writer workers
5) Db writer worker - a worker node that writes events to db
6) Event db - a db to store events sharded by evemt source )EventShard1 - EventShardN)
]7)EventQuery Service - applies event rules to filter events and categorized them by severity, type, etc.
8) RuleDB - a db of event rules
9) MostFrequentRUlesCache - a cache of the most frequent rules
10) CleanupJob - a cron job that cleans events older than 30 days
11) Notification service - a service that schedules notification workers that send various types of event notificaiotns
]12) Client - a client that subsrecibes to notifications
13)AlertSystem - a system that displays alerts for sever events
]14) Dashboard - a dashboard that displays events
15) RertryQueue - a queue of failed notifictions htat await to be retreid
]16) Dead lettter queue - a queue of failed notifications that deemed to failed to be retreid
17)QueueeServie2 - a queeu service that schedules notificaiton workes that esnd out notifications to various seources using NotificationWorkers1-N
Push flow:
]1) An event srouce connects to the system using HTTPS protocol and uses api to connect push notiificaiotns
]2) Events come to the system and are rate limited based on settings
3) Events are buffered in event buffer queue, if queue overflows we have an overflow temp storage
]4) Events are collected and aggregated in event collecteor
5) Event collector schedules db writesr through db writer job
6) Events are written into a n event DB
]7) EventQuerySrvice periodically scans event db and applies event rules from RUlesDB
]8) If events pass the rules EventQuery ERvice passes them to notification service along with metadata
9) Based on type notification service schedules event workers through EventWorkerqueue that send out notificaiotn to subscibed clients
]10) Clients receive notifications through various channels like Email, sms or custom
11) Periodically old events are cleanud up by a CleanupJob
]
Pull flow:
]1) OUr system connects to an event source though event source 3rd aprty api
]2) Events are pulled from the event srouce by our system and are aggregated on a client side using client agent
3) Events are buffered in event buffer queue, if queue overflows we have an overflow temp storage
]4) Events are collected and aggregated in event collecteor
5) Event collector schedules db writesr through db writer job
6) Events are written into a n event DB
]7) EventQuerySrvice periodically scans event db and applies event rules from RUlesDB
]8) If events pass the rules EventQuery ERvice passes them to notification service along with metadata
9) Based on type notification service schedules event workers through EventWorkerqueue that send out notificaiotn to subscibed clients
]10) Clients receive notifications through various channels like Email, sms or custom
11) Periodically old events are cleanud up by a CleanupJob
Event collection consists of:
1) Client agent - a client program prvided by us that collects events and performs simple aggregations
2) Rate limiter - in case we decide to use push notification and the amount of event is too big said service will etiehter answer with 429 in case of overflow or use backpressure mechanisc to tell the event source to send less events
3) RateLimiterQUeue - a second line of defence agains to many events, we use a dedicated queue to buffer teh events for EventCOllector, we also use a small separte temp storage OVerflow1 which can be scaled to hold overflowed events
4) Event collector - a service that aggregates evetns and schedules db writer workesr that perisst events in long term storage (30days)
]5) CLanupJob - a cron job that periodically scans Event DB a deletes satle events (older than 30 days)
Event filtering consists of:
1) Rule db a database to hold rules for filtering events
2) Event filtering service - a service that reads Event db and applies a filtering rules producing events grouped by severity or state
Notificaiton service conssits of:
1) Notification service - a service that receives events and schedules notifcaiton workesr that send events
2) Notificaiton quqeue - a queu of notification events that acats as a buffer for notification workers
3) Notification workers - worker node that sends out different types of events (sms, mail, push, etc)
To store event rules we choose mongo db as it is a well -tested documnet oriented db and our rules are dynamic in nature. The read and write numbers are not big fir the rules so we don't need to scale it that much
Toi store events we use custom time-series based DB( InfluxDB or TimescaleDB), because the number of writes for events is really huge in this system and SQL won't fit the throughput number. To scale the storage we shard the database by event source (like from a bank, sms, etc.) and then by event_id if events will overflow
]TO track the replicas we use consistenta hashing schema and Event collector is reposnible for tracking all the replicase
]To buffer the events after the rate limiter with use rabbit MQ because in this part of the system we use quqeu as a buffer and not as a separtion of producer/consumer
To scale our notifications we user apache kafka to decouple our producers of events (notiticationa service) with out our consumers (notification workesr)
1) If event is failed to be sent in our notitifacion service we user a dedeicated retry queue. If event reaches a certain retry limit it is put into a dead letter queue for further inspection by an operator
]2) If notitifcaiotn worker is dead we can easily spin another one as they are stateless
]3) If event queue overflos after a rate limiter we can use multiple temp storages or signla 429 to the client to stop sending events
]4) If event throughput becomes too much we can send 429 to a client or rate limit the client or sernd backpressure argument to tell the client to limit events
5) if event shard becomes full we can create another one and shard by event type, the number of rehashings wont be big becuase we use consistent hashing schema
6) If db worker crasehs we can spin another one as they are stateless
1) User ML for event filtering instead of predefined rules
2) Usr ML to predict which events are going to happen in the next time window