System requirements

Functional:

• Store logs
• Search logs
• Read logs
• Notification system for alerts
• Analysis pipeline

Non-Functional:

• Scalable
• Available
• User Privacy
• Eventually consistent model
• Low latency

Capacity estimation

Lets say we are a log collection platform for SaaS businesses.

• We have 1000 paying customers
• Each customer logs around 1M log events per day on average.
• Each log event is 1KB.

Our system will write 1B events per day.

Storage = 1B * 1KB ... = 1TB per day of storage.

Storage for a 2 year period = 1TB * 2 * 365 = 730TB --> 1PB adding an extra buffer for growth.

API design

StoreLogs():

Input: id, timestamp, type

Output: 201 created

SearchLogs():

Input: id, range

Output: List of Events logs

ReadLogs():

Input: type

Output: List of Events logs

Database design

User: id, websiteUrl

LogEvent: id, type, timestamp, ownedBy

Quick Access Pattern -->

Given a User, read all log events

Given the heavy write workload that our system will have to bear, I think I will go with a NoSQL database, more precisely Cassandra, given that all its nodes are masters and able to write.

For reading event log data, I will leverage the capabilities of a search engine, which will enable the scalability, high availability and fault tolerance requirements of the system.

High-level design

Request flows

Store logs: client -> LB -> API -> Data collection -> NoSQL DB

Search logs: Publisher reads -> NoSQL DB -> publish event to message broker -> data is processed -> feed into the search engine

Read logs: Publisher reads -> NoSQL DB -> publish event to message broker -> data is processed -> feed into the search engine

Detailed component design

The data collection pipeline:

1. Client application writes logs into our system through the API
2. The event log data is collected through a service that validates and cleans up any unnecessary information
3. Then the data is stored in the database

The data processing service:

1. A publisher asynchronously reads data from NoSQL and publishes it to a message broker for a decoupled process
2. The data processing service processes the information in whatever ways necessary and feeds it into a Search Engine. Also, raises an Alert Event to send notifications in a certain criteria is met

The Alert notification service:

1. This service is triggered by an Alert Event published by the data processing pipeline.
2. Builds the notifications (email, sms, slack, etc)
3. Sends the notification

Trade offs/Tech choices

1. Cassandra is great for write heavy workloads and brings flexibility to the data structures that might be saved in the data store now and in future iterations, however lacks advanced querying capabilities so we had to implement a data processing pipeline separately adding necessary complexity to the system.

Failure scenarios/bottlenecks

Failure: Load Balancer

Failure: API Gateway

Failure: Network connection to NoSQL database, potential data loss

Future improvements

• Horizontally scale LBs

• Horizontally scale API Gateways

• Database replication or retry mechanism of data inserts