System requirements

Functional:

1. The system supports collecting logs from various sources.
2. The system should support analysis tools
3. The system should support monitoring and alerting

Non-Functional:

1. The system supports real-time processing and alerting.
2. Handle large volume of data, DAU 100M
3. The storage should be scalable
4. The latency to retrive the analytics results should be fast < 200ms
5. Availability >> consistency
6. Data should be accurate, data should not be collected twice.

Capacity estimation

1. 100M DAU, 100M * 100 /80000 = 125 K QPS for log ingestions
2. 100M * 100 * 365 * 100B = 400PB storage of logs yearly if we retend the data for one year.

API design

• POST /api/log -> 200

{

key,

value

}

• GET /api/log/logKey=?&startTime=?&endTime=?

-> Log[]

• GET /api/log/query=? -> Log[]

Database design

1. Log storage

There are couple of choices.

• • S3 - cheap and scalable, but hard to make analytics query
  • NoSQL - scalable and have good throughput, but limited to query index for analytics
  • TimeSeries DB - good for time range query, can be scalable by partitioning by log_key
  • RDBMS with materialized view - good for fast analytics query, can be scalable by partition, but will require some data duplication
  • ElasticSearch - good for text based indexing and query

Mornitor and alerting storage:

• We can store in RDMBS and partition.

High-level design

Firstly, we should have some log collecting server, which can be partitioned by log_key.

We have two options, push model vs pull model.

In application server that will send logs, we can have a daemon or side-car container to push logs to log server.

We can also let log collecting server to pull the log in X amount of time.

Secondly, we can persists the log injection request to a message queue like kafka.

Then, we will have consumer group subscribed to different topics which is the log_key.

Then the workers will inject the data into different data sinks like NoSQL and TimeSeries DB as well as Elastic Search.

We will have a Mornitoring Service which notified by CDC(change data capture) in NoSQL db and also check the correspoding alerting rules in RDMS. If condition is met, we can notify an Alerting service to send alert to the subscribers.

Request flows

App server -> log collecting server -> kafka -> nosql data sink

-> mornitoring service -> alerting service

Detailed component design

1. In kafka, if a log_key is popular, we will need to handle the hot shard. We can partition further and split the traffic
2. In mornitoring service, it can be notified by a cron job in a minute level, but our alert will experince some delay, so we choose to be notified by CDC to get real-time alert and mornitoring.
3. We can also have a dashboard which consumes data from the time series db to get fast updates on some time series data.
4. For the noSql log store, we can have multiple leader structure to allow good read throughput. For the read side, we can have read only replica to allow fast reads.
5. For kafka, we need to enable log retention to ensure data persistence. It can be used when we want to recover the data.

Trade offs/Tech choices

1. Data duplication vs Fast analytics
   1. we need to prioritize fast data analytics, so we decided to write data into multiple data sinks with duplication.
2. Push vs Pull model in log ingestion
   1. push can ensure that all the data will be collected.
   2. If we use pull, log collecting server don't know what instances to send request to and need to maintain a mapping. If instances goes down and new instances are added in, we need to update the mapping.
3. Using kafka vs Let server handle the log ingestion request directly
   1. kafka is high available and provides log replication
   2. server may be down and request will lost if we don't persist them
4. Different data sinks vs single data sink
   1. different data sinkes provides high perforance in reading the data but introduces more system complexity and data replication
   2. single data sink will save resources but the performace of reading the data will downgrade and we need to do computation on the fly

Failure scenarios/bottlenecks

1. If log collecting server failed. The request will failed and client can retry. We can have server clusters to load balance the requests.
2. If log storage failed, we can leveagre the replica and elect the new leader.
3. If kafka failed, we can restore the data by leveraging the logs.
4. If log processing worker failed, we can assign new workers and adjust the mapping of consumer groups.
5. If mornitoring server failed, we can levearge server replica.
6. If alerting service failed, the alert request will failed, and we will retry. We can also replicate the alert server to ensure high availability.

Future improvements

1. Based on our analytics query requirements, we can build cache layer to speed up the query time.
2. We can scale our system by partition by log_key. We can reduce the network latency further by using schema like ProtoBuf. If we define the log schema ahead, we can save some bytes in transferring the log data and also include the field number and data without giving the field name. This can help us speed up the log ingestion performance.