Loading...
Estimate the scale of the system you are going to design...
GET /logs {
query, // optional full-text search query
logLevel, // optional log level filter
serviceId, // optional service ID to filter logs
start, // required start time
end, // required end time
page, // optional page number (default: 1)
limit // optional page size (default: 100)
}
GET /services/{serviceId}/metrics {
metricType, // required (e.g., error_count, request_rate)
start, // required start time
end, // required end time
interval // required interval size (e.g., 1m, 5m, 1h)
}
Cassandra is perfect data base for string logs, because was designed for horizontal salability and handling a lot of write operations. We can partitionate data by service_id
Data model in Cassandra
{
service_id
log_level
timestamp
log_message
log_id
}
Log source - produces of logs.
Message queue - handles increasing numbers of logs requests. Kafka is the best option for logs.
Log collections service - pulls logs from Kafka , process them and store in DB.
Aggregation Service - pulls logs from Kafka and collects different metrics and store them in DB.
Cassandra DB - nonSql DB that stores short-term logs and metrics.
Search service - pulls logs from Kafka and store and handle search requests from user.
Log view service - fetches user requested data from DB service that visualize logs/statistics/notifications.
Elasticsearch - uses for full text search and storing long tern logs in S3.
Write path:
Log sources - produces logs in writes in kafka. Then Log collection service pull logs, proceed them and store on DB. Data aggregation service calculates metrics and statistic. Search service periodically pulls data and puts in Elasticsearch.
Read path:
In case of Query search data will be retrieve from Cassandra DB
In case of FullText seach - from Elasticsearch.
Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...
Explain any trade offs you have made and why you made certain tech choices...
Try to discuss as many failure scenarios/bottlenecks as possible.
What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?