System requirements

Functional:

• Fast search in logs.
• Fast storing and retrieving logs.
• Visualizations of logs (graph, diagrams).
• Provide users with statistic and different metrics(for instance error counts).

Non-Functional:

• Scalability - the system should be able to scale vertically and horizontally to handle burst of the log from some produce..
• Availability - system should support hight availability feature.
• Eventual consistency - latency should be small, to allows user relays on logs fro trouble shooting.

Capacity estimation

Estimate the scale of the system you are going to design...

API design

GET /logs {

query, // optional full-text search query

logLevel, // optional log level filter

serviceId, // optional service ID to filter logs

start, // required start time

end, // required end time

page, // optional page number (default: 1)

limit // optional page size (default: 100)

}

GET /services/{serviceId}/metrics {

metricType, // required (e.g., error_count, request_rate)

start, // required start time

end, // required end time

interval // required interval size (e.g., 1m, 5m, 1h)

}

Database design

Cassandra is perfect data base for string logs, because was designed for horizontal salability and handling a lot of write operations. We can partitionate data by service_id

Data model in Cassandra

{

service_id

log_level

timestamp

log_message

log_id

}

High-level design

Log producer - pushes logs.

Message queue - handles increasing numbers of logs requests. Kafka is the best option for logs.

Log collections service - pulls logs from Kafka , process them and store in DB.

Statistic service - pulls logs from Kafka and collects different metrics and store them in DB.

Cassandra DB - nonSql DB that stores logs and metrics.

Log search service - pulls logs from Kafka and store and handle search requests from user.

Log view service - fetches user requested data from DB service that visualize logs/statistics/notifications.

Request flows

Explain how the request flows from end to end in your high level design. Also you could draw a sequence diagram using the diagramming tool to enhance your explanation...

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?