System requirements

Functional:

• Fast querying and filtering.
• Fast storing and retrieving logs.
• Log aggregation and visualization.
• Access-control.
• Real-time alerts and notification.

Non-Functional:

• Performance.
• Reliability and fault tolerance.
• Security.
• Scalability - the system should be able to scale vertically and horizontally to handle burst of the log from some produce..
• Availability - system should support hight availability feature.
• Eventual consistency - latency should be small, to allows user relays on logs fro trouble shooting.

Capacity estimation

Estimate the scale of the system you are going to design...

API design

GET /logs {

query, // optional full-text search query

logLevel, // optional log level filter

serviceId, // optional service ID to filter logs

start, // required start time

end, // required end time

page, // optional page number (default: 1)

limit // optional page size (default: 100)

}

GET /services/{serviceId}/metrics {

metricType, // required (e.g., error_count, request_rate)

start, // required start time

end, // required end time

interval // required interval size (e.g., 1m, 5m, 1h)

}

Database design

Cassandra is perfect data base for string logs, because was designed for horizontal salability and handling a lot of write operations. We can partitionate data by service_id

Data model in Cassandra

{

service_id

log_level

timestamp

log_message

log_id

}

High-level design

Log Ingestion

• Log sources send data to the Message Queue (Kafka).
• Kafka acts as a buffer and ensures high-throughput, decoupled ingestion.

Processing Layer

Kafka feeds logs into multiple services:

• Log Collection Service:
  • Pulls raw log data from Kafka.
  • Updates the Database (Cassandra) with the collected logs for durable storage.
• Flink (Data Aggregation):
  • Pulls logs from Kafka for real-time aggregation (e.g., computing metrics, summaries).
  • Also updates Cassandra, likely with enriched or transformed data.
• Search Service:
  • Performs batch pulls from Kafka (probably for indexing).
  • Updates Elasticsearch for fast search/query operations.

Storage Layer

• Cassandra:
  • Stores the full log data (raw and/or aggregated).
  • Durable, scalable, good for time-series data.
• Elasticsearch:
  • Stores indexed logs for fast full-text search.
  • Also updates Cassandra and can archive to S3 for long-term storage.

User Access Layer

• Client accesses the system via the API Gateway.
  • API Gateway routes requests to:
    • Search Service for querying Elasticsearch.
    • Log View Service for retrieving logs from Elasticsearch or Cassandra.

Archival Layer

• Elasticsearch → S3:
• Periodically offloads data to Amazon S3 for cost-effective, long-term storage and backup.

Request flows

Logs → Kafka.

Kafka → various consumers:

• Log Collection → Cassandra
• Flink (aggregation) → Cassandra
• Search Service → Elasticsearch

Search Service ↔ Elasticsearch ↔ Cassandra

Elasticsearch → S3

Clients access logs/search via API Gateway + Log View Service.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

1. Kafka (Message Queue)

Failure Scenarios:

• Broker crash or partition leader failure.
• Retention limit exceeded before consumers read messages.
• High write/read throughput causing lag or OOM errors.

Bottlenecks:

• Under-provisioned Kafka cluster (too few partitions/brokers).
• Slow consumers causing topic backlog.

Mitigations:

• Replication and monitoring (e.g., via Prometheus/Grafana).
• Increase partition count for parallelism.

2. Log Collection Service

Failure Scenarios:

• Crashes or hangs while pulling from Kafka
• Failure to write to Cassandra

Bottlenecks:

• Single-threaded or unoptimized processing
• Dependency on Cassandra write performance

Mitigations:

• Use retries with exponential backoff
• Horizontal scaling with stateless workers
• Queue overflow alerts

3. Flink (Data Aggregation)

Failure Scenarios:

• Flink job failure or out-of-memory errors
• Incorrect checkpointing configuration causing data loss

Bottlenecks:

• High-latency operations or complex aggregation logic
• Stateful processing without adequate memory

Mitigations:

• Enable checkpointing with durable backends
• Optimize operators and parallelism
• Auto-scaling and performance profiling

4. Cassandra (Database)

Failure Scenarios:

• Node failures or ring imbalance
• Disk space exhaustion
• Read/write latency spikes under load

Bottlenecks:

• High write volume without proper compaction tuning
• Large partition sizes

Mitigations:

• Use proper data modeling (e.g., time bucketing)
• Tune GC, compaction, and caching settings
• Monitor with nodetool and alert on queue lengths

5. Elasticsearch

Failure Scenarios:

• Node crashes or cluster instability under heavy indexing
• Shard allocation failures

Bottlenecks:

• Write-heavy loads can slow down queries
• Expensive queries (wildcards, aggregations)

Mitigations:

• Use index lifecycle management (ILM)
• Optimize mappings and use time-based indices
• Implement query rate limiting and circuit breakers

6. Search Service

Failure Scenarios:

• Latency due to heavy reads/writes
• Dependency failures (e.g., Elasticsearch down)

Bottlenecks:

• CPU-intensive queries or lack of caching

Mitigations:

• Add request throttling and query caching
• Handle fallback logic (e.g., to Cassandra if ES fails)

7. Log View Service

Failure Scenarios:

• High client load → API timeouts or crashes
• Dependency on Elasticsearch for fast responses

Bottlenecks:

• Poor pagination or inefficient queries

Mitigations:

• Paginate results and use backend caching
• Front-end rate limits

8. S3 (Archival Storage)

Failure Scenarios:

• Intermittent write failures due to rate limiting or permissions

Bottlenecks:

• Large file writes or infrequent batch jobs that overflow memory

Mitigations:

• Use multipart uploads and retry logic
• Schedule S3 writes during low-load periods

9. API Gateway

Failure Scenarios:

• Misconfigured routing or throttling rules
• TLS handshake overload under load

Bottlenecks:

• Rate limits reached, blocking legitimate users

Mitigations:

• Auto-scale and enforce quotas
• Circuit breakers and load balancing

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?