System requirements

Functional:

• Fast querying and filtering.
• Fast storing and retrieving logs.
• Log aggregation and visualization.
• Access-control.
• Real-time alerts and notification.

Non-Functional:

• Performance.
• Reliability and fault tolerance.
• Security.
• Scalability - the system should be able to scale vertically and horizontally to handle burst of the log from some produce..
• Availability - system should support hight availability feature.
• Eventual consistency - latency should be small, to allows user relays on logs fro trouble shooting.

Capacity estimation

Log Ingestion Volume

• Total: 1 GB/s peak (1,000 sources × 500 logs/s × 1 KB)
• Daily: ~86 TB/day

Kafka (Message Queue)

• 20 brokers
• 100+ partitions
• 86 TB/day storage for 24-hour retention

Log Collection Service

• 20+ instances
• Each handles ~50 MB/s
• Must support retries and buffering

Flink (Data Aggregation)

• 20–30 task slots/parallelism
• Needs 8+ vCPUs, 16+ GB RAM per task manager
• RocksDB for stateful operations

Cassandra (Database)

• 30–50 nodes
• Handles ~1 million writes/sec
• Use time-bucketed partitioning

Elasticsearch

• 50+ data nodes
• Handles 500 MB/s (50% of total logs)
• Requires ~300 TB for 7-day retention
• Use rollover indices + ILM

S3 (Archival Storage)

• Up to 2.6 PB/month
• Use gzip or Parquet to compress logs
• Lifecycle policies to Glacier

API Gateway & Services

• 2–5 stateless instances
• Peak load: ~20 RPS from clients
• Backed by Elasticsearch/Cassandra

Headroom & Buffering

• Add 15–20% capacity headroom

API design

GET /logs {

query, // optional full-text search query

logLevel, // optional log level filter

serviceId, // optional service ID to filter logs

start, // required start time

end, // required end time

page, // optional page number (default: 1)

limit // optional page size (default: 100)

}

GET /services/{serviceId}/metrics {

metricType, // required (e.g., error_count, request_rate)

start, // required start time

end, // required end time

interval // required interval size (e.g., 1m, 5m, 1h)

}

Database design

Cassandra is perfect data base for string logs, because was designed for horizontal salability and handling a lot of write operations. We can partitionate data by service_id

Data model in Cassandra

{

service_id

log_level

timestamp

log_message

log_id

}

High-level design

Log Ingestion

• Log sources send data to the Message Queue (Kafka).
• Kafka acts as a buffer and ensures high-throughput, decoupled ingestion.

Processing Layer

Kafka feeds logs into multiple services:

• Log Collection Service:
  • Pulls raw log data from Kafka.
  • Updates the Database (Cassandra) with the collected logs for durable storage.
• Flink (Data Aggregation):
  • Pulls logs from Kafka for real-time aggregation (e.g., computing metrics, summaries).
  • Also updates Cassandra, likely with enriched or transformed data.
• Search Service:
  • Performs batch pulls from Kafka (probably for indexing).
  • Updates Elasticsearch for fast search/query operations.

Storage Layer

• Cassandra:
  • Stores the full log data (raw and/or aggregated).
  • Durable, scalable, good for time-series data.
• Elasticsearch:
  • Stores indexed logs for fast full-text search.
  • Also updates Cassandra and can archive to S3 for long-term storage.

User Access Layer

• Client accesses the system via the API Gateway.
  • API Gateway routes requests to:
    • Search Service for querying Elasticsearch.
    • Log View Service for retrieving logs from Elasticsearch or Cassandra.

Archival Layer

• Elasticsearch → S3:
• Periodically offloads data to Amazon S3 for cost-effective, long-term storage and backup.

Request flows

Logs → Kafka.

Kafka → various consumers:

• Log Collection → Cassandra
• Flink (aggregation) → Cassandra
• Search Service → Elasticsearch

Search Service ↔ Elasticsearch ↔ Cassandra

Elasticsearch → S3

Clients access logs/search via API Gateway + Log View Service.

Detailed component design

Dig deeper into 2-3 components and explain in detail how they work. For example, how well does each component scale? Any relevant algorithm or data structure you like to use for a component? Also you could draw a diagram using the diagramming tool to enhance your design...

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

1. Kafka (Message Queue)

Failure Scenarios:

• Broker crash or partition leader failure.
• Retention limit exceeded before consumers read messages.
• High write/read throughput causing lag or OOM errors.

Bottlenecks:

• Under-provisioned Kafka cluster (too few partitions/brokers).
• Slow consumers causing topic backlog.

Mitigations:

• Replication and monitoring (e.g., via Prometheus/Grafana).
• Increase partition count for parallelism.

2. Log Collection Service

Failure Scenarios:

• Crashes or hangs while pulling from Kafka
• Failure to write to Cassandra

Bottlenecks:

• Single-threaded or unoptimized processing
• Dependency on Cassandra write performance

Mitigations:

• Use retries with exponential backoff
• Horizontal scaling with stateless workers
• Queue overflow alerts

3. Flink (Data Aggregation)

Failure Scenarios:

• Flink job failure or out-of-memory errors
• Incorrect checkpointing configuration causing data loss

Bottlenecks:

• High-latency operations or complex aggregation logic
• Stateful processing without adequate memory

Mitigations:

• Enable checkpointing with durable backends
• Optimize operators and parallelism
• Auto-scaling and performance profiling

4. Cassandra (Database)

Failure Scenarios:

• Node failures or ring imbalance
• Disk space exhaustion
• Read/write latency spikes under load

Bottlenecks:

• High write volume without proper compaction tuning
• Large partition sizes

Mitigations:

• Use proper data modeling (e.g., time bucketing)
• Tune GC, compaction, and caching settings
• Monitor with nodetool and alert on queue lengths

5. Elasticsearch

Failure Scenarios:

• Node crashes or cluster instability under heavy indexing
• Shard allocation failures

Bottlenecks:

• Write-heavy loads can slow down queries
• Expensive queries (wildcards, aggregations)

Mitigations:

• Use index lifecycle management (ILM)
• Optimize mappings and use time-based indices
• Implement query rate limiting and circuit breakers

6. Search Service

Failure Scenarios:

• Latency due to heavy reads/writes
• Dependency failures (e.g., Elasticsearch down)

Bottlenecks:

• CPU-intensive queries or lack of caching

Mitigations:

• Add request throttling and query caching
• Handle fallback logic (e.g., to Cassandra if ES fails)

7. Log View Service

Failure Scenarios:

• High client load → API timeouts or crashes
• Dependency on Elasticsearch for fast responses

Bottlenecks:

• Poor pagination or inefficient queries

Mitigations:

• Paginate results and use backend caching
• Front-end rate limits

8. S3 (Archival Storage)

Failure Scenarios:

• Intermittent write failures due to rate limiting or permissions

Bottlenecks:

• Large file writes or infrequent batch jobs that overflow memory

Mitigations:

• Use multipart uploads and retry logic
• Schedule S3 writes during low-load periods

9. API Gateway

Failure Scenarios:

• Misconfigured routing or throttling rules
• TLS handshake overload under load

Bottlenecks:

• Rate limits reached, blocking legitimate users

Mitigations:

• Auto-scale and enforce quotas
• Circuit breakers and load balancing

Future improvements

1. Scalability Enhancements

Auto-Scaling for Core Components

• Kafka Consumers (Log Collection, Flink, Search Service): Enable dynamic scaling based on lag metrics.
• Elasticsearch Nodes: Auto-scale data nodes based on storage or query load.

Use Tiered Storage

• Offload older Kafka logs or Elasticsearch indices to S3 or Glacier to reduce hot storage cost.

Optimize Cassandra Data Model

• Time-bucketed partitioning (e.g., per day/hour) to prevent hotspotting and enable TTL-based expiry.

2. Intelligent Processing

ML-based Log Anomaly Detection

• Integrate anomaly detection (e.g., via Flink or Spark) to proactively flag unusual patterns or spikes.

Dynamic Indexing in Elasticsearch

• Use smart logic to decide which logs need to be indexed (e.g., based on severity or source).

3. Reliability and Resilience

Implement Dead Letter Queues (DLQs)

• For Kafka consumers that fail parsing or transformation—store problematic logs for review.

Multi-region Deployment

• Use multi-region Kafka, Cassandra, and Elasticsearch clusters for high availability and disaster recovery.

Checkpointed Retry Pipelines

• For Flink: Configure durable checkpointing and restore capabilities using tools like RocksDB or filesystem backends.

4. Observability Improvements

Centralized Monitoring & Alerting

• Use Prometheus + Grafana, ELK, or Datadog to monitor:
  • Kafka lag
  • Cassandra write latency
  • Flink job health
  • Elasticsearch query performance

Distributed Tracing

• Implement OpenTelemetry or Jaeger to trace request flows from the client → API → backend services.

5. API and Client Improvements

GraphQL or gRPC APIs

• Replace or supplement REST for more efficient data fetching and schema evolution.

Real-time Log Streaming to UI

• Use WebSockets or server-sent events (SSE) to push real-time logs to clients.

6. Cost Optimization

Cold Storage Offload

• Archive older logs from Elasticsearch to S3 with Athena/Glacier using ILM policies or Curator.

Lambda for Lightweight Ingestion

• Replace simple log processors with serverless functions (e.g., AWS Lambda) for lower infra cost.

7. Modularization & CI/CD

Microservices Split

• Break large services (e.g., Log View or Search Service) into microservices for better scaling and fault isolation.

CI/CD Integration

• Automate deployment using GitHub Actions, ArgoCD, or Jenkins with canary/staged releases.