System requirements

Functional:

• Track and analyze web requests to identify top-K most frequent requests
• Support configurable time intervals (minutes to months)
• Allow dynamic adjustment of K value
• Provide historical analysis capabilities
• Support filtering by request attributes (URL paths, status codes, etc.)
• Generate real-time and historical visualizations

Non-Functional:

Scalability: Handle millions of requests per minute

Performance: Sub-second query response time for recent data

Availability: 99.9%+ uptime with fault tolerance

Flexibility: Support various time window configurations

Consistency: Provide accurate counts with minimal error rates

Cost-efficiency: Optimize storage and compute resources

Capacity estimation

Traffic volume: 10M requests/minute (167K requests/second)

Data size: ~200 bytes per request record

Storage requirements:

• 2GB per minute
• 2.88TB per day
• ~86.4TB per month (without compression)

API design

Define what APIs are expected from the system...

GET /api/v1/topk

Parameters:

- k: integer (default: 10)

- start_time: timestamp

- end_time: timestamp

- granularity: [minute|hour|day]

- filters: JSON object of attribute filters

Database design

Two-tier storage approach:

1. Hot Storage (Redis):
   • Sliding window counters using sorted sets
   • Structure: request_counters:<window> → {request_hash: count}
   • TTL-based expiration for time windows
2. Warm Storage (Time-Series DB):
   • Schema:

1. Cold Storage (Object Store):

• Archived data in columnar format (Parquet)
• Organized by time partitions

High-level design

[Web Servers] → [Load Balancers] → [Data Collectors]

↓

[Kafka] ← [Stream Processors] → [Redis] ← [Query Service] ← [API Gateway] ← [Clients]

↓ ↓

[HDFS/S3] ← [Batch Processors] → [TimescaleDB]

Request flows

Data Ingestion Flow:

1. Web servers log requests to local buffers
2. Collectors forward log entries to Kafka
3. Stream processors consume from Kafka:
   • Update Redis counters for real-time view
   • Aggregate and write to time-series DB
4. Batch processors archive data to cold storage

Query Flow:

1. Client submits top-K query with parameters
2. For recent data: Query Redis directly
3. For historical data:
   • Query time-series DB for aggregated counts
   • For very old data, query cold storage
4. Apply filters and return formatted results

Detailed component design

Stream Processing Engine:

• Implementation: Apache Flink
• Key algorithms:
  • Sliding window counting with efficient state management
  • Count-Min Sketch for approximate counting
  • HyperLogLog for cardinality estimation

In-Memory Counter Manager:

• Redis sorted sets for each time window
• Periodic pruning of expired counters
• Sharding by request hash for scalability

Trade offs/Tech choices

Counting Accuracy vs. Performance:

• Exact counting: Maintains precise counts but requires more memory
• Probabilistic counting: Space-efficient but introduces small error rate (~2%)
• Recommendation: Use probabilistic structures (Count-Min Sketch) for high-cardinality datasets

Storage Tiers:

• In-memory → Time-series DB → Object storage
• Trade-off: cost vs. query performance
• Data sampling strategies for extremely large datasets

Failure scenarios/bottlenecks

Potential Failures:

1. Kafka partition failures: Mitigated by replication factor ≥ 3
2. Redis node failures: Use Redis Sentinel or Redis Cluster
3. Processing node crashes: Stateful processing with checkpointing

Bottlenecks:

1. Ingestion bottlenecks: Horizontal scaling with partitioning
2. Memory pressure in Redis: Implement intelligent eviction policies
3. Query latency for large time ranges: Implement tiered aggregations

Future improvements

Machine learning integration:

• Anomaly detection for unusual request patterns
• Predictive analytics for traffic forecasting

Enhanced visualization capabilities:

• Interactive dashboards with drill-down capabilities
• Correlation analysis with business metrics

Advanced analytics:

• Request sequence analysis
• User session tracking
• Geographic distribution visualization