Design a Top-K Request Analysis System

Functional

A top-K request analysis system identifies the most frequently accessed URLs or API endpoints across a high-volume web traffic stream. Think of it as a real-time leaderboard for your backend, at any moment, you can ask "what are the 10 busiest endpoints right now?" and get an answer within seconds. The system ingests every HTTP request, counts URL frequencies within time windows, and serves ranked results on demand.

• Identify top-K most frequent requests: Track and return the K most frequently requested URLs within configurable time windows (1 minute, 10 minutes, 1 hour). K is dynamically adjustable. The system maintains running counts as requests stream in, not just at window boundaries.
• Near real-time querying: Clients can query top-K results at any time and receive a response within a few seconds of the current traffic state. Results reflect the most recently closed time window plus an approximation of the in-progress window.
• Large-scale data ingestion: Handle 10 million requests per minute (~167K events/sec). The ingestion pipeline must sustain this throughput without dropping events, even during traffic spikes.
• Historical analysis: Store top-K results for past time windows so operators can query "what were the top 10 endpoints yesterday at 3 PM?" Historical data is retained for a configurable period (default 30 days for aggregated results, 24 hours for raw events).

Scale

• Request volume: 10 million requests per minute = ~167K events/sec. Each event contains URL (avg 60 bytes), timestamp (8 bytes), request metadata (source IP, user agent hash, ~50 bytes) = ~120 bytes per event.
• Unique URLs: Assume 1 million unique URLs across the system. In any given 1-minute window, typically 100K-500K distinct URLs appear.
• K value: Default K=100 (top 100 endpoints). Configurable up to K=1000.

Throughput

Top-K Query APIs

GET /v1/top-requests?k=10&interval=1min

1{
2  "interval": "1min",
3  "windowStart": "2026-03-01T14:37:00Z",
4  "windowEnd": "2026-03-01T14:38:00Z",
5  "k": 10,
6  "results": [
7    {"rank": 1, "url": "/api/feed", "count": 47823},
8    {"rank": 2, "url": "/api/search", "count": 31456},
9    {"rank": 3, "url": "/api/notifications", "count": 28901}
10  ]
11}

Schema

Storage Engine Choices

Components

Load Balancer: Distributes incoming HTTP requests across ingestion nodes using round-robin. Health checks remove failed ingestion nodes from the rotation. The load balancer itself is stateless and horizontally scalable (DNS-based or L4 hardware).

Ingestion Flow: Request to Aggregation

1. An HTTP request arrives at the load balancer. The load balancer routes it to an ingestion node via round-robin.
2. The ingestion node extracts the URL, timestamp, source IP, and user agent hash. It assigns a unique event_id (UUID) for idempotency tracking.
3. The ingestion node produces the event to Kafka topic request-events, partition determined by hash(url) mod 12. The event is batched with up to 99 others for efficient Kafka producing.
4. Kafka replicates the event to 2 follower brokers (RF=3) and acknowledges the write.
5. A Flink task instance consumes the event from its assigned partition. It updates the Count-Min Sketch: hash the URL through 5 hash functions, increment the counter at each position.
6. Flink checks if the URL's estimated frequency (minimum of 5 counters) exceeds the minimum element in the K-sized min-heap. If yes, the URL replaces the heap minimum.
7. When the time window closes (watermark advances past window end), Flink emits the min-heap contents as the finalized top-K for that window. Results are written to Cassandra (topk_results) and Redis (topk_cache).
8. Raw events are also written to Cassandra (raw_events) asynchronously for retention.

Count-Min Sketch + Min-Heap: The Core Challenge

The core algorithmic challenge is computing the top-K most frequent URLs from a stream of 167K events/sec without storing per-URL counters for all 1 million unique URLs. A hash map approach would work but consumes O(N) memory and requires a full sort at window close. The Count-Min Sketch + min-heap combination solves this in O(w×d + K) memory with O(1) amortized updates.

Approximate vs Exact Counting

Approximate counting (chosen, Count-Min Sketch): O(w×d) memory regardless of URL cardinality. Updates in O(d) time. The sketch is a fixed-size structure: 200 KB whether tracking 1,000 or 10 million unique URLs. Error is bounded and predictable. The trade-off: some URLs may be overcounted, potentially causing a URL ranked #101 to appear in the top 100 while the true #100 is pushed out.

Exact counting (rejected for streaming): A hash map of all URL counts is precise but grows linearly with unique URLs. At 1M unique URLs × ~100 bytes = 100 MB per window. Three concurrent windows: 300 MB. Still feasible for this scale, but: 1) At window close, sorting 1M entries takes O(N log N) = ~20M comparisons. 2) Memory is unpredictable. A DDoS with 50M unique URLs would consume 5 GB. 3) The hash map doesn't fit in L2 cache, so every update incurs a cache miss (~100ns vs ~3ns).

Ingestion Node Failure

An ingestion node crashes. Impact: events routed to this node are lost during the brief period before the load balancer detects the failure (health check interval, typically 5-10 seconds). Recovery: the load balancer removes the failed node from rotation and distributes its traffic to remaining nodes. If 4 nodes were handling 167K events/sec (42K/node), the remaining 3 nodes each handle 56K/sec. A 33% increase. Auto-scaling replaces the failed node within minutes. Data lost during the 5-10 second gap: ~42K × 10 = 420K events. Impact on top-K accuracy: negligible: 420K out of 10M events per minute is 4.2%.

Anomaly Detection on Top-K Patterns

The top-K data is a natural input for anomaly detection. A baseline model tracks the typical top-K composition over time (e.g., /api/feed is usually #1 with 45K-55K requests per minute). When the pattern deviates. A new URL enters the top 10 that has never appeared before, or a URL's count jumps 5x. An anomaly alert fires. Implementation: a lightweight ML model (isolation forest or Z-score) runs on each window's top-K results. The model trains on 7 days of historical top-K data. Alerts are surfaced via webhook or PagerDuty integration. This catches DDoS attacks (sudden new URLs at high volume), broken client code (one endpoint called in a loop), and viral content (organic traffic spikes).

Multi-Dimensional Top-K