0%
System requirements
Functional
Capacity estimation
Traffic
API design
Event Ingestion (Write Path)
Database design
ClickHouse: Event Storage
High-level design
Request flows
Write Flow: Event Ingestion
Detailed component design
Event Ingestion Pipeline
Tradeoffs
ClickHouse vs Time-Series Database (InfluxDB, TimescaleDB)
Failure scenarios
Kafka Partition Failure
Future improvements
Real-Time Streaming Analytics
System requirements
Functional
Capacity estimation
Traffic
API design
Event Ingestion (Write Path)
Database design
ClickHouse: Event Storage
High-level design
Request flows
Write Flow: Event Ingestion
Detailed component design
Event Ingestion Pipeline
Tradeoffs
ClickHouse vs Time-Series Database (InfluxDB, TimescaleDB)
Failure scenarios
Kafka Partition Failure
Future improvements
Real-Time Streaming Analytics
A web analytics tool like Google Analytics has a deceptively simple user-facing feature set: count page views, track where visitors come from, measure conversions. The engineering challenge is in the sheer volume of writes. Every page view, every click, every scroll event on every tracked website generates a write to your system. This is not a read-heavy social feed or a balanced e-commerce checkout. It is a write-dominated pipeline, and that single fact shapes every design decision that follows.
Event tracking: Capture page views, clicks, scroll depth, and custom events (such as "add to cart") on client websites via a lightweight JavaScript snippet embedded in their pages. This snippet is the primary data collection mechanism. It must load fast, run silently, and never interfere with the host website.
The capacity numbers for a web analytics system are dominated by one insight: the write volume is enormous, but the read volume is modest. This asymmetry drives every infrastructure decision. Designing for the wrong workload profile wastes money and creates bottlenecks where none need to exist.
Tracked websites: 100K active sites, ranging from small blogs generating 1-5 events per minute to high-traffic e-commerce sites producing 100K+ events per minute. Averaged across all sites, the rate is roughly 70 events per minute per site.
The API has two fundamentally different workloads: high-volume event ingestion (fire-and-forget from tracking scripts) and low-volume dashboard queries (interactive, latency-sensitive). Recognizing this split early is essential because the two paths have completely different scaling requirements.
1POST /api/collect
2Content-Type: application/json
3
4{
5"site_id": "s_abc123",
6"events": [
7{
8"event_id": "evt_uuid",
9"type": "pageview",
10"url": "/products/shoes",
11"referrer": "https://google.com/search",
12"timestamp": 1710000000,
13"session_id": "sess_xyz",
14"metadata": {
15"viewport": "1920x1080",
16"load_time_ms": 1200
17}
18}
19]
20}
21Response: 204 No ContentThe system uses polyglot persistence: different storage technologies for different access patterns. This is not added complexity for its own sake. Each store is chosen because it excels at a specific workload that the others handle poorly.
At 115K writes per second and 3TB per day, the database must be built for high-volume analytical writes. ClickHouse is a columnar database that batches inserts into sorted, compressed column files. Analytical queries (such as "count page views by country for the last 30 days") scan only the columns they need, not entire rows.
The architecture follows a CQRS (Command Query Responsibility Segregation) pattern, splitting the system into a write path and a read path. This is not a theoretical pattern applied for elegance. It directly addresses the 100:1 write-to-read ratio. The write path (event ingestion) has completely different scaling needs, latency tolerances, and failure modes than the read path (dashboard queries).
Mid-level: Describe the write and read paths separately and name the key components (Kafka, ClickHouse, Redis).
Senior: Explain why CQRS is the right pattern for the 100:1 write-to-read ratio, and how Kafka decouples ingestion throughput from processing speed.
Staff: Design the backpressure propagation chain (ClickHouse slowdown causes worker pause causes Kafka buffering) and explain how this prevents data loss under cascading partial failures.
Two request flows define the system: the write flow (recording events) and the read flow (serving dashboards). Understanding the exact sequence of steps reveals why each architectural decision was made.
The core engineering challenge of this problem (the GATE) is building an event ingestion pipeline that sustains 115K to 500K events per second without losing data. Every component in this section exists to solve some aspect of that challenge: how to absorb the write volume, how to deduplicate at speed, how to partition for parallelism, and how to recover from failures without gaps in analytics data.
The GATE insight: the entire system hinges on the event ingestion pipeline. Tracking scriptlets embedded in client websites capture page loads, clicks, and conversion events, streaming them to regional ingestion services that write to columnar storage. The pipeline must handle 115K sustained writes per second with 500K peaks, which means every stage (validation, buffering, deduplication, enrichment, storage) must be designed for throughput first and latency second.
Every architectural decision in this system involves a trade-off. Understanding these trade-offs and being able to articulate them in an interview demonstrates that you made deliberate choices rather than blindly adopting patterns.
Time-series databases seem like a natural fit since analytics events are timestamped data points. But the queries are not simple time-series lookups. "Show me page views grouped by country, filtered by traffic source, for the last 30 days" requires GROUP BY across multiple dimensions and filtering on string columns. ClickHouse's SQL support, columnar storage, and vectorized execution engine handle these analytical queries 10-100x faster than time-series databases. The trade-off is that ClickHouse requires more operational expertise and does not auto-downsample old data the way InfluxDB does.
Designing for the happy path is the easy part. What makes a system production-ready is how it handles failures. Each failure scenario maps to a specific architectural decision made earlier.
If a Kafka broker hosting a partition goes down, the partition becomes unavailable until the controller reassigns leadership to a replica. During this window (typically 5-30 seconds with proper configuration), events for sites mapped to that partition fail to enqueue. Mitigation: replication factor 3 ensures two replicas survive. The Beacon API retries Kafka produce for 5 seconds before returning a 503. The tracking script will retry on the next flush cycle. With proper configuration, a single broker failure causes zero data loss and only a brief delay.
These improvements extend the system beyond its current capabilities without changing the core architecture. Each builds on the existing Kafka pipeline and ClickHouse storage.
Replace batch aggregation with Apache Flink or Kafka Streams for sub-second dashboard updates. This enables live conversion monitoring, real-time alerting on traffic anomalies, and instant A/B test results. The cost is 3-5x higher infrastructure spend and significant operational complexity. Start with streaming for a few high-value metrics (live conversion count, real-time error rate) while keeping batch aggregation for the rest.
Reading a solution is not the same as producing one under interview pressure. Design it on the whiteboard and get scored feedback.
0%
System requirements
Functional
Capacity estimation
Traffic
API design
Event Ingestion (Write Path)
Database design
ClickHouse: Event Storage
High-level design
Request flows
Write Flow: Event Ingestion
Detailed component design
Event Ingestion Pipeline
Tradeoffs
ClickHouse vs Time-Series Database (InfluxDB, TimescaleDB)
Failure scenarios
Kafka Partition Failure
Future improvements
Real-Time Streaming Analytics
System requirements
Functional
Capacity estimation
Traffic
API design
Event Ingestion (Write Path)
Database design
ClickHouse: Event Storage
High-level design
Request flows
Write Flow: Event Ingestion
Detailed component design
Event Ingestion Pipeline
Tradeoffs
ClickHouse vs Time-Series Database (InfluxDB, TimescaleDB)
Failure scenarios
Kafka Partition Failure
Future improvements
Real-Time Streaming Analytics