Requirements

Functional Requirements

Data Ingestion: Must support diverse sources (HTTP logs, IoT sensors, CDC from databases).

Processing: Must support both real-time aggregations

Queryability: Analysts must be able to run ad-hoc SQL queries on the data.

Persistence

Non-Functional Requirements:

Scalability: Horizontal scaling to handle spikes (e.g., Black Friday).

High Availability: 99.99% uptime; no single point of failure.

Low Latency: End-to-end latency for the "speed layer" should be $< 1$ second.

Data Integrity: Exactly-once processing semantics to avoid double-counting transactions.

API Design

The Write Path (Ingestion API)

POST /v1/events

Payload: { "event_type": "string", "payload": "JSON", "timestamp": "long" }

Response: 202 Accepted (Async processing is key for high throughput).

The Read Path (Analytics API)

GET /v1/analytics/metrics?metric=total_sales&window=1h

Response: { "metric": "total_sales", "value": 500000, "unit": "USD" }

High-Level Design

The architecture follows a Lambda/Kappa hybrid pattern to ensure both speed and accuracy.

1. Ingestion: Load Balancer $\rightarrow$ API Gateway $\rightarrow$ Apache Kafka.
2. Streaming Layer: Apache Flink or Spark Streaming reads from Kafka for real-time alerts.
3. Batch Layer: Kafka connects to S3 (Data Lake) via Kafka Connect. Apache Spark runs heavy ETL on S3.
4. Serving Layer: Processed data is stored in a Druid or ClickHouse cluster for sub-second OLAP queries.

Detailed Component Design

Deep Dive 1: The Message Bus (Apache Kafka)How it scales: Kafka uses Partitioning. An "Orders" topic can be split into 100 partitions across 10 brokers. This allows 100 consumers to process data in parallel.Trade-offs: Increasing partitions improves parallelism but increases overhead on the Zookeeper/Controller for leader election.Data Integrity: Use acks=all and idempotent producers to ensure that even if a broker fails, data is neither lost nor duplicated.

Deep Dive 2: The Stream Processor (Apache Flink)State Management: Flink keeps "State" (e.g., a running sum of sales) in memory. It uses Checkpointing to save this state to S3 periodically.Algorithm: Watermarking. In big data, events often arrive out of order. Watermarks allow Flink to "wait" for late-arriving data before finalizing a window calculation.Scalability: If the processing lag increases, we increase the Parallelism of the Flink job, which redistributes the keys across more Task Managers.

Deep Dive 3: The Storage Layer (Columnar Format)Mechanism: Instead of storing data row-by-row (like PostgreSQL), we use Apache Parquet.Why it works: If a query only asks for SUM(price), a columnar store only reads the "Price" column from disk, skipping "User_ID," "Address," etc. This reduces I/O by $80\%-90\%$.Trade-off: Columnar formats are expensive for single-row lookups but unbeatable for massive aggregations.