Design a Big Data Processing Pipeline

A big data processing pipeline ingests massive volumes of raw data from diverse sources, transforms it through progressive refinement stages, and delivers analytics-ready datasets for business intelligence. Think of what powers Spotify Wrapped or Uber's surge pricing: billions of raw events flow in, get cleaned and enriched, and emerge as actionable insights within minutes or hours.

Functional

• Data ingestion: Ingest data from diverse sources in both batch and real-time streaming modes. Batch sources include databases (JDBC), log files, and periodic API pulls. Streaming sources include Kafka topics, IoT sensors via MQTT, and clickstream events. The system handles structured, semi-structured, and unstructured data formats.
• Data transformation and enrichment: Transform, normalize, and enrich raw data through a multi-stage processing pipeline. Apply schema validation, data cleansing, type coercion, and join enrichment from reference datasets. Support both batch ETL jobs and continuous stream processing.
• Scalable multi-tier storage: Store raw and processed data at petabyte scale across tiered storage. Raw data preserved in original format for reprocessing. Processed data in optimized columnar formats (Parquet, ORC) for efficient analytical queries.
• Pipeline orchestration: Schedule, coordinate, and monitor pipeline jobs through DAG-based orchestration. Manage dependencies between pipeline stages, handle retries on failure, and alert on SLA breaches.

Let us size a pipeline that ingests clickstream, IoT, and transactional data for a mid-size e-commerce platform.

Ingestion Volume

• Streaming: 500K events/sec average from clickstream and IoT sensors. Each event averages 500 bytes. Daily streaming volume: 500K x 500B x 86,400 = roughly 21TB/day raw.
• Batch: Nightly database extracts total 2TB from transactional systems via JDBC.
• Combined: roughly 23TB/day raw ingestion.

The pipeline exposes APIs for job management, data source registration, and query access to processed data.

Pipeline Job Management

1POST /v1/jobs
2Body: { "pipeline_id": "clickstream_etl", "mode": "batch", "source": "s3://bronze/clickstream/2026-03-17/", "target": "s3://silver/clickstream/", "config": { "parallelism": 20 } }
3Response: { "job_id": "j-abc123", "status": "queued", "created_at": "..." }
4
5GET /v1/jobs/:job_id
6Response: { "job_id": "j-abc123", "status": "running", "progress": 0.65, "records_processed": 42000000, "errors": 12, "started_at": "...", "estimated_completion": "..." }
7
8POST /v1/jobs/:job_id/cancel
9Response: { "job_id": "j-abc123", "status": "cancelling" }

The pipeline uses different storage technologies for each layer, optimized for their access patterns.

Bronze Layer - Raw Ingestion Store

Object storage (S3, GCS) partitioned by source and date:

The pipeline follows a layered architecture with five core components, each independently scalable.

Ingestion Layer

Let us trace data through the pipeline end-to-end for both batch and streaming paths.

Batch Ingestion Flow

The Medallion architecture is the central design pattern of this pipeline. Understanding how data transforms through Bronze, Silver, and Gold layers is the core interview question.

Medallion Architecture: Bronze, Silver, Gold

Lambda vs Kappa Architecture

Lambda runs separate batch and stream processing paths. The batch layer (Spark) processes complete datasets for accuracy. The speed layer (Flink) processes real-time streams for low latency. A serving layer merges results from both. Benefit: batch provides "correct" results while stream provides "fast" results. Cost: maintaining two codebases with identical business logic is expensive and error-prone. When a transformation rule changes, you must update it in both Spark and Flink.

Kappa eliminates the batch layer entirely. Everything is a stream. Historical reprocessing is done by replaying the Kafka log. Benefit: single codebase, simpler architecture. Cost: replaying years of data through a stream processor is slow, and complex batch transformations (large multi-way joins, ML feature engineering) are harder to express in stream processing APIs.

Kafka Broker Failure

A Kafka broker holding partitions for the clickstream topic goes down. With replication factor 3, two other brokers hold replicas of every partition. Kafka's controller promotes an in-sync replica (ISR) to partition leader within seconds. Producers and consumers experience a brief pause (typically under 5 seconds) during leader election, then resume. No data is lost because the new leader has a complete copy.

If the replication factor were 1, the failure would cause data loss for events on the failed broker's partitions. This is why production Kafka clusters always use replication factor 3.

Real-Time ML Feature Store

Integrate a feature store (Feast, Tecton) that serves pre-computed features from both batch and real-time pipelines. Batch features (user lifetime value, 30-day purchase history) are computed in Gold and pushed to the feature store daily. Real-time features (current session activity, last-click timestamp) are computed by Flink and pushed continuously. ML models query the feature store for a unified feature vector at prediction time.

Incremental Processing