Build a high-throughput Data Pipeline Pipeline

Question

Design a high-throughput data pipeline system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - The system must ingest and process millions of data requests per second, focusing on high throughput.
  - It should provide near real-time data processing with low latency for downstream analytics.
  - Must support various data formats (JSON, CSV, Avro) for incoming requests.
  - Data should be retrievable for batch processing and real-time queries by various consumers.

- **Non-Functional Requirements**:
  - **Scalability**: The system should scale horizontally to handle increased data loads without degradation in performance.
  - **Availability**: The system should be designed for high availability with a target uptime of 99.99%.
  - **Consistency**: Strong consistency is preferred for critical data, while eventual consistency can be acceptable for less critical data.
  - **Monitoring**: Implement robust monitoring and alerting for system health and performance metrics.

- **Estimating QPS**:
  - Assume 10 million requests per day, which breaks down to approximately 115 QPS (10 million / (24 * 60 * 60)).
  - Given peak load patterns, we should design for 5x this estimate, targeting about 600 QPS.

- **Storage Requirements**:
  - If each request generates an average of 1KB of data and we retain data for 30 days, this results in:
    - Daily data generation: 10 million KB (10 GB)
    - Monthly storage requirement: 300 GB (10 GB/day * 30 days).
  - Plan for 3x growth, leading to a total storage requirement of approximately 900 GB for 30 days of data retention.

- **Bandwidth**:
  - Assume each request requires 10 KB for processing and response, leading to a bandwidth requirement of: 600 QPS * 10 KB = 6 MB/s.

The architecture consists of the following components:

1. **Load Balancer** (e.g., NGINX):
   - Distributes incoming requests across multiple API servers to handle high throughput.

2. **API Gateway** (e.g., Kong):
   - Manages request routing and provides authentication, rate limiting, and monitoring.

3. **Data Ingestion Layer**:
   - **Message Queue** (e.g., Apache Kafka):
     - Decouples the data producers from consumers, enabling asynchronous processing and high throughput.

4. **Processing Layer**:
   - **Stream Processing** (e.g., Apache Flink or Apache Spark):
     - Processes data in real-time as it arrives via Kafka, allowing for transformations and aggregations.

5. **Database** (e.g., Oracle Autonomous Database):
   - Stores processed data with strong consistency guarantees for real-time queries.
   - Replication for read scalability and high availability.

6. **Cache** (e.g., Redis):
   - Caches frequently accessed data to reduce latency for read operations.

7. **Monitoring and Alerting** (e.g., Prometheus + Grafana):
   - To track system health and performance metrics.

The schema design should cater to the access patterns expected:

- **Data Ingestion Table**:
  - `DataRequests`:
    - `request_id` (UUID, Primary Key)
    - `user_id` (UUID)
    - `request_data` (JSON)
    - `timestamp` (TIMESTAMP)

- **Processed Data Table**:
  - `ProcessedData`:
    - `data_id` (UUID, Primary Key)
    - `request_id` (UUID, Foreign Key)
    - `processed_data` (JSON)
    - `processing_timestamp` (TIMESTAMP)

- **Indexing**:
  - Create indexes on `user_id` and `timestamp` for efficient querying.

- **Partitioning Strategy**:
  - Use time-based partitioning for the `ProcessedData` table to manage large datasets effectively.

- **Consistency vs. Availability**:
  - Opting for eventual consistency for non-critical data (e.g., logs) allows for higher availability and throughput.
  - For critical transactions (e.g., user actions), strong consistency is enforced, which may introduce latency.

- **Caching vs. Database Load**:
  - Using Redis for caching reduces database load but requires cache invalidation strategies to maintain freshness.

- **Message Queue Choice**:
  - Choosing Kafka for its high throughput and durability guarantees, but requires managing consumer offsets and potential complexities in stream processing.

- **Scalability**:
  - Horizontal scaling is essential for API servers and processing nodes, but this introduces complexities in managing state and session consistency.

- **Monitoring**:
  - A robust monitoring solution is necessary to quickly identify and resolve issues, but adds operational overhead.

Build a high-throughput Data Pipeline Pipeline

Oracle

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Oracle Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search