Design a geo-distributed Data Pipeline System

Question

Design a geo-distributed data pipeline system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

- **Functional Requirements**:
  - The system must ingest data from multiple geo-distributed sources in real-time.
  - It should support transformations and aggregations of incoming data.
  - The system must provide APIs for querying processed data with low latency.
  - It should allow for data enrichment from external services.
  - The pipeline should ensure data delivery guarantees (e.g., at-least-once delivery).

- **Non-Functional Requirements**:
  - The system should be highly available with 99.9% uptime.
  - Latency for data processing and querying should be under 100 ms for 95% of requests.
  - It must handle up to 10 million requests per day across all geo-regions.
  - Data consistency should be managed effectively to ensure accuracy across distributed regions.
  - The system must be secure, with data encryption in transit and at rest.

- **Estimation**:
  - Assuming 10 million requests per day, this translates to roughly 115 requests per second (RPS).
  - If each request generates 100 KB of data, the total data volume would be approximately 1 TB per day.
  - Considering a retention period of 30 days, we would need around 30 TB of storage capacity.
  - Factoring in growth, let's estimate a 20% increase annually, leading to around 36 TB needed by next year.

The proposed architecture consists of the following components:

- **Data Ingestion Layer**: Utilizes Apache Kafka for handling real-time data streams from multiple sources, providing high throughput and fault tolerance.
- **Processing Layer**: Apache Flink will be used for stream processing, allowing for real-time transformations and aggregations.
- **Storage Layer**: A combination of:
  - **Cassandra** for distributed and scalable storage of processed data with eventual consistency.
  - **Amazon S3** for raw data storage and batch processing jobs.
- **Query Layer**: Use of **Elasticsearch** for fast querying capabilities on processed data.
- **Load Balancer**: AWS Application Load Balancer to distribute incoming requests among multiple instances.
- **Monitoring**: Prometheus and Grafana for monitoring system health and alerting.

The schema design will include:

- **Source Data Schema**: 
  ```
  SourceData {
      id: UUID,
      source_type: String,
      payload: JSON,
      timestamp: Date,
      region: String
  }
  ```
  - This captures the raw data with necessary metadata for processing.

- **Processed Data Schema**: 
  ```
  ProcessedData {
      id: UUID,
      aggregated_value: Double,
      source_id: UUID,
      processed_at: Date,
      region: String
  }
  ```
  - Designed for efficient querying and analytics, with a focus on key aggregations.

- **Consistency vs. Availability**: Choosing Cassandra allows for high availability and partition tolerance but with eventual consistency. This trade-off works well for scenarios where slight delays in data accuracy are acceptable.
- **Latency vs. Data Freshness**: Real-time processing with Apache Flink ensures low latency but may introduce complexity in handling out-of-order events. Implementing a buffer can help manage this trade-off.
- **Storage Costs**: While S3 offers cost-effective storage for raw data, accessing this data could be slower compared to using a dedicated database like Cassandra, necessitating a caching strategy for frequently accessed data.
- **Security**: Implementing robust security measures such as encryption and access controls may add overhead, but they are crucial for compliance in handling sensitive data.

Design a geo-distributed Data Pipeline System

Workday

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Workday Machine Learning Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search