Design a event-driven Data Pipeline System

Question

Design a event-driven data pipeline system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Event Ingestion**: The system must support millions of event requests per second, allowing producers to push events into the pipeline.
2. **Real-time Processing**: Events should be processed in near real-time to ensure low latency for downstream consumers.
3. **Data Transformation**: The ability to transform data as it flows through the pipeline, including filtering, aggregation, and enrichment.
4. **Storage**: Processed events must be stored in a scalable manner, allowing for querying and analytics.
5. **Multi-region Support**: The system should support multi-region deployments to ensure low-latency access and disaster recovery.

### Non-Functional Requirements
1. **Scalability**: The system should scale horizontally to handle increasing loads.
2. **Fault Tolerance**: The system should ensure high availability and automatically recover from failures.
3. **Consistency**: Acceptable levels of eventual consistency must be defined, especially in multi-region setups.
4. **Performance**: The system must maintain low latency (sub-second) for event processing.
5. **Monitoring and Logging**: Robust monitoring and logging must be in place to track system health and performance.

Assuming we handle **10 million events per second**:
- **Data Size per Event**: Average size of an event is 1KB.
- **Total Data Ingestion**: 10 million events/second * 1KB/event = **10 GB/second**.

### Daily Data Volume
- **Daily Volume**: 10 GB/second * 86400 seconds/day = **864 TB/day**.

### Storage Requirements
If we retain data for 30 days:
- **Total Storage**: 864 TB/day * 30 days = **25.92 PB**.

### Processing Capability
To process events in real-time, let's assume:
- Each processing node can handle 50,000 events/second.
- **Nodes Required**: 10 million events/second / 50,000 events/node = **200 nodes**.

### Scaling Considerations
- Each region may require a similar setup to ensure low latency globally.

### Component Diagram
1. **Event Producers**: Various sources (IoT devices, web applications) that generate events.
2. **Load Balancer**: Distributes incoming event requests across the event ingestion service.
3. **Event Ingestion Service**: A microservice (e.g., built with Node.js) that receives and queues events using a message broker (e.g., Apache Kafka).
4. **Processing Layer**: Stream processing framework (e.g., Apache Flink) that consumes events for transformation and enrichment.
5. **Storage Layer**: Use MongoDB Atlas for storing processed events; sharded clusters ensure scalability.
6. **Monitoring & Logging**: Tools like Prometheus and Grafana for metrics; ELK stack for logging.

### Technology Choices
- **Message Broker**: Apache Kafka for high-throughput event streaming.
- **Processing Framework**: Apache Flink for real-time data processing.
- **Database**: MongoDB for flexible schema and horizontal scaling.

### Schema Design
1. **Events Collection**: Store event data with fields such as:
   - `event_id`: String (unique identifier)
   - `timestamp`: Date (event occurrence)
   - `data`: Object (dynamic key-value pairs representing event details)
   - `metadata`: Object (optional metadata for processing context)

2. **Processing States Collection**: Track processing state for each event:
   - `event_id`: String (reference to Events Collection)
   - `status`: String (e.g., 'processed', 'failed')
   - `retry_count`: Integer (number of retries)
   - `error_details`: String (if applicable)

### Access Patterns
- Frequent queries based on `timestamp` and `status` to monitor event processing.

### Architectural Decisions
1. **Eventual Consistency vs. Strong Consistency**: Opt for eventual consistency for high availability and performance, especially in multi-region setups. This allows faster processing but may lead to temporary inconsistencies in data.
2. **CAP Theorem**: Prioritize availability and partition tolerance over consistency. Use conflict resolution strategies (e.g., last-write-wins) to handle discrepancies.
3. **Choice of Storage**: Using MongoDB allows flexibility in data structure but introduces challenges in consistency management across regions. Sharding strategies must be carefully planned to optimize read/write operations.

### Operational Excellence
- **Deployment Strategy**: Implement a blue-green deployment strategy for smooth updates.
- **Chaos Engineering**: Introduce chaos experiments to validate system resilience under failure scenarios.
- **SLOs/SLIs**: Define clear service level objectives for availability (99.99%) and latency (< 1 second for event processing).

Design a event-driven Data Pipeline System

MongoDB

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

MongoDB Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search