Design a real-time Data Pipeline System

Question

Design a real-time data pipeline system handling millions of requests. Discuss consistency, availability, and performance trade-offs.

Codemia · Accepted Answer

- **Functional Requirements**:
  - Ingest real-time data from multiple sources such as inventory systems, sales transactions, and customer interactions at Walmart stores and online.
  - Provide event streaming capabilities to process and analyze data in real-time for immediate insights and actions.
  - Support data enrichment, transformation, and filtering to enhance the data quality for downstream services.
  - Enable data storage for historical analysis while maintaining real-time processing capabilities.

- **Non-Functional Requirements**:
  - **Scalability**: The system must handle millions of events per second during peak hours (e.g., Black Friday).
  - **Latency**: Aim for low-latency processing (sub-second) to support real-time analytics and operational decisions.
  - **Availability**: Ensure high availability with at least 99.99% uptime to support critical business operations.
  - **Consistency**: Maintain eventual consistency for data processing while allowing for real-time availability.

To estimate the capacity for Walmart's real-time data pipeline:
- **Daily Transactions**: Assume 1 million transactions per day across stores and online.
- **Average Event Size**: Each transaction generates an average of 500 bytes of data.
- **Peak Hour Estimation**: During peak hours, let's say 10% of the daily transactions occur. Thus, we expect:
  - Peak Transactions = 1,000,000 / 24 hours * 2 hours = ~83,333 transactions/hour = ~23 transactions/second.
  - Total Data = 83,333 transactions * 500 bytes = ~41.67 MB/hour or ~11.5 KB/second.

Considering a 5x overhead for spikes and additional sources, we can estimate:
- **Throughput Requirement**: 60 KB/second (to account for other data sources like inventory updates).
- This requires a system capable of handling around 80-100 KB/second of data ingestion.

The high-level architecture for Walmart's real-time data pipeline can be represented as follows:

- **Components**:
  - **Data Ingestion Layer**: Use Apache Kafka for high-throughput, fault-tolerant event streaming.
  - **Stream Processing Layer**: Use Apache Flink for real-time data processing and transformation.
  - **Data Storage Layer**: Use Amazon S3 for raw data storage and Amazon Redshift for analytical queries.
  - **Monitoring & Alerting**: Use Prometheus and Grafana to monitor system health and performance metrics.

- **Component Diagram**:
  ```
  [Data Sources] --> [Kafka] --> [Flink] --> [S3/Redshift] --> [Analytics/Reporting]
  ```
  This architecture allows for scalable ingestion, processing, and storage, ensuring low latency and high availability.

The data model for Walmart's real-time data pipeline should accommodate various types of data:

- **Schema Design**:
  - **Transaction Event**:
    - `transaction_id`: String (Primary Key)
    - `store_id`: String
    - `timestamp`: DateTime
    - `customer_id`: String
    - `items`: List of Objects (each with `item_id`, `quantity`, `price`)
    - `total_amount`: Decimal

- **Inventory Update Event**:
    - `item_id`: String (Primary Key)
    - `store_id`: String
    - `timestamp`: DateTime
    - `quantity`: Integer
    - `status`: String

- **Access Patterns**: The model supports querying by transaction_id for detailed transaction history and by item_id for inventory monitoring.

In designing the real-time data pipeline for Walmart, several key trade-offs must be considered:

- **Consistency vs. Availability**: Given the requirement for real-time analytics, we opt for eventual consistency in favor of high availability and low latency. This means that immediate reads may not reflect the latest writes.

- **Complexity vs. Performance**: Using Apache Flink for stream processing introduces some complexity in deployment and maintenance. However, its ability to handle high throughput and low latency makes it a preferable choice over simpler systems.

- **Cost vs. Scalability**: Utilizing cloud services like AWS can enhance scalability but may incur higher costs. Balancing the trade-off between on-premise solutions and cloud services will be critical based on Walmart's budget and scalability needs.

Design a real-time Data Pipeline System

Walmart

How to Approach This

Sharpen Your Skills on Codemia

Sample Answer

Requirements

Capacity Estimation

Submit Your Answer

Walmart Machine Learning Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search