Build a event-driven Task Scheduling Pipeline

Question

Design a event-driven task scheduling system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Event Processing**: The system should process various event types (e.g., order creation, payment confirmations) and trigger corresponding tasks (e.g., sending notifications, updating inventory).
2. **Task Scheduling**: Allow scheduling of tasks with different execution priorities and timeframes (e.g., immediate, delayed).
3. **Retry Mechanism**: Implement a retry mechanism for failed tasks with exponential backoff.
4. **Monitoring and Alerts**: Provide monitoring dashboards for real-time metrics and alerting mechanisms for task failures or delays.

### Non-Functional Requirements
1. **Scalability**: Handle millions of event requests per second, scaling horizontally as needed.
2. **Availability**: The system must be highly available, with a target uptime of 99.99%.
3. **Consistency**: Event ordering must be preserved, but strict consistency is not required for all tasks (eventual consistency is acceptable).
4. **Latency**: The system should maintain low latency (under 100ms) for task execution after event receipt.

Assuming Shopify handles approximately 1 million events per day, let's estimate the requirements:
- **QPS Calculation**: 
  - 1 million events/day / 86400 seconds/day = ~11.57 QPS (average)
  - However, we expect traffic spikes, especially during sales or promotions. Assuming a peak of 1000 QPS (10x average).

- **Storage Calculation**: 
  - Each task/event might require 1 KB of data. 
  - For 1 million events, that's ~1 GB/day. 
  - Over a month, that's ~30 GB. To allow for growth, we should plan for 1 TB of storage.

- **Bandwidth Estimation**: 
  - If each event requires sending a response of 500 bytes, the bandwidth for 1000 QPS would be:
  - 1000 requests/second * 0.5 KB/request = 500 KB/s, with peaks reaching several MB/s.

### High-Level Architecture
- **Event Producers**: Various Shopify services that generate events (e.g., order service).
- **Event Queue**: Use **Apache Kafka** as the message broker for handling incoming events. It provides durability and scalability.
- **Task Scheduler**: A microservice that consumes events from Kafka and schedules tasks. This could be implemented using **Spring Boot** or **Node.js**.
- **Database**: Use **Cassandra** for task storage due to its high availability and horizontal scalability properties. The schema can be designed for efficient querying based on task status and execution time.
- **Caching Layer**: Implement **Redis** for caching frequently accessed data, reducing load on the database and improving latency.
- **Monitoring**: Use **Prometheus** and **Grafana** for monitoring the task scheduling system and alerting on failures.

### Schema Design
- **Tasks Table** (Cassandra):
  - `task_id` (UUID, PRIMARY KEY)
  - `event_type` (TEXT)
  - `status` (TEXT, e.g., PENDING, IN_PROGRESS, FAILED, COMPLETED)
  - `scheduled_time` (TIMESTAMP)
  - `execution_time` (TIMESTAMP)
  - `retry_count` (INT)
  - `payload` (TEXT) - JSON representation of the task data.

### Access Patterns
- Query by `status` to fetch pending tasks for processing.
- Query by `event_type` to aggregate metrics or perform analytics on task execution.

### Architectural Decisions
- **Consistency vs. Availability**: We favor availability and partition tolerance (AP) over strong consistency. Eventual consistency is acceptable for task execution, allowing us to use Cassandra effectively.
- **Database Choice**: Choosing Cassandra over traditional SQL databases allows for horizontal scaling, but it comes with challenges in terms of query complexity and eventual consistency.
- **Caching Strategy**: Using Redis introduces complexity but significantly reduces database load and latency. The trade-off is potential cache invalidation issues, which we mitigate by ensuring the cache is updated whenever tasks are modified.
- **Event Ordering**: While Kafka guarantees ordering within partitions, we may need to design our task scheduler to handle parallel processing of different event types, ensuring that tasks dependent on one another are processed in the correct order.

Build a event-driven Task Scheduling Pipeline

Shopify

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Shopify Machine Learning Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search