Design a Task Scheduling for Mastercard

Question

Design a high-throughput task scheduling system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Task Creation**: Users can create tasks with a specific priority and execution time.
2. **Task Scheduling**: The system must schedule tasks based on their priorities and execution times.
3. **Task Execution**: Execute tasks reliably, ensuring that tasks are processed in the order of their priority.
4. **Task Status Tracking**: Provide real-time updates on task status (Pending, In Progress, Completed, Failed).
5. **Retry Mechanism**: Automatically retry failed tasks based on predefined rules.

### Non-Functional Requirements
1. **High Throughput**: The system must handle millions of requests per second (QPS).
2. **Low Latency**: Task scheduling and execution should have sub-second response times.
3. **Scalability**: The system should scale horizontally to accommodate increasing numbers of tasks and users.
4. **Availability**: System uptime should be at least 99.99%.
5. **Security**: Ensure secure authentication for users and tasks, complying with PCI DSS standards.

Assuming Mastercard handles approximately 1 billion transactions a day, let’s estimate the task scheduling system's capacity:

1. **Daily Requests**: 1 billion transactions / day = ~11,574 transactions/second (TPS).
2. **Task Creation Rate**: Assuming 10% of these transactions generate tasks, we have:
   - **Task Creation**: 1 billion * 0.10 = 100 million tasks/day = ~1,157 tasks/second.
3. **Task Execution Rate**: Each task may take an average of 1 second to execute, leading to:
   - **Task Execution**: 1,157 tasks/second needing execution.
4. **Storage Needs**: If each task record is approximately 1KB:
   - **Daily Storage**: 100 million tasks * 1KB = ~100 GB/day.
5. **Total QPS**: Considering read requests (status checks, updates), we can estimate around 3,000 QPS for a high-load scenario.

### Component Diagram
- **API Gateway**: Handles incoming requests, load balancing, and routing.
- **Task Scheduler**: A service that manages task creation, scheduling, and execution. Implemented using **Apache Kafka** for message queuing to ensure high throughput.
- **Worker Nodes**: Microservices that pull tasks from Kafka and execute them. Each worker can be scaled horizontally.
- **Data Store**: **Cassandra** for storing task metadata due to its high write throughput and availability.
- **Cache**: **Redis** for caching frequently accessed task statuses to reduce read latency.
- **Monitoring**: **Prometheus** and **Grafana** for real-time monitoring and alerting on task execution metrics.

### Schema Design
- **Task Table** (in Cassandra):
  - `task_id`: UUID (Primary Key)
  - `user_id`: UUID (Foreign Key referencing user)
  - `status`: ENUM ('Pending', 'In Progress', 'Completed', 'Failed')
  - `priority`: INT (Higher number means higher priority)
  - `creation_time`: TIMESTAMP
  - `execution_time`: TIMESTAMP
  - `retry_count`: INT
  - `error_message`: TEXT (nullable)

### Access Patterns
- **Task Retrieval**: Query by `task_id` for specific task details and by `user_id` for task history.
- **Task Status Update**: Frequent updates to the `status` and `retry_count` fields.

### Architectural Decisions
1. **Cassandra vs. SQL Database**: Chose Cassandra for its scalability and write-heavy workload handling, sacrificing some transactional consistency (eventual consistency) for availability and partition tolerance.
2. **Kafka for Task Queue**: Utilizing Kafka introduces some latency in task execution but provides high throughput and durability, allowing for decoupled task processing.
3. **Consistency vs. Availability**: Opted for eventual consistency in task status updates to allow high availability and responsiveness in a distributed system.
4. **Caching vs. Freshness**: Implemented Redis caching for task status to provide low-latency reads but must manage cache invalidation to avoid stale data.
5. **Monitoring and Alerting**: Prioritized robust monitoring using Prometheus to catch failures early, which adds operational overhead but is crucial for maintaining SLAs.

Design a Task Scheduling for Mastercard

Mastercard

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Submit Your Answer

Mastercard Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search