Design a multi-tenant Task Scheduling System

Question

Design a multi-tenant task scheduling system that handles millions of requests. Discuss trade-offs in consistency, availability, and performance.

Codemia · Accepted Answer

### Functional Requirements
1. **Multi-Tenancy Support**: The system should support multiple tenants, where each tenant can have its own tasks and scheduling policies.
2. **Task Creation and Management**: Users should be able to create, update, and delete tasks with specific parameters including execution time, recurrence, priority, and status.
3. **Task Execution**: The system should execute tasks based on their schedules and handle concurrent executions.
4. **Notifications**: Users should receive notifications upon task completion or failure.
5. **API Access**: Provide RESTful APIs for task management and status retrieval.

### Non-Functional Requirements
1. **Scalability**: The system should handle millions of requests per day with low latency (under 100ms for API calls).
2. **Reliability**: Task execution should be guaranteed, with retries in case of failure.
3. **Security**: Ensure data isolation between tenants and implement authentication and authorization mechanisms.
4. **Monitoring and Logging**: The system must have robust logging and monitoring to track task executions and system health.

### Back-of-Envelope Calculations
1. **QPS**: Assume 10 million tasks scheduled per day across 1,000 tenants. Each tenant schedules an average of 10 tasks per hour, leading to:
   - 10 million tasks / 24 hours = ~416,667 tasks/hour = ~116,667 tasks/minute = ~1,944 tasks/second.

2. **Storage**: Assuming each task requires an average of 1 KB of storage for its metadata:
   - 10 million tasks * 1 KB = 10 GB of storage per day.
   - Over a month, this translates to about 300 GB.

3. **Bandwidth**: If each task results in a notification (e.g., 200 bytes per notification), and assuming 10% of tasks are executed:
   - 1 million notifications * 200 bytes = ~200 MB per day for notifications.

### Component Diagram and Technology Choices
- **API Gateway**: Use **NGINX** for handling incoming requests, load balancing, and providing a single entry point for the API.
- **Task Scheduler Service**: Implemented using **Spring Boot** for task CRUD operations and scheduling logic.
- **Database**: Use **PostgreSQL** for transactional data storage with multi-tenancy support through a shared schema.
- **Task Queue**: Use **RabbitMQ** to manage the execution of tasks asynchronously.
- **Notification Service**: Use **AWS SNS** for sending notifications to users based on task status.
- **Monitoring**: Implement **Prometheus** for metrics collection and **Grafana** for visualization.

### Schema Design
1. **Tasks Table**:
   - **task_id** (UUID, Primary Key)
   - **tenant_id** (UUID, Foreign Key)
   - **task_name** (VARCHAR)
   - **execution_time** (TIMESTAMP)
   - **recurrence_pattern** (VARCHAR)
   - **status** (ENUM: 'PENDING', 'RUNNING', 'COMPLETED', 'FAILED')
   - **created_at** (TIMESTAMP)
   - **updated_at** (TIMESTAMP)

2. **Tenants Table**:
   - **tenant_id** (UUID, Primary Key)
   - **tenant_name** (VARCHAR)
   - **created_at** (TIMESTAMP)

This schema supports efficient querying by tenant and task status, enabling quick retrieval of tasks for execution.

### Architectural Decisions
1. **Consistency vs. Availability**: Given the nature of task scheduling, we prioritize availability over strong consistency. Tasks might execute with some delay, but we ensure eventual consistency by implementing retries and logging task status.
2. **Database Choice**: Choosing PostgreSQL allows us to leverage ACID properties for transactional integrity, albeit at the cost of some performance during high load situations. NoSQL alternatives were considered but dismissed due to their weaker consistency guarantees.
3. **Task Execution Strategy**: Opting for an asynchronous model using RabbitMQ allows for better scalability as tasks can be processed in parallel, but introduces complexity in handling failures and ensuring idempotency.
4. **Monitoring and Alerting**: Implementing Prometheus and Grafana introduces additional overhead but is essential for maintaining system reliability and performance insights.

Design a multi-tenant Task Scheduling System

Oracle

What the Interviewer Expects

Key Topics to Cover

How to Approach This

Possible Follow-up Questions

Practice a Similar Problem on Codemia

Sample Answer

Requirements

Functional Requirements

Capacity Estimation

Back-of-Envelope Calculations

Submit Your Answer

Oracle Software Engineer Interview Guide

Related Questions

Design a high-throughput Inventory Management System

Design a low-latency Rate Limiting System

Design a fault-tolerant Payment System

Design a fault-tolerant Messaging System

Design Walmart Product Search