Design a GPU Resource Scheduling System
Last updated: March 3, 2025
Quick Overview
Schedule ML training and inference jobs across GPU clusters with bin-packing, fairness policies, preemption, and fault tolerance for long-running jobs.
ByteDance
System Design
Software Engineer
ByteDance
March 3, 2025Software Engineer
System Design Round
System Design
Hard
7
4
4,170 solved
Schedule ML training and inference jobs across GPU clusters with bin-packing, fairness policies, preemption, and fault tolerance for long-running jobs.
ByteDance invests heavily in ML infrastructure. This tests understanding of GPU cluster management for model training and inference workloads.
What the Interviewer Expects
- Design job scheduling with bin-packing optimization
- Implement fairness policies across teams
- Handle preemption for priority jobs
- Design fault tolerance for long-running training jobs
- Address resource fragmentation
Key Topics to Cover
GPU scheduling
Bin packing
Job preemption
Fault tolerance
Resource management
How to Approach This
- Start by clarifying functional and non-functional requirements with the interviewer.
- Estimate the scale: QPS, storage, bandwidth. This drives your design decisions.
- Draw a high-level architecture first, then deep dive into 1-2 critical components.
- Discuss trade-offs explicitly (e.g., consistency vs availability, SQL vs NoSQL).
- Address failure scenarios, monitoring, and how the system handles 10x traffic spikes.
Possible Follow-up Questions
- How do you handle jobs that need multi-node distributed training?
- What is your strategy for GPU memory fragmentation?
- How do you measure and improve cluster utilization?
Sharpen Your Skills on Codemia
Practice similar problems with our interactive workspace, get AI feedback, and track your progress.
Practice System Design ProblemsSample Answer
Requirements
Functional Requirements
- Job Submission: Users can submit ML training and inference jobs specifying resource requirements (e.g., GPU type, memory).
- Scheduling Algorithm: Implement a b...
Capacity Estimation
Assuming ByteDance has 1,000 GPUs distributed across various clusters:
- Average Job Size: Let's say each job typically requires 2 GPUs and runs for an average of 4 hours.
- **Daily Job Submission...
Submit Your Answer
Markdown supported