Design a GPU cluster scheduler for large-scale ML training workloads like Meta AI Infra or Anthropics training platform. Support gang scheduling (all N pods start together or none), topology-aware placement across NVLink pairs and same-rack GPUs for AllReduce bandwidth, hierarchical fair-share quotas across orgs/teams/users, priority preemption with coordinated checkpointing, spot-instance handling with graceful drain, resource fragmentation mitigation, and integration with Kubernetes or Ray as the execution layer. Handle thousands of GPUs, queue times under 10 minutes at p50, and fault recovery without restart-from-scratch.
Suggested time: 90 minutes
Loading the workspace...
A GPU cluster scheduler for ML training looks superficially like "Kubernetes but for GPUs," and that surface reading is why most first attempts fail. The real problem is dispatching distributed training jobs that need N GPUs simultaneously, placed on the right network topology, preempting correctly under priority pressure, enforcing per-team quotas, and recovering from node failures mid-run without losing weeks of training progress. The generic Kubernetes default scheduler cannot do this, and that is the reason systems like Volcano, YuniKorn, Slurm, Kueue, and the internal schedulers at Meta, Google, Microsoft, and OpenAI exist as distinct pieces of infrastructure.
Distributed training job
Priority preemption
Topology-aware placement
Spot/preemptible handling
Fair share across teams
Design a GPU Cluster Scheduler for ML Training: editorial solution
Community solutions for Design a GPU Cluster Scheduler for ML Training
Design Sora (hard)
Design a Hybrid Cloud Workload Orchestrator (advanced)
Design a Large-Scale Graph Processing System (advanced)
Design a Big Data Processing Pipeline (advanced)
Design a High-Performance Computing Cluster (advanced)