Design a Container Orchestration System

A container orchestration system sounds like a deployment tool: push a container image and it runs somewhere. The real challenge is what happens after deployment. When a container crashes at 3am, the system must detect the failure, spin up a replacement, and route traffic away from the dead instance, all without waking anyone up. The system is not a deployment tool; it is a self-healing runtime that keeps hundreds of thousands of containers alive across thousands of machines.

Functional

Automate the full container lifecycle: Deploy, schedule, monitor, restart, and remove containers. Users declare an application's image, CPU/memory requirements, and desired replica count. The system makes it happen and keeps it that way. If you say "run 5 copies," there will always be 5 copies running, even if machines fail.

The capacity numbers for a container orchestration system are not about user-facing traffic like web requests. They are about the control plane's ability to manage cluster state: scheduling decisions, heartbeat processing, and state store throughput. Designing for the average case means the system fails during the exact moments it matters most, such as when a node goes down and 100 containers need rescheduling simultaneously.

Cluster Scale

5,000 worker nodes in a single cluster: this is the proven upper limit for Kubernetes. Each node runs 10-100 containers depending on workload density. At 50 containers per node average, the cluster manages 250,000 containers. Google's Borg operates clusters with tens of thousands of machines, but 5,000 nodes covers the vast majority of production deployments.

The API follows a declarative model: users describe the desired state ("run 5 replicas of image X") and the system continuously reconciles actual state toward that target. This is fundamentally different from imperative APIs ("start a container on node Y"). Declarative APIs are idempotent: submitting the same spec twice produces no additional changes, making them safe for CI/CD retries and automated workflows.

Deployment Management

1POST /api/v1/deployments
2Body: {
3name: string,
4image: string,
5replicas: number,
6cpu_request: string, // e.g., "500m" (0.5 cores)
7memory_request: string, // e.g., "256Mi"
8ports: [number],
9env: [{name, value}],
10health_check: {
11path: string,
12port: number,
13interval_seconds: number
14}
15}
16Response: 201 Created { deployment_id: string, status: "pending" }

The data layer is a single strongly consistent key-value store (etcd) that holds all cluster state. The choice of a single store rather than multiple databases is deliberate: every scheduling decision, every health check evaluation, and every reconciliation loop depends on a consistent view of the entire cluster. Splitting state across databases would require distributed transactions between them, adding latency and failure modes to every operation.

Why etcd

etcd provides three properties that a container orchestrator requires:

The architecture splits into two planes: the control plane (brain) and the data plane (muscle). The control plane decides what should run where; the data plane actually runs the containers. This separation is the key to resilience: if the control plane goes down, already-running containers keep running. New deployments pause, but existing services continue serving traffic.

Two request flows define the system: the deployment flow (creating containers) and the self-healing flow (replacing failed containers). Understanding the exact sequence of steps, and where failures can occur at each step, reveals why the architecture uses event-driven reconciliation rather than direct commands.

Container Deployment Flow

The core engineering challenge (the GATE of this problem) is the scheduler's filter-score pipeline. In a cluster of 5,000 nodes, the scheduler must make hundreds of placement decisions per second, each considering resource availability, constraint satisfaction, and failure domain spreading. The scheduler's design determines whether the system achieves efficient resource utilization or wastes capacity through poor placement.

Scheduler Filter-Score Pipeline

Every architectural decision in this system involves a trade-off. Understanding these trade-offs, and being able to articulate why you chose one side, demonstrates that you made deliberate design choices rather than blindly following Kubernetes patterns.

Strong Consistency vs. Availability for Cluster State

Strong consistency (our choice): etcd uses Raft consensus. Every write requires majority acknowledgment. If the majority of etcd nodes are down, writes are blocked entirely.

The system's reliability comes from layered failure handling: each component can fail independently without bringing down the entire cluster. Understanding failure modes is critical in interviews because it demonstrates you have thought beyond the happy path.

Control Plane Master Failure

Scenario: One of three master nodes running the API server, scheduler, and controller manager crashes.

These improvements extend the system without changing the core architecture. Each addresses a real limitation that organizations encounter as they scale.

Multi-Cluster Federation

A single cluster has practical limits (5,000 nodes, one failure domain). Federation coordinates multiple clusters under a unified API. A deployment can span clusters for geographic redundancy: "run this service in US-East, EU-West, and AP-South." The federation controller distributes replicas across clusters based on latency, cost, and regulatory requirements. Each cluster remains independently operational, so a federation control plane failure does not affect individual clusters.