System requirements

Functional:

1. Container Lifecycle Management
2. Scheduling
3. Scaling
4. Service Discovery and Load Balancing
5. Self-Healing
6. Resource Monitoring and Reporting
7. Security and Access Control
8. Integration with CI/CD Pipelines
9. Namespace and Multi-Tenancy Support

Non-Functional:

1. High Availability
2. Scalability
3. Performance
4. Reliability

Capacity estimation

• Cluster size: 1000 nodes
• Containers per node: 100 containers
• Total containers: 1000 * 100 = 100,000 containers

API design

1. Authentication

POST /auth/login

Request:

2. Deployments

Create Deployment

POST /deployments

List Deployments

GET /deployments

Response:

Get Deployment Details

Update Deployment (e.g., rolling update)

PUT /deployments/{id}

Delete Deployment

3. Nodes

List Nodes

GET /nodes

Get Node Details

4. Services & Load Balancing

List Services

Create Service

POST /services

5. Monitoring & Health

Cluster Metrics

GET /metrics/cluster

Deployment Metrics

Node Metrics

Health Check

GET /healthz

7. CI/CD Integration

Trigger Deployment via Webhook

POST /webhooks/deploy

Database design

We need to capture:

1. Cluster Metadata — clusters, nodes, regions (etcd).
2. Workload Definitions — apps, deployments, container specs(etcd).
3. Runtime State — pods/containers, scheduling, health (etcd).
4. Networking & Services — services, endpoints, load balancers.
5. Security — users, roles, namespaces, RBAC rules.
6. Events & Monitoring — container lifecycle events, metrics references.
7. Audit Logs — every control-plane API action.

An example of Cluster Metadata:

An example of Workload Definitions:

An example of Runtime State:

An example of Networking & Services:

Events & Monitoring

Audit Logs

High-level design

• API Server: Acts as a gateway for all interactions with the system, ensuring secure communication and consistent state updates.
• Cluster State Database (etcd): Acts as a central hub for storing all critical cluster data. All components interact with this to some extent to ensure seamless operation.
• Scheduler and Controller Manager: Coordinate to ensure pods are correctly scheduled to nodes and the state is as intended.
• Node Agent and Container Runtime: Manage containers on individual nodes, ensuring containers' lifecycle is maintained.
• Auto-scaling: Helps dynamically adjust workloads based on the current load, reducing manual intervention and optimizing resource usage.
• Networking & Service Discovery: Facilitates communication between containers and services, handling DNS registration and load balancing.

Request flows

Deployment Flow Steps

1. Client Request:
   • The user or a CI/CD pipeline initiates a deployment by sending a request to the API Server, typically using a POST request to the /deployments endpoint.
   • This request includes specifications such as the container image, desired number of replicas, and resource requirements (e.g., CPU, memory).
2. API Server Processing:
   • The API Server authenticates the request, checks for authorization, validates the input, and then writes the desired state to the Cluster State Database (etcd).
   • A Deployment object is created in the database, specifying the intended number of running instances and the configuration details.
3. Deployment Controller Reaction:
   • The Deployment Controller, part of the Controller Manager, is continuously watching for new or updated Deployment objects.
   • Upon detecting the new deployment request, it creates the requisite number of Pod objects to match the desired state. These Pods initially have no assigned node.
4. Scheduling:
   • The Scheduler constantly monitors the Cluster State Database for unassigned Pods.
   • It selects nodes for these Pods based on current cluster resource availability and scheduling constraints, and updates the Pod objects in the database to reflect the node assignment.
5. Node Agent Activation:
   • Node Agents running on the selected nodes are notified of new Pod assignments.
   • They pull the required container images (if not already cached) and initiate container creation via the Container Runtime (e.g., Docker or containerd).
6. Container Startup:
   • The Node Agent starts the container with the specified configurations (e.g., network settings, environment variables).
   • The container enters a running state upon successful startup.
7. Service Registration (if applicable):
   • If a corresponding Service exists for the application, the Service Controller updates the service registry to include network locations for the new containers.
   • This setup ensures that incoming requests are properly routed to the running instances.
8. Status Update:
   • The Node Agent updates the Pod status in the Cluster State Database to "Running" and records vital information such as IP address and availability.
   • The API Server can now provide the user with feedback stating the deployment was successful and is active.

Scheduling Flow Steps

1. Monitor for Pending Pods:
   • The Scheduler constantly monitors the Cluster State Database (etcd) for Pods that are in a "Pending" state, meaning they haven't yet been assigned to any node.
2. Filter Nodes:
   • The Scheduler evaluates available nodes and filters out those that do not meet the Pod's specific resource requirements (e.g., CPU, memory) or constraints such as node labels or affinity rules.
3. Score Nodes:
   • The feasible nodes (those not filtered out) are then scored based on various criteria, such as:
     • Current resource utilization: to balance load across the cluster.
     • Affinity and anti-affinity policies: to keep or separate certain pods.
     • Customizable business rules or priorities.
4. Select Optimal Node:
   • The Scheduler selects the node with the highest score for the Pod. This step balances resource usage and efficiency.
5. Bind Pod to Node:
   • The Scheduler updates the Cluster State Database, indicating that the Pod is now assigned to a specific node, effectively binding the Pod to that node.
6. Node Agent Execution:
   • The Node Agent on the selected node is notified of the new Pod assignment and pulls necessary container images if they are not cached.
   • The Node Agent then starts the container using the Container Runtime.
7. Status Update:
   • Once the container is running successfully, the Node Agent updates the Pod status in the Cluster State Database to "Running."

Self-Healing Flow Steps

1. Failure Detection:
   • Node Agents continuously monitor the health of containers using liveness and readiness probes.
   • If a container fails a health check or crashes, the Node Agent marks its status as "Failed" in the Cluster State Database (etcd).
2. Controller Intervention:
   • The relevant controller (e.g., ReplicaSet, StatefulSet) is watching the database for changes in the status of pods.
   • When a pod is marked as "Failed", the controller identifies the discrepancy between the desired and actual states (e.g., fewer running replicas than desired).
3. Pod Replacement:
   • The controller creates a new Pod object to replace the failed instance. This Pod is set to a "Pending" state until it can be scheduled.
4. Scheduling:
   • The Scheduler detects the newly created (pending) Pod and assigns it to an appropriate node based on available resources and constraints, modifying the Pod’s state in the database to reflect this assignment.
5. Node Agent Activation:
   • On the selected node, the Node Agent is notified of the new Pod assignment. The agent pulls the necessary container image (if not cached) and starts the container using the Container Runtime.
6. Status Update:
   • Once the new container is running successfully, the Node Agent updates the Pod’s status in the Cluster State Database to "Running."

Node Management Flow Steps

1. Adding a Node:
   • Node Registration:
     • A new node is introduced to the cluster. The Node Agent on this node registers itself with the Cluster State Database (etcd), providing details such as resource capacity and node status.
   • Cluster Database Update:
     • The API Server registers the node metadata, updating the Cluster State Database with the new information.
2. Node Configuration:
   • Initial Configuration:
     • The Node Agent configures the necessary runtime environment, networking setup, and any required security settings.
   • Scheduler and Controller Notification:
     • The Scheduler and Controller components are notified of the new node so they can consider it for scheduling decisions.
3. Updating a Node:
   • Resource Changes:
     • If resource capacities (e.g., CPU, memory) or labels on an existing node change (due to hardware upgrades or reallocation), the Node Agent updates these changes in the Cluster State Database.
   • Configuration Changes:
     • Any adjustments in node configurations, such as network settings or taints/tolerations, are also updated in the database.
   • Re-evaluation by Scheduler:
     • The Scheduler re-evaluates its scheduling decisions, considering the updated capacities and configurations.
4. Removing a Node:
   • Node Decommissioning Request:
     • A node is either proactively removed from the cluster (e.g., maintenance or resource reallocation) or marked for removal due to a failure.
   • Eviction of Workloads:
     • The Controller Manager might initiate Pod evictions, gracefully moving workloads from this node to other nodes.
   • Node Deregistration:
     • The Node Agent updates the Cluster State Database to reflect that the node is offline or removed. The API Server then removes the node's metadata from the cluster registry.

Scheduling Flow Steps

1. Monitor for Pending Pods:
   • The Scheduler continuously monitors the Cluster State Database (etcd) for Pods that are in a "Pending" state, indicating that they have not yet been assigned to any node.
2. Filter Nodes:
   • For each pending Pod, the Scheduler applies filters to the available nodes to eliminate those that cannot meet the Pod’s resource requirements, such as insufficient CPU or memory, or that do not meet specific scheduling constraints (e.g., node labels, taints, or affinity rules).
3. Score Nodes:
   • The remaining candidate nodes are evaluated further by scoring them based on various criteria, such as:
     • Resource availability and current usage.
     • Pod affinity and anti-affinity rules.
     • Custom scoring functions that take into account specific deployment strategies.
4. Select Optimal Node:
   • Based on the scores, the Scheduler selects the node with the highest score for the Pod. The selected node optimizes factors like load balancing and resource utilization.
5. Bind Pod to Node:
   • The Scheduler updates the Cluster State Database to bind the Pod to the chosen node. This step finalizes the assignment, officially making the node responsible for running the Pod.
6. Node Agent Execution:
   • The Node Agent on the assigned node is notified of the new Pod assignment. It proceeds to pull the required container image (if not cached locally) and starts the container using the Container Runtime.
7. Status Update:
   • Once the container starts successfully, the Node Agent updates the Pod's status in the Cluster State Database to "Running."

Service Discovery and Load Balancing Flow Steps

1. Service Definition:
   • When a user defines a Service, they specify details such as the service type (ClusterIP, NodePort, LoadBalancer), port configurations, and selector labels that identify which Pods should be part of the service.
2. Pod Registration:
   • As Pods matching the service's selector labels become "Running," they are automatically registered as endpoints for the Service.
3. Endpoint Management:
   • The Endpoint Controller constantly watches for Pod status changes. When Pods matching a service's selectors become available or unavailable, the controller updates the endpoint lists for those services in the Cluster State Database.
4. Service Proxy Configuration:
   • Components like kube-proxy (in Kubernetes) running on each node watch these endpoint changes and update IP tables or other routing mechanisms to ensure that traffic directed to a Service is routed to the appropriate Pod.
5. Load Balancing:
   • The service proxy on each node helps distribute incoming traffic among the available Pods. This ensures efficient use of resources and maintains high availability and reliability even as Pods start, stop, or are replaced.
6. External Access (if applicable):
   • For services of type "LoadBalancer," integration with cloud provider APIs may be used to provision external load balancers. Traffic directed to these load balancers is distributed across various nodes where service Pods are running.
7. DNS Registration:
   • Internal DNS manages service names, resolving them to the appropriate service IPs or external load balancer addresses, facilitating service discovery within the cluster.

CI/CD Integration Flow Steps

1. Code Commit:
   • Developers push changes to the source code repository, triggering a CI/CD pipeline.
2. Build and Test:
   • The pipeline builds the application, creating new container images.
   • Automated tests are run to ensure the changes are functioning as expected.
3. Image Push:
   • Upon successful testing, the new image is pushed to a container registry.
4. Deployment Trigger:
   • A webhook or API call triggers the deployment of the new image in the container orchestration system.
5. Update Deployment:
   • The API Server receives the deployment update request. It updates the deployment object in the Cluster State Database to reflect the new image and desired state.
6. Rolling Update:
   • The Deployment Controller initiates a rolling update, carefully updating Pods to the new image version with minimal downtime.
   • Pods are updated incrementally, monitored for health, and verified before further updates.
7. Status Feedback:
   • The orchestration system provides status feedback to the CI/CD system, indicating the success or failure of the deployment.
8. Rollback (if necessary):
   • If issues are detected with the new deployment, the system can automatically revert to the previous stable version to maintain service continuity.

Detailed component design

1. API Server

Responsibilities:

• Acts as the front-end interface for all operations within the orchestration system.
• Handles authentication, authorization, and data validation.
• Interacts with the Cluster State Database to read and write system state.

Design Considerations:

• Stateless Design: Can be run in multiple instances behind a load balancer for higher availability.
• Secure Communication: Use TLS for secure API transactions and enforce Role-Based Access Control (RBAC) for user permissions.
• Performance: Caches frequently accessed data to reduce direct database queries.

2. Scheduler

Responsibilities:

• Assigns unscheduled Pods to Nodes based on resource requirements and policy constraints.

Design Considerations:

• Resource Filtering and Scoring: Implements multi-stage filtering to identify eligible nodes and scores them based on a weighted criteria (e.g., resource availability, affinity, and anti-affinity rules).
• Extensibility: Use a plugin-based architecture to allow custom scheduling policies.
• Concurrency Handling: Utilize optimistic locking when updating Pod assignments to prevent race conditions.

3. Controller Manager

Responsibilities:

• Runs various control loops to ensure the desired state of the cluster matches the actual state.
• Manages components like Deployment, ReplicaSet, and Node controllers.

Design Considerations:

• Control Loops: Each controller operates based on watch events and adjusts each resource toward its desired state.
• State Reconciliation: Corrects state discrepancies by creating, updating, deleting Pods, nodes, or services as necessary.
• Modular Architecture: Enable each controller to be updated or enhanced independently.

4. Node Agent (like Kubelet)

Responsibilities:

• Manages the lifecycle of Pods scheduled on the node.
• Monitors node resources and health.

Design Considerations:

• Resource Isolation: Relies on container runtimes like containerd or Docker, utilizing cgroups and namespaces for resource isolation.
• Local Health Checks: Runs liveness and readiness probes to determine the health of containers.
• Self-recovery: Automatically tries to restart containers that fail within tolerance limits locally before alerting the control plane.

5. Container Runtime

Responsibilities:

• Provides the environment where containers are executed.
• Interfaces with operating system-level resources.

Design Considerations:

• Compatibility: Supports OCI-compliant images and runtimes for interoperability.
• Security: Implement process isolation (e.g., GVisor or Firecracker for microVMs) to prevent privilege escalation.
• Performance: Optimized for fast container startup, image pulling, and caching mechanisms.

6. Service Discovery and Networking

Responsibilities:

• Discovers services dynamically and manages the routing of network traffic to the appropriate Pods.

Design Considerations:

• Networking Layer: Use Container Networking Interface (CNI) for managing network configurations, enabling flexible plugin use.
• Service Mesh Integration: Allows seamless integration of service mesh components (e.g., Istio) for enhanced observability, security, and traffic management features.
• Load Balancing: Ensure balanced traffic distribution across Pods with mechanisms like round-robin or least connections.

7. Storage Management

Responsibilities:

• Facilitates attaching persistent volumes for stateful applications.

Design Considerations:

• Storage Classes: Define policies for provisioning storage (e.g., local SSDs, networked storage solutions).
• Dynamic Provisioning: Automate volume creation and attachment to Pods as needed.
• Data Protection: Support backups and replication policies for data integrity.

Trade offs/Tech choices

Explain any trade offs you have made and why you made certain tech choices...

Failure scenarios/bottlenecks

Try to discuss as many failure scenarios/bottlenecks as possible.

Future improvements

What are some future improvements you would make? How would you mitigate the failure scenario(s) you described above?