A cluster is six or seven processes with a strict rule about who may talk to whom, and almost every operational question becomes easy once that rule and the ownership boundaries are clear.

The one architectural rule

Everything goes through the interface server, and only the interface server touches the datastore.

No scheduler, controller, node agent or proxy opens a connection to the datastore. They read and write objects through the interface server, which performs authentication, authorisation, validation and admission on every request. The datastore's credentials exist in exactly one place.

This single rule explains a great deal. It is why the interface server is the availability bottleneck for control operations. It is why every extension point is an interface server feature rather than a datastore feature. And it is why an audit log at the interface server is a complete record of every mutation, which would be impossible if components wrote directly.

The control plane processes

Four processes make up the control plane proper, and each owns a distinct job.

The interface server validates and serves the object model, and it is stateless, so any number of instances can run behind a load balancer. The datastore holds the objects and provides consistency and watch semantics. The scheduler assigns pods to nodes and does nothing else. The controller manager runs several dozen controllers in one process, each reconciling one kind of object towards its declared state.

A fifth process, the cloud controller manager, exists where the cluster runs on an infrastructure provider, and it owns everything that calls the provider's interface: node lifecycle, load balancers and routes. It was split out of the controller manager precisely so that provider code is not in the core components.

The node processes

Two processes run on every node, with one of them optional depending on the networking implementation.

The node agent owns the node: it registers it, reports its status, and is the only thing that starts and stops containers. It watches the interface server for pods assigned to its node and makes reality match. It reports back through the pod's status.

The proxy implements service addressing by programming the node's packet handling, so that traffic to a service address reaches one of its endpoints. Some networking implementations replace it entirely, which is why it is the one component that may be absent.

What each component does not own

Stating the negatives is more useful than the positives, because mistaken ownership is behind most misdirected debugging.

The scheduler does not start containers; it writes a node name into a pod and stops. The node agent does not decide where pods go; it only acts on pods already assigned to it. The controller manager does not schedule; it creates pods and lets the scheduler place them. The interface server does not reconcile anything; it stores and serves objects, and controllers do the work.

So a pod stuck in a pending state is a scheduler question, a pod stuck in a creating state is a node agent question, and a deployment that never creates pods at all is a controller manager question. Three different components, three different log files, and the pod's own state tells you which.

The watch mechanism everything depends on

Components do not poll, and understanding what they do instead explains both the efficiency and the failure modes.

A component opens a long-lived watch request for a kind of object, optionally filtered, and the interface server streams changes as they occur. Each component maintains an in-memory cache built from an initial list plus the subsequent stream, so its reads are local and free. The interface server in turn watches the datastore and fans out to every watcher.

The consequence worth knowing is that every component's view is eventually consistent and slightly stale, by the time it takes a change to propagate. Controllers are written to tolerate this by being idempotent and reconciling repeatedly, which is why the model works. A controller written as if its cache were current is the source of a whole class of bug.

Seeing the components from outside

The state of the control plane is readable through the same interface as everything else, which is worth demonstrating.

1kubectl get --raw='/readyz?verbose'            # per-check readiness of this API server
2kubectl get leases -n kube-system              # who holds scheduler and controller leadership
3kubectl get --raw='/metrics' | grep -E '^etcd_|^apiserver_request_total' | head
4kubectl get componentstatuses                  # deprecated, frequently misleading

The first is the honest health check, since it names which subsystem is failing rather than reporting a single value. The second answers which instance is actually doing the work. The last is deprecated and reports health for a topology that no longer reflects how most clusters are deployed, so a healthy result there means little.

The aggregation and extension surfaces

Because every extension is an interface server feature, it is worth naming the three that exist.

Custom resource definitions add new kinds served by the same interface server, stored in the same datastore, with the same authorisation and admission. This is how almost every extension is built, and the resulting kinds are indistinguishable from built-in ones to a client.

The aggregation layer registers an external server to serve a particular group, with the main interface server proxying requests to it. This is how the metrics interface works, and it is the right mechanism when the data should not be stored as objects, such as live measurements.

Admission webhooks and authorisation modules extend the request pipeline rather than the object model. Together these three cover extension of what exists, of what is served, and of what is permitted, which is a tidy division worth remembering.

What runs where, in practice

Deployment topology varies and the variation causes confusion when reading documentation.

A self-managed cluster typically runs the control plane components as static pods on dedicated nodes, which is why they appear as pods in the system namespace while having been started by the node agent directly. A managed cluster runs them on infrastructure the operator never sees, so they do not appear as pods at all and their logs are reached through the provider's interface.

The practical consequence is that half the debugging advice available assumes access that a managed cluster does not provide. Knowing which half applies saves time, and the dividing line is simple: anything requiring a control plane host's filesystem or process list is unavailable on a managed cluster, and everything reachable through the interface server is available on both.

The resource model underneath

One more property of the interface server explains a lot of behaviour.

Every object has a group, a version and a kind, a name scoped either to a namespace or to the cluster, a specification expressing intent and a status expressing observation. Writes to specification and status are separable, which is what allows a controller to own one and a user the other.

Objects also carry a resource version, which is the datastore's revision at which they were last written. Every watch is resumable from a version, which is how a component reconnects without relisting, and every update can be made conditional on a version, which is how concurrent writers avoid clobbering each other. These two uses of one field are the mechanism behind both the watch machinery and leader election.

Namespaces, and what they are not

Since the object model is central, one frequent misunderstanding is worth clearing up.

A namespace is a name scope with an attached policy surface: authorisation rules, resource quotas and limit ranges apply to it, and most object kinds live inside one. It is not an isolation boundary in any stronger sense. Pods in different namespaces reach each other over the network by default, share the nodes they land on, and share the node's kernel.

So a namespace per team is a reasonable administrative division and a weak security boundary. Making it stronger requires network policy to restrict traffic, quotas to restrict consumption, and separate node pools where kernel-level isolation matters. Treating a namespace as a tenant boundary without those is a common and consequential mistake.