Every behaviour in a cluster that is not a direct response to a request comes from a controller, and all of them obey one contract whose properties explain both their resilience and their failure modes.

Level triggered, not edge triggered

A controller does not process events; it processes the current state of the world.

An edge-triggered design would react to each change: a pod was deleted, so create one. A level-triggered design reads the current desired state and the current actual state, computes the difference and acts to close it. The event only says that something may have changed and is otherwise discarded.

This is why controllers survive losing events, restarting, being paused for an hour and being run after a datastore restore. None of those are special cases, because the controller never relied on the sequence of changes in the first place. The entire tolerance of the system to disruption comes from this one design choice, and a controller written to react to events instead of to state loses all of it.

Idempotence, and what it demands

Because a reconcile may run any number of times for one change, every action in it must be safe to repeat.

Concretely: creating an object must tolerate it already existing, deleting must tolerate it being gone, and an external call must either be naturally repeatable or be guarded by a recorded marker that makes the second call a no-op. A reconcile that counts, appends or increments is almost always wrong, because it will run twice and produce two of whatever it counted.

The test to apply when reviewing a reconcile is to ask what happens if the process is killed immediately after each individual action and then restarted. If any answer is wrong, the reconcile is not idempotent, and the bug will appear eventually, because that sequence happens naturally during deploys and node failures.

The shape of a reconcile

Almost every controller's main function has the same five steps in the same order.

Read the object by key from the local store, and if it is absent treat that as a deletion rather than an error. Read the actual state, which means the objects or external resources this one owns. Compute what is needed. Apply the difference, creating, updating or deleting. Write status describing what was observed, including the generation that was observed.

Deviations from this shape are where bugs live. A reconcile that writes status before acting reports intent as observation. One that reads actual state from the server rather than its store generates load. One that returns early on a partial failure without requeueing silently stops converging, which is the most common way a controller appears to work and does not.

Generation, observed generation and drift

Two counters make a controller's progress observable, which matters more than it sounds.

The generation increments on specification changes only. A controller records the generation it acted on in the status. The difference between them is the controller's lag, and a lag that is non-zero and growing is the signal that the controller is stuck, which is otherwise invisible because a stuck controller produces no errors.

This is the single most useful thing to monitor about any controller, custom or built in. A cluster-wide check for objects whose observed generation has been behind for more than a few minutes catches a stalled controller, a lost leader lease and a permanently failing reconcile, all of which otherwise present as nothing happening.

Why controllers do not coordinate

Many controllers act on the same objects with no locking between them, which works for a specific reason.

Each one owns a disjoint set of fields and makes decisions that are correct given whatever it observes. The scheduler writes a node assignment; the replica set controller writes pod counts; the endpoint controller writes addresses. None reads another's intent, and none needs to, because each is closing its own difference.

Where two controllers do write the same field, the system does not save them, and the result is the fighting described later in this lesson. The absence of coordination is a deliberate simplification that holds only as long as ownership is disjoint, which is why field ownership is a design question rather than an implementation detail.

The controllers that exist, and what each watches

Naming the set makes the pattern concrete, because the same shape repeats with different subjects.

The node lifecycle controller watches node leases and conditions and taints or evicts. The endpoint controllers watch services and pods and maintain address lists. The namespace controller watches namespaces with deletion timestamps and empties them. The service account and token controllers maintain credentials. The persistent volume controllers bind claims to volumes and handle reclaim. The job and cron job controllers manage batch work. The garbage collector maintains the ownership graph. The resource quota controller aggregates usage. And the certificate controllers approve and sign signing requests.

That is most of what the controller manager runs, and every one of them is the same loop over a different pair of desired and actual states. Once the pattern is visible, a cluster stops looking like a collection of features and starts looking like one mechanism applied repeatedly.

Events as a side channel

A controller's reasoning is often published as events, which are worth understanding as a deliberate design.

An event is a short record naming an involved object, a reason, a message and a count, written by the controller when something noteworthy happens. Events are deliberately not the controller's state: they are a human-readable trail, retained for about an hour and then discarded regardless of value.

The implication for a controller author is that events are for explaining, not for recording. Anything the controller needs later belongs in status. Anything a human will want while diagnosing belongs in an event, with the caveat that events are a shared cluster resource, so one event per state transition is correct and one per retry attempt is a denial of service against everyone else's events.

Why reconciles are kept small

There is a design preference for narrow controllers and the reasons are practical rather than aesthetic.

A controller that owns one transformation has a small state table, which is testable. Its permissions are narrow, which bounds the damage if it is compromised. It can be upgraded without touching anything else. And its failure stops one behaviour rather than many, which is the property that makes a cluster degrade rather than break.

The deployment chain in the next section is the canonical example: three controllers where one would have worked, chosen so that each is trivially correct. The cost is that diagnosing a problem means following the chain, which is why knowing the chain matters more than knowing any one controller in it.

What a controller must never assume

A short list of assumptions, each of which produces a real bug when relied upon.

That an event will arrive for every change, which fails on reconnect. That events arrive in order relative to other resources, which they do not across different watches. That its cache is current, which it is not during the window after a write. That it is the only writer, which it rarely is. That an object it created still exists, which it may not. And that its previous reconcile completed, since the process may have died mid-way.

Every one of those is handled by the same discipline: read current state, compute the difference, act idempotently, and never carry assumptions between reconciles.

Where the term reconcile comes from

One analogy is worth keeping because it predicts behaviour correctly.

A thermostat does not react to the event of a door opening. It reads the current temperature, compares it against the target, and turns the heating on or off. It does not know why the temperature changed, it cannot be confused by a change it did not observe, and it needs no memory of previous readings. Restarting it loses nothing.

Every controller in a cluster is that thermostat with a different sensor and a different actuator. The questions worth asking of any controller follow from the analogy: what is it measuring, what is it comparing against, what can it actuate, and how quickly does it read again. A controller whose author cannot answer those four is usually one that drifted into being edge triggered.