0%
Containers and Kubernetes: From cgroups to Operators
What a Container Actually Is
Working with Docker
Kubernetes Architecture
Workloads
Scheduling and Resources
Networking and Storage
Custom Resources and the Controller Pattern
Extending the cluster's object model is a single declarative act: you register a kind, the interface server starts serving it, and from that moment it behaves like every built-in kind, with the same validation, authorisation, versioning and watch semantics. Getting the definition right is most of the work, because almost all of it is immutable in practice.
What a definition buys you
One object produces a complete interface surface.
Registering a definition gives you a set of paths, the full verb set including watch, list pagination, field and label selectors, authorisation integration, audit logging, admission control, and storage in the cluster's datastore with optimistic concurrency. Every client and tool that works with built-in kinds works with yours.
That is a large amount of machinery for one manifest, and it is why the pattern dominates: the alternative, a separate service with its own interface and storage, re-implements all of it and integrates with none of the cluster's authorisation or tooling.
Naming, and why it is hard to change
One block of names is effectively permanent.
A definition declares a group, a plural and singular name, a kind, optional short names, and a scope. The object's name is the plural plus the group. Changing any of it means a new definition and a migration of every object, because the stored objects are keyed by the path.
So the group should be a domain you control, the kind should read naturally in a manifest, and short names should not collide with anything built in. Spending twenty minutes on names before the first version ships is cheaper than any later alternative, and reviewing them with someone who will write the manifests is the useful check.
Scope decides the authorisation model
One field has consequences far beyond storage.
A resource is namespaced or cluster-scoped. Namespaced objects live in a namespace, are deleted with it, and are reachable by namespace-scoped permissions, so a team with access to its namespace can create them.
Cluster-scoped objects need cluster-wide permissions, which means either a platform team creates them or every team gets cluster-level access, and that second outcome is how extension objects become a privilege escalation path. The rule of thumb is that anything a team should manage for itself is namespaced, and anything representing cluster configuration is cluster-scoped, and the choice cannot be changed later.
The schema is the validation
One section does most of the quality work.
The definition carries a structural schema per version, describing properties, types, required fields, formats, and constraints like minimum, maximum, pattern and enumerated values. The interface server rejects objects that do not conform, before any controller sees them.
Writing it properly is the single highest-value investment in a custom resource, because it moves errors from a controller's log, where a user may never look, to the moment of submission, where the message is attached to the thing they just typed. A schema that permits any object is a schema that guarantees confusing failures later.
Validation beyond types
One newer facility expresses real rules.
Expression-based validation rules attached to the schema evaluate against the object, so you can require that one field is less than another, that a field is immutable after creation by comparing against the old object, or that a list has unique entries. The rules are cost-limited so they cannot be expensive.
This closes the gap that previously required an admission webhook for simple invariants, which means a definition can enforce most of its own rules with no running code. Reaching for a webhook should now be the exception, reserved for rules needing information the object does not contain.
Pruning and what happens to unknown fields
One default behaviour surprises people once.
With a structural schema, fields not described in it are removed on write, silently. A manifest with a misspelled field is accepted and the field disappears, so the controller never sees it and the user's intent is lost without an error.
Describing every field is therefore not optional, and the characteristic bug report is that a setting has no effect, where the cause is a typo pruned on submission. Comparing the submitted manifest against the stored object is the diagnostic, and it takes one command.
Printer columns and the user experience
One small section determines whether the resource is pleasant.
Printer columns declare which fields appear in a listing, with a name, a type and a path into the object. Without them a listing shows only the name and age, which is nearly useless for anything with interesting state.
Three or four columns showing the important specification values and the current status make the resource legible at a glance, and this is the cheapest usability improvement available. The categories field is worth setting too, since it is what makes the resource appear in aggregate listings alongside built-in kinds.
Subresources, and why status is separate
One pair of subresources changes the permission model.
Enabling the status subresource splits the object: a write to the main path cannot change status, and a write to the status path cannot change the specification. Enabling the scale subresource exposes a replica count at a standard path.
The status split is important for two reasons. It lets authorisation grant users the specification and controllers the status, which is the correct division. And it means a controller updating status does not conflict with a user editing the specification, removing a whole class of write conflicts. Any resource with a controller should enable it.
Specification and status as a contract
One convention is worth following strictly.
The specification is the user's desired state and only the user writes it. The status is the observed state and only the controller writes it. The controller never writes the specification, even to normalise or default it, because that makes the object fight whatever manages the manifest.
Breaking this is the commonest design error in a first custom resource, and the symptom is a controller and a configuration management tool overwriting each other indefinitely. Defaults belong in the schema, not in a controller's write-back.
Conditions as the status vocabulary
One standard shape makes status readable.
Conditions are a list of entries with a type, a boolean-like status, a reason, a message and timestamps. The convention is a Ready condition summarising usability, plus specific conditions for the interesting sub-states, each with a machine-readable reason and a human-readable message.
Following it means existing tooling can wait on your resource and operators can read it without documentation. An ad-hoc status with a free-text phase field works initially and becomes a parsing problem for everyone downstream, which is why the condition shape is worth adopting from the first version.
Aggregated interfaces, and when they are the answer
One alternative exists and is rarely right.
Instead of a definition, an extension interface server can be registered to serve a group, with its own storage and logic. That is the right choice when objects must not be stored in the cluster's datastore, when they are numerous or high-churn enough to threaten it, or when the semantics genuinely differ from object storage.
The cost is an entire service to build, secure and operate, with its own certificates and availability. For nearly every use, a definition is correct, and the honest test is whether you can name a specific requirement that a definition cannot meet.
Labels, annotations and where extra data goes
One convention keeps the object clean.
Information that tools select on belongs in labels, which are indexed and selectable but constrained in format and length. Information that is merely carried belongs in annotations, which are unconstrained and not selectable.
The temptation with a custom resource is to use annotations as an informal extension of the specification, which defeats the schema entirely: an annotation is a string with no validation, no documentation and no pruning, so a feature expressed that way has none of the safety the definition provides. If it is part of the interface, it belongs in the schema with a type and a constraint.
A strong answer treats the schema as the product: structural validation plus expression rules move errors to submission time, where the message lands on the manifest the user just wrote, rather than into a controller log nobody reads. The characteristic mistake is a permissive schema, since unknown fields are pruned silently, so a misspelled setting is accepted and vanishes and the bug report says the setting has no effect.