Raw manifests are honest and they do not scale. The moment the same workload must exist in three environments with four differences each, or thirty services must share one standard shape, something has to generate the manifests, and every tool in this space is an answer to that one problem.

What actually duplicates

One count makes the pressure concrete.

A service needs a deployment, a service, a configuration map, a claim, an ingress rule, a disruption budget, a horizontal scaler and a service account. Across environments and services, the number of manifest files grows as the product:

With thirty services, four environments and eight objects each, that is nearly a thousand files, almost all identical. The differences are typically a dozen values: replica counts, resource sizes, hostnames, image tags, feature flags. So the real shape of the problem is a small amount of variation buried in a large amount of repetition, and every tool is a different way of factoring that out.

The three families

Three approaches cover everything in use.

Templating takes a parameterised document and substitutes values, producing manifests from a package plus a values file. Overlays take complete, valid manifests as a base and apply structured patches for each variant. Programmatic generation writes manifests from code in a general-purpose language, with types and abstraction.

Each trades differently. Templating is flexible and operates on text, so it can produce anything and can also produce nonsense. Overlays operate on structured objects, so they cannot produce invalid structure but can only express what the patch mechanisms support. Programmatic generation gives the full power of a language along with the full cost of a build.

Why text templating hurts

One property explains most of the pain.

A template is text with substitution and control flow, so the tool does not know it is producing an object graph. Indentation is significant in the output format but invisible to the template, control flow can produce syntactically broken documents, and no type checking is possible because nothing is typed until the output is parsed.

The practical consequence is the characteristic error: a conditional that emits a block at the wrong indentation, producing a document that parses into the wrong shape rather than failing. Rendering the output and reading it is the only reliable check, which is why rendering locally before applying is a habit worth building early.

Why overlays constrain you

One property explains their pain.

An overlay works on parsed objects, so the base must be valid and every patch is a structured operation. That makes malformed output impossible and makes expressing some changes awkward: conditionally including an object, generating a variable number of objects, or computing a value from two others.

So an overlay tool handles the common cases beautifully, substituting an image, adjusting replicas, adding a label everywhere, patching a field in one environment, and resists the unusual ones. Teams hitting that wall usually reach for a template, which is how clusters end up with both.

Drift, and what it really is

One concept is often stated loosely.

Drift is a difference between what the repository says and what the cluster holds. It has three sources, and treating them identically is a mistake. Someone edited the object directly, which is the case people mean. A controller or mutating webhook changed it, which is not drift but expected behaviour. Or the cluster defaulted a field the manifest did not set, which is also not drift.

So a tool reporting drift continuously is usually reporting the second and third kinds, and the response is to teach the tool to ignore fields it does not own rather than to fight the cluster. Distinguishing the three is the difference between a useful drift report and an alert everyone silences.

The idempotent-apply requirement

One property every approach must have.

Applying the same configuration twice must produce no change the second time. That sounds obvious and is routinely violated: a template that embeds a timestamp or a random value, a generated name that changes each run, a patch that appends rather than sets.

The symptom is a delivery pipeline that reports changes on every run with nothing having changed, which trains everyone to ignore its output. Checking that a second apply is a no-op is the single most valuable test for any configuration setup, and it takes one command.

Where environment differences belong

One decision shapes the whole repository.

Differences can live in values files per environment, in overlay directories per environment, in branches per environment, or in a parameter object in the cluster. The first two are equivalent in effect and both work. Branches per environment are the one to avoid, because the difference between environments becomes a merge rather than a file, so the question of what differs has no answer you can read.

The useful test is whether someone can answer what differs between staging and production by reading one file. If the answer requires a diff between branches or a tool invocation, the structure is wrong, and that becomes acute during an incident.

Configuration that is not manifests

One category needs separate thought.

Application settings, feature flags, connection strings and tuning parameters are not deployment configuration, even though they often arrive as configuration maps. They change on a different cadence, are owned by different people, and a change to one should not require a deployment.

So the sound arrangement is to keep genuinely deployment-shaped configuration in the repository and runtime settings somewhere a change does not roll pods, whether a settings service, a configuration map the application re-reads, or a feature flag system. Conflating them produces a deployment for every flag change, which is how deployment pipelines become the bottleneck for product decisions.

Secrets do not belong in the repository

One rule and three ways to honour it.

Secret values cannot be committed in plaintext, so either they are encrypted in the repository and decrypted at apply time, or they are referenced in the repository and fetched from an external store by a controller, or they are created outside the pipeline entirely and merely referenced.

Each works and they differ in failure mode. Encrypted-in-repository keeps everything in one place and makes rotation a commit, with the decryption key as the thing to protect. External references keep secrets out of the repository entirely at the cost of a controller and a dependency. Creating by hand is simplest and leaves no record of what exists, which fails on the day someone must recreate a cluster.

The composition question

One architectural choice is worth making deliberately.

Should every service have its own package, or should there be one shared package parameterised per service? A per-service package gives freedom and duplicates the standard. A shared package gives consistency and becomes a constrained interface everyone fights occasionally.

The arrangement that works in most organisations is a shared base for the standard shape, with a documented escape hatch for the services that genuinely differ, and a strong default that new services use the base. The failure mode of the other extreme, thirty independently maintained packages, is that a change to the standard requires thirty pull requests, which means the standard stops moving.

What to optimise for

One criterion beats the others.

The thing to optimise is how quickly someone unfamiliar can see what will be applied and why. Not expressiveness, not brevity, not the elegance of the abstraction. During an incident, the question is always what is actually deployed and what changed, and a setup that answers it in one command is worth a great deal of inconvenience elsewhere.

That criterion argues for fewer layers, rendered output committed or easily produced, and differences visible in files rather than computed. It argues against clever abstraction, deep template inheritance and anything requiring a mental model to read.