0%
Observability and Delivery: Running Production Systems
The Signals
Metrics Systems
Logs and Traces
Reliability Targets
Incidents
Feature Flags and Progressive Delivery
A feature flag is a conditional that moves a decision out of the build and into runtime. That is a small mechanism with a large consequence: the moment a feature is behind a flag, shipping the code and turning the feature on become two separate events that can happen days apart and can be undone independently.
Most of the value people attribute to flags comes from that separation and not from the conditional itself. Deployment becomes boring because every deployment is a no op for users. Release becomes reversible because turning something off does not require a build. And the blast radius of a change becomes a dial rather than a switch, because the condition can depend on who is asking.
Four kinds of flag, with different lifetimes
Lumping all flags together is the first mistake, because the right policy for each kind is different.
A release flag exists to separate deploy from release for one feature. It is temporary by definition, it should be removed within weeks of the feature being fully on, and leaving it in place is the main source of flag debt.
An operational flag exists to let somebody change system behaviour without a deployment: shed load, disable an expensive code path, switch to a degraded mode. It is permanent on purpose, it is exercised during incidents, and it should be tested precisely because it is rarely used.
An experiment flag exists to split traffic and compare outcomes. Its lifetime is the length of the experiment, its correctness requirements are statistical rather than operational, and it needs sticky assignment in a way the other kinds do not.
A permission flag exists to gate functionality by plan, contract or account. It is not really a flag at all; it is product logic wearing a flag costume, and treating it as one means it will never be cleaned up and should never be in the same inventory as release flags.
What a flag costs you immediately
Every flag doubles the number of code paths through the region it guards, and the growth is multiplicative rather than additive when flags overlap. Three independent flags around the same request produce eight combinations, of which your tests probably exercise two: all off, and all on.
That combinatorial reality is the real limit on how many flags a team can carry, and it is why the useful discipline is not "fewer flags" but "fewer simultaneously live flags in the same code path". Two flags in different services cost almost nothing. Two flags nested inside the same function cost an afternoon every time somebody reads it.
The inventory nobody keeps
Flags accumulate because creating one is a one line change and removing one is a conversation. The predictable shape is an inventory where most entries are permanently on, still referenced, and owned by nobody.
The number worth tracking is not how many flags exist but how many are in a terminal state and still in the code. A flag that has been one hundred percent on for six months is not a flag; it is a branch that will never be taken, plus a dependency on a service that must answer before your request can proceed.
Progressive delivery is what you do with the dial
Once the decision is at runtime and can depend on the requester, you can expose a change to one percent of traffic, watch, and widen. That is progressive delivery, and it is worth being clear that it is a technique built on flags rather than a synonym for them.
The reason it matters is that it changes what you learn. A staging environment tells you whether the code works on the data and traffic you could arrange. One percent of production tells you whether it works on the data and traffic you actually have, including the account with four hundred thousand line items and the client still sending a deprecated field.
The flag is a contract between two teams
A flag that only one team touches is a local variable with extra steps. The ones that earn their complexity are the ones where the person who writes the code and the person who changes the exposure are different people, often on different schedules, and that is worth being explicit about because it changes what the flag has to provide.
The engineer who wrote the feature needs the flag to be precise: keyed on the right identifier, evaluated at the right place, and impossible to put a single user on both sides of. The person running the rollout needs something different. They need to change exposure without reading code, see what the current value is without asking, and know what happens if they set it to zero. Those are interface requirements, and a flag that satisfies the first set and not the second gets used exactly once, by its author, and then becomes debt.
This is also why naming matters more than it seems. A flag called new_checkout_v2 tells a person at three in the morning nothing about whether turning it off is safe. A flag called checkout_use_new_pricing_service tells them what moves and roughly what breaks. The name is the only documentation most flags will ever have, so it should describe what the flag controls rather than which project created it.
The same reasoning applies to who is allowed to change it. A rollout flag that anybody can set to one hundred percent from a web interface is a production change with no review, which is appropriate for a rollout and inappropriate for a flag that gates access to a payment path. Flag systems that let you require a second approver on specific flags are worth the configuration effort for the small set where it matters.
Naming conventions help here only if they are enforced at creation time, since a convention that lives in a wiki page is a convention that half the flags will ignore within a year.
What flags do not fix
Three things, and each one burns teams that expected otherwise.
A flag does not make a schema change reversible. If the new code path wrote rows in a new shape, turning the flag off stops new writes but leaves the rows, and the old path has to tolerate them. Data changes need their own reversibility story, which the previous lesson on migrations covered, and the flag is irrelevant to it.
A flag does not isolate failure in a shared dependency. If both paths hit the same database and the new one issues a query that locks a hot table, the one percent cohort takes down the hundred percent cohort. Progressive exposure limits who sees the new behaviour, not who is affected by its side effects.
A flag does not substitute for a decision. A feature that has been at fifty percent for four months because nobody wants to own the choice is worse than either outcome, because you are maintaining both paths and learning nothing.
Where the flag check belongs
A flag evaluated in one place per feature is maintainable. The same flag evaluated in eleven places across four services is a distributed consistency problem, because those eleven evaluations can disagree within a single request if the value changes mid flight.
The pattern that holds up is to evaluate once at the edge of the request, attach the result to the request context, and pass it down. Every component then sees one consistent answer, the number of evaluation sites stays at one, and removing the flag later is a single deletion rather than a search.