The name is unfortunate. Chaos engineering sounds like a licence to break production on a Friday afternoon, and that reading has cost the practice more credibility than any failed experiment ever did. The actual discipline is narrower and considerably more boring: you state what you believe about how a system responds to a specific failure, you cause that failure under controlled conditions, and you find out whether you were right.

That framing matters because it determines what counts as success. An experiment that injects a failure and nothing bad happens is not a wasted afternoon. It is a confirmed hypothesis, and it means the next person who argues that the system cannot survive a lost replica has evidence to argue against. An experiment that causes an outage is also not a failure of the practice. It is the discovery of a weakness that was already there, found at a moment when the people who understand it were watching, rather than at three in the morning during a product launch.

Breaking things is not the goal

The distinction between an experiment and an act of vandalism is the hypothesis. If you terminate a node and you cannot say in advance what you expect to observe, you have not run an experiment; you have generated an incident with extra steps. The value of chaos work comes almost entirely from the gap between what you predicted and what happened, and that gap only exists if you wrote the prediction down first.

This also means the interesting candidates are not the dramatic failures. Everybody knows what happens if the primary database disappears, because that scenario gets rehearsed in architecture reviews until it is folklore. The productive experiments target the places where the team's mental model is vague: what happens when the identity provider is slow rather than down, what happens when one availability zone's network is degraded rather than partitioned, what happens when a dependency returns correct responses at ten times its usual latency. Those are the cases where a team's confidence is highest and its evidence is thinnest.

Steady state is a measurement, not a feeling

Every experiment needs a definition of normal that is specific enough to be violated. "The site works" is not usable, because it is true during a slow partial degradation that is costing real money. "Checkout completion rate stays within two percent of the trailing hour, and p99 checkout latency stays under eight hundred milliseconds" is usable, because a graph can tell you whether it held.

The best steady state metrics are business facing rather than infrastructure facing. Orders per minute, logins per minute, messages delivered per minute: these are the quantities the experiment is ultimately asking about, and they are robust to the thing you just broke being irrelevant. Infrastructure metrics make poor steady state definitions precisely because the experiment is designed to disturb them. If you terminate a third of the fleet, CPU utilisation on the survivors will change, and that tells you nothing about whether users noticed.

The hypothesis has to be falsifiable

A usable hypothesis names the injection, the steady state metric, the expected magnitude of the deviation and the time within which the system should return to normal. "If we terminate one of the three cache nodes, checkout completion stays within two percent and p99 latency rises by no more than forty percent, recovering within ninety seconds as the client library redistributes keys" is a hypothesis. Every clause in it can be checked against a graph afterwards, and each one is a separate opportunity to be wrong.

Teams discover something uncomfortable the first time they write one of these properly: they often cannot predict the magnitude at all. They are confident the system survives, and they have no idea whether the latency impact is five percent or five hundred. That uncertainty is itself the finding, and it is worth recording before the experiment runs, because afterwards everybody will remember having expected whatever happened.

Blast radius is a design input, not a hope

Controlled means two things. First, the injection is scoped: one node and not the fleet, one percent of requests and not all of them, one zone and not the region, a single tenant's traffic and not everybody's. Second, you can undo it faster than the system can degrade. An experiment that depends on a human noticing a dashboard and typing a command is not controlled, because the dashboard is thirty seconds behind and the human is in a meeting.

In practice this means every experiment carries an abort condition written in the same specific terms as the hypothesis, and an automated mechanism to apply it. If checkout completion drops more than five percent, or error rate exceeds one percent, the injection stops on its own. The abort condition is not an admission that the experiment might be dangerous; it is what makes running the experiment a reasonable thing to do at all.

Why production, and why that is not recklessness

The uncomfortable part of the practice is that staging environments mostly cannot answer the questions you care about. The failure modes that hurt are the ones that depend on real traffic shapes, real data volumes, real cache hit rates, real client retry behaviour and real dependency latency. A staging environment has none of those, and the experiments you run there confirm that your configuration is syntactically plausible rather than that your system is resilient.

That does not mean the first experiment belongs in production. The sensible progression is to run a new injection type in a pre-production environment to shake out the tooling, then in production against a small scoped slice during a low traffic window with the team watching, then on a schedule once the result is boring. What you are building up is not courage but confidence in the apparatus: the injection works as documented, the abort works, the observability shows you what happened.

What you learn that a test cannot tell you

An integration test asserts that a known input produces a known output. A chaos experiment asks what the system does in a state nobody designed for, and the answers tend to be about interactions rather than components. The circuit breaker opened correctly and the fallback path had no cache warming, so the fallback was slower than the failure it replaced. The retry logic was correct in isolation and three layers of it compounded into a sixteen times amplification. The node drained gracefully and the load balancer's health check interval meant traffic kept arriving for eleven seconds after it stopped accepting.

None of those are bugs in a unit. They are properties of the assembled system under a condition the assembly was never exercised in, and no amount of component level testing surfaces them. This is the honest argument for the practice: it is the only technique that tests the parts of the system that exist between the parts.

Where a programme usually starts, and where it stalls

The first experiment in most organisations is terminating an instance, because the tooling is trivial and the expected result is well understood. The second is usually injecting latency into a dependency, which is where the surprises start. The third, if the programme survives, is something about a shared resource: a connection pool, a queue, a rate limit.

Programmes stall for predictable reasons. The injection tooling only works in one environment. The results live in somebody's notebook and nobody can find last quarter's findings. The experiments are run by one enthusiast and stop when that person changes team. Each of these is an organisational problem rather than a technical one, which is why the later sections of this lesson spend as much time on how results are recorded and acted on as on what to inject.