0%
AWS in Practice: Services, Limits, and Failure Modes
Account Foundations
Compute
Storage
Databases
Networking
Limits, Cost and Recovery
CloudFormation, CDK, and Terraform
Infrastructure as code describes cloud resources in files that are reviewed, versioned and applied by tools, rather than created by hand in a console. The benefits are well known: environments can be reproduced, changes are reviewed like code, history shows who changed what, and recovery after a disaster starts from a known definition. On AWS, three tools dominate: CloudFormation, the native service; the Cloud Development Kit, which generates CloudFormation from general purpose code; and Terraform, a multi cloud tool with its own state model. This lesson covers how each works, where each breaks, and how to run them safely.
Templates and stacks
A CloudFormation template is a YAML or JSON document listing resources, each with a logical ID, a type such as AWS::S3::Bucket and properties. Parameters let one template serve several environments, outputs expose values such as a load balancer's DNS name, and intrinsic functions such as Ref, GetAtt and Sub connect resources to one another. Deploying a template creates a stack: a set of resources that CloudFormation creates, updates and deletes together, tracking which physical resource corresponds to each logical ID so that later deployments know what to change.
CloudFormation works out the order of operations from references between resources. A security group that references a VPC is created after the VPC, and an instance that references the security group after that. Resources with no dependency between them are created in parallel. When a dependency is implicit, for example an application that needs a policy attached before it starts, a DependsOn attribute makes it explicit. Getting dependencies right matters most during deletion, when CloudFormation removes resources in reverse order, and a missing dependency can cause a delete to fail on a resource still in use.
A useful habit is to read a template the way CloudFormation does: as a list of logical IDs, each mapped to one real resource. The logical ID is the identity. Changing a property changes the resource, but changing the logical ID itself, for example renaming OrdersTable to OrderTable during a tidy up, tells CloudFormation that one resource has been removed and another added. It will create the new one and delete the old one, with no warning beyond the change set. Teams that treat logical IDs as permanent names, never renamed once deployed, avoid a whole class of accidental deletions.
Updates and replacement
When a template changes, CloudFormation compares it with the current stack and decides, per resource, whether the change can be applied in place, needs an interruption, or requires replacement. Replacement means creating a new physical resource and deleting the old one, and for stateful resources it can be catastrophic: changing the name of a DynamoDB table, the engine of an RDS instance or certain properties of an EBS volume replaces the resource, and with it the data. The documentation for each resource type lists which properties trigger replacement, and experienced teams check it for every stateful change.
Some replacements are surprising. Changing the subnet of an EC2 instance, the availability zone of a volume, or the name of a security group can all force replacement, and for a named resource the replacement can fail outright because the new resource cannot use a name the old one still holds. This is one reason many teams avoid setting physical names on resources at all, letting CloudFormation generate unique names. A generated name makes replacement possible without a name collision, and code that needs the name reads it from an output or parameter instead of hard coding it.
Change sets
A change set previews what an update will do before it runs: which resources will be added, modified or removed, and for modifications, whether replacement will occur. Running deployments through change sets, and reviewing them for any replacement or removal of stateful resources, is the single most effective safeguard against accidental data loss. Many pipelines fail automatically if a change set contains a replacement of a database, table or bucket, requiring an explicit approval to continue.
Change sets also reveal changes nobody intended. A template generated by a tool, or edited during a merge, can carry an unrelated modification, such as a reduced retention period on a log group or a removed tag used for cost allocation. Reviewers who only look at the lines they changed in the pull request can miss these, but the change set lists every resource that will be touched. Posting a short summary of the change set into the pull request, with replacements and removals at the top, puts the real effect of the change in front of the person approving it.
Rollback
If any resource fails to create or update, CloudFormation by default rolls the whole stack back to its previous state, deleting newly created resources and reverting modified ones. This keeps stacks consistent but can take a long time, and rollback itself can fail, for example when a resource was modified outside CloudFormation, leaving the stack in UPDATE_ROLLBACK_FAILED. Recovering requires continuing the rollback while skipping the problematic resources, then reconciling them by hand. Options such as disabling rollback during development, so a failed stack can be inspected and fixed, speed up iteration but should not be used in production.
Long rollbacks usually come from resources that wait for stabilisation, such as ECS services, Auto Scaling groups and custom resources. An ECS service that cannot reach a healthy state keeps CloudFormation waiting until a timeout, and only then does the rollback start. Enabling the ECS deployment circuit breaker makes the service fail fast and roll back on its own, and setting sensible timeouts on custom resources avoids hour long hangs when a Lambda backing the resource never responds. Fast failure keeps a bad deployment short and frees the stack for the fix.
Stack size and boundaries
Stacks have limits, such as 500 resources per stack and a maximum template size, and large stacks are slow to deploy and risky to change because every update touches one big unit. Splitting infrastructure into stacks along lifecycles works well: a network stack that rarely changes, a data stack holding databases and buckets with strict protection, and application stacks that deploy many times a day. Cross stack references through exports, or values passed through parameters and SSM Parameter Store, connect them. Exports create hard dependencies, though: an exported value cannot change while another stack imports it, which can block refactoring later.
Stack policies add another guard inside a single stack. A stack policy is a JSON document that denies certain update actions on certain logical IDs, for example denying Update:Replace and Update:Delete on the production database. Any update that would replace or delete those resources fails until someone deliberately overrides the policy for that one update. Combined with termination protection on the stack, which blocks deleting the whole stack, these settings make the dangerous operations require two explicit steps rather than one careless one.
StackSets and multiple accounts
Organisations with many accounts and regions use StackSets to deploy the same template everywhere, such as baseline IAM roles, logging configuration or guardrail resources. A StackSet integrated with AWS Organizations can deploy automatically to every new account added to an organisational unit. Rollouts are controlled with settings for how many accounts or regions to update in parallel and how many failures to tolerate before stopping, which allows a cautious, staged rollout of changes that would be dangerous to apply everywhere at once.
None of these safeguards help if engineers bypass them in a hurry. A deployment run from a laptop with --no-execute-changeset skipped, or a console update of a stack outside the pipeline, avoids every review step. Restricting who can call UpdateStack and ExecuteChangeSet in production, so that only the pipeline role can, closes that path and makes the reviewed change set the only route to production for every stack.
Review change sets for replacements and removals, especially of databases, tables and buckets, and block them in pipelines without explicit approval. Split stacks by lifecycle, keep stateful resources in protected stacks, and know how to recover a stack stuck in a failed rollback.