0%
AWS in Practice: Services, Limits, and Failure Modes
Account Foundations
Compute
Storage
Databases
Networking
Limits, Cost and Recovery
Deployment Strategies on AWS
Every deployment is a controlled change to a running system, and most production incidents follow one. The strategy chosen for shipping new code decides how many users a bad release can reach, how quickly it is noticed, and how long it takes to undo. AWS offers several mechanisms, from rolling updates in ECS and Auto Scaling groups, to blue green and canary deployments through CodeDeploy, to weighted aliases for Lambda. This lesson covers how each works, what each protects against, and the parts that no deployment tool handles for you, such as database changes and configuration.
What a rolling deployment does
A rolling deployment replaces old instances or tasks with new ones in batches. In ECS, the service's deployment configuration sets minimumHealthyPercent and maximumPercent. With a desired count of ten, a minimum of 100 percent and a maximum of 200 percent, ECS starts up to ten new tasks, waits for them to pass health checks, and then stops old ones. With a minimum of 50 percent, it may stop five old tasks first to make room, which is cheaper but halves capacity during the deployment. Auto Scaling groups do the same with instance refresh, replacing a percentage of instances at a time.
Deployment speed matters too. Each batch must start, pass health checks and register with the load balancer, and old tasks must drain their connections before they stop. The target group's deregistration delay, three hundred seconds by default, keeps old tasks alive while in flight requests finish, which is safe but slow for services whose requests take milliseconds. Lowering it to something close to the longest real request, such as thirty seconds, shortens each deployment considerably. Health check intervals and thresholds add up similarly, so a service with a ten second interval and five healthy checks needed waits nearly a minute per batch before it can continue.
Capacity during the roll
The settings trade speed and cost against capacity. Starting new tasks before stopping old ones needs spare room in the cluster, either free capacity on EC2 instances or simply more Fargate tasks billed for a few minutes. Stopping old tasks first needs no spare room but reduces capacity, and if the deployment happens at peak traffic the remaining tasks may be overloaded. A common failure is a deployment during a busy period with a low minimum healthy percent, which causes latency spikes that look like a bug in the new code but are really a capacity shortfall.
Capacity planning for deployments also includes quotas and placement. On EC2 backed clusters, starting extra tasks needs free CPU and memory on existing instances, or a capacity provider that adds instances, which takes minutes. On Fargate, a large service starting a hundred extra tasks at once can hit the account's task launch rate or vCPU quota, and the deployment stalls with tasks stuck in provisioning. Checking these limits for the largest services, and requesting increases ahead of time, prevents a deployment from failing for reasons that have nothing to do with the code being shipped.
Health checks decide progress
A rolling deployment only moves forward when new tasks are healthy, so health checks are the deployment's safety mechanism. The load balancer's target group health check confirms a task responds, and container health checks confirm the process is alive. A health check that only returns 200 from a static path will pass even when the application cannot reach its database, so the deployment will roll a broken release across every task. Better checks confirm the service can do useful work, without becoming so strict that a slow dependency marks every task unhealthy at once.
The grace period matters as well. A new task that takes ninety seconds to warm its caches will fail a health check that starts after ten, and ECS will kill it and start another, which also fails. Setting healthCheckGracePeriodSeconds on the service gives tasks time to start before failed load balancer checks count against them. Applications that start slowly can also report not ready until they are, and the readiness signal should flip only when the task can actually serve a request at normal latency, not merely when the process has started.
Mixed versions
During a rolling deployment, old and new versions serve traffic at the same time, sometimes for many minutes. Every change must therefore be compatible in both directions: a new version must handle requests and messages produced by the old one, and the old version must tolerate data written by the new one. An API that renames a field, a queue message with a new required attribute, or a cache entry in a new format can all break during the overlap. This constraint applies to every gradual strategy, and it is the reason database and schema changes need their own discipline, covered later in this lesson.
Message formats deserve particular care because queues hold messages for a long time. A message written by the new version may sit in an SQS queue for minutes or days and be consumed after a rollback, by the old version. A message written by the old version may arrive at the new one long after the deployment completed. Versioned message schemas, optional new fields with defaults, and consumers that ignore unknown fields make these cases harmless. The same applies to anything persisted, from cache entries to files in S3, that one version writes and another reads.
The circuit breaker
ECS deployment circuit breaker watches a deployment and marks it failed when tasks repeatedly fail to start or fail health checks. With rollback enabled, ECS automatically returns the service to the last completed deployment. Without the circuit breaker, a service with a bad image can keep launching and killing tasks indefinitely, and a CloudFormation update waiting on that service can hang for hours. Enabling the circuit breaker with rollback is a low cost default for every ECS service, and CloudWatch alarms can also be attached so a deployment rolls back when error rates rise even though tasks are healthy.
The circuit breaker counts failures to reach a steady state, so it does not notice a release that runs fine but returns errors to users. That gap is covered by attaching CloudWatch alarms to the deployment configuration. ECS watches the listed alarms while the deployment is in progress and for a bake period afterwards, and rolls back if any of them enters the alarm state. Alarms on the target group's 5xx rate and on p99 latency are the usual starting points, and the bake period should be long enough for real traffic to exercise the new code paths.
When rolling is enough
Rolling deployments suit many services: they need no duplicate environment, they are built into ECS and Auto Scaling, and with good health checks and a circuit breaker they catch releases that crash or fail checks. Their weakness is releases that start and pass health checks but behave badly, such as returning wrong results, raising errors on one endpoint or slowing down under real load. By the time those problems are visible in metrics, the rollout may already be complete, and rolling back means another full rolling deployment of the previous version.
Whatever the strategy, every deployment should be easy to identify in metrics and logs. Tagging tasks with the image digest or release version, logging the version on each request, and annotating dashboards with deployment start and end times make it possible to tell whether a change in error rate began with a release. Without that, teams waste the first minutes of an incident establishing what changed and when, time that a simple version label on every log line and metric would have saved.
Choose minimum and maximum percentages that keep capacity at peak, write health checks that prove useful work, enable the ECS circuit breaker with rollback, and make every change compatible with the old version running beside it.