0%
AWS in Practice: Services, Limits, and Failure Modes
Account Foundations
Compute
Storage
Databases
Networking
Delivery and Visibility
Multi Region and Disaster Recovery
Most AWS architectures run in one region across several availability zones, which protects against the failure of a data centre or a zone. A region wide problem, a regional service outage, a corrupted database, deleted data or a ransomware attack needs a different kind of protection. Disaster recovery is the plan for those events: how much data the business can lose, how long it can be down, and what infrastructure and practice it takes to meet those numbers. This lesson covers the objectives, the four common strategies, replicating data between regions, moving traffic, performing a failover, and testing the whole thing so it works when needed.
RTO and RPO
Two numbers define a recovery requirement. The recovery time objective, RTO, is how long the service may be unavailable after a disaster before the impact becomes unacceptable. The recovery point objective, RPO, is how much data, measured in time, the business can afford to lose: an RPO of one hour means losing up to the last hour of writes is acceptable. These are business decisions, informed by the cost of downtime and lost data, and different parts of a system often warrant different objectives. A payment ledger may need an RPO of seconds, while a recommendations model can be rebuilt from scratch over a day.
Setting the objectives well takes a conversation with the business rather than an engineering guess. The useful questions are concrete: what happens to revenue, customers and contracts for each hour the system is down, and what happens if the last ten minutes, hour or day of data disappears. Some answers are surprising. An internal tool may turn out to block a warehouse from shipping, making its downtime expensive, while a customer facing feature may be tolerable to lose for a day. Writing the agreed RTO and RPO next to each system in the service catalogue, along with who agreed them and when, makes the numbers something the organisation owns and revisits rather than an assumption buried in an architecture document that nobody reads after launch.
Four strategies
AWS describes four strategies along a spectrum of cost and recovery speed. Backup and restore keeps backups in another region and rebuilds everything after a disaster, which is cheapest but takes hours. Pilot light keeps data replicated and the core infrastructure defined but switched off or minimal, ready to scale up, giving recovery in tens of minutes. Warm standby runs a smaller but fully working copy of the system in the second region that can scale up and take traffic within minutes. Multi site active active serves traffic from several regions at once, so a regional failure only removes capacity, with recovery in seconds to minutes and the highest cost and complexity.
The strategies differ mostly in what is already running when the disaster happens. In backup and restore, nothing runs in the second region except storage for backups. In pilot light, the data layer runs, with databases replicating continuously, but application servers and most other infrastructure are off or absent, defined in code and ready to deploy. In warm standby, every tier runs at reduced size, so the system can serve real requests immediately and only needs to scale. In active active, every region runs at enough size to carry its share, and often enough headroom to absorb another region's share as well. The more that is already running, the less has to go right during the stressful minutes of a recovery, which is the real reason faster strategies are more reliable as well as quicker.
Matching strategy to objective
The strategy follows from the objectives and the cost of meeting them. An RTO of a day and an RPO of a day can be met with nightly backups copied to another region. An RTO of an hour needs infrastructure as code that can rebuild quickly and data that is already replicated, which is pilot light. An RTO of minutes needs running capacity in the second region, which is warm standby. Near zero downtime needs active active. Each step up costs more, both in infrastructure and in engineering effort, so organisations usually apply the expensive strategies only to the systems whose downtime costs the most.
Active active designs carry costs beyond infrastructure. Every data store must handle writes from several regions, which means either partitioning data so each item has a home region or accepting conflict resolution rules such as last writer wins. Features that depend on global ordering or uniqueness, such as usernames, inventory counts or financial balances, need careful design. Deployments must roll out region by region without breaking compatibility between versions running at the same time. Monitoring must compare regions as well as watch each one. Teams sometimes adopt active active for resilience and find most of their effort goes into these consistency problems, so it is worth confirming that the business case really requires recovery in seconds before taking them on.
Backup and restore in practice
Backup and restore sounds simple, but restoring a whole system is slow and full of surprises. Large databases take hours to restore from snapshots, and the restored volumes are slower until their blocks are loaded. Infrastructure must be recreated from code, which only works if the code is complete and has been tested in the recovery region. DNS records, certificates, secrets and configuration all need to exist there. Teams that have never practised often find that a restore estimated at four hours takes ten. For systems where that is acceptable, it remains the most cost effective strategy, and every system needs backups regardless of what other strategy it uses.
Restore speed can be improved without changing strategy. Keeping infrastructure code deployable to the recovery region, with a pipeline that has been run there recently, removes most of the rebuilding uncertainty. Smaller databases restore faster, so archiving old data reduces restore time as well as cost. Fast snapshot restore for EBS removes the slow first read penalty on restored volumes, at an hourly price. Restoring in parallel rather than sequentially, and automating the order of steps, compresses the timeline further. Each restore exercise should record how long every step took, because the slowest step is the one to work on next, and that is rarely the step a team would have guessed in advance.
Disasters that are not regional
Many of the most likely disasters do not involve a region going down at all. An engineer drops a production table, a deployment corrupts data, a compromised credential deletes resources, or ransomware encrypts a database. Replication faithfully copies all of these to the other region within seconds, so a multi region active active system offers no protection against them. Point in time recovery, versioned backups, deletion protection, and backups held in a separate account that production credentials cannot touch are the defence. A complete recovery plan covers both the region that fails and the data that is destroyed while every region is healthy.
Point in time recovery deserves particular attention because it is the fastest way back from the most common data disasters. RDS, Aurora and DynamoDB can restore to any second within their retention window, typically up to thirty five days, into a new database or table. Restoring to a moment just before a bad deployment or an accidental delete recovers the lost data without touching the live system, and the restored copy can be used to repair only the affected records. Knowing how to perform that restore quickly, which requires knowing which tables to restore, how to compare them with production and how to merge the repaired rows back, is a skill worth practising before it is needed at three in the morning.
Set RTO and RPO per system from the cost of downtime and data loss, choose backup and restore, pilot light, warm standby or active active to meet them at acceptable cost, and remember that replication copies deletions and corruption, so isolated point in time backups are needed whatever the strategy.