Failure scenarios/bottlenecks

It's possible that our cache will fail and then leave us without cached records as we fill up the new cache. For this reason it makes sense to run multiple instances of the cache and to replicate data between them.

When our batching service fails we'll lose the views that are in memory. We can attempt to build in a graceful shut down to the application logic to dump this info to blob storage if our needs prohibit us from losing that data. We can also increase how often the batching service writes to Dynamo to reduce this risk.

If our MySQL leader goes down then we'll have the read replicas elect a new leader automatically. If we need to scale up we can look into partitioning our MySQL database. That way we could still have a single leader for each partition of the data. We could partition on ranges of userIds or ranges of tinyUrls depending on how we generate tinyUrls. We'd want to make sure we choose a partition key with high cardinality to avoid hot spots.

Our API Gateway could fail so we'll want to have a primary-secondary failover setup there where we have a passive secondary ready to go when the primary heartbeat fails.

If our read or write services go down then we'll take them out of the pool of services we route to from our API Gateway. These services are stateless so we can seamlessly redirect traffic. If a request to one of those services 500s then we can retry it once on a healthy node.

If our batching service goes down then we'll lose the records in memory. We could use application logic to attempt graceful shutdowns where we dump the views in memory to blob storage that we read on startup. We could also write to dynamo more often to reduce the risk of data loss at the cost of batching less records that could have been batched.

If our cleanup job fails to run then subsequent runs should be able to handle the backlog. We can also run extra cron jobs at periods of low traffic in the case of any rare excessive backlogs

Submissions

Displaying Solutions with Scores Ranging from 1 to 10

Submissions

Displaying Solutions with Scores Ranging from 1 to 10