Don’t let CloudFormation delete your RDS!

Once an RDS instance is restored from a snapshot, the snapshot name must remain in the CF template on subsequent deployments. Removing it will tear down the database and create a new, empty database. For Aurora, both the cluster and all its instances will get replaced if the snapshot name changes or is removed.

Attempting to delete a database via CLI results in:

AmazonRDSException Please specify FinalSnapshotIdentifier or SkipFinalSnapshot (Service: AmazonRDS; Status Code: 400; Error Code: InvalidParameterCombination)

That is to say, you cannot programmatically delete a database from the RDS CLI without this ‘are you sure’ check. Rather it is the CF template where you are most likely to accidentally request a database deletion via replacement, by accidentally removing the snapshot id from the database definition.

In the Cloudformation CLI, you can list changesets for a CF stack and then describe the proposed changeset, which will show which resources are going to be replaced (replacement: true), which gives you a change to back out of a replacement if incorporated into the CF template execution process.

Do not rely on DeletionPolicy for RDS to mitigate against this. An RDS deletion policy of ‘retain’ on the database only retains it when the stack is deleted. It does NOT retain it when just the database is deleted. So DeletionPolicy Retain does not protect you against snapshot name changes tearing down your database.

An RDS deletion policy of snapshot will create a snapshot when the CF stack is deleted so you can at least restore to a new database. Snapshot will be of type ‘manual’. But again, this does not protect you against snapshot name changes tearing down your database, because it only works when the whole CF stack is deleted.

TLDR; a CF changeset that causes an RDS teardown and setup does not respect a ‘Retain’ or ‘Snapshot’ deletion policy, either for the cluster or any of the instances, because it is not technically a delete, but a modify.

Therefore, RDS databases should always protect themselves against deletion and modification by using IAM deny policies over and above RDS deletion policies.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s