Automated Amazon EBS Snapshots

EBS snapshots is a powerful tool that can be used as a base for your backups. Snapshots are stored on S3 which provides
99.999999999% of durability. Wort to remember is that if an EC2 instance is up and running during snapshot creation, there may be applications working with data in memory or pending I/O operations that cannot be retrieved. To make you backups consistent you should quiet your application, unmount the volume or power off instance. EBS snapshots are not recommended for database servers and you should use native database backups.

If you want to automate snapshot creation process you can achieve it in several ways within AWS. First, lets gather sample requirements for snapshotting solution.

Requirements

• Volumes to be snapshotted: all EBS volumes with specific tag.
• Schedule: every 48 hours, between 01:00-02:00 AM.
• Retention time: 30 days.
• Snapshot creation/removal: fully automated.

Data Lifecycle Manager

You can use Amazon Data Lifecycle Manager (Amazon DLM) to automate the creation, retention, and deletion of snapshots taken to back up your EBS volumes.

It is very convenient solution that you can just turn on. It can snapshot all volumes with specific tag.

Unfortunately there is no possibility to set up snapshots every 48 hours, so it doesn’t fulfil our requirements. Maximum time between snapshots can be set to 24 hours. In the AWS news blog page they are planning to add it, but it was mentioned in June 2018 and still it is not added:

In the Works – As you might have guessed from the name, we plan to add support for additional AWS data sources over time. We also plan to support policies that will let you do weekly and monthly snapshots, and also expect to give you additional scheduling flexibility.

Source: https://aws.amazon.com/blogs/aws/new-lifecycle-management-for-amazon-ebs-snapshots/

CloudWatch Events

Another way how to automate snapshot creation is via CloudWatch Events.
You can run rules according to a schedule and choose a fixed rate to create a snapshot every few hours or use a cron expression to specify that the snapshot is made at a specific time of day.

This approach has some drawbacks though:

• You can set snapshots on predefined volumes only, tags doesn’t matter here.
• You cannot set automatic retention policy. CloudWatch will not remove your snapshot and you need to handle it in other way.

Step Functions and Lambda

Well, so how to achieve our goal to have fully automated snapshots every 48 hour, with 30 days retention policy and based on volume tags? In the end, solution is not that simple…

I used ready made solution that uses Step Functions in conjunction with CloudWatch Events, Lambda, and CloudFormation. It can be found on Gtihub. Of course it can be done in other ways, but this look quite nice, with DR option included.

First you must set up something to trigger snapshot creation, as it was described in previous sections, like CloudWatch Event rule. Then you must create another event rule that will be triggered on the completion of the snapshot creation. The target for the CloudWatch event is an AWS Step Functions state machine. The state machine coordinates different steps in the EBS snapshot management, including deleting snapshots past the retention period specified, and copying snapshots to a Disaster Recovery region. Another state machine is deployed in the DR region that performs similar steps for the snapshots that are copied into the DR region.

Materials

EBS explained
Automating the Amazon EBS Snapshot Lifecycle
Tutorial: Schedule Automated Amazon EBS Snapshots Using CloudWatch Events