AWS – Automating EBS Snapshots (Create & Delete after X Days) using Bash and Cron

This guide will show you how you can automate the AWS EBS volume snapshots on specified EC2 Instances whilst also checking and deleting any volume snapshots older than X days using a simple Bash script with AWS CLI.


Pre-Requisites

You will need some pre-requisites before starting:

  • AWS CLI
  • An EC2 Instance with EBS Storage
  • Python 2 version 2.6.5+ or Python 3 version 3.3+
  • Windows, Linux, macOS, or Unix

AWS CLI Service installation

Will be using Centos for this guide. Let’s start by installing the necessary services that will be required by the Bash script. More info here: AWS CLI OS Guides

  • SSH to your Linux box and run the following commands to install both python and the AWS CLI
    yum -y install epel-release
    yum -y install python-pip
    pip install --upgrade pip
    pip install awscli
  • Next you will need to configure the AWS Credentials so that anything that will be run on that machine can automatically pick up these configs.
    aws configure

awscli1

  • This will ask for the Access Key, Secret Key and Region for a specific AWS User. Provide the necessary details accordingly or use the following script:
    #!/bin/bash
    #Created by Elton Atkins - MintOpsBlog
    
    read -p "This will override any AWS credential file you might have on the current user. Do you want to continue? <y/N> " prompt
    if [[ $prompt == "y" || $prompt == "Y" || $prompt == "yes" || $prompt == "Yes" ]]
    then
    ######## CREATING AWS CONFIGURE FILE ########
    AWSFILE=".aws/credentials"
    mkdir .aws
    read -p "Please specify the AWS_ACCESS_KEY: " AWS_ACCESS_KEY
    read -p "Please specify the AWS_SECRET_KEY: " AWS_SECRET_KEY
    cat > $AWSFILE <<EOF
    [awsbkup]
    aws_access_key_id = $AWS_ACCESS_KEY
    aws_secret_access_key = $AWS_SECRET_KEY
    EOF
    
    else
     exit 0
    fi
  • The following is the minimum policy requirement for the AWS User to have. This is needed to create and delete volume snapshots on EC2 instances.
    {
    
     "Version": "2012-10-17",
     "Statement": [
    
     {
     "Sid": "Stmt1426256275000",
     "Effect": "Allow",
     "Action": [
     "ec2:CreateSnapshot",
     "ec2:CreateTags",
     "ec2:DeleteSnapshot",
     "ec2:Describe*",
     "ec2:DescribeSnapshots",
     "ec2:DescribeVolumes"
    
     ],
    
     "Resource": [
     "*"
     ]
     }
     ]
    }
  • Once that is done, you can then copy the below script and run it on the EC2 Instances. Make sure to create a cronjob to schedule the script to run depending on each use case. (below the script runs daily at 10pm)
    crontab -e

    awscli2

  • N.B. Change the Variable AWSPROFILE depending on how you configured the .aws/credentials. If you used the above script, the AWSPROFILE will be [awsbkup]
    #!/bin/bash
    #Created by Elton Atkins - MintOpsBlog
    
    ###COMMON VARIABLES
    HOSTNAME=$(hostname)
    INSTANCEID=$(wget -q -O- http://169.254.169.254/latest/meta-data/instance-id)
    DATE=$(date +%Y-%m-%d)
    RETENTIONDAYS=7
    RETENTIONDATE=$(date +%Y-%m-%d --date "$RETENTIONDAYS days ago")
    DESCRIPTION=$(hostname)-$DATE
    AWSPROFILE=awsbkup
    AWSREGION=$(wget -q -O- http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -e 's/\([1-9]\).$/\1/g')
    
    ######## CREATING NEW VOLUME SNAPSHOTS ########
     ###VOLUME FILE VARIABLES
     BKUPPATH="/tmp/snapshot"
     VOLUMEID="/tmp/snapshot/volumeid"
     VOLUMESTATS="/tmp/snapshot/volumestats"
     mkdir -p $BKUPPATH
     
     ###Get the volume details of the current instance
     aws ec2 describe-volumes --profile $AWSPROFILE --region $AWSREGION --filters Name=attachment.instance-id,Values="$INSTANCEID" | grep -i volumeid | awk '{ gsub(/[",]/,""); print $2 }' > $VOLUMESTATS
    
    ###Remove duplicate details from the volumestats file
     awk '!seen[$0]++' $VOLUMESTATS > $VOLUMEID
    
    ###Separate the Volume IDs in separate files
     INCREMENT=$(grep -c $ $VOLUMEID)
     for ((i=1;i<=$INCREMENT;i++));
     do
     sed -n "$i"p $VOLUMEID > $VOLUMEID$i
     done
    
    ###Create volume snapshots for backup
     for ((i=1;i<=$INCREMENT;i++));
     do
     aws ec2 create-snapshot --profile $AWSPROFILE --region $AWSREGION --volume-id "`cat $VOLUMEID$i`" --description $DESCRIPTION
     done
    
    ######## DELETING OLD VOLUME SNAPSHOTS ########
     ###Get snapshot information of each EBS volume. 
     ###All snapshots of each volume and also start time of when the snapshots were taken
     SNAPSHOTS="/tmp/snapshot/snapshots"
     SNAPSHOTID="/tmp/snapshot/snapshotid"
     SNAPSHOTTIME="/tmp/snapshot/snapshotstime"
     SNAPSHOT_ID_FILE="/tmp/snapshot/snapshot_id_file"
     SNAPSHOT_TIME_FILE="/tmp/snapshot/snapshot_time_file"
     
     for ((i=1;i<=$INCREMENT;i++));
     do
     aws ec2 describe-snapshots --profile $AWSPROFILE --region $AWSREGION --filters "Name=volume-id,Values=`cat $VOLUMEID$i`" | grep -i snapshotid | awk '{ gsub(/[",]/,""); print $2 }' >> $SNAPSHOT_ID_FILE
     aws ec2 describe-snapshots --profile $AWSPROFILE --region $AWSREGION --filters "Name=volume-id,Values=`cat $VOLUMEID$i`" | grep -i starttime | awk '{ gsub(/[",]/,""); print $2 }' | cut -c -10,25- >> $SNAPSHOT_TIME_FILE
     done
     
     SNAPSHOTIDINCREMENT=$(grep -c $ $SNAPSHOT_ID_FILE)
     SNAPSHOTTIMEINCREMENT=$(grep -c $ $SNAPSHOT_TIME_FILE)
     ###Gets all the snapshot ids of the data volumes and puts them in separate files
     for ((i=1;i<=$SNAPSHOTIDINCREMENT;i++));
     do
     sed -n "$i"p $SNAPSHOT_ID_FILE > $SNAPSHOTID$i
     echo -e -n "`cat $SNAPSHOTID$i` " >> $SNAPSHOTS$i
     done
     
     ###Gets all the snapshot start times and puts them in separate files
     for ((i=1;i<=$SNAPSHOTTIMEINCREMENT;i++));
     do
     sed -n "$i"p $SNAPSHOT_TIME_FILE > $SNAPSHOTTIME$i
     echo -e -n "`cat $SNAPSHOTTIME$i`" >> $SNAPSHOTS$i
     ### Checks each file inside the for loop to see whether it is older than 7 days
     if [[ $RETENTIONDATE -lt $(cat $SNAPSHOTTIME$i) ]];
     then
     echo -e "The snapshot `cat $SNAPSHOTID$i` with date `cat $SNAPSHOTTIME$i` is older then $RETENTIONDATE"
     echo -e "Will start deletion for `cat $SNAPSHOTID$i`"
     aws ec2 delete-snapshot --profile $AWSPROFILE --region $AWSREGION --snapshot-id $(cat $SNAPSHOTID$i)
     else
     echo -e "No snapshots older then $RETENTIONDATE"
     fi
     done
    
    rm -rf $BKUPPATH
     echo -e "Backups and cleanup done"
    ############################

Hope this guide helps you out, if you have any difficulties don’t hesitate to post a comment. Also, any needed improvements or mistakes done in the guides feel free to point them out.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s

Advertisements
Advertisements
%d bloggers like this: