Amazon’s EC2, EBS And MySQL Backups – Part 2 – Creating a Snapshot Of The XFS MySQL Volume

In Part 1 I explained how to move your MySQL instance onto an XFS volume. Now I show you the frankenstein script I created, stealing bits and adapting from other people’s attempts.

Below is the full script. It’s quite self explanatory. It firstly collects the instance i.d. which it’ll then use in all the other calls. It then finds the MySQL volume based on the device mounted where the data files are. To get an adequate snapshot it then connects the the MySQL daemon and causes the databases to go read only before it freezes the filesystem. Next it calls to the EC2 API for a snapshot to be generated of that volume and once the call has been completed (which doesn’t take long) it unfreezes the filesystem and restores the databases to read/write.

That’s the hard work done. The next step is to loop through the remaining volumes and just take a straight snapshot.

Finally it finds the snapshots for all the volumes and clears out any over the amount defined by MY_SNAPSHOTS.

#!/bin/bash

export EC2_HOME=<put your EC2 tools home here>
export JAVA_HOME=<put your jre home here>
export EC2_URL=<put the amazon api url here e.g. https://ec2.eu-west-1.amazonaws.com>
export PATH=$PATH:$EC2_HOME/bin
export EC2_CERT=<path to amazon cert here>
export EC2_PRIVATE_KEY=<path to amazon key here>

MY_MYSQL_MOUNT_POINT=<mysql data directory here e.g. /var/lib/mysql>
MY_SNAPSHOTS=<days of snapshots you want to keey e.g. 7>

MY_MYSQL_VOLUME=$(cat /proc/mounts |grep ${MY_MYSQL_MOUNT_POINT} |awk {'print $1'})
MY_INSTANCE_ID=$(curl -s http://169.254.169.254/latest/meta-data/instance-id)

echo "Finding MySQL Volumes";
VOLUME_LIST=$(ec2-describe-volumes --filter "attachment.instance-id=${MY_INSTANCE_ID}" | grep ATTACHMENT | grep ${MY_MYSQL_VOLUME} | awk {'print $2'})

for VOLUME in $(echo $VOLUME_LIST); do
    VOLUME_NAME=$(ec2-describe-volumes $VOLUME | sed -n 3p | awk {'print $5'})
    echo "Connecting and locking MYSQL"
    mysql -u root <<-EOFMYSQL
    flush tables with read lock;
    show master status;
    system echo "Freezing system for backup"
    system xfs_freeze -f /var/lib/mysql
    system echo "Creating MySQL Snapshot..."
    system ec2-create-snapshot -d "$VOLUME_NAME daily backup" $VOLUME
    system echo "Unfreeze XFS"
    system xfs_freeze -u /var/lib/mysql
    unlock tables;
    exit
    EOFMYSQL
done

echo "Finding Other Volumes";
VOLUME_LIST=$(ec2-describe-volumes --filter "attachment.instance-id=${MY_INSTANCE_ID}" | grep ATTACHMENT | grep -v ${MY_MYSQL_VOLUME} | awk {'print $2'})

for VOLUME in $(echo $VOLUME_LIST); do
    VOLUME_NAME=$(ec2-describe-volumes $VOLUME | sed -n 3p | awk {'print $5'})
    echo "Creating Snapshot..."
    ec2-create-snapshot -d "$VOLUME_NAME daily backup" $VOLUME
done

echo "Clearing Old Snapshots";
VOLUME_LIST=$(ec2-describe-volumes --filter "attachment.instance-id=${MY_INSTANCE_ID}" | grep ATTACHMENT | awk '{ print $2 }')

for VOLUME in $(echo $VOLUME_LIST); do
    NUMBER_SNAPSHOTS=`ec2-describe-snapshots | grep $VOLUME | grep -v "Created by CreateImage" | wc -l`
    echo "Snapshots: $NUMBER_SNAPSHOTS"
    if [ "$NUMBER_SNAPSHOTS" -gt "$MY_SNAPSHOTS" ]; then
        ec2-describe-snapshots --filter "volume-id=$VOLUME" | grep -v "Created by CreateImage" | sort -k 5 | head -$(($NUMBER_SNAPSHOTS-$MY_SNAPSHOTS)) | awk '{print "Deleting: $2; system("ec2-delete-snapshot " $2)}'
    fi
done

echo "Done!"

Amazon’s EC2, EBS And MySQL Backups – Part 1 – Moving To XFS

After the recent outage at Amazon’s Ireland data centre I decided it was time to actually start using snapshot features they kindly provide. If I had implemented an automated daily snapshot routine it would have meant my server could have been back up and running, in a different availability zone, within minutes of the issue becoming apparent rather than hoping for the best for almost a day and then restoring from a combination of a casual snapshot and some jungle disk backups.

Most of my volumes are pretty snapshottable without issue. The problem volume was my EXT2 MySQL volume. Once snapshotted I have no idea what state the data is, the server could have been half way through updating a series of foreign key related fields which will obviously leave the daemon in a confused state. I don’t want to be shutting down apache and MySQL every time I snapshot so I needed a new solution.

A quick google pointed me towards XFS and several pages provided me with a few scenarios/example scripts which I merged into the setup/script I currently use.

The server is Debian based so you’ll need to adapt your package manager commands as needed.

Create The MySQL Volume (XFS)

Install the XFS tools if they aren’t already available and load the module:

aptitude install xfsprogs
grep -q xfs /proc/filesystems || sudo modprobe xfs

Create a new EBS volume through the AWS control panel and attach the volume to your instance, I will use ‘/dev/sdh’ as the device for the purpose of this post.

Create an XFS filesystem on the new partition. This is the first bit that tripped me up. On older kernels the default log version of 2 for XFS filesystems can cause a kernel panic so I had to specify version 1 to be able to use the volume at all. You can try without ‘-l version=1′ but if the machine hangs while trying to access the new volume’s filesystem then give it a go:

mkfs.xfs -l version=1 /dev/sdh

Mount the new volume ready for copying the data over. My MySQL data is stored in /var/lib/mysql so I mounted it in /var/lib/mysql-new for simplicity. Once that’s done you will need to shut MySQL down and copy the data over:

mount -t xfs /dev/sdh /var/lib/mysql-new
/etc/init.d/mysql stop
cp -rp /var/lib/mysql/* /var/lib/mysql-new/

I am assuming that, like me, you had MySQL on it’s own volume in the next step, if not then obviously there’s no need to unmount the current MySQL data directory. Move it to /var/lib/mysql-old and create a new /var/lib/mysql

Unmount both volumes and remount the new volume where the old volume was and make sure the owner is correct.

umount /var/lib/mysql
umount /var/lib/mysql-new
mount -t xfs /dev/sdh /var/lib/mysql
chown -R mysql:mysql /var/lib/mysql

You will need to edit ‘/etc/fstab’ to make sure the correct volume is mounted:

/dev/sdh /var/lib/mysql xfs defaults 0 0

Restart MySQL and you should now have your MySQL database running on an XFS filesystem ready for snapshotting.

In Part 2 I will remind myself of the script I use to generate and manage the snapshots.