Backing up google cloud platform instances

I’ve used Google Cloud for a long time now, and using it often has made me mighty fond of Google’s gcloud command line API. It lets me backup my 200 instances I look over.
If you want to back up GCE instances, all you need is a windows or linux machine, either in cloud or outside, with gcloud sdk installed.

I won’t go over installing it yet, as there are plenty of writeups around, including on Google’s own documentation, that covers all operating systems. Being a 100% linux user, I’m not very windowsey. This writeup is written with a linux user in mind, but feel free to ask any question if you have any, I’ll try to help!

I’ll explain things as I go.

To backup GCE instances:

backing up google instances is super easy!

I’m in this for business, and I want to make sure my customers data is safe, I keep 30 daily backups (30 days of daily), 2 months of weekly backups, and 6 months of weekly backups of all my machines.

daily

[code]

#!/bin/bash

dt=date +%Y-%m-%d
echo $dt
#Delimiter to exclude is “|”
exclude_list=”NAME”
all_disks=gcloud compute disks list |awk '{print $1"="$2}'|grep -v $exclude_list
for line in echo -e $all_disks ;do
zone=echo $line |awk -F"=" '{print $2}'
disks=echo $line |awk -F"=" '{print $1}'
gcloud compute disks snapshot $disks –zone $zone –snapshot-name “internal-daily-$disks-$dt”
done
[/code]

weekly

[code]

#!/bin/bash

dt=date +%Y-%m-%d
echo $dt
#Delimiter to exclude is “|”
exclude_list=”NAME”
all_disks=gcloud compute disks list |awk '{print $1"="$2}'|grep -v $exclude_list
for line in echo -e $all_disks ;do
zone=echo $line |awk -F"=" '{print $2}'
disks=echo $line |awk -F"=" '{print $1}'
gcloud compute disks snapshot $disks –zone $zone –snapshot-name “internal-weekly-$disks-$dt”
done

[/code]

monthly:

[code]
#!/bin/bash

dt=date +%Y-%m-%d
echo $dt
#Delimiter to exclude is “|”
exclude_list=”NAME”
all_disks=gcloud compute disks list |awk '{print $1"="$2}'|grep -v $exclude_list
for line in echo -e $all_disks ;do
zone=echo $line |awk -F"=" '{print $2}'
disks=echo $line |awk -F"=" '{print $1}'
gcloud compute disks snapshot $disks –zone $zone –snapshot-name “internal-monthly-$disks-$dt”
done

[/code]

The cleanup script

My cleanups are manual, but this could be scripted

when I clean up I specify the date I want to delete:
example: ./vm_snapshot_deleter.sh 2018-02 2018-01 2017-12
would delete all machine snapshots, with the date string included.

This could be scripted just doing some date math. I’ll update this post if I feel froggy.

[code]

#!/bin/bash

#gcloud cleanup script by Justin Reiners
#cleans google cloud backups.
#will also use command line options as well.

echo “Usage ./vm_snapshot_delete.sh (date in YYYY-MM format)”
echo “Daily delete is assummed, but weekly and monthly can be specified

SEARCHSTRINGD=$1
SEARCHSTRINGW=$2
SEARCHSTRINGM=$3

echo $SEARCHSTRING

#start daily cleanup
gcloud compute snapshots list | grep internal-daily | grep $SEARCHSTRINGD | cut -f 1 -d ” ” | while read line; do gcloud compute snapshots delete $line –quiet ;done

#start weekly cleanup
gcloud compute snapshots list | grep internal-weekly | grep $SEARCHSTRINGW | cut -f 1 -d ” ” | while read line; do gcloud compute snapshots delete $line –quiet ;done

#start monthly cleanup
gcloud compute snapshots list | grep internal-monthly | grep $SEARCHSTRINGM | cut -f 1 -d ” ” | while read line; do gcloud compute snapshots delete $line –quiet ;done

[/code]

Backups SQL servers (Hosted mysql)

Google makes and auto deletes about 5-6 backups of hosted SQL, which didnt really work for us, as we like keeping longer retention times for our data.

The following script will allow you to back up 3 instances, databasedb, databasedb2, databasedb3, and will not auto delete. The delete script will follow:

[code]
echo “backing up databasedb…”
gcloud beta sql backups create –instance databasedb –description “production dbdb db $datedfile” –async
echo “done.”

sleep 2

echo “backing up databasedb2…”
gcloud beta sql backups create –instance databasedb2 –description “production dbdb db $datedfile” –async
echo “done.”

sleep 2
echo “backing up databasedb3…”
gcloud beta sql backups create –instance databasedb3 –description “production dbdb db $datedfile” –async
echo “done.”

[/code]

Trimming SQL backups

[code]
#!/bin/bash
#logfile trim cron scripts for cleaning google cloud SQL backups.
#this file can be run daily or weekly, it just keeps the bill down.
#pass sql instance name as first command line parameter.
#trimsql.sh

example:

./trimsql.sh development +60

will keep 60 days of backups for the “development” instance

Assign variables

backups=$2
outfile=”/tmp/sql-trim-output-file.tmp” # temp file for process use.
sql_instance=$1 # this is the master instance name on cloud SQL.
gcloud_location=”/usr/bin/gcloud” # gcloud executable location.
echo “”
echo “Cleaning up Google Cloud SQL instance named ” $1
echo “”
echo “trimming snapshots to: ” $2 ” snapshots, including autosnaps.”
#

gcloud needs to be installed and in a path accessable by the user. This user or account must have the appropriate permissions.

#

gcloud beta sql backups list (lists backups) of $sql_instance | skips deleted backups because you cant delete them twice

the tail command skips $dayskip amount | the cut command cuts the first column (f1) with a file delimited by spaces.

it then save the gcloud clensed output to a file we can ingest into the next step.

#
#$gcloud_location beta sql backups list –instance $sql_instance | grep -v DELETED | tail -n +$dayskip | cut -f1 -d ” ” > $outfile

$gcloud_location beta sql backups list –instance $sql_instance | grep -v DELETED | grep -v UNKNOWN_STATUS | grep -v OVERDUE | tail -n $backups > $outfile

echo “–”
echo “”
echo “You will be deleting the following snapshots in 30 seconds:”
echo “if this is not what you want, please exit now.”

cat $outfile && sleep 30

we cat the $outfile into a while loop, that loops over the file created by first step line by line until we’re done with the file. |

we use the quiet option to not ask you each line, and async to not wait for each command to completely erase.

tac $outfile | cut -f1 -d ” ” | while read line; do $gcloud_location sql backups delete $line --instance $sql_instance --quiet --async; done

delete file in

rm -f $outfile && echo “” && echo “temp file deleted successfully”

[/code]

Leave a Reply

Your email address will not be published. Required fields are marked *