Dory backup scripts

I made a backup script that is working pretty nicely for me so far. It has several function, the last one being the essence of it.

  • Make a dump of all databases (postgres & mysql) to zipped sql files. It deletes any that haven't changed since the last backup.
  • Bring a list of git repositories up-to-date with with their remotes on the current branch, by cloning or pulling. (This is useful to have a backup of a non-bare repository).
  • Make a copy of each in a list of remote directories. It does this by taking the last backup, hardlinking all the files to a new location and then bringing that one up-to-date with the remote.

This way, you'll end up with a copy of each directory you want to preserve, for each time you run the script. If your machine has any problem, you just grab any backup and you have your data back in the original format. And because they are all hardlinks, files only take space if they're changed.

It'd be best to run this every day from a remote computer that can login to (all) the machine(s) you want to backup. If those machines get compromised, the attacker can't get to your backups or logging. And you can easily have the backup computer elsewhere so that your data is safe even if your room burns down. Maybe it's better if you don't burn your room down though.

You can define your own logging functions, so that you'll noticed if backups stop for any reason. Have it send emails on success and failure or make a quick POST request to your logging server.

My setup looks like this:

# include settings and functions
source ~/shcripts/command_logging.sh
source conf_logging.sh
source inc_db_dump.sh
source inc_git_nonbare.sh
source inc_paxsync.sh

# dump databases
printf "Dumping databases. Please wait a little for output to appear...\n"
log_err_all -a -t remote_dump_all_dbs "ursula" "\$HOME/dbdumps"

# pull repositories
# note: git tends to keep running for a while if you ctrl+C the connection
printf "Git pull/cloning. This might take a while to produce any output for large repos.\n"
log_err_all -a -t remote_find_and_clone "ursula" "/repo" "\$HOME/repos"

# create incremental backups
backup_dir="/backup"
declare -a source_dirs=('ursula:/home/backuper/repos' 'ursula:/home/backuper/dbdumps' 'ursula:/etc' 'ursula:/data')
log_err_all_named -a -t -c 'paxsync' do_pax_sync_all "$backup_dir" true

There are some logging functions from shcripts but you can easily replace them if you wish. And conf_logging.sh has functions like this:

function log_failure () {
        printf "$@\n" 1>&2
        curl --silent --show-error --request POST 'https://markv.nl/log/add' \
                --data-urlencode "source=dory" --data-urlencode "description=$@" \
                --data-urlencode "status=error" --data-urlencode 'key=IAmNotTellingYouSorry';
}

There are some requirements though. Like a basic understanding of bash and linux. All your machines should also be reachable through ssh. You should have some machine that can run the backups. A Raspberry Pi is ideal if you can get one to run stably, which I can't (it always gets corrupted after some weeks). In my case it runs on a VPS from Contabo, whose service I am very happy with. Furthermore the backup machine needs git, ssh, pax and rsync. They're all in apt-get and except for pax I think they're very common.

[It's called Dory because my original backup machine was called dory. All of them are named after Disney characters, and this character seemed particularly appropriate.]

Comments

You need to be logged in to comment.