After installing Debian buster on my GnuBee, I set it up for receiving backups from my other computers.
Software setup
I started by configuring it like a typical server but without a few packages that either take a lot of memory or CPU:
I changed the default hostname:
/etc/hostname:foobar/etc/mailname:foobar.example.com/etc/hosts:127.0.0.1 foobar.example.com foobar localhost
and then installed the avahi-daemon package to be able to reach this box
using foobar.local.
I noticed the presence of a world-writable
directory and so I
tightened the security of some of the default mount points by putting the following
in /etc/rc.local:
chmod 755 /etc/network
exit 0
Hardware setup
My OS drive (/dev/sda) is a small SSD so that the GnuBee can run silently when the
spinning disks aren't needed. To hold the backup data on the other hand, I
got three 4-TB drives drives which I setup in a
RAID-5 array.
If the data were valuable, I'd use
RAID-6 instead
since it can survive two drives failing at the same time, but in this case
since it's only holding backups, I'd have to lose the original machine at
the same time as two of the 3 drives, a very unlikely scenario.
I created new gpt partition tables on /dev/sdb, /dev/sdbc, /dev/sdd
and used fdisk to create a single partition of type 29 (Linux RAID) on
each of them.
Then I created the RAID array:
mdadm /dev/md127 --create -n 3 --level=raid5 /dev/sdb1 /dev/sdc1 /dev/sdd1
and waited more than 24 hours for that operation to finish. Next, I formatted the array:
mkfs.ext4 -m 0 /dev/md127
and added the following to /etc/fstab:
/dev/md127 /mnt/data/ ext4 noatime,nodiratime 0 2
Keeping a copy of the root partition
In order to survive a failing SSD drive, I could have bought a second SSD and gone for a RAID-1 setup. Instead, I went for a cheaper option, a poor man's RAID-1, where I will have to reinstall the machine but it will be very quick and I won't lose any of my configuration.
The way that it works is that I periodically sync the contents of the root
partition onto the RAID-5 array using a cronjob in /etc/cron.d/hdd-sync:
0 10 * * * root /usr/local/sbin/ssd_root_backup
which runs the /usr/local/sbin/ssd_root_backup script:
#!/bin/sh
nice ionice -c3 rsync -aHx --delete --exclude=/dev/* --exclude=/proc/* --exclude=/sys/* --exclude=/tmp/* --exclude=/mnt/* --exclude=/lost+found/* --exclude=/media/* --exclude=/var/tmp/* /* /mnt/data/root/
Drive spin down
To reduce unnecessary noise and reduce power consumption, I also installed hdparm:
apt install hdparm
and configured all spinning drives to spin down after being idle for 2
minutes and for maximum power saving by putting the following in /etc/hdparm.conf:
/dev/sdb {
apm = 1
spindown_time = 24
}
/dev/sdc {
apm = 1
spindown_time = 24
}
/dev/sdd {
apm = 1
spindown_time = 24
}
and then reloaded the configuration:
/usr/lib/pm-utils/power.d/95hdparm-apm resume
Monitoring drive health
Finally I setup smartmontools by putting
the following in /etc/smartd.conf:
/dev/sda -a -o on -S on -s (S/../.././02|L/../../6/03)
/dev/sdb -a -o on -S on -s (S/../.././02|L/../../6/03)
/dev/sdc -a -o on -S on -s (S/../.././02|L/../../6/03)
/dev/sdd -a -o on -S on -s (S/../.././02|L/../../6/03)
and restarting the daemon:
systemctl restart smartd.service
Some of these errors reported by this tool are good predictors of imminent failure.
Backup setup
I started by using duplicity since I have been using that tool for many years, but a 190GB backup took around 15 hours on the GnuBee with gigabit ethernet.
After a friend suggested it, I took a look at restic and I have to say that I am impressed. The same backup finished in about half the time.
User and ssh setup
After hardening the ssh setup as I usually do, I created a user account for each machine needing to backup onto the GnuBee:
adduser machine1
adduser machine1 sshuser
adduser machine1 sftponly
chsh machine1 -s /bin/false
and then matching directories under /mnt/data/home/:
mkdir /mnt/data/home/machine1
chown machine1:machine1 /mnt/data/home/machine1
chmod 700 /mnt/data/home/machine1
Then I created a custom passwordless ssh key for each machine:
ssh-keygen -f /root/.ssh/foobar_backups -t ed25519
and placed it in /home/machine1/.ssh/authorized_keys on the GnuBee.
Then I added the restrict prefix in front of that key so that it looked like:
restrict ssh-ed25519 AAAAC3N... root@machine1
On each machine, I added the following to /root/.ssh/config:
Host foobar.local
User machine1
Compression no
Ciphers aes128-ctr
IdentityFile /root/backup/foobar_backups
IdentitiesOnly yes
ServerAliveInterval 60
ServerAliveCountMax 240
The reason for setting the ssh cipher and disabling compression is to speed up the ssh connection as much as possible given that the GnuBee has a very small RAM bandwidth.
Another performance-related change I made on the GnuBee was switching to the internal sftp
server
by putting the following in /etc/ssh/sshd_config:
Subsystem sftp internal-sftp
Restic script
After reading through the excellent restic documentation, I wrote the following backup script, based on my old duplicity script, to reuse on all of my computers:
# Configure for each host
PASSWORD="XXXX" # use `pwgen -s 64` to generate a good random password
BACKUP_HOME="/root/backup"
REMOTE_URL="sftp:foobar.local:"
RETENTION_POLICY="--keep-daily 7 --keep-weekly 4 --keep-monthly 12 --keep-yearly 2"
# Internal variables
SSH_IDENTITY="IdentityFile=$BACKUP_HOME/foobar_backups"
EXCLUDE_FILE="$BACKUP_HOME/exclude"
PKG_FILE="$BACKUP_HOME/dpkg-selections"
PARTITION_FILE="$BACKUP_HOME/partitions"
# If the list of files has been requested, only do that
if [ "$1" = "--list-current-files" ]; then
RESTIC_PASSWORD=$PASSWORD restic --quiet -r $REMOTE_URL ls latest
exit 0
# Show list of available snapshots
elif [ "$1" = "--list-snapshots" ]; then
RESTIC_PASSWORD=$GPG_PASSWORD restic --quiet -r $REMOTE_URL snapshots
exit 0
# Restore the given file
elif [ "$1" = "--file-to-restore" ]; then
if [ "$2" = "" ]; then
echo "You must specify a file to restore"
exit 2
fi
RESTORE_DIR="$(mktemp -d ./restored_XXXXXXXX)"
RESTIC_PASSWORD=$PASSWORD restic --quiet -r $REMOTE_URL restore latest --target "$RESTORE_DIR" --include "$2" || exit 1
echo "$2 was restored to $RESTORE_DIR"
exit 0
# Delete old backups
elif [ "$1" = "--prune" ]; then
# Expire old backups
RESTIC_PASSWORD=$PASSWORD restic --quiet -r $REMOTE_URL forget $RETENTION_POLICY
# Delete files which are no longer necessary (slow)
RESTIC_PASSWORD=$PASSWORD restic --quiet -r $REMOTE_URL prune
exit 0
# Unlock the repository
elif [ "$1" = "--unlock" ]; then
RESTIC_PASSWORD=$PASSWORD restic -r $REMOTE_URL unlock
exit 0
# Catch invalid arguments
elif [ "$1" != "" ]; then
echo "Invalid argument: $1"
exit 1
fi
# Check the integrity of existing backups
CHECK_CACHE_DIR="$(mktemp -d /var/tmp/restic-check-XXXXXXXX)"
RESTIC_PASSWORD=$PASSWORD restic --quiet --cache-dir=$CHECK_CACHE_DIR -r $REMOTE_URL check || exit 1
rmdir "$CHECK_CACHE_DIR"
# Dump list of Debian packages
dpkg --get-selections > $PKG_FILE
# Dump partition tables from harddrives
/sbin/fdisk -l /dev/sda > $PARTITION_FILE
/sbin/fdisk -l /dev/sdb > $PARTITION_FILE
# Do the actual backup
RESTIC_PASSWORD=$PASSWORD restic --quiet --cleanup-cache -r $REMOTE_URL backup / --exclude-file $EXCLUDE_FILE
I run it with the following cronjob in /etc/cron.d/backups:
30 8 * * * root ionice nice /root/backup/backup-machine1-to-foobar
30 2 * * Sun root ionice nice /root/backup/backup-machine1-to-foobar --prune
in a way that doesn't impact the rest of the system too much.
I also put the following in my /etc/rc.local to cleanup any leftover temp
directories for aborted backups:
rmdir --ignore-fail-on-non-empty /var/tmp/restic-check-*
Finally, I printed a copy of each of my backup script, using enscript, to stash in a safe place:
enscript --highlight=bash --style=emacs --output=- backup-machine1-to-foobar | ps2pdf - > foobar.pdf
This is actually a pretty important step since without the password, you won't be able to decrypt and restore what's on the GnuBee.