I use Ansible and a dotfiles manager called yadm to store changes I make to my system so that I can rebuild my machine easily whenever I want to. This takes care of programs I use and their settings, plus system settings such as thinkfan config, plasma desktop tweaks and emacs lisp scripts.
I also use git for many of my content repos for software I write/contribute to, creative works I write and wikis/blogs I manage - including this one. This takes care of persisting my files in the cloud so that I can easily restore a working environment on any computer whenever I need.
But this is only the case for files I want to store in git and only whenever I’m ready to commit and push them into a remote repo. So what about the files I’ve work in progress and haven’t pushed or files I simply don’t want in those repos such as photos, videos and some documents?
In this post I’ll document my use of Borg, a backup tool and how I try to make sure I’ve captured those files so I can restore them if I need to.
Installing Borg
I already have my ansible script installing Borg, I use pacman (Arch linux package manager).
I could have done this manually with $ sudo pacman -Syu borg and
I’m pretty sure it would be available in most package managers. Doing
so should install most of the dependencies too.
Other options include building from source and using pip install. Borg is python based so it should run on most operating systems and architectures as long as the dependencies are available.
Setting Up the Borg Repo
I need a directory to initialise a borg repo and I have an mSata drive in my T430 so I’m going to use this for all backups. I should have a remote backup repo too but for now I’ll concentrate on getting a local one set up.
I have a mount to the disk and a directory there for the borg repo, I initialise the repo only once and set up the encryption keys and passcode. There are many ways to encrypt the borg repo and I’ve decided to use BLAKE2b with keyfile. I’ll have to make sure the keyfile isn’t readable by anyone else.
When I initialise the repo it will ask me for a passphrase. I’ll need to store this somewhere and use it whenever I make a backup. Again there are many ways of storing and using secrets like this. For my purposes I’m going to store it in a keepass database.
First I’ll create a keepass file and generate a passphrase to use:
$ keepassxc-cli create -k /secret/path/for/key /path/for/kdbx
Enter password to encrypt database (optional):
Successfully created new database.
I edit the file permissions for the created key and db so that only I can read and write them.
$ chmod 0400 /secret/path/for/key /path/for/kdbx
Then I generate the passphrase that will be used for the borg repo
$ keepassxc-cli add -p --no-password -k /secret/path/for/key /path/for/kdbx borg
Enter password for new entry:
Successfully added entry borg.
Finally I’ll initialise the borg repo using this passphrase for the key.
$borg init --encryption=keyfile-blake2 /path/to/repo
Enter new passphrase:
Enter same passphrase again:
Do you want your passphrase to be displayed for verification? [yN]: n
By default repositories initialized with this version will produce security
errors if written to with an older version (up to and including Borg 1.0.8).
If you want to use these older versions, you can disable the check by running:
borg upgrade --disable-tam /path/to/repo
See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.
IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
Use "borg key export" to export the key, optionally in printable format.
Write down the passphrase. Store both at safe place(s).
Configuring the Backups
I am using a wrapper for borg called
Borgmatic which
consists mainly of a conig.yml file. To generate this file I ran:
$ generate-borgmatic-config --destination ~/path/to/config.yml
$ chmod 0600 ~/path/to/config.yml
I won’t go over all the settings, reading through the file is quite
obvious - it’s well documented and I could use man borg or man borgmatic to quickly reference further info. The source directories
I’ve just set to my home dir and the repository to the path where I
initialised the borg repo.
I’ve added my own preferences throughout the file like using an
exclude files and listing paths such as /*/.cache. It’s this file
I’ll need to change when I want to configure anything to do with my
backups and I imagine I’ll be tweaking it periodically to exclude and
include paths as I install new software etc.
To help me with the tweaking process I’ve set the exclude_if_present
so that if I have a dir with .noborg in then the backup should see
that and not back up that dir.
The config file breaks down borg commands into vars and within the
storage section I have set the encryption details. Using
encryption_passcommand allows me to use a command to access the
passphrase for the repo. I set this in keepass file and I use a cli
tool to access that password using the key and db I created earlier.
encryption_passcommand: keepassxc-cli show -a Password --no-password
-k /secret/path/for/key /path/for/kdbx borg
Timing and Automated Backups
The other important set of variables within the borgmatic is the
retention section. This is where the time periods are set to define
what should be pruned out of the repo. Again I’ll probably tweaks this
over time and I’m going to start out backing up every hour, retaining a
copy for each hour over the previous 24h, one copy per day for the
past 7 days, one copy per week for the past 4 weeks and one copy per
month for the past 6 months.
To run the backups I am using systemd so I created a borgmatic.service file:
[Unit]
Description=borgmatic backup
Wants=network-online.target
After=network-online.target
#ConditionACPower=true
[Service]
Type=oneshot
#running as user tom, permission denied for systemd-inhibit so using runuser in command instead
#User=tom
# Lower CPU and I/O priority.
Nice=19
CPUSchedulingPolicy=batch
IOSchedulingClass=best-effort
IOSchedulingPriority=7
IOWeight=100
Restart=no
# Prevent rate limiting of borgmatic log events. If you are using an older version of systemd that
# doesn't support this (pre-240 or so), you may have to remove this option.
LogRateLimitIntervalSec=0
# Delay start to prevent backups running during boot.
ExecStartPre=sleep 1m
ExecStart=systemd-inhibit --who="borgmatic" --why="Prevent
interrupting scheduled backup" runuser tom -c 'borgmatic --syslog-verbosity 1 --config=/full/path/to/borgmatic/config.yml'
In the comments notice I want to run the service as a my own user but
my user doesn’t have permission to run systemd-inhibit and so I
can’t use the setting User=tom instead I use runuser in the
ExecStart command.
I also have a borgmatic.timer for this, I’ve created it so it will run 5 minutes
after booting up and every hour after that.
[Unit]
Description=Run borgmatic backup
[Timer]
Persistent=true
OnBootSec=5min
OnUnitActiveSec=1h
[Install]
WantedBy=timers.target
I then enable these and start the timer.
$ sudo systemctl enable borgmatic.service
$ sudo systemctl enable borgmatic.timer
$ sudo systemctl start borgmatic.timer
$ sudo systemctl daemon-reload
Listing the Repo
To view the backups taken in a repo I run the following:
borgmatic --config /home/tom/.config/borgmatic/config.yml list
Security
I’ve gone to some effort to secure the backup process by using keys and passphrases. I’ve stored my passphrase in a password manager but it could have also been stored in a key managers and it’s on my todo list to see if there’s a more secure way such as KDE kwallet-query, though the password manager I have chosen is cross platform as is borg.
I have backups of my keys on usb stick and have put my passphrases in a safe place.
I have encrypted keys again and other files containing references to the keys and I have stored these in my favourite dot files manager. My home directory is also encrypted at rest so it won’t be easy to access these files should my laptop get stolen for example.
I’m not certain that I have covered everything to secure my backups but at least I believe I have made it difficult for anyone who wants to access this data.
Conclusion
This was quite a challenging set up but now that it’s backing my files up frequently I feel a bit happier about being able to roll back files to a previous state even if they’re not in source control.
I need to review and experiment with querying the repo next and maybe do a dry run recovery so I know that it works end to end - it’s no use having a backup if you can’t recover it.
Also having the backup on my local machine isn’t really a backup, I need to have that backup also being written to a remote repo somewhere.
I’ll need to tween the configs and expand the include/exclude paths and files to backup as I progress. I think Borg is a great bit of software, it compresses and dedupes to make for small repos and has some interesting querying options to explore.