Duplicity is a space- and bandwidth-efficient backup software utility built atop the rsync and GnuPG technologies. It provides a relatively simple way to create full or incremental backups, either from a command line or using a variety of graphical user interfaces. By default, it uses password-based (symmetric) encryption to protect data at rest, and is compatible with numerous storage backends including Free Software and even technocapitalist SAASS offerings.

We recommend the use of duplicity(1) as a simple and effective means of being prepared for disaster recovery.

Installing

GNU/Linux users may already have duplicity installed on their systems. If you do not, on Debian-derived GNU/Linux distributions, install duplicity with:

sudo apt install duplicity

Using

This section describes several basic uses of duplicity.

Automated symmetrically encrypted backup via rsync over Tor

In this configuration, cron is used to schedule an unattended backup of a given important directory, which will be encrypted locally before being stored on the destination server. The remote host is accessed as a Tor Onion service exposing an rsync daemon. When the configuration is complete, a single cron job of the following form will automatically back up the important directory:

PASSPHRASE="$(/bin/cat $HOME/.duplicity.secret)" RSYNC_CONNECT_PROG='/bin/nc -x 127.0.0.1:9050 \%H 873' /usr/bin/duplicity --gpg-options="--cipher-algo=AES256" --rsync-options="--password-file=$HOME/.rsync.secret --partial --compress" /path/to/important/directory rsync://rsync_user@abcdef0123456789.onion::backups/ >/dev/null 2>&1

Defining a backup schedule is the responsibility of the data owner, as is determining how many backups to keep and for how long. In this configuration, the following backup policy is assumed, but you should change this to suit your needs by editing the relevant cron job schedules and duplicity command options:

Procedure

Assumptions:

Do this to set up automated symmetrically encrypted backups using duplicity that are stored on a (hopefully off-site) rsync Tor Onion service.

  1. Terminal in to the server and become the user who will be performing the backup, say backup_user.
    ssh the.server.local
sudo su --shell /bin/bash --login backup_user
  2. Generate a password to use to encrypt the backup archive, and write this password to a file (for example, .duplicity.secret). This will be used to ensure the remote host never receives a plaintext copy of any filesystem data. Be certain to record this password somewhere safe (such as a password/secret manager application), as you will need it to restore the (encrypted!) backup archive.
    # Collect 45 random bytes (from `/dev/random`?) and base64 encode them.
gpg --armor --gen-random 2 45 > .duplicity.secret
# Ensure the password file is not readable by any other users.
chmod 600 .duplicity.secret
  3. Obtain the password for your rsync user and write it to a file, as well.
    vim .rsync.secret # Put the password to your rsync user account in this file.
# Ensure the password file is not readable by any other users.
chmod 600 .rsync.secret
  4. Create or edit a crontab(5) file for the user that will be running the backups:
    crontab -e
  5. Add the following entry in the backup user’s cron table file. Note that any literal percent signs (%) must be escaped, as they have special meaning to most implementations of the cron daemon.
    # For more information see the manual pages of crontab(5) and cron(8)
# Cron entry fields:
#
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                       7 is also Sunday on some systems)
# │ │ │ │ │
# * * * * *  command to execute

# Run (incremental) backups to a remote Onion at 3:05am every Tuesday.
5 3 * * 2 PASSPHRASE="$(/bin/cat $HOME/.duplicity.secret)" RSYNC_CONNECT_PROG='/bin/nc -x 127.0.0.1:9050 \%H 873' /usr/bin/duplicity --full-if-older-than 2M --gpg-options="--cipher-algo=AES256" --rsync-options="--password-file=$HOME/.rsync.secret --partial --compress" /path/to/important/directory rsync://rsync_user@abcdef0123456789.onion::backups/ >/dev/null 2>&1

# Remove the 13th (and older) full backups, i.e., retain the most recent 12 full backups and their incremental diffs.
5 4 * * 2 RSYNC_CONNECT_PROG='/bin/nc -x 127.0.0.1:9050 \%H 873' /usr/bin/duplicity remove-all-but-n-full 12 --force --rsync-options="--password-file=$HOME/.rsync.secret --partial --compress" rsync://rsync_user@abcdef0123456789.onion::backups/ >/dev/null 2>&1

A breakdown of the command invocations:

  1. PASSPHRASE="$(/bin/cat/ $HOME/.duplicity.secret)" – Save the contents of the file $HOME/.duplicity.secret as the environment variable PASSPHRASE. This variable is checked by duplicity(1), which will pass it to gpg to use as the encryption password. This is only required when encrypting (backing up).
  2. RSYNC_CONNECT_PROG='/bin/nc -x 127.0.0.1:9050 \%H 873' – Set the environment variable RSYNC_CONNECT_PROG to a netcat (nc) proxy (-x) command. The sequence %H is replaced by rsync itself with the address of the remote host. It is backslash-escaped because % characters are treated specially in crontab(5) files.
  3. /usr/bin/duplicity invoke duplicity(1), with the following options and arguments:
    1. --full-if-older-than 2M – If the last full backup is older than 2 months, do a full backup instead of an incremental one.
    2. --gpg-options"--cipher-algo=AES256" – Instruct the gpg executable to use the AES256 cipher algorithm. This arguably improves the security of the backup archive by using a more modern cipher (AES256) than the gpg default (CAST5).
    3. --rsync-options="--password-file="$HOME/.rsync.secret" --partial --compress" – Instruct rsync, when invoked, to read its user account’s password from the file $HOME/.rsync.secret, to record partial transfers, and to compress the datastream before being sent to the network. The --password-file is needed to allow for non-interactive execution. Meanwhile, partial transfers ensure that interruptions caused by, e.g., network issues, can be resumed on the next run, and compression speeds the transfer by minimizing the amount of data that needs to be emitted on the network link.
    4. /path/to/important/directory – The duplicity backup source directory.
    5. rsync://rsync_user@abcdef0123456789.onion::backups/ – The duplicity target URL, i.e., the location where the backup archive will be sent.
  4. In the second command, invoke /usr/bin/duplicity again, but with these alternate instructions:
    1. remove-all-but-n-full 12 – Remove all old backups except the most recent 12 full backups.
    2. --force – Actually do the removal; this is required by duplicity in order to ensure you mean what you say.
  5. >/dev/null 2>&1 – Redirect the command’s STDOUT to /dev/null, and redirect the command’s STDERR to STDOUT (which is going to /dev/null).

If you need to use sudo(1) to run the backup command as another user, say, other_user, you can pass the RSYNC_CONNECT_PROG environment variable through the sudo invocation like so:

sudo RSYNC_CONNECT_PROG='nc -x 127.0.0.1:9050 %H 873' -u other_user duplicity --rsync-options="--password-file=/path/to/password/file" rsync://rsync_user@abcdef0123456789.onion::module_name/ /path/to/restoration/directory

Restoring

To restore a Duplicity backup archive, re-run the same command as that used to create the archive but swap the positions of the source URL and the target directory. For example:

Hardening

Use strong GPG ciphers

Duplicity makes use of a locally installed gpg to perform encryption and decryption on its behalf. By default, most versions of gpg use the CAST5 cipher, despite more modern ciphers being available. View the available cipher list by invoking gpg --version. A good cipher is AES256.

Then instruct duplicity(1) to use this cipher when calling gpg by passing --gpg-options="--cipher-algo=AES256".

Provisioning

The AnarchoTech NYC collective provides an Ansible role for automating the setup of Duplicity backup jobs that run on a Raspberry Pi. It can be installed in your local $ANSIBLE_ROLES_PATH (see Ansible Configuration Settings) for use with an Ansible project with:

ansible-galaxy install https://github.com/AnarchoTechNYC/ansible-role-duplicity/archive/master.tar.gz

See also

ArchWiki: Duplicity
The Arch Linux community’s own documentation for use of duplicity, including an example backup script.