Daily encrypted IMAP backups

A simple flow to make daily email backups that are compressed, GPG encrypted and safely stored. This article assumes there is an mbox for each user at /var/mail/$USER and a directory in /home/$USER/mail. If you have a setup using Maildir, just change the home directory to /home/$USER/Maildir.

The following paragraphs go over the script step by step. To view the whole script at once, have a look at the last section: Putting it all together

Retrieving the user list through doveadm

We can get a list of users with mailboxes by querying the Dovecot administration utility like so:

doveadm user "*"

However, some of the returned users will not hold mailboxes. Often there is a user called nobody, or other system users. We can eliminate these useless accounts by checking whether they really have a mailbox:

for BU_USER in $(doveadm user "*");
do
    BU_INBOX="/var/mail/$BU_USER";
    BU_MLDIR="/home/$BU_USER/mail";
    if [ -d "$BU_MLDIR" ] && [ -f "$BU_INBOX" ];
    then
        ## Make the backup
    fi
done

Creating the compressed backup

First we will bundle all the necessary files for a user in a compressed tar file, and then create a gpg encrypted version of that archive.

Note that we’ll be using gzip for compression (that is the z flag for tar). While compression rates are a bit better with xz than with regular gzip, the resulting files are not remarkably smaller and the compression takes about 16x longer on my simple mailserver VPS. Using bzip2 also produces slightly smaller files, but takes three times as long as gzip. If time isn’t an issue, and storage size very much is, then by all means go for xz. But I’ll opt for gzip here. A simple comparison is shown below:

du -h -d1 | grep -i maildir 
# 5.2G ./Maildir

find Maildir -type f | wc -l 
40871

time tar czf /tmp/test.tar.gz ~/Maildir && ls -l /tmp/test.tar.gz
real    5m17.062s
user    4m50.937s
sys     0m18.397s
-rw-rw-r-- 1 jethro jethro 3509456433 Jul 29 14:27 /tmp/test.tar.gz


time tar cjf /tmp/test.tar.bz2 ~/Maildir && ls -al /tmp/test.tar.bz2
real    16m47.331s
user    16m23.790s
sys     0m17.083s
-rw-rw-r-- 1 jethro jethro 3481734115 Jul 29 16:13 /tmp/test.tar.bz2


time tar cJf /tmp/test.tar.xz ~/Maildir && ls -l /tmp/test.tar.xz
real    81m33.066s
user    80m30.880s
sys     0m52.150s
-rw-rw-r-- 1 jethro jethro 3281076332 Jul 29 15:50 /tmp/test.tar.xz

With that comparison in mind, let’s draw some sane default options:

TAR_CMD="cz";
TAR_OPTS="--atime-preserve --preserve-permissions";

For the GPG encryption, I like to use a different key than my default one, so basically a key specifically for backups. Just replace <RECIPIENT_KEY_ID> here with the key you would like to use.

GPG_KEY="<RECIPIENT_KEY_ID>"
GPG_OPTS="--encrypt --sign --batch --yes --recipient $GPG_KEY"

(Also note that the key must be trusted by gpg. Perform a testrun by encrypting a small file using the desired recipient key, watch carefully if gpg barfs out anything about trust, and fix that first.)

With our basic options set, we can now create the compressed backup archive (holding both $BU_MLDIR and $BU_INBOX), and encrypt it with gpg:

TIMESTAMP=$(date +%Y-%m-%d);
BU_NAME="$TIMESTAMP.mail.$BU_USER.tar.gz.gpg";
BU_TARGET="/tmp/$BU_NAME";

tar "$TAR_CMD" "$BU_MLDIR" "$BU_INBOX" $TAR_OPTS \
    | gpg $GPG_OPTS -o "$BU_TARGET";

Assuming today is the 29th of June 2017, and our user is called Bob, we have now ended up with a file called /tmp/2017-07-29.mail.bob.tar.gz.gpg, holding the contents of /var/mail/bob and /home/bob/mail.

And now that our backup is encrypted, let’s get reckless transfer it to the cloud.

Uploading the file to Backblaze

Using the b2 command line tool, uploading stuff to Backblaze is uncomplicated. By setting the lifecycle of files to let’s say one month, no manual cleanup or pruning of old backup files is needed.

BUCKET="my-bucket-name-at-backblaze";
FOLDER="email-daily";
b2 upload-file "$BUCKET" "$BU_TARGET" "$FOLDER/$BU_NAME";
rm -f "$BU_TARGET";

Putting it all together

#!/bin/bash

### PERSONAL SETTINGS ##############################

    BUCKET="my-bucket-name-at-backblaze";
    FOLDER="email-daily";
    GPG_KEY="0x41abbfab0934c20a";

### END PERSONAL SETTINGS ##########################


TAR_CMD="cz";
TAR_OPTS="--atime-preserve --preserve-permissions";
GPG_OPTS="--encrypt --sign --batch --yes --recipient $GPG_KEY";
TIMESTAMP=$(date +%Y-%m-%d);

for BU_USER in $(doveadm user "*");
do
    BU_INBOX=/var/mail/$BU_USER;
    BU_MLDIR=/home/$BU_USER/mail;

    if [ -d "$BU_MLDIR" ] && [ -f "$BU_INBOX" ];
    then
        BU_NAME="$TIMESTAMP.mail.$BU_USER.tar.gz.gpg";
        BU_TARGET="/tmp/$BU_NAME";

        tar "$TAR_CMD" "$BU_MLDIR" "$BU_INBOX" $TAR_OPTS \
            | gpg $GPG_OPTS -o "$BU_TARGET";

        b2 upload-file "$BUCKET" "$BU_TARGET" "$FOLDER/$BU_NAME";

        rm -f "$BU_TARGET";
    fi;
done
~jvt,