Jump to content
Sign in to follow this  
sdub

[Support] borgmatic

10 posts in this topic Last Reply

Recommended Posts

Application: borgmatic

Docker Hub: https://hub.docker.com/r/b3vis/borgmatic

Github: https://github.com/b3vis/docker-borgmatic

Template's repo: https://github.com/Sdub76/unraid_docker_templates

 

An Alpine Linux Docker container for witten's borgmatic by b3vis. Protect your files with client-side encryption. Backup your databases too. Monitor it all with integrated third-party services.

 

Getting Started:

  • It is recommended that your Borg repo and cache be located on a drive outside of your array (via unassigned devices plugin)
  • Before you backup to a new repo, you need to initialize it first. Examples at https://borgbackup.readthedocs.io/en/stable/usage/init.html
  • Place your crontab.txt and config.yaml in the "Borgmatic config" folder specified in the docker config. See examples below.
  • A mounted repo can be accessed within Unraid using the "Fuse mount point" folder specified in the docker config. Example of how to mount a Borg archive at https://borgbackup.readthedocs.io/en/stable/usage/mount.html

 

Support:

Your best bet for Borg/Borgmatic support is to refer to the following links, as the template author does not maintain the application

 

Why use this image?

Borgmatic is a simple, configuration-driven front-end to the excellent BorgBackup.  BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption.  The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets.

 

Other Unraid/Borg solutions require installation of software to the base Unraid image.  Running these tools along with their dependencies is what Docker was built for.  This particular image does not support rclone, but does support remote repositories via SSH.  This docker can be used with the Unraid rclone plugin if you wish to mirror your repo to a supported cloud service.

Edited by sdub

Share this post


Link to post

Here are example crontab and config files with some descriptions.  Both files should be placed in the appdata/borgmatic/config folder

 

Example crontab:

  • Twice Daily backups @ 1a, 1p
  • Repo & archives checked weekly Wed @ 6a
  • My repo is rather large (~5TB, 1M files) so it was sensible to separate the prune/create and checks to separate schedules
    • The prune/create tasks take about 1 hr per repo to complete with minimal changes (for reference)
    • The repo/archive check tasks takes about 9hr per repo to complete (for reference)

crontab.txt:

0 1,13 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1
0 6    * * 3 PATH=$PATH:/usr/bin /usr/bin/borgmatic check -v 1 2>&1

 

 

Example Borgmatic config:

  • Several source directories are included (read-only):
    • Flash drive and appdata are incrementally backed up (alternative to CA backup utility)
    • Backup share acts like a funnel for other data to be backed up
      • Other machines on my network back themselves up to an unRAID "backup" share (Windows 10 backup, time machine, etc.)
      • Docker images that use mysqlite are configured to place their DB backups in the "backup" share
    • Other irreplaceable user shares
  • Two repos are updated in succession:
    • /mnt/borg-repository - Docker mapped volume NOT part of my array
    • remote.mydomain.net:/mnt/disks/borg_remote/repo - A repo that resides on a family member's Linux box with borg installed
  • Files cache set to use "mtime,size" - Very important as unRAID does not have persistent inode values
  • Folders with a ".nobackup" file are ignored, "cache" and "trash" folders are ignored.
  • There are many options for how to maintain your repo passphrase/keys.  I opted for a simple passphrase that I specify in the config file
  • Compression options are available, but I don't bother since 95% of my data is binary compressed data (MP4, JPG, etc)
  • If you're backing up to a remote repo, you'll need to make sure that your SSH keypairs are working for password-less login.  Don't forget to set the SSH folder permissions properly, or your keyfiles won't work.  
  • I have a MariaDB that runs as a database for Nextcloud and Bookstack.  A full database dump is included in every backup
  • Healthchecks.io monitors the whole thing and notifies me if a backup doesn't complete
  • My retention policy is 2 hourly, 7 daily, 4 weekly, 12 monthly, 10 yearly

I deleted the comments for brevity in the example below, but I recommend you start with the official reference template and make your edits from there.

 

config.yaml:

location:
    source_directories:
        - /boot
        - /mnt/user/appdata
        - /mnt/user/backup
        - /mnt/user/nextcloud
        - /mnt/user/music
        - /mnt/user/pictures
    repositories:
        - /mnt/borg-repository
        - remote.mydomain.net:/mnt/disks/borg_remote/repo
    one_file_system: true
    files_cache: mtime,size
    patterns:
        - '- [Tt]rash'
        - '- [Cc]ache'
    exclude_if_present:
        - .nobackup
        - .NOBACKUP

storage:
    encryption_passphrase: "MYREPOPASSWORD"
    compression: none
    ssh_command: ssh -i /root/.ssh/id_rsa
    archive_name_format: 'backup-{now}'

retention:
    keep_hourly: 2
    keep_daily: 7
    keep_weekly: 4
    keep_monthly: 12
    keep_yearly: 10
    prefix: 'backup-'

consistency:
    checks:
        - repository
        - archives
    prefix: 'backup-'

hooks:
    before_backup:
        - echo "Starting a backup."
    after_backup:
        - echo "Finished a backup."
    on_error:
        - echo "Error during prune/create/check."
    mysql_databases:
        - name: all
          hostname: 192.168.200.37
          password: MYSQLPASSWD
    healthchecks: https://hc-ping.com/MYUUID

 

Edited by sdub

Share this post


Link to post

Hi sdub,

 

this Borg integration looks promising to me. Thanks for taking the time creating the container and making it available to the community. I will definitely check it out and may consider it as a replacement for my local rsync and remote rclone offsite backup. Will report back!

Quote

Flash drive and appdata are incrementally backed up (alternative to CA backup utility)

 

How do you make sure that files are not getting written by your docker containers while the backup is running? The CA backup stops containers to prevent file corruption AFAIK. I cannot see such a mechanism in your solution. Technically, this would be possible with the before_backup and after_backup hooks.

 

Not sure, if any further/similar steps needs to be taken into account for the flash drive. May be worth looking into the CA backup code to review the protection mechanisms.

Edited by T0a

Share this post


Link to post
9 hours ago, T0a said:

How do you make sure that files are not getting written by your docker containers while the backup is running?

Borgmatic supports hooks, so you could likely use those to interact with docker.

I wrote alternative solution to borgmatic with less dependencies that addresses this. Although borgmatic absolutely is a nice tool to work with.

 


Share this post


Link to post
3 hours ago, cheesemarathon said:

Hi, im having issues with SSH key permissions. What should i set them to?

find -L "/root/.ssh" \( -type f -o -type d \) -exec chmod 'u=rwX,g=,o=' -- '{}' \+

(this will guarantee the dirs will get the executable bit set)

Edited by laur

Share this post


Link to post

> This docker can be used with the Unraid rclone plugin if you wish to mirror your repo to a supported cloud service.

Note this goes against borg recommendation.

 

> Files cache set to use "mtime,size" - Very important as unRAID does not have persistent inode values
That's a great point! Will amend my setup. Why did you change the default 'ctime' to 'mtime' though?

Edited by laur

Share this post


Link to post
On 11/23/2020 at 10:17 PM, laur said:

find -L "/root/.ssh" \( -type f -o -type d \) -exec chmod 'u=rwX,g=,o=' -- '{}' \+

(this will guarantee the dirs will get the executable bit set)

Still having issues:

crond: USER root pid 8 cmd PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1
/etc/borgmatic.d/config.yaml: Running command for pre-backup hook
Wed Nov 25 19:19:00 UTC 2020 - Starting backup
ba627szn@ba627szn.repo.borgbase.com:repo: Pruning archives
Remote: ba627szn@ba627szn.repo.borgbase.com: Permission denied (publickey).
Connection closed by remote host. Is borg working on the server?
ba627szn@ba627szn.repo.borgbase.com:repo: Error running actions for repository

Command 'borg prune --keep-daily 3 --keep-weekly 4 --keep-monthly 12 --keep-yearly 2 --prefix {hostname}- --stats --info ba627szn@ba627szn.repo.borgbase.com:repo' returned non-zero exit status 2.
/etc/borgmatic.d/config.yaml: Error running configuration file


summary:
/etc/borgmatic.d/config.yaml: Error running configuration file

ba627szn@ba627szn.repo.borgbase.com:repo: Error running actions for repository

Remote: ba627szn@ba627szn.repo.borgbase.com: Permission denied (publickey).
Connection closed by remote host. Is borg working on the server?
Command 'borg prune --keep-daily 3 --keep-weekly 4 --keep-monthly 12 --keep-yearly 2 --prefix {hostname}- --stats --info ba627szn@ba627szn.repo.borgbase.com:repo' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

File perms:

~/.ssh # ls -al
total 8
drwx------    1 99       users           40 Nov 22 20:24 .
drwx------    1 root     root            58 Nov 25 19:13 ..
-rw-------    1 root     root           464 Nov 22 19:39 borgmatic
-rw-------    1 root     root           202 Nov 22 19:44 known_hosts

 

Share this post


Link to post

> ~/.ssh # ls -al

You don't have any keys in ~/.ssh/

Either generate them, or mount /boot/config/ssh/ or /root/.ssh (ie wherever on host you have the keys) to container's /root/.ssh

Note in the parent post example you can see the example expects key to be found at ~/.ssh/id_rsa:

storage:
    ssh_command: ssh -i /root/.ssh/id_rsa   <-- keyfile for ssh

Yet your dir listing above shows there's no key to be found.

Edited by laur

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this