sdub Posted November 22, 2020 Share Posted November 22, 2020 (edited) Application: borgmatic Docker Hub: https://hub.docker.com/r/b3vis/borgmatic Github: https://github.com/b3vis/docker-borgmatic Template's repo: https://github.com/Sdub76/unraid_docker_templates An Alpine Linux Docker container for witten's borgmatic by b3vis. Protect your files with client-side encryption. Backup your databases too. Monitor it all with integrated third-party services. Getting Started: It is recommended that your Borg repo and cache be located on a drive outside of your array (via unassigned devices plugin) Before you backup to a new repo, you need to initialize it first. Examples at https://borgbackup.readthedocs.io/en/stable/usage/init.html Place your crontab.txt and config.yaml in the "Borgmatic config" folder specified in the docker config. See examples below. A mounted repo can be accessed within Unraid using the "Fuse mount point" folder specified in the docker config. Example of how to mount a Borg archive at https://borgbackup.readthedocs.io/en/stable/usage/mount.html Support: Your best bet for Borg/Borgmatic support is to refer to the following links, as the template author does not maintain the application Borgmatic Source: https://github.com/witten/borgmatic Borgmatic Reference: https://torsion.org/borgmatic Borgmatic Issues: https://projects.torsion.org/witten/borgmatic/issues BorgBackup Reference: https://borgbackup.readthedocs.io Why use this image? Borgmatic is a simple, configuration-driven front-end to the excellent BorgBackup. BorgBackup (short: Borg) is a deduplicating backup program. Optionally, it supports compression and authenticated encryption. The main goal of Borg is to provide an efficient and secure way to backup data. The data deduplication technique used makes Borg suitable for daily backups since only changes are stored. The authenticated encryption technique makes it suitable for backups to not fully trusted targets. Other Unraid/Borg solutions require installation of software to the base Unraid image. Running these tools along with their dependencies is what Docker was built for. This particular image does not support rclone, but does support remote repositories via SSH. This docker can be used with the Unraid rclone plugin if you wish to mirror your repo to a supported cloud service. Edited November 22, 2020 by sdub 3 Quote Link to comment
sdub Posted November 22, 2020 Author Share Posted November 22, 2020 (edited) Here are example crontab and config files with some descriptions. Both files should be placed in the appdata/borgmatic/config folder Example crontab: Twice Daily backups @ 1a, 1p Repo & archives checked weekly Wed @ 6a My repo is rather large (~5TB, 1M files) so it was sensible to separate the prune/create and checks to separate schedules The prune/create tasks take about 1 hr per repo to complete with minimal changes (for reference) The repo/archive check tasks takes about 9hr per repo to complete (for reference) crontab.txt: 0 1,13 * * * borgmatic prune create -v 1 --stats 2>&1 0 6 * * 3 borgmatic check -v 1 2>&1 Example Borgmatic config: Several source directories are included (read-only): Flash drive and appdata are incrementally backed up (alternative to CA backup utility) Backup share acts like a funnel for other data to be backed up Other machines on my network back themselves up to an unRAID "backup" share (Windows 10 backup, time machine, etc.) Docker images that use mysqlite are configured to place their DB backups in the "backup" share Other irreplaceable user shares Two repos are updated in succession: /mnt/borg-repository - Docker mapped volume NOT part of my array remote.mydomain.net:/mnt/disks/borg_remote/repo - A repo that resides on a family member's Linux box with borg installed Files cache set to use "mtime,size" - Very important as unRAID does not have persistent inode values Folders with a ".nobackup" file are ignored, "cache" and "trash" folders are ignored. There are many options for how to maintain your repo passphrase/keys. I opted for a simple passphrase that I specify in the config file Compression options are available, but I don't bother since 95% of my data is binary compressed data (MP4, JPG, etc) If you're backing up to a remote repo, you'll need to make sure that your SSH keypairs are working for password-less login. Don't forget to set the SSH folder permissions properly, or your keyfiles won't work. I have a MariaDB that runs as a database for Nextcloud and Bookstack. A full database dump is included in every backup Healthchecks.io monitors the whole thing and notifies me if a backup doesn't complete My retention policy is 2 hourly, 7 daily, 4 weekly, 12 monthly, 10 yearly I deleted the comments for brevity in the example below, but I recommend you start with the official reference template and make your edits from there. config.yaml: location: source_directories: - /boot - /mnt/user/appdata - /mnt/user/backup - /mnt/user/nextcloud - /mnt/user/music - /mnt/user/pictures repositories: - /mnt/borg-repository - remote.mydomain.net:/mnt/disks/borg_remote/repo one_file_system: true files_cache: mtime,size patterns: - '- [Tt]rash' - '- [Cc]ache' exclude_if_present: - .nobackup - .NOBACKUP storage: encryption_passphrase: "MYREPOPASSWORD" compression: none ssh_command: ssh -i /root/.ssh/id_rsa archive_name_format: 'backup-{now}' retention: keep_hourly: 2 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 10 prefix: 'backup-' consistency: checks: - repository - archives prefix: 'backup-' hooks: before_backup: - echo "Starting a backup." after_backup: - echo "Finished a backup." on_error: - echo "Error during prune/create/check." mysql_databases: - name: all hostname: 192.168.200.37 password: MYSQLPASSWD healthchecks: https://hc-ping.com/MYUUID Edited August 22, 2022 by sdub 5 Quote Link to comment
sdub Posted November 22, 2020 Author Share Posted November 22, 2020 [Tutorial] Borgmatic now available in CA Appstore (aka the NEW best method to back up your data) : unRAID (reddit.com) 1 Quote Link to comment
T0a Posted November 22, 2020 Share Posted November 22, 2020 (edited) Hi sdub, this Borg integration looks promising to me. Thanks for taking the time creating the container and making it available to the community. I will definitely check it out and may consider it as a replacement for my local rsync and remote rclone offsite backup. Will report back! Quote Flash drive and appdata are incrementally backed up (alternative to CA backup utility) How do you make sure that files are not getting written by your docker containers while the backup is running? The CA backup stops containers to prevent file corruption AFAIK. I cannot see such a mechanism in your solution. Technically, this would be possible with the before_backup and after_backup hooks. Not sure, if any further/similar steps needs to be taken into account for the flash drive. May be worth looking into the CA backup code to review the protection mechanisms. Edited November 22, 2020 by T0a Quote Link to comment
laur Posted November 23, 2020 Share Posted November 23, 2020 9 hours ago, T0a said: How do you make sure that files are not getting written by your docker containers while the backup is running? Borgmatic supports hooks, so you could likely use those to interact with docker. I wrote alternative solution to borgmatic with less dependencies that addresses this. Although borgmatic absolutely is a nice tool to work with. Quote Link to comment
cheesemarathon Posted November 23, 2020 Share Posted November 23, 2020 Hi, im having issues with SSH key permissions. What should i set them to? Quote Link to comment
laur Posted November 23, 2020 Share Posted November 23, 2020 (edited) 3 hours ago, cheesemarathon said: Hi, im having issues with SSH key permissions. What should i set them to? find -L "/root/.ssh" \( -type f -o -type d \) -exec chmod 'u=rwX,g=,o=' -- '{}' \+ (this will guarantee the dirs will get the executable bit set) Edited November 24, 2020 by laur Quote Link to comment
laur Posted November 24, 2020 Share Posted November 24, 2020 (edited) > This docker can be used with the Unraid rclone plugin if you wish to mirror your repo to a supported cloud service. Note this goes against borg recommendation. > Files cache set to use "mtime,size" - Very important as unRAID does not have persistent inode values That's a great point! Will amend my setup. Why did you change the default 'ctime' to 'mtime' though? Edited November 24, 2020 by laur Quote Link to comment
cheesemarathon Posted November 25, 2020 Share Posted November 25, 2020 On 11/23/2020 at 10:17 PM, laur said: find -L "/root/.ssh" \( -type f -o -type d \) -exec chmod 'u=rwX,g=,o=' -- '{}' \+ (this will guarantee the dirs will get the executable bit set) Still having issues: crond: USER root pid 8 cmd PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1 /etc/borgmatic.d/config.yaml: Running command for pre-backup hook Wed Nov 25 19:19:00 UTC 2020 - Starting backup [email protected]:repo: Pruning archives Remote: [email protected]: Permission denied (publickey). Connection closed by remote host. Is borg working on the server? [email protected]:repo: Error running actions for repository Command 'borg prune --keep-daily 3 --keep-weekly 4 --keep-monthly 12 --keep-yearly 2 --prefix {hostname}- --stats --info [email protected]:repo' returned non-zero exit status 2. /etc/borgmatic.d/config.yaml: Error running configuration file summary: /etc/borgmatic.d/config.yaml: Error running configuration file [email protected]:repo: Error running actions for repository Remote: [email protected]: Permission denied (publickey). Connection closed by remote host. Is borg working on the server? Command 'borg prune --keep-daily 3 --keep-weekly 4 --keep-monthly 12 --keep-yearly 2 --prefix {hostname}- --stats --info [email protected]:repo' returned non-zero exit status 2. Need some help? https://torsion.org/borgmatic/#issues File perms: ~/.ssh # ls -al total 8 drwx------ 1 99 users 40 Nov 22 20:24 . drwx------ 1 root root 58 Nov 25 19:13 .. -rw------- 1 root root 464 Nov 22 19:39 borgmatic -rw------- 1 root root 202 Nov 22 19:44 known_hosts Quote Link to comment
laur Posted November 28, 2020 Share Posted November 28, 2020 (edited) > ~/.ssh # ls -al You don't have any keys in ~/.ssh/ Either generate them, or mount /boot/config/ssh/ or /root/.ssh (ie wherever on host you have the keys) to container's /root/.ssh Note in the parent post example you can see the example expects key to be found at ~/.ssh/id_rsa: storage: ssh_command: ssh -i /root/.ssh/id_rsa <-- keyfile for ssh Yet your dir listing above shows there's no key to be found. Edited November 28, 2020 by laur Quote Link to comment
sdub Posted December 12, 2020 Author Share Posted December 12, 2020 (edited) Sorry for the slow replies... for some reason I stopped getting post notifications. just turned them back on. On 11/24/2020 at 6:50 AM, laur said: > This docker can be used with the Unraid rclone plugin if you wish to mirror your repo to a supported cloud service. Note this goes against borg recommendation. Yes, I realize that... just listed that as an option for those using a remote cloud service without borg installed. The only real option for the borg recommended solution is to backup to something like rsync.net/borgbase or to backup to a family/friend's server runnnig borg. For everyone else, the only option is to use rsync/rclone and hope you don't propagate errors. I personally backup to a server I set up at a family member's house for my remote backups. Quote > Files cache set to use "mtime,size" - Very important as unRAID does not have persistent inode values That's a great point! Will amend my setup. Why did you change the default 'ctime' to 'mtime' though? I based that on the original tutorial from ds-unraid. I suppose the rationale is that you care more about when the files contents have been modified vs when the file properties have changed. ctime is a superset of mtime, so I suppose you could use that and it should also work, though I'm not sure why you'd want to re-backup a file whose contents haven't been modified. I'm sure there is a scenario where that makes more sense though. Edited December 12, 2020 by sdub Quote Link to comment
sdub Posted December 12, 2020 Author Share Posted December 12, 2020 On 11/25/2020 at 2:22 PM, cheesemarathon said: Still having issues: Hopefully you got your SSH issues sorted... it sounds like laur had the right advice. For anyone else that finds this, I'd recommend opening a shell into the borgmatic container and try SSH'ing from there. If password-less login doesn't work from there, Borgmatic isn't going to work either. Quote Link to comment
sdub Posted December 12, 2020 Author Share Posted December 12, 2020 (edited) Sorry for not answering sooner... my post notifications were accidentally disabled. On 11/22/2020 at 3:51 PM, T0a said: How do you make sure that files are not getting written by your docker containers while the backup is running? The CA backup stops containers to prevent file corruption AFAIK. I cannot see such a mechanism in your solution. Technically, this would be possible with the before_backup and after_backup hooks. That's a great point that I could start/stop docker using the hooks, but I don't want my system down for 4 hrs a day (I run an hour long local and hour long remote backup 2x daily) Not having docker downtime was a significant reason I didn't want to use the CA backup solution. In theory I could minimize this by having a separate borg archive for JUST appdata so the backup would be quicker but with Plex I have a huge number of small files, so it's still longer than I prefer. My rationalization for my approach is twofold... I'm not sure what the odds are that the file in the backup could get corrupted, but it's somewhere between "unlikely" and "possible". The only files that I'd be really worried about are the filesystem-based databases like SQLite. Since I'm doing 2x daily backups, the odds of having consecutive corrupted files backed up seems very, very small. Most of the programs that use SQLite (Plex, 'arrs, etc.) have the option in-app for periodic database backups. Those backups get ingested into the archives, and I don't have to worry about them being corrupted. About the only one that doesn't have this is my Grafana/InfluxDB docker, but I'm not particularly concerned about losing this data. If I were concerned, I'm sure I could find a way to have it dump periodic DB images. If this seems dubious to you, please let me know why... it's just my thought process. Quote Not sure, if any further/similar steps needs to be taken into account for the flash drive. May be worth looking into the CA backup code to review the protection mechanisms. I'm not really sure either... I understand that flash backups will be coming in Unraid 6.9 when it releases, so I'll probably just take my chances until then. Edited December 12, 2020 by sdub Quote Link to comment
T0a Posted December 16, 2020 Share Posted December 16, 2020 Quote If this seems dubious to you, please let me know why... it's just my thought process. No, I totally get your point and your arguments seems reasonable to me. I run a docker container with a filesystem-based SQLite database that has no build-in database export like you mentioned. That's why I asked this question in the first place. I installed your container today and I really like it so far. Up until now, I did my offsite backups to Google drive via rclone. However, this solution didn't let me sleep well to be honest - especially the Google drive part. So, I ordered a storage box from Hetzner today and did my first backup to it with borgmatic! The last piece missing is stopping the docker container I mentioned above. The plan is to use "HA dockermon" from within the borgmatic container. Would you mind adding curl to the docker container for me? Then, I would be able to stop any container via: curl -v -X POST <ha_dockermon_ip>:8126/container/container_name --header 'content-type: application/octet-stream' --data '{"state": "stop"}' Thanks for bringing borgmatic to my attention Quote Link to comment
sdub Posted December 16, 2020 Author Share Posted December 16, 2020 1 hour ago, T0a said: The last piece missing is stopping the docker container I mentioned above. The plan is to use "HA dockermon" from within the borgmatic container. Would you mind adding curl to the docker container for me? Then, I would be able to stop any container via: curl -v -X POST <ha_dockermon_ip>:8126/container/container_name --header 'content-type: application/octet-stream' --data '{"state": "stop"}' I submitted a feature request to the docker maintainer for this... seems pretty straightforward. I could fork the docker, but I'd rather stay tied to the base image. Quote Link to comment
Greygoose Posted December 19, 2020 Share Posted December 19, 2020 (edited) @sdub Testing this now, I will report back. Thank you I have this working using a local machine as a backup location. I now want to implement rclone, to have borgmatic upload to the remote folder, how can I implement this using the inbuilt fuse mount capabilities of borgmatic instead of using the USERSCRIPTS pluging to mount rclone. Edited December 20, 2020 by Greygoose Quote Link to comment
sdub Posted December 20, 2020 Author Share Posted December 20, 2020 On 12/16/2020 at 1:04 PM, T0a said: The last piece missing is stopping the docker container I mentioned above. The plan is to use "HA dockermon" from within the borgmatic container. Would you mind adding curl to the docker container for me? OK, curl is now part of the docker. You will need to go to the Docker tab and do a "force update" to get it. Quote Link to comment
sdub Posted December 20, 2020 Author Share Posted December 20, 2020 On 12/19/2020 at 9:31 AM, Greygoose said: I have this working using a local machine as a backup location. I now want to implement rclone, to have borgmatic upload to the remote folder, how can I implement this using the inbuilt fuse mount capabilities of borgmatic instead of using the USERSCRIPTS pluging to mount rclone. I'm far from an rclone expert, but I'm not sure how you're proposing to use fuse... in borg, it's used to expose the contents of a Borg repo to extract files. If you indeed want to do this for whatever reason, you can follow the borg mount documentation. Mount the repo to /mnt/fuse and you should be able to see it from the Unraid host via the "Fuse mount point" path in the borgmatic container config. What I assume you really want to do is sync the borg repo to a cloud storage provider. I'm afraid rclone is not part of this container, so you will need to do it separately. I can think of 3 options: 1. There's a good SpaceinvaderOne tutorial out there if you want to use the rclone plugin available in the CA store (waseh), but that's basically no different than installing a user script (not my preference). If you go this route, you could invoke rclone to start on the Unraid host from the docker container via an ssh command. Something like "ssh root@hostname rclone sync [opts]" 2. (Preferred) If you really want to avoid installing stuff to your base image altogether, I'd recommend either installing a dedicated rclone docker like "pfidr34/docker-rclone" or the one available in the CA store (thomast_88). You could then perform the rclone asynchonously on it's own cron schedule, but you need to be careful that they don't run at the same time. If you want to automatically run rclone using the "after_backup" hook, You could execute a command that invokes the rclone command in another container from within the borgmatic container. Something like "ssh root@hostname docker exec "rclone" rclone sync [opts]". 3. A final option is to install a single docker container with both borgmatic and rclone installed. There isn't one in the CA store, so you'll need to install from docker hub with a custom template, but it's totally doable. Here's one that looks like it would work: https://hub.docker.com/r/knthmn/borgmatic-rclone/ Quote Link to comment
sdub Posted December 20, 2020 Author Share Posted December 20, 2020 (edited) 1 hour ago, sdub said: OK, curl is now part of the docker. You will need to go to the Docker tab and do a "force update" to get it. @T0a It occurred to me that you could also accomplish this without using HA Dockermon or curl by just executing an "ssh root@host docker stop CONTAINER" command directly. Edited December 20, 2020 by sdub Quote Link to comment
Greygoose Posted December 20, 2020 Share Posted December 20, 2020 @sdub Thank you so much, you have given me many options and I appreciate you taking the time for such a detailed reply. I will post back once I have it working Quote Link to comment
Greygoose Posted December 21, 2020 Share Posted December 21, 2020 @sdub I have an issue which i'm not sure is my configuration or is not possible at all. I have the waseh rclone plugin installed, unraid is set to boot and mounts the OneDrive folder using the user scripts. I then have the folder set as a repository in the Borgmatic docker, I can see the folder mounts correctly in Borgmatic console., When I start the backup in borgmatic after init a new repo on the OneDrive rclone folder it backs up for a few minutes then the array crashes and I need to reboot. When I login to onedrive I can see the backup has started to upload onto the microsoft servers. I have tried a few times and always the same outcome. Quote Link to comment
Greygoose Posted December 22, 2020 Share Posted December 22, 2020 (edited) @sdub I am in the process of setting this up with borgbase. I have registered an account (free) and setup a test repository, I have uploaded my SSH key but having no luck with the connection within the docker console. I have attached 2 images below. The console image shows me trying to connect to the Edited December 22, 2020 by Greygoose Quote Link to comment
Greygoose Posted December 24, 2020 Share Posted December 24, 2020 The above post was my error. The creation of the key needs to be done in the docker container, as explained on the borgbase site. https://docs.borgbase.com/faq/ssh/ Now working with borgbase without any issues. Quote Link to comment
Greygoose Posted December 27, 2020 Share Posted December 27, 2020 Please can you confirm the crontab.txt is just copied into the config folder and nothing else is required for automatic backups? It isn't working my end. crontab.txt 0 1,16 * * * PATH=$PATH:/usr/bin /usr/bin/borgmatic prune create -v 1 --stats 2>&1 0 6 * * 3 PATH=$PATH:/usr/bin /usr/bin/borgmatic check -v 1 2>&1 Quote Link to comment
T0a Posted December 27, 2020 Share Posted December 27, 2020 (edited) 32 minutes ago, Greygoose said: Please can you confirm the crontab.txt is just copied into the config folder and nothing else is required for automatic backups? It isn't working my end. What does the borgmatic Docker log tell you? For me the crontab.txt in the right place is all I need for the automation (despite the config.yaml and keys for sure) Edited December 27, 2020 by T0a Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.