[Support] borgmatic

[sdub]

Thanks for this… I updated the first post to remove the path statement from the suggested crontab config. 
 

Underlines the importance of monitoring your backups with something like Healthchecks (can be self hosted) so you know if something breaks ASAP!

Edited August 22, 2022 by sdub

[Solverz]

8 minutes ago, sdub said:

Thanks for this… I updated the first post to remove the path statement from the suggested crontab config. 
 

Underlines the importance of monitoring your backups with something like Healthchecks (can be self hosted) so you know if something breaks ASAP!

No worries!

 

Yeah healthchecks is my favourite out of the options definitely and bonus it can be self hosted ☺

 

Oh BTW, I correct the cron schedule expression syntax, there was an extra few spaces I noticed. So it should be like the below:

 

0 6 * * 3

 

[Greygoose]

Thanks Solverz, I appreciate the help.

 

sdub, thanks again for the support and time spent on this container.

[TrikkStar]

I'd recommend also adding a regular compact action, as I just ran it for the first time in over a year and.... `compaction freed about 267.57 GB repository space.`

[Solverz]

16 hours ago, TrikkStar said:

I'd recommend also adding a regular compact action, as I just ran it for the first time in over a year and.... `compaction freed about 267.57 GB repository space.`

The compaction feature has been out since Borg 1.2+, so if you only now aware of this, I would definitely recommend reading through the borgmatic and borg documentation as quite a few new features and changes have been added which may trigger you to change your method using borgmatic.

 

For example, the syntax you use to specify remote repos and the consistency checks scheduling.

[Howboys]

Do i need the `/mnt/fuse/:mnt/borg` mount for the container? Fix Common Problems says borg should not exist in /mnt.

 

These are the mounts I have:

Edited September 9, 2022 by Howboys

[sdub]

15 minutes ago, Howboys said:

Do i need the `/mnt/fuse/:mnt/borg` mount for the container? Fix Common Problems says borg should not exist in /mnt.

 

These are the mounts I have:

 

 

 

 

no it’s not necessary. The idea was to have a mount point for mounting archives somewhere you could get to them outside of the container and /mnt seems like the proper place. Either ignore the error, delete the mount point, or pick another mount point that fox common problems doesn’t complain about. 

[Solverz]

1 hour ago, Howboys said:

Do i need the `/mnt/fuse/:mnt/borg` mount for the container? Fix Common Problems says borg should not exist in /mnt.

 

These are the mounts I have:

 

1 hour ago, sdub said:

 

 

no it’s not necessary. The idea was to have a mount point for mounting archives somewhere you could get to them outside of the container and /mnt seems like the proper place. Either ignore the error, delete the mount point, or pick another mount point that fox common problems doesn’t complain about. 

/mnt/borg makes sense in normal distros but for unraid, it is best to put it in /mnt/disks/borg

[stridemat]

With the recent demise of Nerdpack I am looking to use this container for my Borg usage. Though I am having trouble trying to configure. 

 

A few questions which I hope someone could answer:

 

1. It is possible to 'reuse' existing repositories from Borg installed via Nerdpack? Or am I better just starting again?
2. My existing repo is located /mnt/disks/WD_Elements_2620/UnraidBackup (I also have another repo at the following location /mnt/disks/WD_Elements_2620/DocsBackup. I have read this thread and can see the links as to how to have multiple repo's etc.) Config page from the Docker tab below. The Borg Repo path takes you to the location of both repos, meaning that I need to specify the specific repo each time. 
3. Config .yaml file text. This looks ok to me, albeit appreciating that I am backing up my CA Back up folder. Once I get this working I plan to use Borg for all rather the CA Backup:

   location:
           - /mnt/user/backups
   repositories:
           - /mnt/borg-repository/UnraidBackup
           - ssh://[email protected]/mnt/mydisk/UnraidBackup
   one_file_system: true
   files_cache: mtime,size
   patterns:
           - '- [Tt]rash'
           - '- [Cc]ache'
   exclude_if_present:
           - .nobackup
           - .NOBACKUP
   storage:
         encryption_passphrase: "NOTMYPASSWORD"
         compression: lz4
         ssh_command: ssh -i /root/.ssh/id_rsa
         archive_name_format: 'backup-{now}'
   retention:
         keep_weekly: 4
   consistency:
       checks:
           - repository
           - archives
       prefix: 'backup-'
   hooks:
       before_backup:
           - echo "Starting Unraid backup."
       after_backup:
           - echo "Finished Unraid backup."
       on_error:
           - echo "Error during Unraid backup."

4. Assuming I can use existing repo's, then how could I dryrun. This seems to kick out an error however I try it: borg create --list --dry-run /mnt/borg-repository/UnraidBackup ~/UnraidBackup
5. Anything else that I could be doing?

[Roi]

Hello,

 

as Nerdpack is gone I wanted to ask if it is possible to add borgbackup and an SSH server to this Docker.

I used the borgbackup package from Nerdpack and the system SSH server to offer a Borgbackupserver to some of my servers on the internet. Unfortunately with Nerdpack gone and especially because this was not the best idea from the start using the system SSH server itself I am looking for a better solution here.

 

Maybe @stridemat questions/issues are heading into the same direction.

 

Regards,

Roi

[RinxKninks]

17 hours ago, Roi said:

Hello,

 

as Nerdpack is gone I wanted to ask if it is possible to add borgbackup and an SSH server to this Docker.

I used the borgbackup package from Nerdpack and the system SSH server to offer a Borgbackupserver to some of my servers on the internet. Unfortunately with Nerdpack gone and especially because this was not the best idea from the start using the system SSH server itself I am looking for a better solution here.

 

Maybe @stridemat questions/issues are heading into the same direction.

 

Regards,

Roi

NerdPack is gone and after a while I found NerdTools to replace them on UR 6.11.x.
Yet I also decided on getting into and using borgmatic.

[Roi]

3 hours ago, RinxKninks said:

NerdPack is gone and after a while I found NerdTools to replace them on UR 6.11.x.
Yet I also decided on getting into and using borgmatic.

Ah wow thank you. That did not come to my attention, yet. Great!

 

So I can use borgbackup in the meantime like before.

 

As described it would be better to have borgbackup inside a Docker container for security reasons, so I would like to stick to my suggestion adding an SSH daemon and borgbackup to this container.

[stridemat]

Hi all,

 

So I have managed to get a two config set up going how I want manually (yay) but cannot work out how to auto start each one on a different schedule using cron. 

 

Ideally they would run on different schedules due to the nature of what each one is backing up and how often it changes. 
 

Pease can someone point me in the right direction?

[stridemat]

On 10/25/2022 at 4:35 AM, stridemat said:

Hi all,

 

So I have managed to get a two config set up going how I want manually (yay) but cannot work out how to auto start each one on a different schedule using cron. 

 

Ideally they would run on different schedules due to the nature of what each one is backing up and how often it changes. 
 

Pease can someone point me in the right direction?

So have resolved. Posting here for others in the future. Crontab text as follows to call specific config files. Obviously change the paths / file names as required:

 

0 2 * * 1 borgmatic -c /etc/borgmatic.d/UnraidBackup.yaml prune create -v 1 --stats 2>&1
0 4 * * * borgmatic -c /etc/borgmatic.d/DocsBackup.yaml prune create -v 1 --stats 2>&1


0 6 15 * * borgmatic check -v 1 2>&1

 

[stridemat]

On 3/23/2022 at 2:29 PM, sdub said:

 

The package maintainer declined to add ssh-server to the borgmatic docker container... see the discussion in the link.  You'll either need to install this simple borgserver docker or you'll need the borg binary installed on the machine you're trying to remotely backup to.  If the target is also running Unraid this is easy enough with nerdpack.

 

 

 

On 10/19/2022 at 11:35 AM, Roi said:

Ah wow thank you. That did not come to my attention, yet. Great!

 

So I can use borgbackup in the meantime like before.

 

As described it would be better to have borgbackup inside a Docker container for security reasons, so I would like to stick to my suggestion adding an SSH daemon and borgbackup to this container.

Is the above post from sdub the accepted way of making Unraid the target for backups without altering the base Unraid install? 

Edited November 14, 2022 by stridemat

[sdub]

I think that's

On 11/14/2022 at 2:49 AM, stridemat said:

 

Is the above post from sdub the accepted way of making Unraid the target for backups without altering the base Unraid install? 

 

I think the options are:

1. Install a docker container with sshd and borgbackup, like borgserver (here's one that appears to be more frequently updated)

2. Install borgbackup on just about any Linux or FreeBSD VM under Unraid

3. Install the borgbackup binary somewhere on your host system, either manually in /boot/extra or using NerdTools in 6.11+.

 

Either way, you'll need the associated SSH daemon exposed on a port that's accessible via LAN or VPN if you don't want it open to the internet, which I would never recommend. 

 

I'm not sure any particular method is preferred, though I suppose options 1 or 2 could be marginally more secure and keep your Unraid install closer to "stock" if you're concerned about that.  I personally have borgbackup installed through NerdTools

 

Edited November 28, 2022 by sdub

[tranterspace]

I am getting the following error when running borg within the container. Correspondingly, borgmatic breaks with a non-zero exit status 2. Anybody else seeing this?

 

/ # borg
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 38, in <module>
    from . import helpers
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/__init__.py", line 9, in <module>
    from .checks import *  # NOQA
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/checks.py", line 4, in <module>
    from .errors import Error
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/errors.py", line 3, in <module>
    import borg.crypto.low_level
ImportError: Error loading shared library libcrypto.so.1.1: No such file or directory (needed by /usr/local/lib/python3.10/site-packages/borg/crypto/low_level.cpython-310-x86_64-linux-gnu.so)

 

[xlogiic]

Same Issue here.

[SirBilliamson]

23 hours ago, tranterspace said:

I am getting the following error when running borg within the container. Correspondingly, borgmatic breaks with a non-zero exit status 2. Anybody else seeing this?

 

/ # borg
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 38, in <module>
    from . import helpers
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/__init__.py", line 9, in <module>
    from .checks import *  # NOQA
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/checks.py", line 4, in <module>
    from .errors import Error
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/errors.py", line 3, in <module>
    import borg.crypto.low_level
ImportError: Error loading shared library libcrypto.so.1.1: No such file or directory (needed by /usr/local/lib/python3.10/site-packages/borg/crypto/low_level.cpython-310-x86_64-linux-gnu.so)

 

 

7 minutes ago, xlogiic said:

Same Issue here.

It appears to be an issue with the latest docker image. Temporary fix can be found here - https://github.com/borgmatic-collective/docker-borgmatic/issues/184#issuecomment-1333599796

[sdub]

45 minutes ago, SirBilliamson said:

 

It appears to be an issue with the latest docker image. Temporary fix can be found here - https://github.com/borgmatic-collective/docker-borgmatic/issues/184#issuecomment-1333599796

Yes I’m seeing it now too. Best thing to do in this case is to specify a specific older version until the container is fixed. In the docker config change “Repository:” to read “b3vis/borgmatic:1.7.5”. This will pin it to that specific version. Once it’s fixed change it back to :latest or just leave that part off. 
 

In this specific instance the workaround described at the link seems easy enough but isn’t persistent if the docker container updates itself. Assuming the fix is in the next docker image, it doesn’t really matter. 

Edited December 1, 2022 by sdub

[Roi]

I am seeing this for two days now. The container seems to be updates. So this must be a new problem.

 

borgmatic 1.7.5

apprise 1.2.0
crond: crond (busybox 1.35.0) started, log level 8
crond: USER root pid  14 cmd borgmatic -c /etc/borgmatic.d/config.yaml 2>&1
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 38, in <module>
    from . import helpers
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/__init__.py", line 9, in <module>
    from .checks import *  # NOQA
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/checks.py", line 4, in <module>
    from .errors import Error
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/errors.py", line 3, in <module>
    import borg.crypto.low_level
ImportError: Error loading shared library libcrypto.so.1.1: No such file or directory (needed by /usr/local/lib/python3.10/site-packages/borg/crypto/low_level.cpython-310-x86_64-linux-gnu.so)
/etc/borgmatic.d/config.yaml: Error getting local Borg version
Command '('borg', '--version')' returned non-zero exit status 2.
/etc/borgmatic.d/config.yaml: Error running configuration file

summary:
/etc/borgmatic.d/config.yaml: Error running configuration file
/etc/borgmatic.d/config.yaml: Error getting local Borg version
Command '('borg', '--version')' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

 

[sdub]

3 hours ago, Roi said:

I am seeing this for two days now. The container seems to be updates. So this must be a new problem.

 

borgmatic 1.7.5

apprise 1.2.0
crond: crond (busybox 1.35.0) started, log level 8
crond: USER root pid  14 cmd borgmatic -c /etc/borgmatic.d/config.yaml 2>&1
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/borg/archiver.py", line 38, in <module>
    from . import helpers
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/__init__.py", line 9, in <module>
    from .checks import *  # NOQA
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/checks.py", line 4, in <module>
    from .errors import Error
  File "/usr/local/lib/python3.10/site-packages/borg/helpers/errors.py", line 3, in <module>
    import borg.crypto.low_level
ImportError: Error loading shared library libcrypto.so.1.1: No such file or directory (needed by /usr/local/lib/python3.10/site-packages/borg/crypto/low_level.cpython-310-x86_64-linux-gnu.so)
/etc/borgmatic.d/config.yaml: Error getting local Borg version
Command '('borg', '--version')' returned non-zero exit status 2.
/etc/borgmatic.d/config.yaml: Error running configuration file

summary:
/etc/borgmatic.d/config.yaml: Error running configuration file
/etc/borgmatic.d/config.yaml: Error getting local Borg version
Command '('borg', '--version')' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

 

Did you read the last few posts before posting?

[Roi]

11 minutes ago, sdub said:

Did you read the last few posts before posting?

 

Ehm... No... Thank you! Temporary fix applied...

[Greygoose]

Hello fellow users of this docker,

 

I just want to say thanks to everyone thats contributed to this thread, especially sdub.

 

Borg is rock solid and this docker is superb for unraid.

 

Merry Christmas

[Solverz]

19 minutes ago, RinxKninks said:

So the incredible nice borgmatic does a lot (and more), yet I'd like to

add a "hook". The first hook doing a cp works perfectly an then I'd like to add a rsnapshot provided to unraid via NerdTools.

I can see why this neither works (security) and copying the file to the container's bin (there is busybox stuff inside) also doesen't help either.

 

Why getting such Idea? Well, beside handling the main part of the data by borg(matic) I'd like to avoid to include already deduplicated, encrypted and compressed data coming from other systems into the process of borgmatic by using rsnapshot by "hook".

 

My goal is getting the data (borg+other) to rotating target disks while not using too many scripts, leading to the desire using borgmatic and rsnapshot with one call. Do you guys have ideas on how to orchestrate my wishes elegantly, yet without programming abilities?

 

Please just say what you exactly want to do without using prolonged sentences and fancy words. Your post is like a riddle that we'd need to break down to understand what you are talking about.