[Support] borgmatic

[Greygoose]

Ive been having issues with borgmatic not running.

 

What do these logs mean

 

 

[witten]

Those logs look pretty normal to me. 🤷‍♂️ Maybe increase your borgmatic verbosity in your crontab to get more details about what might be going on?

[DeanA]

On 4/15/2023 at 2:05 PM, darkside40 said:

Was something recently changed in this Image?

Using it for over a year but in the past couple of weeks it seems that the cron runs quite random. Normally it should rum at 18:00 every day, but now it sometimes simply runs the script when the server comes from sleep etc.

 

Anyone also experienced this?

I have had Borgmatic running weekly backups for about a year thanks to all of the setup tips in this forum. Over the last couple of weeks I had noticed that my backups have stopped running regulary. When I checked in more detail, seems like this started back in Feb. Also, at about the same time, I have been unable to update Borgmatic version without using "force update". I am a real novice at this stuff, so any suggestions on how to trouble shoot would be most welcome. Thanks in advance.

[sdub]

2 hours ago, DeanA said:

I have had Borgmatic running weekly backups for about a year thanks to all of the setup tips in this forum. Over the last couple of weeks I had noticed that my backups have stopped running regulary. When I checked in more detail, seems like this started back in Feb. Also, at about the same time, I have been unable to update Borgmatic version without using "force update". I am a real novice at this stuff, so any suggestions on how to trouble shoot would be most welcome. Thanks in advance.

 

It's going to be hard to help debug this without some specifics from the error logs.  Does it simply not run, or does it occasionally fail? If it fails, how does it fail?  If the folks on this thread can't help, maybe a support ticket over at Borgmatic docker github could help.  Remember I only maintain the docker template in the CA Appstore. @witten is the actual developer of the docker image.

 

The automatic docker updates in Unraid are handled by the CA APPLICATION AUTO UPDATE plugin. As long as your repository is set to "ghcr.io/borgmatic-collective/borgmatic" it should work.  Note that this location changed somewhat recently, but shouldn't explain why "auto-update" vs "forced update" behaves differently.  Maybe consult that support thread for help.

 

 

[DeanA]

On 4/23/2023 at 8:08 PM, sdub said:

 

It's going to be hard to help debug this without some specifics from the error logs.  Does it simply not run, or does it occasionally fail? If it fails, how does it fail?  If the folks on this thread can't help, maybe a support ticket over at Borgmatic docker github could help.  Remember I only maintain the docker template in the CA Appstore. @witten is the actual developer of the docker image.

 

The automatic docker updates in Unraid are handled by the CA APPLICATION AUTO UPDATE plugin. As long as your repository is set to "ghcr.io/borgmatic-collective/borgmatic" it should work.  Note that this location changed somewhat recently, but shouldn't explain why "auto-update" vs "forced update" behaves differently.  Maybe consult that support thread for help.

 

 

@sdub thanks for the response. I have multiple .yaml files that run to back up different data. I seem to be getting the same error. I have added it here just incase anyone can offer some advice on a best next step?

 

 

 

Edited April 25, 2023 by DeanA

[sdub]

Borg creates a lock to make sure it has exclusive access to the backup archive and cache to avoid corruption.  When a backup gets  interrupted, that lock is sometimes left in place, blocking subsequent backups.  Whenever you see an error about not being able to acquire a lock, the first thing you need to do is break the lock if you are sure that no borg process is running against that repo.

 

From inside the borg container (docker exec -it borgmatic /bin/sh), run following command and try again.

borg break-lock /mnt/borg-repository

 

[witten]

This may also work if you don't want to type the repository path (or you want to use a borgmatic repository label to select the respository):

 

borgmatic break-lock

 

The one thing to look out for when breaking a Borg lock is first make sure Borg isn't actively running! Because if it is then you'll be breaking the lock of a running process and potentially accessing the repository from multiple different processes at the same time. That's probably not what's going on here, but it's just something to look out for.

Edited April 25, 2023 by witten

[DeanA]

6 minutes ago, witten said:

This may also work if you don't want to type the repository path (or you want to use a borgmatic repository label to select the respository):

 

borgmatic break-lock

 

The one thing to look out for when breaking a Borg lock is first make sure Borg isn't actively running! Because if it is then you'll be breaking the lock of a running process and potentially accessing the repository from multiple different processes at the same time. That's probably not what's going on here, but it's just something to look out for.

sorry if this is a really stupid question. How do I see if there is a process already running pls?

[sdub]

ps -ax

 

[Solverz]

4 hours ago, DeanA said:

sorry if this is a really stupid question. How do I see if there is a process already running pls?

 

4 hours ago, sdub said:

ps -ax

 

 

grep for the borg process so you do not have to look through all the lines manually.

 

ps -aux | grep borg

[Greygoose]

is this the default borgmatic repo

 

As i have had this popup in my log

 

[stridemat]

On 4/30/2023 at 10:46 AM, Greygoose said:

is this the default borgmatic repo

 

As i have had this popup in my log

 

Im having the same. Any ideas?

[stridemat]

Is this container still being supported?

Edited June 18, 2023 by stridemat

[sdub]

I haven’t updated the CA container in some time just because it hasn’t been necessary. The docker container it pulls from is still very actively developed and widely used, so yes it’s still active. If needed I’ll update the CA template. 

[sdub]

On 5/24/2023 at 3:52 PM, stridemat said:

Im having the same. Any ideas?

Sorry I missed this issue. I’ll take a look and get back to you… for most issues it’s more effective to ask in the Borgmatic GitHub support page.

 

last year they moved the Borgmatic repo from b3vis’s repo to the main “Borgmatic-collective” repo. I updated the template but maybe something else changed. 
 

that specific error at first glance looks like the link to the icon just changed. Is it causing Borgmatic to not update/work for you?

Edited June 18, 2023 by sdub

[stridemat]

10 minutes ago, sdub said:

Sorry I missed this issue. I’ll take a look and get back to you… for most issues it’s more effective to ask in the Borgmatic GitHub support page.

 

last year they moved the Borgmatic repo from b3vis’s repo to the main “Borgmatic-collective” repo. I updated the template but maybe something else changed. 
 

that specific error at first glance looks like the link to the icon just changed. Is it causing Borgmatic to not update/work for you?

Container is still working great (apart from the icon), but I was hoping it wasn’t the first sign of the container no longer being supported moving forward. 
 

Thanks for you work  

[Johnny4233]

hi, thank you for this.

 

I managed to get this up. Works good for me.

 

only one thing is left: I can extract the backups to an specific unraid location. but if I try to mount from a repo, the files are accessible in the docker container shell, but I can not see it in unraid file system. 

 

device dev/fuse in docker template is set, and the folder is mounted to the container.

 

anything I forgot?

 

regards

Jan

[sdub]

3 hours ago, Johnny4233 said:

hi, thank you for this.

 

I managed to get this up. Works good for me.

 

only one thing is left: I can extract the backups to an specific unraid location. but if I try to mount from a repo, the files are accessible in the docker container shell, but I can not see it in unraid file system. 

 

device dev/fuse in docker template is set, and the folder is mounted to the container.

 

anything I forgot?

 

regards

Jan

I’m afraid I’ve never gotten this to work in the docker container. I’ve tried passing the fuse device in, but I can’t access the share externally. I may have the permissions wrong or maybe it’s something else. 
 

what I’ve always done is use this image to create the backup image then use the Borgmatic binary on the host system (either Unraid or some other host OS that can access the borg repo on the LAN) to browse and extract the files I need. Borg and llfuse can be installed from the Nerdpack in Unraid. 
 

if someone else has been able to expose a mounted repo from outside the docker container I’d love to hear how. 

Edited June 18, 2023 by sdub

[sdub]

OK, @Greygoose and @stridemat, I updated the CA template to point to a static location for the borgmatic icon.  Future pulls should get this, but I'm not sure if it will automatically update for you.  

 

To manually fix, go to advanced view in the Borgmatic docker container config.  Change "Icon URL" to 

https://raw.githubusercontent.com/Sdub76/unraid_docker_templates/main/images/borgmatic.png

 

This is a static copy that I have in the CA repo, so it shouldn't change unless github changes it's static URLs.

[Revan335]

What's the Different between yours and this Borg Backup?

 

 

[sdub]

2 hours ago, Revan335 said:

What's the Different between yours and this Borg Backup?

 

 

 

To reduce data transmission, Borg requires the borg binary to be present on both the local machine and the remote machine when performing remote backups.  That way the remote machine can do the work of comparing what's changed from the cache.  

 

If you don't want to (or can't) install borg to the host OS, you can run nold360's borgserver container on the remote machine.  This image has just the plain "borg" binary and an SSH server to allow it to act as a borg "target".  No borgmatic, but allows you to connect and perform a backup "to" that remotely running docker container.

 

If you were running a backup scheme where computer A backs up to computer B and vice versa, an ideal scenario would be just having the 2 "borgmatic" docker containers talk to each other.  Unfortunately, the official borgmatic container lacks an SSH server so you can't.  It's been requested to add SSH server support but they've declined to do so to keep the image simple.   This CA application uses the "official" borgmatic container published by the borgmatic team.

 

The alternative to using the "borgserver" container in Unraid is to use the Nerdpack to install borg directly to the host Unraid OS, since it's already running the requred SSH server.

 

If you were just using vanilla borg and not borgmatic, you certainly could run the "borgserver" container on both the local and remote systems and need nothing else.  

 

Hope that clarifies.  

Edited June 21, 2023 by sdub
clarification

[stridemat]

3 hours ago, sdub said:

OK, @Greygoose and @stridemat, I updated the CA template to point to a static location for the borgmatic icon.  Future pulls should get this, but I'm not sure if it will automatically update for you.  

 

To manually fix, go to advanced view in the Borgmatic docker container config.  Change "Icon URL" to 

https://raw.githubusercontent.com/Sdub76/unraid_docker_templates/main/images/borgmatic.png

 

This is a static copy that I have in the CA repo, so it shouldn't change unless github changes it's static URLs.

That worked. Thank you. 

[witten]

On 6/18/2023 at 10:56 AM, Johnny4233 said:

only one thing is left: I can extract the backups to an specific unraid location. but if I try to mount from a repo, the files are accessible in the docker container shell, but I can not see it in unraid file system.

 

I don't know how easy the following is to do with Unraid, but here's an example of this working:

# mkdir -p /root/test/mnt
# docker run -it --privileged --mount type=bind,source=/root/test,target=/test,bind-propagation=rshared --device /dev/fuse -v /root/repo:/repo b3vis/borgmatic bash
f069c33ce2cc:/# borg mount /repo/repo.borg::test /test/mnt

After that, the mounted archive show up within /test/mnt on the host. The main downside is you need to the run container as privileged. There might be a way to make this work with --add-cap instead, but that's beyond my Docker skills.

 

More background here: https://stackoverflow.com/a/53632103

[Killabee44]

Hi all,

 

Can someone please give me a hand in figuring this error out? I take it to mean that borg can't get a lock? I saw a "read only file system" error so I tried changing the permissions to the share:

chmod -R 775 /mnt/user/disks/easystore264D

But still get the error:

summary:
/etc/borgmatic.d/config.yml: Loading configuration file
/etc/borgmatic.d/config.yml: An error occurred
/mnt/user/disks/easystore264D: Error running actions for repository
...
                ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 5144, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 168, in wrapper
    with repository:
  File "/usr/local/lib/python3.11/site-packages/borg/repository.py", line 200, in __enter__
    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
  File "/usr/local/lib/python3.11/site-packages/borg/repository.py", line 432, in open
    self.lock = Lock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait).acquire()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/locking.py", line 384, in acquire
    self._wait_for_readers_finishing(remove, sleep)
  File "/usr/local/lib/python3.11/site-packages/borg/locking.py", line 397, in _wait_for_readers_finishing
    self._lock.acquire()
  File "/usr/local/lib/python3.11/site-packages/borg/locking.py", line 137, in acquire
    raise LockFailed(self.path, str(err)) from None
borg.locking.LockFailed: Failed to create/acquire the lock /mnt/user/disks/easystore264D/lock.exclusive ([Errno 30] Read-only file system: '/mnt/user/disks/easystore264D/lock.exclusive.ofdect2c.tmp').
Platform: Linux 60166bcbb7d8 6.1.34-Unraid #1 SMP PREEMPT_DYNAMIC Fri Jun 16 11:48:38 PDT 2023 x86_64
Linux: Unknown Linux
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.4 [pyfuse3,llfuse]
PID: 67  CWD: /
sys.argv: ['/usr/local/bin/borg', 'create', '--patterns-from', '/tmp/tmpvyvh0ei6', '--exclude-if-present', '.nobackup', '--exclude-if-present', '.NOBACKUP', '--compression', 'none', '--one-file-system', '--read-special', '--files-cache', 'mtime,size', '--list', '--filter', 'AMEx-', '/mnt/user/disks/easystore264D::backup-{now}', '--stats', '--debug', '--show-rc']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
Command 'borg create --patterns-from /tmp/tmpvyvh0ei6 --exclude-if-present .nobackup --exclude-if-present .NOBACKUP --compression none --one-file-system --read-special --files-cache mtime,size --list --filter AMEx- /mnt/user/disks/easystore264D::backup-{now} --stats --debug --show-rc' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

Thanks!

Edited July 9, 2023 by Killabee44

[sdub]

1 hour ago, Killabee44 said:

Hi all,

 

Can someone please give me a hand in figuring this error out? I take it to mean that borg can't get a lock? I saw a "read only file system" error so I tried changing the permissions to the share:

chmod -R 775 /mnt/user/disks/easystore264D

But still get the error:

summary:
/etc/borgmatic.d/config.yml: Loading configuration file
/etc/borgmatic.d/config.yml: An error occurred
/mnt/user/disks/easystore264D: Error running actions for repository
...
                ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 5144, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 168, in wrapper
    with repository:
  File "/usr/local/lib/python3.11/site-packages/borg/repository.py", line 200, in __enter__
    self.open(self.path, bool(self.exclusive), lock_wait=self.lock_wait, lock=self.do_lock)
  File "/usr/local/lib/python3.11/site-packages/borg/repository.py", line 432, in open
    self.lock = Lock(os.path.join(path, 'lock'), exclusive, timeout=lock_wait).acquire()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/borg/locking.py", line 384, in acquire
    self._wait_for_readers_finishing(remove, sleep)
  File "/usr/local/lib/python3.11/site-packages/borg/locking.py", line 397, in _wait_for_readers_finishing
    self._lock.acquire()
  File "/usr/local/lib/python3.11/site-packages/borg/locking.py", line 137, in acquire
    raise LockFailed(self.path, str(err)) from None
borg.locking.LockFailed: Failed to create/acquire the lock /mnt/user/disks/easystore264D/lock.exclusive ([Errno 30] Read-only file system: '/mnt/user/disks/easystore264D/lock.exclusive.ofdect2c.tmp').
Platform: Linux 60166bcbb7d8 6.1.34-Unraid #1 SMP PREEMPT_DYNAMIC Fri Jun 16 11:48:38 PDT 2023 x86_64
Linux: Unknown Linux
Borg: 1.2.4  Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.4 [pyfuse3,llfuse]
PID: 67  CWD: /
sys.argv: ['/usr/local/bin/borg', 'create', '--patterns-from', '/tmp/tmpvyvh0ei6', '--exclude-if-present', '.nobackup', '--exclude-if-present', '.NOBACKUP', '--compression', 'none', '--one-file-system', '--read-special', '--files-cache', 'mtime,size', '--list', '--filter', 'AMEx-', '/mnt/user/disks/easystore264D::backup-{now}', '--stats', '--debug', '--show-rc']
SSH_ORIGINAL_COMMAND: None
terminating with error status, rc 2
Command 'borg create --patterns-from /tmp/tmpvyvh0ei6 --exclude-if-present .nobackup --exclude-if-present .NOBACKUP --compression none --one-file-system --read-special --files-cache mtime,size --list --filter AMEx- /mnt/user/disks/easystore264D::backup-{now} --stats --debug --show-rc' returned non-zero exit status 2.

Need some help? https://torsion.org/borgmatic/#issues

Thanks!

It looks to me like maybe you are mounting the share to Borgmatic as read-only… not that it’s read only in the host filesystem. What’s your Borgmatic docker config look like?