- Closed
I don't really know where to start as I think there are happening a few things that shoudn't happen in a few layers.
I set up unraid 1,5 years ago and installed borgmatic for remote backups to hetzner.
Borgmatic and unraid worked fine for over a year. All of a sudden, it stopped. With some investigation, I found the following in the logs:
Quotebackupserver: Error running actions for repository
Command 'borg prune --keep-daily 10 --keep-hourly 2 --keep-monthly 24 --keep-weekly 5 --keep-yearly 10 --glob-archives backup-* ssh://******:23/./borg-backup' returned non-zero exit status 2.
Error during prune/create/check.
/etc/borgmatic.d/config.yaml: An error occurredsummary:
/etc/borgmatic.d/config.yaml: Configuration sections (like location:, storage:, retention:, consistency:, and hooks:) are deprecated and support will be removed from a future release. To prepare for this, move your options out of sections to the global scope.
/etc/borgmatic.d/config.yaml: The healthchecks hook now expects a key/value pair with "ping_url" as a key. String values for this option are deprecated and support will be removed from a future release.
/etc/borgmatic.d/config.yaml: The checks option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release.
/etc/borgmatic.d/config.yaml: The prefix option is deprecated and support will be removed from a future release. Use archive_name_format or match_archives instead.
/etc/borgmatic.d/config.yaml: An error occurred
backupserver: Error running actions for repository
...
^^^^^^^
File "/usr/local/lib/python3.12/site-packages/borg/cache.py", line 382, in local
return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/borg/cache.py", line 502, in __init__
self.commit()
File "/usr/local/lib/python3.12/site-packages/borg/cache.py", line 640, in commit
self.chunks.write(fd)
File "src/borg/hashindex.pyx", line 121, in borg.hashindex.IndexBase.write
File "/usr/local/lib/python3.12/site-packages/borg/crypto/file_integrity.py", line 29, in write
return self.fd.write(data)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/borg/crypto/file_integrity.py", line 74, in write
n = super().write(data)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/borg/crypto/file_integrity.py", line 29, in write
return self.fd.write(data)
^^^^^^^^^^^^^^^^^^^
OSError: [Errno 28] No space left on device
Platform: Linux 6397b9584465 6.6.68-Unraid #1 SMP PREEMPT_DYNAMIC Tue Dec 31 13:42:37 PST 2024 x86_64
Linux: Unknown Linux
Borg: 1.4.0 Python: CPython 3.12.7 msgpack: 1.1.0 fuse: llfuse 1.5.1 [pyfuse3,llfuse]
PID: 153 CWD: /
sys.argv: ['/usr/local/bin/borg', 'prune', '--keep-daily', '10', '--keep-hourly', '2', '--keep-monthly', '24', '--keep-weekly', '5', '--keep-yearly', '10', '--glob-archives', 'backup-*', 'ssh://*******/./borg-backup']
SSH_ORIGINAL_COMMAND: None
Command 'borg prune --keep-daily 10 --keep-hourly 2 --keep-monthly 24 --keep-weekly 5 --keep-yearly 10 --glob-archives backup-* ssh:/***************/./borg-backup' returned non-zero exit status 2.Need some help? https://torsion.org/borgmatic/#issues
I checked every "space" I could imagine. I upgraded the hetzner remote space, I double checked everything locally but all seemed fine with enough space left on the array and cache.
Concerning to me: at the same time, the whole unraid started to act weird. GUI was unresponsive, I got weird, cryptic error messages from time to time, other docker containers stopped repsonding etc etc.
After fiddeling around, I upgraded my RAM form 12 GB to 16 GB and everything worked great again. So I guess the "device" with no space left was my system memory? Alright, fine. Everything went back to normal.
But sure enough, after a few weeks with more stuff to backup (and I assume a bigger database which is written into memory at every borg action?), it broke again. But now I can't just extend my RAM as I maxed out what my system can handle.
Whats weird to me: after a fresh restart of unraid, everything is working great. When I start a backup or prune, the RAM usage of docker goes up to 65%, borg fails, I stop the whole container and instead of (most of) the 65% just beeing freed again, this amount just stays allocated but instead of beeing part of "docker" usage, it turns into "system" usage. A quick look at TOP I can't find any process allocating that much memory. This is the top line of top with all processes allocating <2.5% memory:
QuoteMiB Mem : 15925.6 total, 1118.5 free, 9673.3 used, 13310.9 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 6252.3 avail Mem
And the worst of all: unraid starts acting weird again. In the Main Tab the devices disappear and appear again, repeating once a second. Temperature values are just "*", Boot device is an empty listing. Connecting to shares by SMB isn't possible anymore and the list goes on and on. Only a restart brings everything back to normal (until the next borg run). At times the system was so irresponsive, that I had to shut it down by pulling the plug as there was no way to initiate a reboot in the GUI. This doesn't happen immidiatly/every time, that's why I just refere to unraid acting "weird"
So my issue consists of two parts:
1. How do I get borg(matic) working again without the prossibility to upgrade the memory again? (Which might not be solved here - fair)
2. Any issue of a container in docker shouldn't affect the system in that way, should it? I seems to me, whatever goes wrong, should be handled more gracefully by the system. Just because a container goes rogue (espacially as "not enough RAM" is a predicatable error?), It shouldn't drag the whole system down? (I know how docker is not a VM and more tied to the system by using the same processes... but still?)
I already asked for help in the borgmatic support thread, but the maintainer asked me to move it here as he agrees that there must be broader issue here. As I said, I don't really know where to start. Any help would be appreciated. Where should I start? What info is necessary?
Attached you'll find some documentation "*_before*" and *"_after*". Between these all I did was to start a "borgmatic prune" command in the console of the borgmatic container and waited for the error to happen (plus tried to connect via smb so this issue is documented as well).
When I tried to generate a diagnostic.zip afterwards, hell broke loose again. The whole GUI disappears (blank page), refresh (after pressing many times ctrl+r!) brings it back up only for it to disappear right away. Sometimes it works for a few seconds, then I can try to give the command to shut something down. Sometimes not. The .zip file never was downloaded which is why I can't attach it. Going back to the tab to try it again is impossible. All I have is a copy of the log which is shown while the .zip is being created, also find attached.
FYI: This is not an issue with 7.0.0 alone, this issue drives me crazy for like 3 months now.
Thanks a lot!
