Killabee44 Posted July 9, 2023 Share Posted July 9, 2023 (edited) 9 hours ago, sdub said: It looks to me like maybe you are mounting the share to Borgmatic as read-only… not that it’s read only in the host filesystem. What’s your Borgmatic docker config look like? Here it is: My config: location: source_directories: - /boot # - /mnt/user/appdata - /mnt/user/bootbackup - /mnt/user/CommunityApplicationsAppdataBackup - /mnt/user/nextcloud # - /mnt/user/downloads repositories: - /mnt/user/disks/easystore264D # - remote.mydomain.net:/mnt/disks/borg_remote/repo one_file_system: true files_cache: mtime,size patterns: - '- [Tt]rash' - '- [Cc]ache' exclude_if_present: - .nobackup - .NOBACKUP storage: encryption_passphrase: "passphrase" compression: none ssh_command: ssh -i /root/.ssh/id_rsa archive_name_format: 'backup-{now}' retention: keep_hourly: 2 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 10 prefix: 'backup-' consistency: checks: - repository - archives prefix: 'backup-' hooks: before_backup: - echo "Starting a backup." after_backup: - echo "Finished a backup." on_error: - echo "Error during prune/create/check." mysql_databases: - name: all hostname: 192.168.1.29 password: "password" Edited July 10, 2023 by Killabee44 Quote Link to comment
Killabee44 Posted July 17, 2023 Share Posted July 17, 2023 On 7/9/2023 at 12:31 PM, Killabee44 said: Here it is: My config: location: source_directories: - /boot # - /mnt/user/appdata - /mnt/user/bootbackup - /mnt/user/CommunityApplicationsAppdataBackup - /mnt/user/nextcloud # - /mnt/user/downloads repositories: - /mnt/user/disks/easystore264D # - remote.mydomain.net:/mnt/disks/borg_remote/repo one_file_system: true files_cache: mtime,size patterns: - '- [Tt]rash' - '- [Cc]ache' exclude_if_present: - .nobackup - .NOBACKUP storage: encryption_passphrase: "passphrase" compression: none ssh_command: ssh -i /root/.ssh/id_rsa archive_name_format: 'backup-{now}' retention: keep_hourly: 2 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 10 prefix: 'backup-' consistency: checks: - repository - archives prefix: 'backup-' hooks: before_backup: - echo "Starting a backup." after_backup: - echo "Finished a backup." on_error: - echo "Error during prune/create/check." mysql_databases: - name: all hostname: 192.168.1.29 password: "password" Anyone? Quote Link to comment
sdub Posted July 17, 2023 Author Share Posted July 17, 2023 (edited) Your repo should be “/mnt/borg-repository” not “/mnt/user/disks/easystore264D”. You have mounted “/mnt/user” as a read-only path, so you can’t write to the repo via that path. Edited July 17, 2023 by sdub Quote Link to comment
Daniel15 Posted September 5, 2023 Share Posted September 5, 2023 (edited) Is there a way to have two separate configs? I want to back up into two different repos: One for my SSDs and one for larger data on my hard drives. I can bind mount multiple things under /mnt/user, but it'd all still go to the same repo. Edit: I just realised that it can take multiple config files, they just have to have different names. I'll try it out. Edited September 5, 2023 by Daniel15 1 Quote Link to comment
sdub Posted September 7, 2023 Author Share Posted September 7, 2023 On 9/5/2023 at 2:56 PM, Daniel15 said: Is there a way to have two separate configs? I want to back up into two different repos: One for my SSDs and one for larger data on my hard drives. I can bind mount multiple things under /mnt/user, but it'd all still go to the same repo. Edit: I just realised that it can take multiple config files, they just have to have different names. I'll try it out. you can also specify the config file with the borgmatic -c /path/to/config syntax. I personally have a local.yaml and remote.yaml that both include a common.yaml for common options. I call the local and remote backups through separate lines in my crontab 1 Quote Link to comment
mkono87 Posted October 5, 2023 Share Posted October 5, 2023 Trying to initilize the local repo, I recieve a bunch of traceback errors, yet I still see files created in the folder. 9b90dcb83a73:/# borg init --encryption=none /mnt/borg-repository/ Local Exception Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main exit_code = archiver.run(args) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run return set_ec(func(args)) ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 183, in wrapper return method(self, args, repository=repository, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 290, in do_init with Cache(repository, key, manifest, warn_if_unencrypted=False): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 387, in __new__ return local() ^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 378, in local return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 478, in __init__ self.path = cache_dir(repository, path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 229, in cache_dir return path or os.path.join(get_cache_dir(), repository.id_str) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/helpers/fs.py", line 100, in get_cache_dir with SaveFile(cache_tag_fn, binary=True) as fd: File "/usr/local/lib/python3.11/site-packages/borg/platform/base.py", line 230, in __enter__ self.tmp_fd, self.tmp_fname = mkstemp_mode(prefix=self.tmp_prefix, suffix='.tmp', dir=self.dir, mode=0o666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/helpers/fs.py", line 408, in mkstemp_mode return _mkstemp_inner(dir, prefix, suffix, flags, output_type, mode) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/helpers/fs.py", line 364, in _mkstemp_inner fd = _os.open(file, flags, mode) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/borg/CACHEDIR.TAG-uryrraek.tmp' Platform: Linux 9b90dcb83a73 6.1.49-Unraid #1 SMP PREEMPT_DYNAMIC Wed Aug 30 09:42:35 PDT 2023 x86_64 Linux: Unknown Linux Borg: 1.2.6 Python: CPython 3.11.5 msgpack: 1.0.5 fuse: llfuse 1.5.0 [pyfuse3,llfuse] PID: 65 CWD: / sys.argv: ['/usr/local/bin/borg', 'init', '--encryption=none', '/mnt/borg-repository/'] SSH_ORIGINAL_COMMAND: None Quote Link to comment
sdub Posted October 5, 2023 Author Share Posted October 5, 2023 19 minutes ago, mkono87 said: Trying to initilize the local repo, I recieve a bunch of traceback errors, yet I still see files created in the folder. 9b90dcb83a73:/# borg init --encryption=none /mnt/borg-repository/ Local Exception Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main exit_code = archiver.run(args) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run return set_ec(func(args)) ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 183, in wrapper return method(self, args, repository=repository, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 290, in do_init with Cache(repository, key, manifest, warn_if_unencrypted=False): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 387, in __new__ return local() ^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 378, in local return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 478, in __init__ self.path = cache_dir(repository, path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 229, in cache_dir return path or os.path.join(get_cache_dir(), repository.id_str) ^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/helpers/fs.py", line 100, in get_cache_dir with SaveFile(cache_tag_fn, binary=True) as fd: File "/usr/local/lib/python3.11/site-packages/borg/platform/base.py", line 230, in __enter__ self.tmp_fd, self.tmp_fname = mkstemp_mode(prefix=self.tmp_prefix, suffix='.tmp', dir=self.dir, mode=0o666) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/helpers/fs.py", line 408, in mkstemp_mode return _mkstemp_inner(dir, prefix, suffix, flags, output_type, mode) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/helpers/fs.py", line 364, in _mkstemp_inner fd = _os.open(file, flags, mode) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/borg/CACHEDIR.TAG-uryrraek.tmp' Platform: Linux 9b90dcb83a73 6.1.49-Unraid #1 SMP PREEMPT_DYNAMIC Wed Aug 30 09:42:35 PDT 2023 x86_64 Linux: Unknown Linux Borg: 1.2.6 Python: CPython 3.11.5 msgpack: 1.0.5 fuse: llfuse 1.5.0 [pyfuse3,llfuse] PID: 65 CWD: / sys.argv: ['/usr/local/bin/borg', 'init', '--encryption=none', '/mnt/borg-repository/'] SSH_ORIGINAL_COMMAND: None Looks like the root issue is in that '/root/.cache/borg' folder. If you open a shell into borg and browse to that folder, does everyhing appear normal? Could be that folder mapping is screwed up in your docker config. Quote Link to comment
mkono87 Posted October 5, 2023 Share Posted October 5, 2023 54 minutes ago, sdub said: Looks like the root issue is in that '/root/.cache/borg' folder. If you open a shell into borg and browse to that folder, does everyhing appear normal? Could be that folder mapping is screwed up in your docker config. It mapped in a subfolder of the repo folder. Is that fine? The cache folder on the host wasnt there so I created it but now when I try to init it says there is something already there. Quote Link to comment
mkono87 Posted October 7, 2023 Share Posted October 7, 2023 (edited) @sdubDo you shutdown your containers before backing up /mnt/user/appdata? If not, no issues have come from that? How is this the easier way to go vs the backup appdata plugin? Edited October 7, 2023 by mkono87 Quote Link to comment
sdub Posted October 7, 2023 Author Share Posted October 7, 2023 No I don’t. I use the database backup functions for things like MariaDB and Postgres and the built-in database backup function for file based databases like Plex. For everything else the risk is realistically minimal. Especially with scheduled backups and a multitude of retained versions. In the off-chance the daily backup happens to be bad, there are other daily, weekly, monthly Archives I can draw from. Losing a day or two of data isn’t a big concern of mine. Quote Link to comment
sonofdbn Posted November 20, 2023 Share Posted November 20, 2023 I'm trying to use Borg to backup from unRAID to a Synology DS414. I attached the DS414 as an SMB share using Unassigned Devices. (Had some problems, but don't think they're relevant to this issue.) When I try to create the repo using borg init --encryption=none /mnt/borg-repository I get an error message: / # borg init --encryption=none /mnt/borg-repository Local Exception OSError: [Errno 28] No space left on device During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main exit_code = archiver.run(args) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run return set_ec(func(args)) ^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 183, in wrapper return method(self, args, repository=repository, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/archiver.py", line 290, in do_init with Cache(repository, key, manifest, warn_if_unencrypted=False): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 387, in __new__ return local() ^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 378, in local return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 485, in __init__ self.create() File "/usr/local/lib/python3.11/site-packages/borg/cache.py", line 513, in create with open(os.path.join(self.path, 'README'), 'w') as fd: OSError: [Errno 28] No space left on device From googling around, it seems like the "No space" error might be caused by the Borg cache being full. I had left it at the default of /mnt/disks/borg_backup/cache, and there's plenty of space on the array. If I run df -h I get this: / # df -h Filesystem Size Used Available Use% Mounted on .... tmpfs 1.0M 1.0M 0 100% /root/.cache/borg shfs 931.1G 48.4G 882.6G 5% /root/.config/borg .... What can I do to fix this? Quote Link to comment
Skotticus Posted December 20, 2023 Share Posted December 20, 2023 Quote Files cache set to use "mtime,size" - Very important as unRAID does not have persistent inode values Is this still best practice? I haven't found any explicit documentation about how unRAID uses inodes in the manual. Inodes in general seem to be something that confuses a lot of people; is it dependent on the filesystem you use or are all filesystems in unRAID implemented without persistent inode values? Or is it because of the flexibility of unRAID's array system? Quote Link to comment
Greygoose Posted April 9 Share Posted April 9 (edited) can someone please paste thier recent working yaml file. im getting lots of errors as things look to have changed in borgmatic itself. ------------------ /etc/borgmatic.d/Photo&Video-Collections.yml: Error parsing configuration file An error occurred while parsing a configuration file at /etc/borgmatic.d/Photo&Video-Collections.yml: did not find expected <document start> in "<unicode string>", line 42, column 1 /etc/borgmatic.d/config_datasync.yml: Error parsing configuration file An error occurred while parsing a configuration file at /etc/borgmatic.d/config_datasync.yml: At the top level: 'repositories' is a required property /etc/borgmatic.d/config_nextcloud.yml: Error parsing configuration file An error occurred while parsing a configuration file at /etc/borgmatic.d/config_nextcloud.yml: did not find expected <document start> in "<unicode string>", line 53, column 1 /etc/borgmatic/config.yaml /etc/borgmatic.d /root/.config/borgmatic/config.yaml /root/.config/borgmatic.d: No valid configuration files found Need some help? https://torsion.org/borgmatic/#issues ------------ /etc/borgmatic.d/Photo&Video-Collections.yml: The checks option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release. /etc/borgmatic.d/Photo&Video-Collections.yml: The repositories option now expects a list of key/value pairs. Lists of strings for this option are deprecated and support will be removed from a future release. /etc/borgmatic.d/Photo&Video-Collections.yml: The prefix option is deprecated and support will be removed from a future release. Use archive_name_format or match_archives instead. Edited April 9 by Greygoose Quote Link to comment
sdub Posted April 9 Author Share Posted April 9 I am not at my server right now but I can say that my backups are still working and I haven’t touched the config in months. the Unicode error makes me wonder… did you possibly edit your config file in windows such that it has a Windows text file format instead of UNIX? Quote Link to comment
Greygoose Posted April 10 Share Posted April 10 (edited) 13 hours ago, sdub said: I am not at my server right now but I can say that my backups are still working and I haven’t touched the config in months. the Unicode error makes me wonder… did you possibly edit your config file in windows such that it has a Windows text file format instead of UNIX? Hi sdub, thanks for the reply mate. yes i edited using notepad++ in windows 11. I have dont this with other files and its been ok but it certainly could be the issue. EDIT: How do you edit the yaml files?, ie what software? Edited April 10 by Greygoose Quote Link to comment
Rick_Sanchez Posted April 10 Share Posted April 10 2 hours ago, Greygoose said: Hi sdub, thanks for the reply mate. yes i edited using notepad++ in windows 11. I have dont this with other files and its been ok but it certainly could be the issue. EDIT: How do you edit the yaml files?, ie what software? I prefer visual studio code Quote Link to comment
Greygoose Posted April 10 Share Posted April 10 5 hours ago, Rick_Sanchez said: I prefer visual studio code Wow this is great, thank you. I have played about a bit more to clear errors but still have two that i can resolve? Quote Link to comment
Solverz Posted April 10 Share Posted April 10 3 minutes ago, Greygoose said: Wow this is great, thank you. I have played about a bit more to clear errors but still have two that i can resolve? My friend, read what the warnings say 😅. The syntax you are using in the config files is deprecated and may not work in future versions. Solution: read the borgmatic docs and use the new syntax. 1 Quote Link to comment
Greygoose Posted April 10 Share Posted April 10 ok sorted the above, the below config shows up no errors if i console into docket and type borgmstic list. I hope this helps someone else. took me a few hours as ive not had to work on borg for a while so was great fun and refresher, I had to add -path: label archive_name_format: 'backup-{now}' -------------------------------------- #location: source_directories: - /mnt/user/data_storage/Photo&Video-Collections repositories: - path: ssh://blarblarblar label: photo_video one_file_system: true files_cache: mtime,size patterns: - '- [Tt]rash' - '- [Cc]ache' exclude_if_present: - .nobackup - .NOBACKUP #storage: encryption_passphrase: "thisisasecret" compression: none ssh_command: ssh -i /root/.ssh/id_ed24419 archive_name_format: 'backup-{now}' #retention: keep_hourly: 2 keep_daily: 7 keep_weekly: 4 keep_monthly: 12 keep_yearly: 10 archive_name_format: 'backup-{now}' #consistency: # checks: # - repository # # - archives # prefix: 'backup-' #hooks: before_backup: - echo "Starting a backup." after_backup: - echo "Finished a backup." on_error: - echo "Error during prune/create/check." #healthchecks: #ping_url: https://hc-ping.com/434343434343 Quote Link to comment
Tainted3605 Posted July 24 Share Posted July 24 I have been using borgmatic for weeks now and I am loving it. Only minimal configuration needed. i am trying to use the “before_actions” and “after_actions” to mount and unmount borg repository to local system. However it seems to hang trying to run the mount command. Any ideas? Unmount command:/usr/local/bin/borgmatic umount --mount-point /mnt/fuse/ Mount command: /usr/local/bin/borgmatic mount --repository /mnt/<path-to-repository> --archive latest --mount-point /mnt/fuse/ Quote Link to comment
HagenS Posted July 24 Share Posted July 24 borgmatic runs as a non-privileged docker container - mounting at host needs more privileges. I use for that an ssh session to unraid host: before_everything: - ssh -i /root/.ssh/id_rsa_unraid 192.168.10.244 '/usr/local/sbin/rc.unassigned mount /dev/sdd' - ln -sf /mnt/borg-repository/Backup-8TB/cache/* /root/.cache/borg/ after_everything: - ssh -i /root/.ssh/id_rsa_unraid 192.168.10.244 '/usr/local/sbin/rc.unassigned umount /dev/sdd' Quote Link to comment
T0a Posted August 11 Share Posted August 11 (edited) Problem statement I discovered that my Borgmatic setup does not backup my cache only appdata directories (see error message below). While investigating, I found that the fuse mount "/mnt/user" symlinks to the cache directory for my appdata share (see code snippets below). This symlink is not available via the mount in the Borgmatic Container. I assume this changed with Unraid 6.12 & the exclusive shares feature. How to you guys handle this situation in your setup? Do you mount "/mnt" instead and adapt the Borgmatic configuration accordingly? Do you create individual mounts for your cache only shares? ... Update: I also find it alarming that my healthcheck hook wasn't called. Seems like the backup still runs. Error message: Creating archive at "<redacted>" ... /mnt/user/appdata/paperless-ngx: stat: [Errno 21] No such file or directory: '/mnt/user/appdata/paperless-ngx' ... Share Configuration: Borgmatic Container configuration: From the Unraid server console: root@server:# ls -ahl /mnt/user total 0 drwxrwxrwx 1 nobody users 107 Aug 11 09:11 ./ drwxr-xr-x 11 root root 220 Aug 11 09:01 ../ lrwxrwxrwx 1 nobody users 356 Jul 13 11:50 appdata -> ../cache/appdata/ root@server:# ls -ahl /mnt/user/appdata/paperless-ngx/ total 4.0K drwxr-xr-x 1 root root 8 Jun 5 2022 ./ drwxrwxrwx 1 nobody users 356 Jul 13 11:50 ../ drwxr-xr-x 1 nobody users 292 Aug 11 09:50 data/ From inside the Borgmatic container: /etc/borgmatic.d # ls -ahl /mnt/user/ total 0 drwxrwxrwx 1 99 users 107 Aug 11 09:11 . drwxr-xr-x 1 root root 46 Jul 7 16:53 .. lrwxrwxrwx 1 99 users 356 Jul 13 11:50 appdata -> ../cache/appdata # Cannot access appdata content, because it is a symlink /etc/borgmatic.d # ls -ahl /mnt/user/appdata/ ls: /mnt/user/appdata/: No such file or directory Edited August 11 by T0a Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.