chrissi5120 Posted April 30, 2022 Share Posted April 30, 2022 2 hours ago, sdub said: I’ve never successfully exposed a fuse mount in one docker container to another. I usually perform browse and restore operations from the command line within the Borgmatic docker A couple of options that may work are: 1) install vorta container, which gives a borg gui so you can at least visually browse the archive, even if it’s not really a file manager. 2) install borg and llfuse to the base Unraid image using nerdpack. I would think this should work but I’ve never gotten the fuse mount to appear via samba or a file manager docker container. Not sure why because I think it should work. 3) install an Ubuntu VM with borg and llfuse and try to mount it there, browsing the archive visually through VNC. I haven’t actually tried this one yet. Let us know if you figure out a working solution! Thank you for your feedback! i opted for midnight commander (command is "mc", its preinstalled), INSIDE a screen session. after thoroughly reading into fuse between containers, i realized, that there is absolutely no "plug and play" way to achive this. This would all need scary permissions and people seem to have actual problems with it even if it is running. midnight commander seems to be the solution for quick copy and paste operations as it is easier on the eye when dealing with spaces in file-/foldernames when i would need to restore a complete volume, i would certainly opt for something like a rsync with recursive search etc. activated. tl;dr: FUSE sucks if you want to move stuff between containers and the host-OS, better use something on the host-shell. Quote Link to comment
xxnumbxx Posted May 4, 2022 Share Posted May 4, 2022 Wondering if someone can give me a hand. I am trying to backup to a remote server also on unraid. I successfully created my ssh keys and init the repo. However the backup fails to run via 'borgmatic create -v 2 --files --stats'. I also tried to just manually create a test backup via 'borg create --stats [email protected]:/mnt/user/borg_backups::Test ~/mnt/user/music' and it fails with similar errors. Not sure what I am doing wrong. I can get borg to work fine via the nerdpack plugin but would like to use this docker. I have attached the errors below. TIA Borgmatic Response: Manual borg create: My Config.yaml Quote Link to comment
sdub Posted May 4, 2022 Author Share Posted May 4, 2022 (edited) I think the issue is that you created the borg repo then moved it, or are trying to access it using a different path. Run this borg command manually and confirm that you’re ok with the move. Then try running Borgmatic again. borg list 10.10.1.184:/mnt/user/borg Edited May 4, 2022 by sdub 1 Quote Link to comment
xxnumbxx Posted May 4, 2022 Share Posted May 4, 2022 33 minutes ago, sdub said: I think the issue is that you created the borg repo then moved it, or are trying to access it using a different path. Run this borg command manually and confirm that you’re ok with the move. Then try running Borgmatic again. borg list 10.10.1.184:/mnt/user/borg This worked although not sure the cause. I never moved the repo. Only difference is omitting the user prior to the ip. Thanks for your help. Quote Link to comment
xxnumbxx Posted May 6, 2022 Share Posted May 6, 2022 How do I get exclude patterns to work? I have the recycle bin plugin installed it adds /.Recycle.Bin to each folder. I have tried every combination I can think of and it still backs it up. exclude_patterns: - '- [Tt]rash' - '- [Cc]ache' - '- */.Recycle.Bin' - '- /.Recycle.Bin' Quote Link to comment
sdub Posted May 6, 2022 Author Share Posted May 6, 2022 I struggled with something similar and i think the issue is once it’s in the cache it won’t stop backing it up. Only applies going forward for new folders. I bet if you do a simple test on a new repo with that folder it will work as expected. Might want to ask the guys over at the borg GitHub repo since it’s a borg behavior that’s causing it. 1 Quote Link to comment
xxnumbxx Posted May 6, 2022 Share Posted May 6, 2022 (edited) 14 minutes ago, sdub said: I struggled with something similar and i think the issue is once it’s in the cache it won’t stop backing it up. Only applies going forward for new folders. I bet if you do a simple test on a new repo with that folder it will work as expected. Might want to ask the guys over at the borg GitHub repo since it’s a borg behavior that’s causing it. I tried this thinking the same thing. I created a new repo and it still backs up the recycle bin folder. SOLVED: for anyone else with this problem I was able to exclude using **/.Recycle.Bin Edited May 6, 2022 by xxnumbxx Solved Quote Link to comment
xxnumbxx Posted May 6, 2022 Share Posted May 6, 2022 Does anyone know how Borg works in regard to deduplication, specific to files of the same name. I am curious of photo backups specifically. Will it determine files of the same name and check if the are different files. My camera repeats the names of files and I want to make sure all photos are retained regardless of file names. Quote Link to comment
sdub Posted May 6, 2022 Author Share Posted May 6, 2022 3 hours ago, xxnumbxx said: Does anyone know how Borg works in regard to deduplication, specific to files of the same name. I am curious of photo backups specifically. Will it determine files of the same name and check if the are different files. My camera repeats the names of files and I want to make sure all photos are retained regardless of file names. The filename makes no difference at all. What matters is the binary content of the files. Borg breaks up files into 2MB "chunks" (by default) that are deduplicated irrespective of metadata like filename or path. In this way even large files that have been partially changed have the possibility of partial deduplication, and renamed files will be fully deduplicated. In your example, different files with the same file name would both be broken into chunks and fully retained in the backup. When the last backup archive that uses a chunk is purged, the chunk is purged also. Reading up on the "chunker" might help you get an appreciation. I've found (as a borg user) it's very efficient. https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#chunker-details 1 Quote Link to comment
T0a Posted May 7, 2022 Share Posted May 7, 2022 I added the progress flag to my borgmatic command (borgmatic create --verbosity 1 --progress --stats) today and noticed that the logs available via the WebUI are incomplete when compared to docker logs from the command line. The left side shows the logs via the build-in web UI log viewer. The right side shows the logs via `docker logs -f <id>`. Not sure whether this is an issue with how Unraid fetches the docker logs for the container. Anybody experienced something similar in the past or can explain that behavior? Quote Link to comment
xxnumbxx Posted May 20, 2022 Share Posted May 20, 2022 Curious what time of speeds people are getting with their backups? I am using borgmatic to backup to a remote server via ssh. Speeds are approx 20Mb/s. I have tested both with iperf and copying files to the remote server and reach speeds of 110+Mb/s. Are there any tips to increase the speeds? I was considering running an additional instance of borgmatic to backup seperate repos but this just seems redundant. Quote Link to comment
sdub Posted May 20, 2022 Author Share Posted May 20, 2022 I backup my server to another server where my connection is ~15Mbps up and the remote server is 200Mbps down. I performed a backup last night that consisted of 5.8GB against a 7.1TB archive. It took 82 minutes. The equivalent local backup took 55 minutes. … so if it all scales linearly I’d say my backups take about 5 minutes per TB (total archive) plus copy time. If it’s local that’s about 1 minute per GB and remote (with my line speed) that’s about 7 minutes per GB changed. Remember that a lot do the time during the backup, it’s scanning files on the remote machine using borg in server mode and not transmitting data. Quote Link to comment
xxnumbxx Posted May 20, 2022 Share Posted May 20, 2022 6 minutes ago, sdub said: I backup my server to another server where my connection is ~15Mbps up and the remote server is 200Mbps down. I performed a backup last night that consisted of 5.8GB against a 7.1TB archive. It took 82 minutes. The equivalent local backup took 55 minutes. … so if it all scales linearly I’d say my backups take about 5 minutes per TB (total archive) plus copy time. If it’s local that’s about 1 minute per GB and remote (with my line speed) that’s about 7 minutes per GB changed. Remember that a lot do the time during the backup, it’s scanning files on the remote machine using borg in server mode and not transmitting data. Ya mine is very slow then. It took about 12hours to do 880GB. I am creating an initial backup not sure if it takes longer the first time. What could I do to increase backup speed? Quote Link to comment
sdub Posted May 20, 2022 Author Share Posted May 20, 2022 The first one is very slow. Not sure there’sa lot you can do about it. I did my initial remote backup locally then copied it to the remote sever in person. Aka sneaker-net. if you felt like borg network performance was really the issue you could create the repo locally like I did then use a lightweight protocol like rsync to copy the initial repo across. When you change how you access the repo it will warn you once but allow it. Quote Link to comment
Greygoose Posted May 24, 2022 Share Posted May 24, 2022 On 5/20/2022 at 6:27 PM, sdub said: The first one is very slow. Not sure there’sa lot you can do about it. I did my initial remote backup locally then copied it to the remote sever in person. Aka sneaker-net. if you felt like borg network performance was really the issue you could create the repo locally like I did then use a lightweight protocol like rsync to copy the initial repo across. When you change how you access the repo it will warn you once but allow it. This is how I do things inititally. I use borgbase (highly recommend) for my crucial data, the first backup takes a while (1TB) but after that its super fast. Quote Link to comment
vonwanja Posted July 25, 2022 Share Posted July 25, 2022 Hello, this is my first post here. unfortunately I fail with borg on Unraid both here with borgmatic, and in the counter attempt with the Vorta docker. Everything still works when backing up, but I fail to restore (extract) the backup. No matter if with borgmatic or vorta, it always fails with the error "makedirs: [Errno 30] Read-only file system". 2022-07-25 21:51:35,791 - vorta.borg.borg_job - WARNING - data/bkp_from/TimeMachine/TreCime_1920x1080.jpg: makedirs: [Errno 30] Read-only file system: '/data/bkp_from/TimeMachine/data' In this case I did it with Vorta, but the result is exactly the same with borgmatic. If I understand correctly, the docker fails because of the appropriate permissions? I feel like I've been digging all over the internet, but haven't found anything yet, nor anyone who has the same problem. This leads me to believe that I have not understood something fundamental. I would be very grateful for any help. Until the backup and restore works well, I don't want to use Unraid seriously. Quote Link to comment
Solverz Posted July 25, 2022 Share Posted July 25, 2022 (edited) 9 minutes ago, vonwanja said: Hello, this is my first post here. unfortunately I fail with borg on Unraid both here with borgmatic, and in the counter attempt with the Vorta docker. Everything still works when backing up, but I fail to restore (extract) the backup. No matter if with borgmatic or vorta, it always fails with the error "makedirs: [Errno 30] Read-only file system". 2022-07-25 21:51:35,791 - vorta.borg.borg_job - WARNING - data/bkp_from/TimeMachine/TreCime_1920x1080.jpg: makedirs: [Errno 30] Read-only file system: '/data/bkp_from/TimeMachine/data' In this case I did it with Vorta, but the result is exactly the same with borgmatic. If I understand correctly, the docker fails because of the appropriate permissions? I feel like I've been digging all over the internet, but haven't found anything yet, nor anyone who has the same problem. This leads me to believe that I have not understood something fundamental. I would be very grateful for any help. Until the backup and restore works well, I don't want to use Unraid seriously. The error says read only file system, so the file system you are restoring too is read only. Have you checked the directory permissions or the docker path mapping? Are these read only or read/write? Edited July 25, 2022 by Solverz More detail Quote Link to comment
vonwanja Posted July 25, 2022 Share Posted July 25, 2022 1 hour ago, Solverz said: Have you checked the directory permissions or the docker path mapping? Are these read only or read/write? This is really embarrassing. It was. I had never had to change the permissions in the path mappings before and didn't have that option on the screen. Thanks a lot! If only I had asked earlier ... Quote Link to comment
Solverz Posted July 25, 2022 Share Posted July 25, 2022 (edited) 6 minutes ago, vonwanja said: This is really embarrassing. It was. I had never had to change the permissions in the path mappings before and didn't have that option on the screen. Thanks a lot! If only I had asked earlier ... No problem! It was probably set to read only in the template by default so that the container can only read data from your mapped directory and not modify or delete it or its contents, security hardening. Edited July 25, 2022 by Solverz Quote Link to comment
Hammy Havoc Posted July 27, 2022 Share Posted July 27, 2022 Does it still make sense to use the Unassigned Devices plugin versus creating a pool specifically for Borg now that we can have multiple pools? Quote Link to comment
Solverz Posted July 27, 2022 Share Posted July 27, 2022 15 minutes ago, Hammy Havoc said: Does it still make sense to use the Unassigned Devices plugin versus creating a pool specifically for Borg now that we can have multiple pools? As I like to say, "it depends". There are advantages and disadvantages to both, for instance, UD devices can be removed and installed adhoc where a pool cannot and unraid will complain if it goes "missing" Pools allow you to have redundancy, UD devices do not from the gui (as far as I am aware, please correct me if otherwise) UD can mount network shares etc and not just local media, pools is for local media only. Etc etc etc, just way up the pros and cons and see what suits your situation and setup the best and use that. ☺ 1 Quote Link to comment
sdub Posted July 27, 2022 Author Share Posted July 27, 2022 2 hours ago, Hammy Havoc said: Does it still make sense to use the Unassigned Devices plugin versus creating a pool specifically for Borg now that we can have multiple pools? 3-2-1 backup means 3 copies, 2 types of media, at least one offsite. When I wrote that my reasoning was just to get as much diversity as possible between your data and the backup in lieu of 2 types of media. Preferably it would be on a different system with a different Os, filesystem, etc. There’s no “right” or “wrong”, just better or worse. 1 Quote Link to comment
Solverz Posted August 17, 2022 Share Posted August 17, 2022 FYI, if anyones backups are failing since the latest update to this container 2 days ago, please see below: The borgmatic binary is no longer at /usr/bin/borgmatic it is now in /usr/local/bin/borgmatic. Of course, this this will only affect you if you are specifiying the full path to the borgmatic binary in your cronjob. /usr/local/bin is actually and to no surpise in the PATH env so you do not need to actually specifiy the full path to borgmatic and you can instead just put write "borgmatic" and if this path ever changes in the future you should not be affected. Quote Link to comment
Greygoose Posted August 21, 2022 Share Posted August 21, 2022 On 8/17/2022 at 7:41 PM, Solverz said: FYI, if anyones backups are failing since the latest update to this container 2 days ago, please see below: The borgmatic binary is no longer at /usr/bin/borgmatic it is now in /usr/local/bin/borgmatic. Of course, this this will only affect you if you are specifiying the full path to the borgmatic binary in your cronjob. /usr/local/bin is actually and to no surpise in the PATH env so you do not need to actually specifiy the full path to borgmatic and you can instead just put write "borgmatic" and if this path ever changes in the future you should not be affected. Hi Solverz, thank you for this post. I have edited my cron file to the new path below, you mention just using borgmatic instead of specifiying the full path. please can you change the cron below to show this please as i'm unsure exactly. Quote 0 1,13 * * * PATH=$PATH:/usr/bin /usr/local/bin/borgmatic prune create -v 1 --stats 2>&1 0 6 * * 3 PATH=$PATH:/usr/bin /usr/local/bin/borgmatic check -v 1 2>&1 Quote Link to comment
Solverz Posted August 22, 2022 Share Posted August 22, 2022 (edited) On 8/21/2022 at 8:36 PM, Greygoose said: Hi Solverz, thank you for this post. I have edited my cron file to the new path below, you mention just using borgmatic instead of specifiying the full path. please can you change the cron below to show this please as i'm unsure exactly. 0 1,13 * * * borgmatic prune create -v 1 --stats 2>&1 0 6 * * 3 borgmatic check -v 1 2>&1 Hope that helps! Edited August 23, 2022 by Solverz Syntax 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.