[Support] borgmatic


sdub

Recommended Posts

2 hours ago, sdub said:

I’ve never successfully exposed a fuse mount in one docker container to another. I usually perform browse and restore operations from the command line within the Borgmatic docker

 

A couple of options that may work are:

 

1) install vorta container, which gives a borg gui so you can at least visually browse the archive, even if it’s not really a file manager.

 

2) install borg and llfuse to the base Unraid image using nerdpack. I would think this should work but I’ve never gotten the fuse mount to appear via samba or a file manager docker container. Not sure why because I think it should work. 
 

3) install an Ubuntu VM with borg and llfuse and try to mount it there, browsing the archive visually through VNC. I haven’t actually tried this one yet. 
 

Let us know if you figure out a working solution!

Thank you for your feedback!

 

i opted for midnight commander (command is "mc", its preinstalled), INSIDE a screen session.

 

after thoroughly reading into fuse between containers, i realized, that there is absolutely no "plug and play" way to achive this.

This would all need scary permissions and people seem to have actual problems with it even if it is running.

 

midnight commander seems to be the solution for quick copy and paste operations as it is easier on the eye when dealing with spaces in file-/foldernames

 

when i would need to restore a complete volume, i would certainly opt for something like a rsync with recursive search etc. activated.

 

tl;dr: FUSE sucks if you want to move stuff between containers and the host-OS, better use something on the host-shell.

Link to comment

Wondering if someone can give me a hand. I am trying to backup to a remote server also on unraid. I successfully created my ssh keys and init the repo. However the backup fails to run via 'borgmatic create -v 2 --files --stats'. I also tried to just manually create a test backup via 'borg create --stats [email protected]:/mnt/user/borg_backups::Test ~/mnt/user/music' and it fails with similar errors. Not sure what I am doing wrong. I can get borg to work fine via the nerdpack plugin but would like to use this docker. I have attached the errors below. TIA

 

Borgmatic Response:

borgmatic.thumb.png.f10d79ccaa198966fd5c6963c700b4eb.png

 

Manual borg create:

borg_manual.png.17d9b469a285286585f452b54eec1ba0.png

 

My Config.yaml

configyaml.png.d823c5a08981bcde297016bd212b7d7e.png

Link to comment

I think the issue is that you created the borg repo then moved it, or are trying to access it using a different path. 

Run this borg command manually and confirm that you’re ok with the move.  Then try running Borgmatic again. 
 

borg list 10.10.1.184:/mnt/user/borg

 

Edited by sdub
  • Upvote 1
Link to comment
33 minutes ago, sdub said:

I think the issue is that you created the borg repo then moved it, or are trying to access it using a different path. 

Run this borg command manually and confirm that you’re ok with the move.  Then try running Borgmatic again. 
 

borg list 10.10.1.184:/mnt/user/borg

 

This worked although not sure the cause. I never moved the repo. Only difference is omitting the user prior to the ip. Thanks for your help.

 

Link to comment

How do I get exclude patterns to work? I have the recycle bin plugin installed it adds /.Recycle.Bin to each folder. I have tried every combination I can think of and it still backs it up.

 

 exclude_patterns:
        - '- [Tt]rash'
        - '- [Cc]ache'
        - '- */.Recycle.Bin'
        - '- /.Recycle.Bin'

 

Link to comment

I struggled with something similar and i think the issue is once it’s in the cache it won’t stop backing it up. Only applies going forward for new folders. 

I bet if you do a simple test on a new repo with that folder it will work as expected. 

 

Might want to ask the guys over at the borg GitHub repo since it’s a borg behavior that’s causing it. 

  • Upvote 1
Link to comment
14 minutes ago, sdub said:

I struggled with something similar and i think the issue is once it’s in the cache it won’t stop backing it up. Only applies going forward for new folders. 

I bet if you do a simple test on a new repo with that folder it will work as expected. 

 

Might want to ask the guys over at the borg GitHub repo since it’s a borg behavior that’s causing it. 

 

I tried this thinking the same thing. I created a new repo and it still backs up the recycle bin folder.

 

SOLVED:

for anyone else with this problem I was able to exclude using **/.Recycle.Bin

Edited by xxnumbxx
Solved
Link to comment

Does anyone know how Borg works in regard to deduplication, specific to files of the same name. I am curious of photo backups specifically. Will it determine files of the same name and check if the are different files. My camera repeats the names of files and I want to make sure all photos are retained regardless of file names.

Link to comment
3 hours ago, xxnumbxx said:

Does anyone know how Borg works in regard to deduplication, specific to files of the same name. I am curious of photo backups specifically. Will it determine files of the same name and check if the are different files. My camera repeats the names of files and I want to make sure all photos are retained regardless of file names.

 

The filename makes no difference at all.  What matters is the binary content of the files.

 

Borg breaks up files into 2MB "chunks" (by default) that are deduplicated irrespective of metadata like filename or path.  In this way even large files that have been partially changed have the possibility of partial deduplication, and renamed files will be fully deduplicated.   In your example, different files with the same file name would both be broken into chunks and fully retained in the backup.  When the last backup archive that uses a chunk is purged, the chunk is purged also.  

 

Reading up on the "chunker" might help you get an appreciation.  I've found (as a borg user) it's very efficient. 

 

https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#chunker-details

 

  • Upvote 1
Link to comment

I added the progress flag to my borgmatic command (borgmatic create --verbosity 1 --progress --stats) today and noticed that the logs available via the WebUI are incomplete when compared to docker logs from the command line.

 

The left side shows the logs via the build-in web UI log viewer. The right side shows the logs via `docker logs -f <id>`. Not sure whether this is an issue with how Unraid fetches the docker logs for the container.

 

Anybody experienced something similar in the past or can explain that behavior?

 

unraid_docker_log_does_not_show_everything.thumb.png.b2e071371ae123abe330bbce3f344677.png

Link to comment
  • 2 weeks later...

Curious what time of speeds people are getting with their backups? I am using borgmatic to backup to a remote server via ssh. Speeds are approx 20Mb/s. I have tested both with iperf and copying files to the remote server and reach speeds of 110+Mb/s. Are there any tips to increase the speeds? I was considering running an additional instance of borgmatic to backup seperate repos but this just seems redundant. 

Link to comment

I backup my server to another server where my connection is ~15Mbps up and the remote server is 200Mbps down. 
 

I performed a backup last night that consisted of 5.8GB against a 7.1TB archive. It took 82 minutes. The equivalent local backup took 55 minutes. 
 

… so if it all scales linearly I’d say my backups take about 5 minutes per TB (total archive) plus copy time.

 

If it’s local that’s about 1 minute per GB and remote (with my line speed) that’s about 7 minutes per GB changed.  
 

Remember that a lot do the time during the backup, it’s scanning files on the remote machine using borg in server mode and not transmitting data. 

Link to comment
6 minutes ago, sdub said:

I backup my server to another server where my connection is ~15Mbps up and the remote server is 200Mbps down. 
 

I performed a backup last night that consisted of 5.8GB against a 7.1TB archive. It took 82 minutes. The equivalent local backup took 55 minutes. 
 

… so if it all scales linearly I’d say my backups take about 5 minutes per TB (total archive) plus copy time.

 

If it’s local that’s about 1 minute per GB and remote (with my line speed) that’s about 7 minutes per GB changed.  
 

Remember that a lot do the time during the backup, it’s scanning files on the remote machine using borg in server mode and not transmitting data. 

Ya mine is very slow then. It took about 12hours to do 880GB. I am creating an initial backup not sure if it takes longer the first time. What could I do to increase backup speed?

Link to comment

The first one is very slow. Not sure there’sa lot you can do about it. I did my initial remote backup locally then copied it to the remote sever in person. Aka sneaker-net.
 

if you felt like borg network performance was really the issue you could create the repo locally like I did then use a lightweight protocol like rsync to copy the initial repo across. When you change how you access the repo it will warn you once but allow it. 

Link to comment
On 5/20/2022 at 6:27 PM, sdub said:

The first one is very slow. Not sure there’sa lot you can do about it. I did my initial remote backup locally then copied it to the remote sever in person. Aka sneaker-net.
 

if you felt like borg network performance was really the issue you could create the repo locally like I did then use a lightweight protocol like rsync to copy the initial repo across. When you change how you access the repo it will warn you once but allow it. 

 

This is how I do things inititally.

 

I use borgbase (highly recommend) for my crucial data, the first backup takes a while (1TB) but after that its super fast.

Link to comment
  • 2 months later...

Hello,

this is my first post here.

 

unfortunately I fail with borg on Unraid both here with borgmatic, and in the counter attempt with the Vorta docker.
Everything still works when backing up, but I fail to restore (extract) the backup. No matter if with borgmatic or vorta, it always fails with the error "makedirs: [Errno 30] Read-only file system".

2022-07-25 21:51:35,791 - vorta.borg.borg_job - WARNING - data/bkp_from/TimeMachine/TreCime_1920x1080.jpg: makedirs: [Errno 30] Read-only file system: '/data/bkp_from/TimeMachine/data'

In this case I did it with Vorta, but the result is exactly the same with borgmatic.
If I understand correctly, the docker fails because of the appropriate permissions?

 

I feel like I've been digging all over the internet, but haven't found anything yet, nor anyone who has the same problem. This leads me to believe that I have not understood something fundamental.

 

I would be very grateful for any help. Until the backup and restore works well, I don't want to use Unraid seriously.

Link to comment
9 minutes ago, vonwanja said:

Hello,

this is my first post here.

 

unfortunately I fail with borg on Unraid both here with borgmatic, and in the counter attempt with the Vorta docker.
Everything still works when backing up, but I fail to restore (extract) the backup. No matter if with borgmatic or vorta, it always fails with the error "makedirs: [Errno 30] Read-only file system".

2022-07-25 21:51:35,791 - vorta.borg.borg_job - WARNING - data/bkp_from/TimeMachine/TreCime_1920x1080.jpg: makedirs: [Errno 30] Read-only file system: '/data/bkp_from/TimeMachine/data'

In this case I did it with Vorta, but the result is exactly the same with borgmatic.
If I understand correctly, the docker fails because of the appropriate permissions?

 

I feel like I've been digging all over the internet, but haven't found anything yet, nor anyone who has the same problem. This leads me to believe that I have not understood something fundamental.

 

I would be very grateful for any help. Until the backup and restore works well, I don't want to use Unraid seriously.

The error says read only file system, so the file system you are restoring too is read only.

 

Have you checked the directory permissions or the docker path mapping? Are these read only or read/write?

Edited by Solverz
More detail
Link to comment
1 hour ago, Solverz said:

Have you checked the directory permissions or the docker path mapping? Are these read only or read/write?

This is really embarrassing. It was. I had never had to change the permissions in the path mappings before and didn't have that option on the screen.

Thanks a lot! If only I had asked earlier ...

Link to comment
6 minutes ago, vonwanja said:

This is really embarrassing. It was. I had never had to change the permissions in the path mappings before and didn't have that option on the screen.

Thanks a lot! If only I had asked earlier ...

No problem!

 

It was probably set to read only in the template by default so that the container can only read data from your mapped directory and not modify or delete it or its contents, security hardening.

Edited by Solverz
Link to comment
15 minutes ago, Hammy Havoc said:

Does it still make sense to use the Unassigned Devices plugin versus creating a pool specifically for Borg now that we can have multiple pools?

As I like to say, "it depends".

 

There are advantages and disadvantages to both, for instance,

UD devices can be removed and installed adhoc where a pool cannot and unraid will complain if it goes "missing"

 

Pools allow you to have redundancy, UD devices do not from the gui (as far as I am aware, please correct me if otherwise)

 

UD can mount network shares etc and not just local media, pools is for local media only.

 

Etc etc etc, just way up the pros and cons and see what suits your situation and setup the best and use that. ☺

  • Like 1
Link to comment
2 hours ago, Hammy Havoc said:

Does it still make sense to use the Unassigned Devices plugin versus creating a pool specifically for Borg now that we can have multiple pools?


3-2-1 backup means 3 copies, 2 types of media, at least one offsite. 
 

When I wrote that my reasoning was just to get as much diversity as possible between your data and the backup in lieu of 2 types of media. Preferably it would be on a different system with a different Os, filesystem, etc.  There’s no “right” or “wrong”, just better or worse. 

 

 

  • Like 1
Link to comment
  • 3 weeks later...

FYI, if anyones backups are failing since the latest update to this container 2 days ago, please see below:

 

The borgmatic binary is no longer at /usr/bin/borgmatic it is now in /usr/local/bin/borgmatic.

 

Of course, this this will only affect you if you are specifiying the full path to the borgmatic binary in your cronjob.

 

/usr/local/bin is actually and to no surpise in the PATH env so you do not need to actually specifiy the full path to borgmatic and you can instead just put write "borgmatic" and if this path ever changes in the future you should not be affected.

Link to comment
On 8/17/2022 at 7:41 PM, Solverz said:

FYI, if anyones backups are failing since the latest update to this container 2 days ago, please see below:

 

The borgmatic binary is no longer at /usr/bin/borgmatic it is now in /usr/local/bin/borgmatic.

 

Of course, this this will only affect you if you are specifiying the full path to the borgmatic binary in your cronjob.

 

/usr/local/bin is actually and to no surpise in the PATH env so you do not need to actually specifiy the full path to borgmatic and you can instead just put write "borgmatic" and if this path ever changes in the future you should not be affected.

 

Hi Solverz, thank you for this post.

 

I have edited my cron file to the new path below, you mention just using borgmatic instead of specifiying the full path.  please can you change the cron below to show this please as i'm unsure exactly.

 

Quote

0 1,13 * * * PATH=$PATH:/usr/bin /usr/local/bin/borgmatic prune create -v 1 --stats 2>&1
0 6    * * 3 PATH=$PATH:/usr/bin /usr/local/bin/borgmatic check -v 1 2>&1

 

Link to comment
On 8/21/2022 at 8:36 PM, Greygoose said:

 

Hi Solverz, thank you for this post.

 

I have edited my cron file to the new path below, you mention just using borgmatic instead of specifiying the full path.  please can you change the cron below to show this please as i'm unsure exactly.

 

 

0 1,13 * * * borgmatic prune create -v 1 --stats 2>&1

0 6 * * 3 borgmatic check -v 1 2>&1

 

 

Hope that helps!

 

Edited by Solverz
Syntax
  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.