so i am rsync'ing my entire array to a back up unraid box - i tried to do it initailly by mounting the unraid share in my primary unraid server and then just rsync-ing wiht in my primary server - but rsync hung and stopped working so i nixed it and switched to rsync'ing via ssh and it worked like a charm ... i mean it took like two  plus days to do the back up but it worked.

 

now i'm running rsync again on my master to push a fresh update via the smb share --- again it got half way through (ish) and crashed

 

setting up a recurring job via ssh would be a pita because unraid doesn't retain ssh keys between boots and i'm lazy and i dont' want to do ssh-key-add or whatever it is 

 

has anyonen else had issues with rsync'ing lots of data across a smb share?

 

i don't want to use a backup docker app because i don't want archived or encrypted local backup - i just want a slave 

15 minutes ago, SeattleBandit said:

i don't want to use a backup docker app because i don't want archived or encrypted local backup - i just want a slave 

It doesn't need to be either of those things.  You can use a docker like Syncthing to make a 1:1 clone and a lot easier and more reliable than it seems to be using rsync manually.  I'm sure there are others, maybe like Duplicati, to do the same thing.

6 hours ago, Energen said:

It doesn't need to be either of those things.  You can use a docker like Syncthing to make a 1:1 clone and a lot easier and more reliable than it seems to be using rsync manually.  I'm sure there are others, maybe like Duplicati, to do the same thing.

tried duplicati - it archives stuff and compresses it 

 

syncthing crapped the bed when i tried to set it up for my entire array 

 

any other suggestions?

12 hours ago, SeattleBandit said:

... i'm lazy ...

There lies the problem. Laziness works fine with small number of files. It tends to crap the bed when things become too big to deal with.

 

Try rsync a small section of the server each type (e.g. one share at a time) to identify the share that tends to cause issue and then break it into smaller chunks. If something as barebone as rsync hangs then there isn't quite any other solution out there that would work properly.

I'd wait for Hoopster to chip in here. He will offer a full blown rsync solution complete with SSH keys and all ...

Sent from my CLT-L09 using Tapatalk

7 hours ago, SeattleBandit said:

syncthing crapped the bed when i tried to set it up for my entire array

I used 2 syncthing dockers simultaneously as the "source" (unraid server) and as the "destination" (external usb client via unassigned devices) to back up entire shares and did not have any problems with that...  syncthing is a little bit of a hassle to set up properly with the shared folders and all that but once it's going.. it goes.

 

I currently use it to pull files from an internet server, similar to ftp... so that I'm always receiving new files.

 

I would personally suggest that you look at syncthing again and try to get it set up because I think it's going to be the most reliable for you..

 

Note that it does take quite a while for syncthing to scan and build a database for an entire array -- perhaps that's where you were not giving it enough time.. when I first did this it easily took more than a day to scan everything up.  It will not begin transferring data until the initial database is built.

 

6 minutes ago, Energen said:

I would personally suggest that you look at syncthing again and try to get it set up because I think it's going to be the most reliable for you..

The guy proudly proclaimed he's lazy. There isn't a software that would solve laziness.

I tried synching and a real pita it was too. I even registered on the ST forum, asked a couple if questions and was told that it wasn't really designed to be used in the way I wanted use a regular backup to another machine. I have a backup unraid server which I wanted to power up, say weekly, and rsync changes with. Hoopster's your man ...

Sent from my CLT-L09 using Tapatalk

49 minutes ago, superloopy1 said:

I tried synching and a real pita it was too. I even registered on the ST forum, asked a couple if questions and was told that it wasn't really designed to be used in the way I wanted use a regular backup to another machine. I have a backup unraid server which I wanted to power up, say weekly, and rsync changes with. Hoopster's your man ...

Sent from my CLT-L09 using Tapatalk
 

thats basically my use case - either daily syncs or weekly syncs - in the interim i'll just rsync the individual shares because it seems to lose its shit when backing up my plex meta stuff - i acutally kicked off a rsync for the entire array last night via ssh and let it run all night (forgot to add the 'time' command so i don't know how long it took) but it actually completed - maybe rsync just needed to relax and lets its emotions go to finish the sync.

 

id love to script it but i haven't found a good solution for ssh keys yet - i've looked at the user scripts plugin (basically a front end for cron it seems like) but the few scripts i've put together to run on a schedule don't seem to work the way i expect them too.

 

i'm eager to hear back from this mythical 'hoopster'  

Drop him a pm, he's v.helpful.

Sent from my CLT-L09 using Tapatalk