tomwhi Posted June 8, 2020 Posted June 8, 2020 Hi, I wondered what the advice would be to sync files from my UnRaid server to another UnRaid server over the internet (my Dad's house). I want to be able to both put our project photos in plain view without modification on both sides (not encrypting or chunking into Zips). I would also like to backup my private data and encrypt it onto his server in mine had a disaster (zip and encrypt would be ok for this). I've seen Duplicati can be used to backup which can encrypt but also zips the files into chunks (which would be ok for the backup part of my project). I've seen rclone can be used to Sync but it looks complicated. I've seen spaceinveder one has done a video on both but mainly talks about putting it into the cloud. Are there any good ideas/advice on how to do this, and any good guides you'd point me at before I start down the wrong road? Quote
Hoopster Posted June 8, 2020 Posted June 8, 2020 22 minutes ago, tomwhi said: I wondered what the advice would be to sync files from my UnRaid server to another UnRaid server over the internet The discussion below may fall into the "too complicated" scenario for you, but, it is worth a read. I use this method and was a complete SSH/rsync novice when I started. It took a couple of days, and lots of questions in that thread, before I got it figured out, but since then, it has worked without issue and completely unattended for over two years. Quote
tomwhi Posted June 8, 2020 Author Posted June 8, 2020 1 hour ago, Hoopster said: The discussion below may fall into the "too complicated" scenario for you, but, it is worth a read. Thank you! I think I saw another one of your posts during my Googlin' referencing this post. I'll deffo give it a read though - because it was quite an old post I didn't want to be missing a trick if there was something else around. In the post I did see I saw it was setup on the same LAN - did you move the server off the WAN and does it just operate over SSH (port 22)? Quote
Hoopster Posted June 8, 2020 Posted June 8, 2020 (edited) 2 hours ago, tomwhi said: In the post I did see I saw it was setup on the same LAN - did you move the server off the WAN and does it just operate over SSH (port 22) Both servers are still on the same LAN; however, the OP started that thread to describe how he was backing up/syncing files between U.S. and China servers. He used a Raspberry Pi (I believe) to power on the backup server over a WAN. Another user in that thread was doing something similar via a router. It will work both on a LAN and over a WAN with the right setup. I just don't have the WAN setup yet so I cannot offer first-hand experience on that configuration. Edited June 8, 2020 by Hoopster Quote
tomwhi Posted June 9, 2020 Author Posted June 9, 2020 12 hours ago, Hoopster said: It will work both on a LAN and over a WAN with the right setup. I just don't have the WAN setup yet so I cannot offer first-hand experience on that configuration. cheers for the info! I'll give the thread a good read now and get cracking :-) Quote
tr0910 Posted June 16, 2020 Posted June 16, 2020 (edited) If you are on the same LAN, no complicated SSH keys are needed. I was backing up over the same LAN for years beginning with unraid version 5 way back in 2011 just by NFS mounting the drives from the other server and using rsync as follows. mkdir /mnt/t2disk1 mount -t nfs tower2:/mnt/disk1/ /mnt/t2disk1 rsync -av --stats --progress /mnt/t2disk1 /mnt/disk1/ >> /mnt/disk1/disk1.log I would do this for each disk, and rsync would backup the servers in unbelieveable speed. There were 250,000 files per disk, and the process over gbit ethernet would be finished in about 60 seconds per disk, provided there were no changes to the disk. This works good for disk to disk rsync where you are on the same LAN, and you are sure the connection is safe. Not so good if you want to go from user share to user share. Edited June 16, 2020 by tr0910 Quote
tomwhi Posted June 17, 2020 Author Posted June 17, 2020 23 hours ago, tr0910 said: If you are on the same LAN, no complicated SSH keys are needed. I was backing up over the same LAN for years beginning with unraid version 5 way back in 2011 just by NFS mounting the drives from the other server and using rsync as follows. I managed to get this "working", and I have to say without you, @Hoopster and @ken-ji's posts in the original thread I would have been lost. So thank you! I say "working", because my keys don't survive a reboot but I've been through all 6 pages of the original thread and worked out what I did wrong so that'll be a doddle to fix. Even so, all I have to do is put in a username and password until I can automate it so the solution is ready to rock. I am going to try and write my solution up at some point (mainly because a mate at work wants to do the same thing) and hopefully it helps others. I am using your original rsync commands from the original post for data I want sync'd, and then I got Duplicati working to encrypt personal files, photos and videos (that I'd rather not be in plain text on my dad's server!!) which are sent using the same idea of SSH/SFTP so it's encrypted in transit too, and tested a restore across the wire and locally to my dad's server in case my server totally died. #sample rsync commands rsync -avu --numeric-ids --progress --stats -e "ssh -i /root/.ssh/Tower-rsync-key -T -o Compression=no -x" "/mnt/user/Audio Books/" [email protected]:"/mnt/user/media/Audio\ Books/" rsync -avu --numeric-ids --progress --stats -e "ssh -i /root/.ssh/Tower-rsync-key -T -o Compression=no -x" "/mnt/user/Ebooks/" [email protected]:"/mnt/user/media/Ebooks/" I installed zerotier this morning and this is just the icing on the already glamorous cake. I tested it with my mate and he was able to access my server once I authorised him access (which was quickly revoked after testing). I selected the IP range of 192.168.192.* so everything I join to that network is on the same network range and can access other nodes on that virtual network - epic, it worked so quickly and easily. Tom Lawrance on YouTube does a good starter video on it. It works amazingly in UnRaid too and passes the virtual network down to the host. So I can setup my rsync and duplicati to use the 192.168.192 range which will never change even if public IP's change. And just for piece of mind the data is still encrypted because of your original SSH idea - which I keep, even though it'll be double encrypted, I'm not fussy about transfer speeds as long as the data gets there. 1 Quote
Hoopster Posted June 17, 2020 Posted June 17, 2020 5 hours ago, tomwhi said: I managed to get this "working" 👍Glad it's working for you. Once you get things set up the way you want them to work, you can just sit back and let the backups happen and not worry about it. This procedure has never become "broken" or stopped working for me in over two years. 1 Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.