how do i avoid this rsync issue


comet424

Recommended Posts

ok sounds good well I have a really slow internet connection anyways so doesn't matter lol 600kbs upload 5megabits download

 

and do you know if user scripts can handle if statesments    so  I wanted to do   if OpenVPN = connected  then run the rsync   if not  then wait 5 min  and reconnect OpenVPN   and do this 3 times  if it cant initialized 3 times just forget it

 

just like a fax machine....  it will try to send a fax 3 times with 2 or 3 min pause before trying again  and then it just quits if it cant establish

is that possible in your user scripts and OpenVPN with rysnc   or am I asking too much 

Link to comment

Should be doable - you can have openvpn establish the connection, the script will ping the remote unraid and if it works... it will run the rsync.

This is a bit beyond me (the openVPN part) as I use router to router GRE tunnels that are up 24/7

Maybe you should open a new thread for the OpenVPN part. so that the other users who use openvpn on unraid / or pfsense can give you help.

Link to comment

I did OpenVPN  was told had to do it in the support page for OpenVPN.. well I asked and no one replied not even the coder for OpenVPN unraid .

ya you helped a lot 

as for the other guy   he said run memtest for days how long do you run it.. and how do you tell if a motherboard or power supply droppin power like you mentioned does unraid monitor this stuff and record it.. is there an app the says power is dropping   etc like the bios shows running voltages..

what is a GRE tunnel.. and how much bandwith does that use..

Link to comment

About the memtest and monitoring of power, I ran memtest for about a few cycles when I first built my machine.

as for power monitoring, I just made sure that my components can never overload my PSU without going way over. then my UPS does report power consumption of the PSU, so I have a fairly good idea if there are issues - but there is no monitoring tool. unless you get expensive industrial component testing tools. My previous PSU did fail without warning and I never figured out why.

 

GRE is a network protocol that like OpenVPN can be used to create virtual network links across the internet. key difference is by it self, there is no encryption, but the link will feel like a slower LAN connection. It doesn't really consume bandwidth unless I try to send traffic across it, mainly because I didn't use the same subnet on both sides. I actually have 3 locations linked up this way.

Link to comment

ah I used memtest for 10 cycles when I first did unraid on my gaming machine..

I have unraid plugged into a ups now.

my power suppy is a 750 watt corsair.  powers 6 hard drives

so the GRE  does that mean your ISP can see what your files are whe you transfer  no encryption...  

oh 3 locations  and how often you run rysnc?   that's my goal...  1 Server sisters house... 1 Server in My house  1 Server in my shop  and they replicated to each other with the rsync…  GRE is that part of unraid?

Link to comment

and doesn't rsync  send encrypted? reason for the private and public key...  I not at home to try it now anyways.. but was thinking instead of root@mitchsserver:/     can I not do  [email protected]:/   

and then it be encrypted from the public and private key of  rysnc?

but probably doesn't work that way

Edited by comet424
Link to comment
14 hours ago, ken-ji said:

@tr0910 How do you to want us to truncate the thread? should we have done the discussion as PMs next time?

Ha, I was just kidding.  It's great to have it all out in the open like this.  But we really need to simplify this process.  I haven't seen many things in unRaid that can get this ugly.  Thank goodness some of the extraneous material you resolved for him wasn't normally required.

 

This thread will really scare off the iwannabackup crowd now.  lol

Link to comment
9 hours ago, comet424 said:

ah I used memtest for 10 cycles when I first did unraid on my gaming machine..

I have unraid plugged into a ups now.

my power suppy is a 750 watt corsair.  powers 6 hard drives

so the GRE  does that mean your ISP can see what your files are whe you transfer  no encryption...  

oh 3 locations  and how often you run rysnc?   that's my goal...  1 Server sisters house... 1 Server in My house  1 Server in my shop  and they replicated to each other with the rsync…  GRE is that part of unraid?

If you did memtest on both machines at both ends, your corruption is strange.  Could be ISP related, but I have never seen that even to China.

Yes, the backup is encrypted over the wire via ssh encryption.  Over the wire you don't need to worry about your data being visible.

GRE is a permanent connection between 2 sites.  Don't worry about it for now.  This is advanced stuff for later.  I would suggest loading TeamViewer on a Windows machine on the destination.  That way you can connect over the net and get in and do all the things you would normally, just like being there.  Later once you are bored and want cooler advanced always connected connections look at GRE or OpenVPN. 

The bunker utiliity I mentioned was the early command line version of the Dynamix File Integrity Plugin (find it on the unRaid APP store).  They are fully compatible and work together nicely.  Originally designed to identify file bit rot, it is super helpful for you in detecting any rsync transfer corruption.  First make sure the file hashes are calculated and stored in the extended attributes before you rsync.  Then make sure that you use "rsync -avuX"  The X tells rsync to transfer the extended attributes where those file hashes are stored.  Then on the destination computer, you can ssh in and execute the bunker command to verify if there were any files that arrived in different form than they should have.  Check it out here:

 

Edited by tr0910
Link to comment

@tr0910ok ill look that stuff up..

as for rsyncing over the net haven't done that yet... what I was doing with all the issues is I had my sisters unraid at my place and I was trying to get rsync to work properly … it was working properly at first.. but we had a few days of power outages on and off and unraid kept shutting down due to no ups ….  and then I was having rysnc issues it wasn't working..  so the I was asking help.. and the @ken-ji  was helpful to help which was great  as I guessing it corrupted files  reason  I couldn't ssh or all that good stuff...

 

so since it was working yesterday.. I took it over to mysisters  house and pluged it in   and went home after  to try it out that's where I wasn't able to rysnc over the internet 

now  maybe your advanced too and may shed light too?  

as I tried to get OpenVPN  to to work on unraid I cant seem to connect to it I figured unraid (my server) be client  and unraid (my sisters) be host for OpenVPN  it would connect and transfer...  but that didn't work

 

I tried changing root@mitchsserver:/mnt/,,,,,    to  [email protected]:/mnt.... in the script file or command line and that failed.

 

 

now the current setup I do to connect to her House and her unraid is..

 

in Windows (my computer ) I have OpenVPN Client

on her network  I have PFsense Router  with OpenVPN Server running...  so I can OpenVPN to her router... and then remote access her unraid and pfsense.. and im guessing that similar to your TeamViewer

 

but if rsync ssh is encrypted.. then I wouldn't need OpenVPN correct? I can skip that .. as I was trying but I found if I have OpenVPN client on unraid and I tried to connect to my sisters pfsense router OpenVPN... it locked out unraid… id have to physical shut it down.. it locked me out of the shares too...

 

but if I rysnc is encrypted data transfer

 

what do I need to change or edit?  if I cant get OpenVPN to connect and then just rsync root@mitchsserver

 

sorry if I sound confusing...  with dyslexia and learning disability my head knows what I saying I just never explain it right for people sometimes

Link to comment

and when rysnc is done  to make sure I never have to run new permissions

would I do  after the rysncs are done

to do … will both these ways work

 

chmod 777 root:mitchsserver:/mnt/*

 

or would I do chmod before rysnc starts  so it makes it full rights of the Tower files before it starts

 

chmod 777 /mnt/*

 

 

Edited by comet424
Link to comment
2 hours ago, comet424 said:

so I got it to rsync  had to get help for port forwarding  on pfsense.. but that worked..

I don't think you should be using port forwarding for rsync over ssh as this would expose the unraid servers ssh to the internet and with the weak password you probably have set (based on our long debugging exercise), your server would get hacked in a matter of minutes.

 

2 hours ago, comet424 said:

now Id like to know @ken-ji have you gotten the email to text you once its complete to send you a email or text if there was errors.. as I cant get notifications  either to email me or text me

This means you need to add to the rsync script some stuff that will email/ notify you. And no, I don't have such a thing setup on my end either. but since I do run something every day to backup to Dropbox, my PC would show a notice about files being added or deleted daily.

 

Finally, you can try running this after you rsync depending on which is the appropriate one.

chmod -R 777 /mnt/user/Mikes Files/Documents

chmod -R 777 /mnt/user/Mitchs Files/Documents

 

Link to comment

Regarding hacking ssh, yes your server will get probed if you have port 22 (default port for ssh) forwarded in your router firewall.  Without a port being forwarded,  you cannot connect via ssh and do ssh rsync to my knowledge.  I use the ssh plugin and change the ssh port to something between 50000 and 60000.  This eliminates the probing.  I also remove password access to ssh once I have this working well and test by rebooting a few times on both servers.  Only then is it safe to remove password login, as you might lock yourself out.  See the plugin on the unRaid app store.  As @ken-ji has warned you, you should not use this plugin, until you have your server working stable and rebooting it a few times using the standard port 22.

Edited by tr0910
Link to comment

@ken-ji@tr0910

as for port forwarding I have no other means of getting behind the router if I don't forward port ...

and if I change the password since yes its weak.. does that mean I gotta change the keys too..

I haven't tested many reboots I did before  I took to my sisters.. and I tried 3 times rysnc over internet  and script so that worked.. I did the rysnc mitchsserver.dnsaddress.com   and did the ssh mitchsserver.dnsaddress.com  and I was able to do that 

 

so ok pick a port in the high numbers not 22.. and I thought using ssh   with private and public key you cant get hacked.. because  you are using the ssh key...

and ok as for the plugin  haven't reinstalled it yet... but what does it have the regular ssh doesn't ?

 

Link to comment

@comet424 Would like to recommend you look up pfsense site to site VPN as that will make your network and your sister's seem like directly connected subnets and allow you to administer stuff as if it was all at home, with just additional latency.

 

Like I said I use three Mikrotik routers providing site to site VPNs. and since we have the same ISP is in the same city, I can access my server content as if I was at home from my parent's and sibling's place. I don't have an offsite backup plan yet and am still working on it, but really important stuff (for me) is backedup to dropbox and google drive so if I loose my NAS I will get annoyed, but its not the end of the world.

Link to comment
On 3/6/2019 at 10:01 PM, comet424 said:

and ok as for the plugin  haven't reinstalled it yet... but what does it have the regular ssh doesn't ?

The plugin is supposed to provide a UI for tweaking the SSH configuration, like preventing SSH access via passwords, which if you port forwarded the SSH service, will prevent access unless the private key is present. There a few more settings, but I never bothered with the plugin, as I know enough to just edit the config files directly.

Link to comment

sorry for delay...

@John_M @ken-ji

so I find that rysncing over internet  to my sisters server doesn't work  if her ip address changes.. I find I have to keep reading the yes  in the SSH command  to add the ip address it doesn't add the dns adder name instead..

 

as for the VPN  I was the one that brought it up  about using a VPN but OPen VPN doesn't work on unraid..

1..  unable  to connect Client to Server Unraid by peter_Sm  so for its been usless it can never connect when I set it up locally on my network  or when I tried to connect over the internet to my sisters house.. 

also there client unraid locks out Unraid when you connect Unraid OpenVPN to PFSENSE OpenVPN Server

 

2.. I tried the Open VPN AS    and web gui fails 99 percent of the time page cant be found  worked once  but tired again gone again..

 

3..  I don't think I can do pfsense to pfsense as it problem drain my internet connection  and I find it mess's up my own network DNS for naming  computer IPS  usually disconnect it just to use my own...  and  I only have a 5Mbps/500k Download/Upload connection  so im sure its just going to kill my connection when I don't need to be connected

 

4..

I have tried couple topics  no one could help me  for a OpenVPN -Connect    run rysnc  - OpenVPN - Disconnect....

no one could help me with that issue so I gave up  trying OpenVPN 

Link to comment
23 minutes ago, comet424 said:

I have tried couple topics  no one could help me  for a OpenVPN -Connect    run rysnc  - OpenVPN - Disconnect....

no one could help me with that issue so I gave up  trying OpenVPN 

Have you tried zerotier? There's a docker for it, very easy to use and configure.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.