how do i avoid this rsync issue

[comet424]

had to add ndpt server names  to fix the time

 

 

[ken-ji]

8 minutes ago, comet424 said:

shouldn't the authrorized keys and id_rsa  time stamps be updated to the new time everytime I reboot

it should...

 

12 minutes ago, comet424 said:

and what is this /usr/local/sbin/emhttp &    I know when I remed it out I couldn't access the server at all  no shares or webgui  and does it matter which is first 

don't ever touch that line - its the unraid core process  - this includes the GUI and the array

 

12 minutes ago, comet424 said:

so does that mean I wanna do chmod 777 after a rysnc  as I find if I copy or had rysnc  I had to no rights to delete or move files in windows from folders

have to do every couple days or so  the new permissions   to all    cuz when I move files from windows to the unraid I don't have rights to change it later it seems

the rsync command is transferring the same permissions from tower->mitchsserver (and back) which might not be correct on the other server.

you can do na ls -l /mnt/user/Documents/{some file} /mnt/user/Mikes\ Files/Documents{some file} so I can advise you.

 

23 minutes ago, comet424 said:

and it didn't work... once i type yes it starts... when i reboot back to gotting type yes again  tried 3 times

we forgot to save known_hosts file I mentioned when I jumped into the thread

 

add the following line to the go file (after the authorized_keys)

on tower: ssh-keyscan mitchsserver > /root/.ssh/known_hosts

on mitchsserver: ssh-keyscan tower > /root/.ssh/known_hosts

 

Edited March 3, 2019 by ken-ji

[comet424]

ok so permissions may no work now or show different cuz I did new permissions lately well couple hours ago..  id have to try again from Windows desktop to unraid  ill do that shortly... and the keyscan  how come a normal  cp wouldn't work??  ill give this a reboot and see what happens lol

so go file now is 

mitchs server

 

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

#Mitchs Server Copy SSH    
    mkdir -p /root/.ssh
    cp /boot/config/ssh/mitchsserver_root /root/.ssh/id_rsa
    cat /boot/config/ssh/tower_root.pub > /root/.ssh/authorized_keys
    ssh-keyscan tower > /root/.ssh/known_hosts 
    chmod g-rwx,o-rwx -R /root/.ssh

 

 

 

tower

--------

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &

# Copy SSH files back to /root/.ssh folder and set permissions for files
    mkdir -p /root/.ssh
    cp /boot/config/ssh/tower_root /root/.ssh/id_rsa
    cat /boot/config/ssh/mitchsserver_root.pub > /root/.ssh/authorized_keys
    ssh-keyscan mitchsserver > /root/.ssh/known_hosts  
    chmod g-rwx,o-rwx -R /root/.ssh
 

 

[comet424]

so it worked one way not the other ugh  getting closer

so you seen the go file above  I also did both reboots of each server

 mitchserver works   but tower doesnt

 

Linux 4.18.20-unRAID.
Last login: Sat Mar  2 22:15:07 -0500 2019 on /dev/pts/0.
root@mitchsserver:~# rsync -avzu -s --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  root@tower:/mnt/user/Documents/ /mnt/user/Mikes\ Files/Documents/
Warning: Permanently added the ECDSA host key for IP address '192.168.0.3' to the list of known hosts.
receiving incremental file list
^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(642) [generator=3.1.3]
rsync error: received SIGUSR1 (code 19) at main.c(1440) [receiver=3.1.3]
rsync: [generator] write error: Broken pipe (32)
root@mitchsserver:~#

I canceled it   for now

 

on tower when I do it

Linux 4.18.20-unRAID.
Last login: Sat Mar  2 22:21:04 -0500 2019 on /dev/pts/2.
root@Tower:~# rsync -avzu -s --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Documents/ root@mitchsserver:/mnt/user/Mikes\ Files/Documents/
The authenticity of host 'mitchsserver (192.168.0.244)' can't be established.
ECDSA key fingerprint is SHA256:APbRWv8D+cAMRGW1g4XnvufCGR/6hSe/lsk07JMVdSQ.
Are you sure you want to continue connecting (yes/no)?

so close yet so far away lol

[comet424]

Linux 4.18.20-unRAID.
Last login: Sat Mar  2 22:21:04 -0500 2019 on /dev/pts/2.
root@Tower:~# rsync -avzu -s --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Documents/ root@mitchsserver:/mnt/user/Mikes\ Files/Documents/
The authenticity of host 'mitchsserver (192.168.0.244)' can't be established.
ECDSA key fingerprint is SHA256:APbRWv8D+cAMRGW1g4XnvufCGR/6hSe/lsk07JMVdSQ.
Are you sure you want to continue connecting (yes/no)? ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(642)[sender=3.1.3]
root@Tower:~# ls -l /root/.ssh
total 12
-rw------- 1 root root  399 Mar  2 22:17 authorized_keys
-rwx------ 1 root root 1823 Mar  2 22:17 id_rsa*
-rw------- 1 root root  662 Mar  2 22:17 known_hosts?
root@Tower:~#

 

[comet424]

and another issue dunno what happened.. other day when the rysnc was working before i asked for help... i was getting like 80 meg/s transfer rate from tower to mitchsserver now max is 31meg/s  like to know how it slowed down  frig lol always something

[ken-ji]

20 minutes ago, comet424 said:

Linux 4.18.20-unRAID.
Last login: Sat Mar  2 22:21:04 -0500 2019 on /dev/pts/2.
root@Tower:~# rsync -avzu -s --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  /mnt/user/Documents/ root@mitchsserver:/mnt/user/Mikes\ Files/Documents/
The authenticity of host 'mitchsserver (192.168.0.244)' can't be established.
ECDSA key fingerprint is SHA256:APbRWv8D+cAMRGW1g4XnvufCGR/6hSe/lsk07JMVdSQ.
Are you sure you want to continue connecting (yes/no)? ^Crsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(642)[sender=3.1.3]
root@Tower:~# ls -l /root/.ssh
total 12
-rw------- 1 root root  399 Mar  2 22:17 authorized_keys
-rwx------ 1 root root 1823 Mar  2 22:17 id_rsa*
-rw------- 1 root root  662 Mar  2 22:17 known_hosts?
root@Tower:~#

 

your known_hosts file has a ? at the end? maybe you have a extended non-printable character in your go file for tower...

 

[comet424]

managed to get tower to send now  had ?  after knownhosts in the go file  not sure why...

so the rysncing is working both ways so far

just cant figure why the speed is not at 80  like it used to now its 30  and I wanna know about progress for rsyncing if there is one  for total completion left 

[comet424]

well I need more help but I too tired for today its rsyncing  so I leave it for now  and ask you some rsyncing questions 

like   if the remote computer runs out of diskspace.. .   tell Rsync to stop  as I noticed it doesn't stop

[ken-ji]

it doesn't - rsync just tries the best to sync across on a file per file level... so disk space is the user's job not the tool. unfortunately.

[comet424]

ah ok I like it cuz  if I transferring files in windows always says out of disk space... but I only wanna copy the files that are updated  not everything.. so I like the rsync in that way..

 

what about the in Freenas  rysnc will tell you the overall completion  like 56% complete … or in windows  you got 5 hours left to copy files... how you do that in unraids rsync  so I know how much time is left...

 

and then there is the OpenVPN issue I having  and no support so far even from the support page I messaged in there

[ken-ji]

one thing you can add to the rsync command is --progress to show you how many files to go rsync thinks needs to be processed and where you are wit all the files

documents/The Ark_ Children of a Dead Earth Book One_B00TCI402K.kfx
        370,745 100%  883.06kB/s    0:00:00 (xfr#39, ir-chk=1225/1267)
documents/The Fourth Element Trilogy_ Boxed Set_B075VSM5X5.kfx
        928,331 100%    1.93MB/s    0:00:00 (xfr#40, ir-chk=1224/1267)
documents/The Glass Gargoyle (The Lost Ancients Book 1)_B00V5JF4GA.kfx
        289,427 100%  565.29kB/s    0:00:00 (xfr#41, ir-chk=1223/1267)
documents/The Grid 1_ Fall of Justice (The Grid Trilogy)_B015XPFZB0.kfx
        163,917 100%  297.54kB/s    0:00:00 (xfr#42, ir-chk=1222/1267)

I guess the UIs just process the info from this.

[tr0910]

@ken-ji after your heroic effort to get comet424 up and running, we sure need to clean up these threads.  Newcomers will be terrified by what you guys just went through.  Surely it doesn't need to be this difficult.

[comet424]

@ken-ji  so its working  

ya I have that progress  but it only shows progressive of over all.. so it doesn't say  Progress of transferring 10TB of data the current stats of overall percent don't and how much longer it will take...  same with windows copy  

 

now I also run in an issues    

1.. its rsyncing the recycle bins  I seen something  -exclude =""  so I guess be -exclude =".recyclebin"

2. is there a way if I change file names on the main server(tower)  for certain things  it when comes to rysnc time  it will rename  the files on mitchsserver  or delete them  and put the new files there..

3..  I cant rsync my enter server as I have 34TB of data.... but from rsync part of it to my sisters server the important stuff.. I got a 

 

rysnc: read error  mapping...

 

I tried to copy file off the Tower (source)  I cant manually copy file  its a video I can watch it off the server  I cant copy  I cant move... I cant even copy to windows

is there a way to fix this... is there a way to test all files  or do I need bigger hard drives on my sisters server to backup the entire 34TB to make sure no other files broken?

[comet424]

I was wrong I had skipped across video  and there was no errors...  watching the video  it crapped out... my bad

and at moment its doing a parity check..

I had replaced the HD's    because I had UDMA CRC check errors on the HDS so I swapped out 

had 1  HD go.. and the Parity Drive had issues  with read write heads 

but I had replaced and rebuilt the array... but  I see the rsync only showed this error so far...

 

 

can rsync or another program  go check every file to see if there is an error so I can fix it?

 

Edited March 4, 2019 by comet424

[comet424]

I re tried rsync a few times and now it synced file to the other server...  could the error issuing be caused from Parity checking... or that a Parity Checked fixed the file  in between rsync me trying and then allowed it to be copied to the other server or to windows desktop

 

[tr0910]

You should expect perfect copies with rsync.

Anything else indicates hardware problems. I will would generate hashes for a huge folder of copied data then see if the files match at dest.

The bunker tool created long ago would send the hashes to file extended attributes. Then rsync -avuX where the X made sure the extended attributes we copied across then bunker run at dest could confirm that the hashes matched the files at destination. But other hash tools will also work.

A single mismatch of even one file should send you into memtest for days on both servers. Could be RAM, PSU, or even MB.

Sent from my chisel, carved into granite

[comet424]

ah ok  I think I understand... I had rebooted Tower(main server)  and when I ran rsync  and still couple copy the file to the other server... and I still couldn't do a Windows File copy of that certain file to the windows desktop had network error...  but I at 77% complete Parity Recheck of 34TB  I figured it some how fixed the file while parity checking went on  ?  

 

so is there a possible test to test this..

 

id also like if there is an error  that rsync could email me or send me a text ….  is that even possible..

I did try to get notifications to email me using a Hotmail account option  and I set the warrnings all array etc  to email me... it never emails me  just shows up in the browser...  

and Id like r sync not to copy the recycling bin  gonna try the -exclude option another date.. as I just took my sisters mitchsserver offline and taking it to her house 2 hours away.... so I wanted to get least the rsyncing to work on my network    so I know it will work over the internet..

once I get OpenVPN to work since it doesn't work over network... and when I use OpenVPN client (unraid) to connect to her pfsense OpenVPN  it locks out unraid…. so I plan to try to get OpenVPN(server) on her server and then OpenVPN client on my unraid to connect..

 

heck if I could figure how to get unraid to get my windows OpenVPN to connect to her pfsense OpenVPN  then rsync data...

but ill play with it later  least I got majority of the bugs out  thanks to @ken-ji  after a few power outs the unraids  lost there rsyncing and corruption happened...  so  now I going to have both unraids on APC UPS's   so wont happen again

[comet424]

and I really appreciated @ken-ji help  as with my dyslexia  I miss read stuff and he helped me with my errors and his too.. but least I saved the procedures in a simple text file for anytime if I had to Format  an unraid  and start fresh... or to setup another unraid which I plan to do ..  have a Backup Server... instead of using hd enclosures...  so I have the steps  

I wish I stuck playing with Linux stuff out of college but I stoped 20 yrs ago at redhat  and just stuck with windows and Visual Basic programming...

 

 

[tr0910]

Parity never fixes anything. No files ever get touched when you run parity

All that happens is your system is very busy checking things while parity is running. So other hardware problems like marginal PSU tends to show up then.

Did you ever run extended memtest on both servers? Several days is a good idea. This should be the first thing you do before loading any data onto the drives.

Sent from my chisel, carved into granite

[ken-ji]

14 hours ago, tr0910 said:

@ken-ji after your heroic effort to get comet424 up and running, we sure need to clean up these threads.  Newcomers will be terrified by what you guys just went through.  Surely it doesn't need to be this difficult.

@tr0910 How do you to want us to truncate the thread? should we have done the discussion as PMs next time?

[ken-ji]

@comet424 what @tr0910 said:

rsync is supposed to make perfect copies of the files.

If there are issues, it typically means: the source cannot be read reliably which in turn means your disks are failing, or the system has errors in memory (caused by bad ram or bad MB) that corrupt data in memory, or your PSU is starting to give and is causing the drives to intermittently return wrong data.

 

Generating checksums is a good idea to see / verify if your files are intact - but checksums should be created when the data is in a known working state - checksums changing on the source is indicative of system problems too.

 

[comet424]

ah ok    how do I do checksums... like  when I rebooted the tower 3 or 4 times as I was doing things    and tried rsync it didn't fix it.. but the source fixed itself then

 

and no never ran  memtest for days.. tower been running almost a year or so...  how can I tell for tests... and if I run memtest how many days do you run it for??   so then strange  about how it fixed itself then ok...

@ken-ji so couple questions do you know how to do the rsync questions I asked above is it possible??  and send I drove couple hours setup mitchsserver on my sisters end...  I home now... I OpenVPN on my windows machine to her PFsense router... and then I access'd her mitchsserver  so I was able to access..  now when I tried running the rsync  it failed..  yet it worked here 

rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.3]
Backing Up Mitchs Install Program To Mitchs Folder (Tower) share ===== Mon Mar 4 21:12:43 EST 2019

rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.3]
Backing Up Mitchs Pictures To Mitchs Folder (Tower) share ===== Mon Mar 4 21:12:44 EST 2019

rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.3]

 

cant see why it doesn't work.. it worked before I left home

[ken-ji]

OpenVPN is on Windows PC, so only your windows PC can connect to mitchsserver (or the remote network for that matter). your OpenVPN client needs to be on your router for all devices to be able to access the remote network.

 

You use a different app to generate the checksums and for comparing. (I use md5sum on the command line)

 

rsync can use

-c option to compare source and destination via checksums but this is only so that the unchanged files but have updated timestamps or vice-versa will sync properly.

--delete to ensure dest matches source ; extra files are all deleted

--exclude pattern to exclude anything that matches the pattern (ps Unraid/Linux is case sensitive); use as many to declare many patterns

 

[comet424]

oh so  when I use OpenVPN (Cilient)  on  my windows PC and connect to her router... then my 1 computer can only see both sides...  I not sure it it will work PfSense OpenVPN Client 

and I trying to get  OpenVPN Server Unraid on Mitchsserver… its started... but I cant seem to get unraid OpenVPN client to connect  and then that's how I figure to rsync 

rsync -avzu -s --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  root@MitchsServer:/mnt/user/Documents/ /mnt/user/Mitchs\ Files/Documents/  

 

that's what I have so far.. where would I add the C...  and there is a Plugin called File Intergrity..  does that check if a file is corrupt?

its weird that I was able to copy again.  but I guess I cant find these errors as it could be whenever?

so  if I add --delete   its not going to accidently delete source..    but just the dest side

 

and do I type    --exclude ".Recycle.Bin"

 

so would it look like

 

 

rsync -avzuc --delete -- exclude ".Recycle.Bin" -s --stats --numeric-ids --progress -e "ssh -i /root/.ssh/id_rsa -T -o Compression=no -x"  root@MitchsServer:/mnt/user/Documents/ /mnt/user/Mitchs\ Files/Documents/  

 

so I trying to get this OpenVPN client unraid to connect to OpenVPN Server Unraid to work...