loady Posted March 20, 2015 Author Share Posted March 20, 2015 DOH!...didnt notice the page needed to scroll down Quote Link to comment
loady Posted March 21, 2015 Author Share Posted March 21, 2015 Well thank you weebo ! im now happily going ahead with the next disk move, thank you ever so much for your support and guidance, i have learned a little bit about linux that i didnt know before, especially interpreting what i am seeing you type and my urge to know what some of those paremeters i typed meant, i see this rsync is very powerful with all its options,,you didnt spoon feed me in that im eager to know more and feel accomplished in this. This is exactly what i like about this community..answering the what might seem stupid questions. i tip my hat to you. Quote Link to comment
WeeboTech Posted March 21, 2015 Share Posted March 21, 2015 It is my pleasure to empower you to help yourself! Quote Link to comment
loady Posted March 22, 2015 Author Share Posted March 22, 2015 so..thinking ahead...whilst this is currently removing the files from the source disk which is going to take a few hours, could i open another screen session and start creating the md5sums for the next filelist on next disk to be done or would it slow down the process that it would even out in time the same ? Quote Link to comment
WeeboTech Posted March 22, 2015 Share Posted March 22, 2015 so..thinking ahead...whilst this is currently removing the files from the source disk which is going to take a few hours, could i open another screen session and start creating the md5sums for the next filelist on next disk to be done or would it slow down the process that it would even out in time the same ? Go ahead, it will all work out and schedule itself accordingly. Considering it's an N40L, my guess it would even out in time the same ? But then again, the rsync checksum algorithm may not be as CPU intense as the md5sum. Also, you wouldn't have to babysit the machine for as long. I.E. you could go out and play longer & harder! <grin> Quote Link to comment
loady Posted March 23, 2015 Author Share Posted March 23, 2015 Must update my sig. ..have an i7 with 8gb ram. I went ahead anyway...servers crashed..telnet died. ..not sure if it finished either task. Quote Link to comment
loady Posted March 23, 2015 Author Share Posted March 23, 2015 dont think this finished removing files from source disk, so i re run that step again and it says sending incrmental filelist but it seems to be hanging... root@warptower:~# rsync --remove-source-files -vv -rcPX /mnt/disk2/ /mnt/disk3/ sending incremental file list been stuck like this for 40mins now, cant hear the drive chattering. not sure how i should proceed, is it hanging because it half removed files and now the filelist doesnt match possibly ? Quote Link to comment
gundamguy Posted March 23, 2015 Share Posted March 23, 2015 dont think this finished removing files from source disk, so i re run that step again and it says sending incrmental filelist but it seems to be hanging... root@warptower:~# rsync --remove-source-files -vv -rcPX /mnt/disk2/ /mnt/disk3/ sending incremental file list been stuck like this for 40mins now, cant hear the drive chattering. not sure how i should proceed, is it hanging because it half removed files and now the filelist doesnt match possibly ? Not sure if it'll help much but you can add additional -v's to your call of rsync to increase the verbosity. If you send it -vvv or -vvvv it might give you more information and help you determine what's causing the transfer to hang. Quote Link to comment
WeeboTech Posted March 23, 2015 Share Posted March 23, 2015 dont think this finished removing files from source disk, so i re run that step again and it says sending incrmental filelist but it seems to be hanging... root@warptower:~# rsync --remove-source-files -vv -rcPX /mnt/disk2/ /mnt/disk3/ sending incremental file list been stuck like this for 40mins now, cant hear the drive chattering. not sure how i should proceed, is it hanging because it half removed files and now the filelist doesnt match possibly ? Check the unraid interface to see if each of the drives is actually being read. This process can take a long time as well. This is one of the reasons to make the md5sum file. You can do the md5sum verification on the destination disk to insure all files are there and match with validity. If the md5sum check passes on the destination disk, then you do not need to remove the files from the source disk. You can format it. However, if you are having issues with these disks, It would be wise to do a smart long test on each disk for peace of mind. Depending on the size of each disk it will take hours. you would do something like the following replaceing ? with the character of your drive. You will need to turn off all spin down timer on the drives being tested or it will cause the smart test to fail. mkdir -p /boot/log smartctl -a /dev/sd? > /boot/log/sd?.start.smart smartctl -t long /dev/sd? a message will be printed with an estimated time of completion.after completion do smartctl -a /dev/sd? > /boot/log/sd?.end.smart compare each file with diff -u /boot/log/sd?.start.smart /boot/log/sd?.end.smart and review the end file for pending, reallocated sectors or a failed smart error. Quote Link to comment
loady Posted March 23, 2015 Author Share Posted March 23, 2015 somethings very seriously wrong...the server is constantly locking now...wonder if i have stepped upon the problem which has come to this XFS conversion..not sure how i can leave it to do a long smart test when the the telnet session is locking and i have no access to the server...the powerdown does not work either EDIT: i could hear a drive making a sound like its parking the heads, it is disk3, i was in the process of remving the files from disk2 which had been copied to disk3 when the server starting freezing...i have unplugged the drive and it has not locked up..problem i have now is that where i have replugged the drive it is wanting to perform a data rebuild...should i let that go ahead ? not sure what data it is rebuilding..would it be rebuilding the data i copied from disk2 to this disk3 or is it going from the parity and trying to rebuild the original data before i formatted it to XFS..thinking i should buy another disk ? Quote Link to comment
WeeboTech Posted March 23, 2015 Share Posted March 23, 2015 What version of unRAID? Can you post a syslog? Did you capture the smart logs? Issue those tests? I could be that disk3 is failing and it's only rearing it's ugly head now. Quote Link to comment
loady Posted March 23, 2015 Author Share Posted March 23, 2015 The problem is that I can't get the sys log as it's completely frozen..someone posted on a thread a command to type at the console that would continuously update the sys log to the flash so I could then plug it into windows and get it off...can't find that thread though. Quote Link to comment
loady Posted March 24, 2015 Author Share Posted March 24, 2015 not sure if you saw this edit i made a few posts back.. EDIT: i could hear a drive making a sound like its parking the heads, it is disk3, i was in the process of remving the files from disk2 which had been copied to disk3 when the server starting freezing...i have unplugged the drive and it has not locked up..problem i have now is that where i have replugged the drive it is wanting to perform a data rebuild...should i let that go ahead ? not sure what data it is rebuilding..would it be rebuilding the data i copied from disk2 to this disk3 or is it going from the parity and trying to rebuild the original data before i formatted it to XFS..thinking i should buy another disk ? im concerned with what the parity rebuild is trying to do ?..was parity following the transfer of files and the start of deleting them from the source disk2 ?..i dont understand what it is trying to rebuild. It is currently doing a check by md5sums (cd'ing into disk3 using disk2 checksums, the server has been off all night and not crashed this morning when i turned it on to perform the md5sum check. this will take a while so hopefully will complete for when i get up. (work nights, sleep in day) i am using v614b, i was seriously beggining to think it was the dockers causing this trouble, didnt have a single crash whilst dockers have been disabled while doing all this untill i started rysync removing source files...like you said..possibly its hitting the drive where the trouble is as might one of the dockers like sabnzb. EDIT: i have also initiated a tailing log..think it was the right command, im hoping that it will catch the point at which it crashes if it does. root@warptower:~# tail -f /var/log/syslog Mar 24 08:05:21 warptower avahi-daemon[2711]: Service group file /services/smb.service changed, reloading. Mar 24 08:05:21 warptower emhttp: shcmd (52): pidof rpc.mountd &> /dev/null Mar 24 08:05:21 warptower emhttp: shcmd (53): /etc/rc.d/rc.atalk status Mar 24 08:05:21 warptower rc.unRAID[3113][3114]: Processing /etc/rc.d/rc.unRAID.d/ start scripts. Mar 24 08:05:22 warptower avahi-daemon[2711]: Service "warptower" (/services/smb.service) successfully established. Mar 24 08:06:00 warptower kernel: mdcmd (42): nocheck Mar 24 08:06:00 warptower kernel: md: md_do_sync: got signal, exit... Mar 24 08:06:00 warptower kernel: md: recovery thread sync completion status: -4 Mar 24 08:06:26 warptower in.telnetd[3853]: connect from 192.168.1.5 (192.168.1.5) Mar 24 08:06:33 warptower login[3856]: ROOT LOGIN on '/dev/pts/0' from '192.168.1.5' Quote Link to comment
WeeboTech Posted March 24, 2015 Share Posted March 24, 2015 Without knowing the hardware, I might suspect the PSU. However if you cannot boot without any drives, that is a pretty disturbing situation. Also, you have a number of plugins and addons that should probably be removed. Mar 23 19:52:05 warptower logger: plugin: installing: /boot/config/plugins/dynamix.plg Mar 23 19:52:05 warptower logger: plugin: not installing older version Mar 23 19:52:05 warptower logger: plugin: installing: /boot/config/plugins/ntfs-3g-x86_64.plg Mar 23 19:52:05 warptower logger: plugin: running: 'anonymous' Mar 23 19:52:05 warptower logger: plugin: skipping: /boot/packages/ntfs-3g-2013.1.13-x86_64-1.txz already exists Mar 23 19:52:05 warptower logger: plugin: running: /boot/packages/ntfs-3g-2013.1.13-x86_64-1.txz Mar 23 19:52:05 warptower logger: Mar 23 19:52:06 warptower logger: plugin: skipping: /boot/config/plugins/ntfs-3g/ntfs-3g.png already exists Mar 23 19:52:06 warptower logger: plugin: creating: /usr/local/emhttp/plugins/ntfs-3g/ntfs-3g.png - copying LOCAL file /boot/config/plugins/ntfs-3g/ntfs-3g.png Mar 23 19:52:06 warptower logger: plugin: creating: /usr/local/emhttp/plugins/ntfs-3g/README.md - from INLINE content Mar 23 19:52:06 warptower logger: plugin: installed Mar 23 19:52:06 warptower logger: plugin: installing: /boot/config/plugins/powerdown-x86_64.plg Mar 23 19:52:06 warptower logger: plugin: skipping: /boot/config/plugins/powerdown/powerdown-2015.02.22.tgz already exists Mar 23 19:52:06 warptower logger: plugin: skipping: /boot/packages/powerdown-2.13-noarch-unRAID.tgz already exists Mar 23 19:52:06 warptower logger: plugin: running: /boot/packages/powerdown-2.13-noarch-unRAID.tgz Mar 23 19:52:06 warptower logger: plugin: running: 'anonymous' Mar 23 19:52:06 warptower logger: plugin: installed Mar 23 19:52:06 warptower logger: plugin: installing: /boot/config/plugins/snap-x86_64.plg Mar 23 19:52:06 warptower logger: plugin: skipping: /boot/packages/inotify-tools-3.14-x86_64-1.txz already exists Mar 23 19:52:06 warptower logger: plugin: running: /boot/packages/inotify-tools-3.14-x86_64-1.txz Quote Link to comment
loady Posted March 24, 2015 Author Share Posted March 24, 2015 I finally formatted the use stick.. still didn't boot..so I done it again this time replacing the 2 core files and it's booting again...maybe the constant hard shutting down corrupted it?...anyway..It wants to keep rebuilding disk3 so I have left it running. isn't dynamic no built into v6 ?...It looks like I possibly have a separate instance of it ?...I will remove all the plug in should I include this to delete them from the flash drive ? Do you suggest i delete those packages from the flash drive ?...all the ones listed in your post, powerdown, ntfs3g, snap, dynamix, the .plg files and the .txz files from config and pllugins folder? Quote Link to comment
WeeboTech Posted March 24, 2015 Share Posted March 24, 2015 dynamix is part of unRAID 6 now, that's why it will not install an older version. I would suggest having as clean a system as possible when doing this. Quote Link to comment
loady Posted March 25, 2015 Author Share Posted March 25, 2015 this keeps crashing...i need the sys.log to be copied to the flash drive or even a data disk or the cache continuously so theres actually a syslog there waiting after i have to hard boot it..the prvious method just leaves it tailing in a telnet session but it crashes and and i cant dump it it or copy..i had a read of the wiki..will this do the job ? tail -f --lines=100 /var/log/syslog >/boot/syslogtail.txt im thinking that will create a continuosly updated syslog of the last 100 lines on the flash drive right up to the point it does a poo ? Quote Link to comment
WeeboTech Posted March 25, 2015 Share Posted March 25, 2015 that will work. This will capture current syslog and new data. The current is good for people to get a full picture after a reboot. mkdir -p /boot/log cp /var/log/syslog /boot/log/syslog tail -f /var/log/syslog > /boot/log/syslog.new OR You can do this so that you can see it on your screen as well. tail -f /var/log/syslog | tee /boot/log/syslog.new Quote Link to comment
loady Posted March 25, 2015 Author Share Posted March 25, 2015 ok..thats throwing an error out on last command... root@warptower:~# mkdir -p /boot/log root@warptower:~# cp /var/log/syslog /boot/log/syslog root@warptower:~# tail -f /var/log/syslog /boot/log/syslog.new ==> /var/log/syslog <== Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection Mar 25 16:08:42 warptower sshd[2104]: Accepted password for root from 192.168.1.5 port 50618 ssh2 tail: cannot open â/boot/log/syslog.newâ for reading: No such file or directory I left the data rebuild running, system has not crashed but the data rebuild is hanging at 19%, i kept refreshing but the writes to the disk remain the same exactly, again i am not sure whats being rebuilt to disk3 as i had implemented to move source files from disk2 and it crashed halfway, i know that disk 2 has files removed and disk3 looked to be as full as disk2 was but because of the crashing i cant seem to to check..im going to pick up another new 3tb hdd and stick that in place of disk3 and see what happens unless you can suggest otherwise..everytime i start the server disk3 is wanting to be rebuilt. Quote Link to comment
WeeboTech Posted March 25, 2015 Share Posted March 25, 2015 My mistake do tail -f /var/log/syslog > /boot/log/syslog.new You can also do this so that you can see it on your screen as well. tail -f /var/log/syslog | tee /boot/log/syslog.new Quote Link to comment
loady Posted March 25, 2015 Author Share Posted March 25, 2015 Well its been another 6 hours, the GUI is responsive but the data rebuild is still stuck at 19%. Dont like the look of the read/write stats for disk3, so far i have successfully copied disk3 to the new disk5, i had successfully copied disk2 to disk3 after reformatting to XFS, then during the removal of source files from disk2 is when all the trouble started to show up, the server crashed, i couldnt redo the cheksum check to see if files had copied from disk2 to disk3 although the disk looked as full as it should have been, i stopped the array, removed the disk and replugged and rebooted server, i selected the disk back into its correct slot and started the array and thats when it kept trying to rebuild disk3, though i dont know what its trying to rebuild from, i have bought a new 3tb drive and i am thinking of preclearing it and then inserting it as disk3 and see what happens, does that sound like a good idea ?..am i at risk of data loss or will the parity correct everything...its looking to me like the disk3 is causing the problem, though you might gleen more information from my secreen shot than i will. https://plus.google.com/photos/115425996052755724168/albums/6130263909186895425?authkey=CNDThNKIy7TnKw Quote Link to comment
WeeboTech Posted March 25, 2015 Share Posted March 25, 2015 During a rebuild there would be more writes then reads. That's normal. Did you look at the smart reports as I asked a few posts prior. You can't really determine the full extent of things without looking at the SMART report. >> i selected the disk back into its correct slot and started the array and thats when it kept trying to rebuild disk3, though i dont know what its trying to rebuild from. It's trying to rebuild disk 3 from parity. If you interrupt this process, I'm not sure you'll be able to recover all the data from disk 3. I would let it continue, If the writes are not progressing at all, then you'll need to look at the syslog and see what's being reported. Quote Link to comment
loady Posted March 25, 2015 Author Share Posted March 25, 2015 How do i go about getting a detailed smart report ?..on the gui they have always shown a thumbs up. I am reading the wiki, is it a short test I need to do initially ? I'm thinking to remove disk3, pre clear another new 3tb drive and then insert that and see if it rebuilds to it?... I certainly think a detailed smart check on disk2 and 3 might rule out disk fault ? I have those two syslogs and ill post them up tomorrow. Quote Link to comment
WeeboTech Posted March 26, 2015 Share Posted March 26, 2015 If you are running beta 14b you can get the smart attributes by going to Main, clicking on the Disk # page and going to the health section of screen. Click on the attributes screen, post those values. You can also run a disk self-test. The short test will complete within 2 minutes, the long test will take a number of hours. So you would need to turn off the spindown timers when doing the long test. However you cannot do the short or long test while this rebuild is in progress. You can grab the attributes, but do not issue the tests until the rebuild has completed. Quote Link to comment
loady Posted March 26, 2015 Author Share Posted March 26, 2015 Ok..Roger that. So is it a good idea to pre clear the new disk and insert it as the replacement of disk 3 and see if the rebuild completes ?..I won't do anything with the original disk 3 so I can put it back if needed. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.