correct commads to copy to XFS


Recommended Posts

  • Replies 83
  • Created
  • Last Reply

Top Posters In This Topic

Well thank you weebo ! im now happily going ahead with the next disk move, thank you ever so much for your support and guidance, i have learned a little bit about linux that i didnt know before, especially interpreting what i am seeing you type and my urge to know what some of those paremeters i typed meant, i see this rsync is very powerful with all its options,,you didnt spoon feed me in that im eager to know more and feel accomplished in this. This is exactly what i like about this community..answering the what might seem stupid questions. i tip my hat to you.

Link to comment

so..thinking ahead...whilst this is currently removing the files from the source disk which is going to take a few hours, could i open another screen session and start creating the md5sums for the next filelist on next disk to be done or would it slow down the process that it would even out in time the same ?

Link to comment

so..thinking ahead...whilst this is currently removing the files from the source disk which is going to take a few hours, could i open another screen session and start creating the md5sums for the next filelist on next disk to be done or would it slow down the process that it would even out in time the same ?

 

Go ahead, it will all work out and schedule itself accordingly.

 

Considering it's an N40L, my guess

it would even out in time the same ?

But then again, the rsync checksum algorithm may not be as CPU intense as the md5sum.

Also, you wouldn't have to babysit the machine for as long.

I.E. you could go out and play longer & harder! <grin>

Link to comment

dont think this finished removing files from source disk, so i re run that step again and it says sending incrmental filelist but it seems to be hanging...

 

root@warptower:~# rsync --remove-source-files -vv -rcPX /mnt/disk2/ /mnt/disk3/
sending incremental file list

 

been stuck like this for 40mins now, cant hear the drive chattering.

 

not sure how i should proceed, is it hanging because it half removed files and now the filelist doesnt match possibly ?

 

Link to comment

dont think this finished removing files from source disk, so i re run that step again and it says sending incrmental filelist but it seems to be hanging...

 

root@warptower:~# rsync --remove-source-files -vv -rcPX /mnt/disk2/ /mnt/disk3/
sending incremental file list

 

been stuck like this for 40mins now, cant hear the drive chattering.

 

not sure how i should proceed, is it hanging because it half removed files and now the filelist doesnt match possibly ?

 

Not sure if it'll help much but you can add additional -v's to your call of rsync to increase the verbosity. If you send it -vvv or -vvvv it might give you more information and help you determine what's causing the transfer to hang.

Link to comment

dont think this finished removing files from source disk, so i re run that step again and it says sending incrmental filelist but it seems to be hanging...

 

root@warptower:~# rsync --remove-source-files -vv -rcPX /mnt/disk2/ /mnt/disk3/
sending incremental file list

 

been stuck like this for 40mins now, cant hear the drive chattering.

 

not sure how i should proceed, is it hanging because it half removed files and now the filelist doesnt match possibly ?

 

Check the unraid interface to see if each of the drives is actually being read. This process can take a long time as well.

 

This is one of the reasons to make the md5sum file.

You can do the md5sum verification on the destination disk to insure all files are there and match with validity.

 

If the md5sum check passes on the destination disk, then you do not need to remove the files from the source disk.

You can format it.

 

However, if you are having issues with these disks, It would be wise to do a smart long test on each disk for peace of mind.

Depending on the size of each disk it will take hours.

 

you would do something like the following replaceing ? with the character of your drive.

You will need to turn off all spin down timer on the drives being tested or it will cause the smart test to fail.

 

mkdir -p /boot/log

smartctl -a /dev/sd? > /boot/log/sd?.start.smart

smartctl -t long /dev/sd?

 

a message will be printed with an estimated time of completion.after completion do smartctl -a /dev/sd? > /boot/log/sd?.end.smart

 

compare each file with

 

diff -u /boot/log/sd?.start.smart /boot/log/sd?.end.smart

 

and review the end file for pending, reallocated sectors or a failed smart error.

Link to comment

somethings very seriously wrong...the server is constantly locking now...wonder if i have stepped upon the problem which has come to this XFS conversion..not sure how i can leave it to do a long smart test when the the telnet session is locking and i have no access to the server...the powerdown does not work either

 

EDIT: i could hear a drive making a sound like its parking the heads, it is disk3, i was in the process of remving the files from disk2 which had been copied to disk3 when the server starting freezing...i have unplugged the drive and it has not locked up..problem i have now is that where i have replugged the drive it is wanting to perform a data rebuild...should i let that go ahead ? not sure what data it is rebuilding..would it be rebuilding the data i copied from disk2 to this disk3 or is it going from the parity and trying to rebuild the original data before i formatted it to XFS..thinking i should buy another disk ?

Link to comment

The problem is that I can't get the sys log as it's completely frozen..someone posted on a thread a command to type at the console that would continuously update the sys log to the flash so I could then plug it into windows and get it off...can't find that thread though.

Link to comment

not sure if you saw this edit i made a few posts back..

EDIT: i could hear a drive making a sound like its parking the heads, it is disk3, i was in the process of remving the files from disk2 which had been copied to disk3 when the server starting freezing...i have unplugged the drive and it has not locked up..problem i have now is that where i have replugged the drive it is wanting to perform a data rebuild...should i let that go ahead ? not sure what data it is rebuilding..would it be rebuilding the data i copied from disk2 to this disk3 or is it going from the parity and trying to rebuild the original data before i formatted it to XFS..thinking i should buy another disk ?

 

im concerned with what the parity rebuild is trying to do ?..was parity following the transfer of files and the start of deleting them from the source disk2 ?..i dont understand what it is trying to rebuild.

It is currently doing a check by md5sums (cd'ing into disk3 using disk2 checksums, the server has been off all night and not crashed this morning when i turned it on to perform the md5sum check. this will take a while so hopefully will complete for when i get up. (work nights, sleep in day)

 

i am using v614b, i was seriously beggining to think it was the dockers causing this trouble, didnt have a single crash whilst dockers have been disabled while doing all this untill i started rysync removing source files...like you said..possibly its hitting the drive where the trouble is as might one of the dockers like sabnzb.

 

EDIT: i have also initiated a tailing log..think it was the right command, im hoping that it will catch the point at which it crashes if it does.

root@warptower:~# tail -f /var/log/syslog
Mar 24 08:05:21 warptower avahi-daemon[2711]: Service group file /services/smb.service changed, reloading.
Mar 24 08:05:21 warptower emhttp: shcmd (52): pidof rpc.mountd &> /dev/null
Mar 24 08:05:21 warptower emhttp: shcmd (53): /etc/rc.d/rc.atalk status
Mar 24 08:05:21 warptower rc.unRAID[3113][3114]: Processing /etc/rc.d/rc.unRAID.d/ start scripts.
Mar 24 08:05:22 warptower avahi-daemon[2711]: Service "warptower" (/services/smb.service) successfully established.
Mar 24 08:06:00 warptower kernel: mdcmd (42): nocheck
Mar 24 08:06:00 warptower kernel: md: md_do_sync: got signal, exit...
Mar 24 08:06:00 warptower kernel: md: recovery thread sync completion status: -4
Mar 24 08:06:26 warptower in.telnetd[3853]: connect from 192.168.1.5 (192.168.1.5)
Mar 24 08:06:33 warptower login[3856]: ROOT LOGIN  on '/dev/pts/0' from '192.168.1.5'

Link to comment

Without knowing the hardware, I might suspect the PSU.

 

 

However if you cannot boot without any drives, that is a pretty disturbing situation.

Also, you have a number of plugins and addons that should probably be removed.

 

 


Mar 23 19:52:05 warptower logger: plugin: installing: /boot/config/plugins/dynamix.plg
Mar 23 19:52:05 warptower logger: plugin: not installing older version
Mar 23 19:52:05 warptower logger: plugin: installing: /boot/config/plugins/ntfs-3g-x86_64.plg
Mar 23 19:52:05 warptower logger: plugin: running: 'anonymous'
Mar 23 19:52:05 warptower logger: plugin: skipping: /boot/packages/ntfs-3g-2013.1.13-x86_64-1.txz already exists
Mar 23 19:52:05 warptower logger: plugin: running: /boot/packages/ntfs-3g-2013.1.13-x86_64-1.txz
Mar 23 19:52:05 warptower logger: 

Mar 23 19:52:06 warptower logger: plugin: skipping: /boot/config/plugins/ntfs-3g/ntfs-3g.png already exists
Mar 23 19:52:06 warptower logger: plugin: creating: /usr/local/emhttp/plugins/ntfs-3g/ntfs-3g.png - copying LOCAL file /boot/config/plugins/ntfs-3g/ntfs-3g.png
Mar 23 19:52:06 warptower logger: plugin: creating: /usr/local/emhttp/plugins/ntfs-3g/README.md - from INLINE content
Mar 23 19:52:06 warptower logger: plugin: installed
Mar 23 19:52:06 warptower logger: plugin: installing: /boot/config/plugins/powerdown-x86_64.plg
Mar 23 19:52:06 warptower logger: plugin: skipping: /boot/config/plugins/powerdown/powerdown-2015.02.22.tgz already exists
Mar 23 19:52:06 warptower logger: plugin: skipping: /boot/packages/powerdown-2.13-noarch-unRAID.tgz already exists
Mar 23 19:52:06 warptower logger: plugin: running: /boot/packages/powerdown-2.13-noarch-unRAID.tgz

Mar 23 19:52:06 warptower logger: plugin: running: 'anonymous'
Mar 23 19:52:06 warptower logger: plugin: installed
Mar 23 19:52:06 warptower logger: plugin: installing: /boot/config/plugins/snap-x86_64.plg
Mar 23 19:52:06 warptower logger: plugin: skipping: /boot/packages/inotify-tools-3.14-x86_64-1.txz already exists
Mar 23 19:52:06 warptower logger: plugin: running: /boot/packages/inotify-tools-3.14-x86_64-1.txz

Link to comment

I finally formatted the use stick.. still didn't boot..so I done it again this time replacing the 2 core files and it's booting again...maybe the constant hard shutting down corrupted it?...anyway..It wants to keep rebuilding disk3 so I have left it running.

 

isn't dynamic no built into v6 ?...It looks like I possibly have a separate instance of it ?...I will remove all the plug in should I include this to delete them from the flash drive ?

 

Do you suggest i delete those packages from the flash drive ?...all the ones listed in your post, powerdown, ntfs3g, snap, dynamix, the .plg files and the .txz files from config and pllugins folder?

Link to comment

this keeps crashing...i need the sys.log to be copied to the flash drive or even a data disk or the cache continuously so theres actually a syslog there waiting after i have to hard boot it..the prvious method just leaves it tailing in a telnet session but it crashes and and i cant dump it it or copy..i had a read of the wiki..will this do the job ?

 

tail -f --lines=100 /var/log/syslog >/boot/syslogtail.txt

 

im thinking that will create a continuosly updated syslog of the last 100 lines on the flash drive right up to the point it does a poo ?

Link to comment

that will work.

 

This will capture current syslog and new data.

The current is good for people to get a full picture after a reboot.

 

mkdir -p /boot/log

cp      /var/log/syslog /boot/log/syslog

tail -f /var/log/syslog > /boot/log/syslog.new

 

OR You can do this so that you can see it on your screen as well.

tail -f /var/log/syslog | tee /boot/log/syslog.new

Link to comment

ok..thats throwing an error out on last command...

 

root@warptower:~# mkdir -p /boot/log
root@warptower:~# cp      /var/log/syslog /boot/log/syslog
root@warptower:~# tail -f /var/log/syslog /boot/log/syslog.new
==> /var/log/syslog <==
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:03:59 warptower emhttp: read_line: client closed the connection
Mar 25 16:08:42 warptower sshd[2104]: Accepted password for root from 192.168.1.5 port 50618 ssh2
tail: cannot open â/boot/log/syslog.newâ for reading: No such file or directory

 

I left the data rebuild running, system has not crashed but the data rebuild is hanging at 19%, i kept refreshing but the writes to the disk remain the same exactly, again i am not sure whats being rebuilt to disk3 as i had implemented to move source files from disk2 and it crashed halfway, i know that disk 2 has files removed and disk3 looked to be as full as disk2 was but because of the crashing i cant seem to to check..im going to pick up another new 3tb hdd and stick that in place of disk3 and see what happens unless you can suggest otherwise..everytime i start the server disk3 is wanting to be rebuilt.

Link to comment

Well its been another 6 hours, the GUI is responsive but the data rebuild is still stuck at 19%. Dont like the look of the read/write stats for disk3, so far i have successfully copied disk3 to the new disk5, i had successfully copied disk2 to disk3 after reformatting to XFS, then during the removal of source files from disk2 is when all the trouble started to show up, the server crashed, i couldnt redo the cheksum check to see if files had copied from disk2 to disk3 although the disk looked as full as it should have been, i stopped the array, removed the disk and replugged and rebooted server, i selected the disk back into its correct slot and started the array and thats when it kept trying to rebuild disk3, though i dont know what its trying to rebuild from, i have bought a new 3tb drive and i am thinking of preclearing it and then inserting it as disk3 and see what happens, does that sound like a good idea ?..am i at risk of data loss or will the parity correct everything...its looking to me like the disk3 is causing the problem, though you might gleen more information from my secreen shot than i will.

 

https://plus.google.com/photos/115425996052755724168/albums/6130263909186895425?authkey=CNDThNKIy7TnKw

Link to comment

During a rebuild there would be more writes then reads. That's normal.

 

Did you look at the smart reports as I asked a few posts prior.

 

You can't really determine the full extent of things without looking at the SMART report.

 

>> i selected the disk back into its correct slot and started the array and thats when it kept trying to rebuild disk3, though i dont know what its trying to rebuild from.

 

It's trying to rebuild disk 3 from parity.

If you interrupt this process, I'm not sure you'll be able to recover all the data from disk 3.

 

I would let it continue, If the writes are not progressing at all, then you'll need to look at the syslog and see what's being reported.

Link to comment

How do i go about getting a detailed smart report ?..on the gui they have always shown a thumbs up.

 

I am reading the wiki, is it a short test I need to do initially ?

 

I'm thinking to remove disk3,  pre clear another new 3tb drive and then insert that and see if it rebuilds to it?... I certainly think a detailed smart check on disk2 and 3 might rule out disk fault ?

 

I have those two syslogs and ill post them up tomorrow.

 

 

Link to comment

If you are running beta 14b you can get the smart attributes by going to

Main,  clicking on the Disk # page and going to the health section of screen.

 

Click on the attributes screen, post those values.

 

You can also run a disk self-test.

The short test will complete within 2 minutes, the long test will take a number of hours.

So you would need to turn off the spindown timers when doing the long test.

 

However you cannot do the short or long test while this rebuild is in progress.

 

You can grab the attributes, but do not issue the tests until the rebuild has completed.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.