Quickest way to copy a drive's contents?


HDRW

Recommended Posts

I'm changing one data drive for another, and changing the file system from ReiserFS to XFS.  I have cleared (15+ hours!) and formatted the new drive, so they're both available on the array.

 

What is the quickest way to copy the entire contents of one drive to the other?  I can obviously do it over the network from a PC, using something like XXCOPY (sic) under Windows, but that seems a bit wasteful when they are both on the same machine.

 

Is there a way to get UnRAID to copy everything over?  (I'm not a Linux person so if it involves using a command line, I'll need detailed instructions, please!  :-)

 

Cheers,

 

Howard

 

(UnRAID 6.2.1 on an HP Microserver)

 

Link to comment

OK, I've followed the instructions up to item 14, with the array stopped and the two drives swapped over (and their formats swapped as well), the disks I've swapped are shown as "Wrong", the Array Operation area shows "Too many wrong and/or missing disks!" and the Start button is greyed-out, so I'm a bit stuck as this doesn't accord with instruction 14.

 

How do I get it going again?

 

Cheers,

 

Howard

 

Link to comment

Not sure what you might have done differently.    I've not used Rob's instructions -- the drives I converted I did entirely "outside" of UnRAID ... a VERY safe process, although it certainly takes longer.

 

If you have "too many wrong or missing disks" that means you've somehow managed to get in a situation where your parity drive is no longer valid.  Hopefully you didn't do anything to mess up your data !!

 

Outline EXACTLY what steps you took; and perhaps post a picture of the Web GUI display as it looks now, and perhaps there will be a clue as to what state you're currently in.    Do NOT do anything else vis-a-vis drive assignments and/or starting the array, as if you do something wrong you could definitely lose some data.

 

Link to comment

Changing disk slots only works with v6.1, at the moment it's not possible with v6.2 with single parity,  it will never be possible with dual parity.

 

Ah ha => that's what happened.    I knew position mattered with dual parity;  wasn't aware it was also an issue with single parity when using 6.2.    That completely invalidates the instructions in Rob's post at that point !!

 

Link to comment

Note that you can basically do as Rob outlined ... i.e. copy all the data from one drive to another;  then change the format of the drive you copied to XFS;  and then copy data from another drive to that one;  etc.

 

... HOWEVER => you have to be CERTAIN that you exclude the target drive from your Global share settings at each step of the way [Rob overcame this by always reassigning the new "target" as the drive # that was already excluded].

 

Note that this is very important, as otherwise you could encounter the "user share copy bug" I've referenced several times.

 

FWIW, the way I did conversions on the disks I chose to convert was as follows:

(a)  Copied the entire contents of the disk to convert to an external location [a disk on my main workstation PC]

(b)  Reformatted the disk I just copied the data from to XFS

©  Copied the data back from the external location to the same disk it came from (which is now XFS)

 

That completely avoids any possibility of data loss due to the user share copy bug and also ensure I have a good copy of the data at all times, since it always exists in at least 2 places until the process is done.  It does, of course, take longer.  [Even longer than it looks like from my description, because I do validations of both of the copies.]

 

 

 

Link to comment

Changing disk slots only works with v6.1, at the moment it's not possible with v6.2 with single parity,  it will never be possible with dual parity.

Arrrggh!  So why did you point me to those instructions?  I updated to 6.2 because 6.1 wouldn't do a Clear with the array started, and now I'm stuck!

 

@garycase:  I haven't found any reference to the Global User Share Bug that you've mentioned, at least on page 26 of that thread, but I have both Disk and User shares turned Off in Global Share Settings, so hopefully that's OK.

Steps I followed:

1. Changed the default format to XFS

2. Stopped the array and powered down, plugged-in the new drive.  Restarted.

3. Per my thread:

https://lime-technology.com/forum/index.php?topic=52948.msg508922#msg508922

I updated to 6.2.1 as 6.1 wouldn't clear the new drive (it wouldn't do anything to it).

4. After the update the new disk was assigned as Disk 4, and Cleared (about 15 hours).

5. Formatted the new disk (as XFS).

6. Following RobJ's item 8:  Accessed the server via SSH and ran:

rsync -avPX  /mnt/disk1/ /mnt/disk4/

which ran for many hours (at least 15).  I'd decided not to remove the source files - that seems like sawing through the branch you're sitting on...

7. Following RobJ's item 9, ran:

rsync -rcvPX  /mnt/disk1/ /mnt/disk4

which ran without copying anything, so I think this proves a good copy.

8. Following RobJ's 10 to 13, swapped over the drives between Disk 1 and Disk 4, and changed their formats to match what they are (Disk 4 now ReiserFS, Disk 1 now XFS).  Both now showing "Wrong" (see attached capture of the Main page at that time - I've sanitised the IP addresses).

9. Now can't Start the array due to "Too many wrong and/or missing disks!" - can't do anything on Main except reboot or power it down.

 

How do I get my disks back?  As far as I know I have all the data still there (two copies of the copied data) so where do I go from here? 

Should I go for a "New Config"?

If so, what should I set for "Retain Configuration"?  (I suspect "Parity slots", and then add in the drives in their new slots, but I'd like confirmation as this seems terribly dangerous!)

 

Finally:  Are the disk "slots" determined by physical location, or Disk ID?  I haven't physically swapped Disk 4 into the Disk 1 slot, but I will do - should I do it before any further recovery, or is physical location irrelevant?

 

Cheers,

Howard

HDRW.zip

Link to comment

Changing disk slots only works with v6.1, at the moment it's not possible with v6.2 with single parity,  it will never be possible with dual parity.

Arrrggh!  So why did you point me to those instructions?  I updated to 6.2 because 6.1 wouldn't do a Clear with the array started, and now I'm stuck!

 

Forgot about that, but you don't to change disk slots to convert, most people don't.

Link to comment

...

Arrrggh!  So why did you point me to those instructions?  I updated to 6.2 because 6.1 wouldn't do a Clear with the array started, and now I'm stuck!

 

Just to be clear, I did NOT point you to those instructions -- Johnnie did  :)

In fact, I gave a strong warning about being CERTAIN you didn't do anything wrong in the process, as it could result in significant data loss (quite a few people have been "bitten" by that issue when doing file conversions and/or moving data around on their servers).    I actually anticipated you might ask about that and was going to provide more detail on safer (but slower) approaches if you did.

 

 

...

Steps I followed:

1. Changed the default format to XFS

2. Stopped the array and powered down, plugged-in the new drive.  Restarted.

3. Per my thread:

https://lime-technology.com/forum/index.php?topic=52948.msg508922#msg508922

I updated to 6.2.1 as 6.1 wouldn't clear the new drive (it wouldn't do anything to it).

4. After the update the new disk was assigned as Disk 4, and Cleared (about 15 hours).

5. Formatted the new disk (as XFS).

6. Following RobJ's item 8:  Accessed the server via SSH and ran:

rsync -avPX  /mnt/disk1/ /mnt/disk4/

which ran for many hours (at least 15).  I'd decided not to remove the source files - that seems like sawing through the branch you're sitting on...

7. Following RobJ's item 9, ran:

rsync -rcvPX  /mnt/disk1/ /mnt/disk4

which ran without copying anything, so I think this proves a good copy.

8. Following RobJ's 10 to 13, swapped over the drives between Disk 1 and Disk 4, and changed their formats to match what they are (Disk 4 now ReiserFS, Disk 1 now XFS).  Both now showing "Wrong" (see attached capture of the Main page at that time - I've sanitised the IP addresses).

9. Now can't Start the array due to "Too many wrong and/or missing disks!" - can't do anything on Main except reboot or power it down.

 

How do I get my disks back?  As far as I know I have all the data still there (two copies of the copied data) so where do I go from here? 

Should I go for a "New Config"?

If so, what should I set for "Retain Configuration"?  (I suspect "Parity slots", and then add in the drives in their new slots, but I'd like confirmation as this seems terribly dangerous!)

 

Finally:  Are the disk "slots" determined by physical location, or Disk ID?  I haven't physically swapped Disk 4 into the Disk 1 slot, but I will do - should I do it before any further recovery, or is physical location irrelevant?

 

Cheers,

Howard

 

You were fine through Step #8 above.  The problem there is, as Johnnie noted, v6 is sensitive to the assigned disk positions => I knew that was true with dual parity, but didn't realize that it had that restriction even with single parity.

 

So ... had you not switched the disks you'd have been fine as long as you avoided the user share copy issue.  What you SHOULD have done -- knowing that the disk positions couldn't be switched -- was simply reformatted disk #1 to XFS after you had copied all the data to disk #4, which would have erased everything from disk #1 and it would then have been ready for the next copy.    Then you could have repeated the process by copying everything from another disk (e.g. #2) to disk #1; and then reformatted that disk ... and then copied another disk to that one; etc.

 

So ... in your current state, there are two ways you can proceed:

 

(1)  You could reassign disks 1 and 4 to their original positions and then look at what the array shows for their file types.  Do NOT Start the array if it shows the wrong file types and indicates it's going to format the disk(s) ... this will erase everything on the disk you copied the data to !!    If all looks good, you can proceed by simply doing what I just outlined above.

 

(2)  Alternatively, you can do a New Config and assign your 4 data drives and the parity drive (Be CERTAIN you assign the correct drive as parity) AND you can check the "parity is already valid" box before you Start the array.    Parity SHOULD in fact be good, based on what you've done to this point -- but just to be certain I'd run a parity check (be sure it's a standard correcting check) immediately after starting the array and before doing anything else.    If any errors are found and corrected, repeat the check to be sure it's error-free before doing anything else with the array.

 

(3)  If you have doubts about parity being good; just do a New Config, but don't check the "already valid" box => this will then put the array "at risk" while you regenerate parity; but that's a relatively minor risk.

 

 

Link to comment

... thinking about it, I'd do either #2 or #3 => #1 has a small risk of wiping out all of the data on both disks; depending on just how you have the file types set for those disks.    As long as you do NOT allow it to format either of them, it'd be okay -- but Murphy has a way of striking at the most inopportune times !!  :)

Link to comment

(I know you didn't point me to those instructions, which is why my comment followed the quote from Johnnie's message  :) )

 

Anyway, thanks - I've gone with number 3, a New Config without known-good Parity, placed the drives as I want them (new drive in, old drive out), and restarted, and it's now rebuilding Parity (another 15 hours!).

 

I wasn't actually doing a cascade of changes of format, the primary objective was to replace a slow (5400rpm) drive with a fast (7200) one, and changing to XFS on the new one was - I thought - a "free" bonus!  I realise I could have just done the "failed drive replacement" procedure, but that would have continued with ReiserFS on the new disk, which seemed to be a wasted opportunity.

 

I still don't understand why features come and go (especially go!) between versions - the absence of clearing while the array is started in 6.1 prompted me to update to 6.2, unaware of the problem with swapping slots.  It seems to me that RobJ's instructions can't work as laid out, since it seems to rely on features that don't all exist in any one UnRAID version!  I may post a warning on that thread.

 

Thinking about it, the disk slots don't all have to be occupied, so I could have just unassigned Disk 1 after copying over, but then all my machines would have to have their access changed from Disk1 to Disk4, and having the physical slots not match the numbers in the software just feels wrong.

 

Anyway, I think I'm now sorted out - I'm not going to touch the thing again for a good long time now!

 

Thanks for all your help - I could probably have perpetrated a disaster without it...

 

Cheers,

 

Howard

 

Link to comment

#3 was certainly the "safest" choice.

 

I was surprised to learn that the positional dependency in 6.2 doesn't depend on whether or not you're using dual parity.  I knew the positions had to be maintained with dual parity systems (because of the way the 2nd parity calculation is done); but with single parity the parity calculation doesn't depend on the drive positions => but the logic in 6.2 for this apparently doesn't factor that in.

 

Link to comment

#3 was certainly the "safest" choice.

 

I was surprised to learn that the positional dependency in 6.2 doesn't depend on whether or not you're using dual parity.  I knew the positions had to be maintained with dual parity systems (because of the way the 2nd parity calculation is done); but with single parity the parity calculation doesn't depend on the drive positions => but the logic in 6.2 for this apparently doesn't factor that in.

Tom is supposed to fix that on an upcoming release.

Link to comment
  • 1 month later...

OK, back again...

 

Anyway, I think I'm now sorted out - I'm not going to touch the thing again for a good long time now!

 

Well, for "good long time", read "about seven weeks"!  :-)

 

I recently got a (rare) bargain on eBay - a Used 2TB WD Re "Enterprise" drive for a good price, and it turned out to be only a few months old, and SMART reports it had only 39 power-on hours!  (Wish they'd had more than one...).

 

I want to swap it in, replacing a drive that's showing some problems (3 reallocated sectors, 419 errors on the main page) and formatting it as XFS (Reiser is current on the existing drive).  Due to changing the format I don't want to follow the "replace a failed drive" procedure.

 

I connected it up, and ran the "Pre-Clear" add-on which took 9 hours.  It's now shown in "Unassigned" as Precleared, and it has "Format" to the left of the Temp column, but it doesn't have a button surround, and clicking it causes a green surround to appear while the button is held down, but nothing happens.

 

Is this a bug in 6.2.4?  How do I format it? 

 

I'd rather not add it to the array at this point in case it triggers a Parity Check which isn't necessary at this point, and puts unnecessary wear on the Parity disk - judging from my earlier experiences, it will be necessary to do a full Parity Check/generation when I swap the drive in anyway.

 

Cheers,

 

Howard

 

Link to comment

Is this a bug in 6.2.4?  How do I format it? 

 

That's the unassigned devices plugin, unRAID doesn't let you format unassigned disks, you can format it by enabling destructive mode on the UD plugin settings but note that unRAID won't recognize it if you ever add it to the array, i.e., you'll need to format it again.

Link to comment

I want to swap it in, replacing a drive that's showing some problems (3 reallocated sectors, 419 errors on the main page) and formatting it as XFS (Reiser is current on the existing drive).  Due to changing the format I don't want to follow the "replace a failed drive" procedure.

So, to be clear, you have all the data from the current ReiserFS drive backed up elsewhere, and you want to put the new drive into that slot with a blank XFS format and copy the data back?

 

Have you read through this thread? http://lime-technology.com/forum/index.php?topic=37490.0

Link to comment

I want to swap it in, replacing a drive that's showing some problems (3 reallocated sectors, 419 errors on the main page) and formatting it as XFS (Reiser is current on the existing drive).  Due to changing the format I don't want to follow the "replace a failed drive" procedure.

So, to be clear, you have all the data from the current ReiserFS drive backed up elsewhere, and you want to put the new drive into that slot with a blank XFS format and copy the data back?

Well this array *is* the backup for others, so I do have the originals, but I didn't want to copy over the network, but between disks on the UnRAID - it still took several hours using rsync.

Yes, but I found last time I wanted to do this (replace a disk and change the format) that the instructions in that thread don't work with the current (6.2.4) version of UnRAID.

 

I've found that the Wiki doesn't cover what I want to do (twice so far, one more to do at some point!) - to pre-clear a new disk, then format it using XFS, then to copy over the data from the Reiser-formatted disk, then swap the latter out for the former.  Seems that it is pretty tricky without it trying to recreate the Parity at least once, and if you're not careful, a couple of times.  There was a period this time when I was running without valid Parity, but probably for less time than it takes to recreate it.

 

It's done this time, but it would be nice to know if there's a better way next time.

 

Cheers,

Howard

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.