Over the next few months I plan on replacing all of my 1TB disks with 1.5TB disks.  At first - I was thinking by keeping the old disk until the rebuild was completed I was protected.  But - I realized there are reads and writes on the disks when the server boots up.  So potentially if I have a server crash I will be out of parity sync with the old drive.

 

Is there a way to remain protected so if something goes wrong during the 36hr window it takes for the server to rebuild the drive?  Is there a way I can "clone" a disk, and "flip a switch" to change the server from whatever disk I am replacing to the new disk?

Over the next few months I plan on replacing all of my 1TB disks with 1.5TB disks.  At first - I was thinking by keeping the old disk until the rebuild was completed I was protected.  But - I realized there are reads and writes on the disks when the server boots up.  So potentially if I have a server crash I will be out of parity sync with the old drive.

 

Is there a way to remain protected so if something goes wrong during the 36hr window it takes for the server to rebuild the drive?  Is there a way I can "clone" a disk, and "flip a switch" to change the server from whatever disk I am replacing to the new disk?

Nothing I can think of will eliminate the few bytes difference when mounting/un-mounting the file-systems.  Those bytes are time-stamps and status bytes in the reiserfs superblock.

 

I went through the whole scenario in my mind and can quickly say that the replacement 1.5TB disk will be very different than the original in its formatting and bitmaps (there are extra bitmaps in the 1.5TB file-system space that will not exist in the 1TB file-system....  You can format the new partition on the 1.5TB drive, create a 1.5TB reiserfs file-system on it, copy the data, and swap drives.  The parity would not even be close to correct.  This will not work at all.

 

The only way to do this is to expand the 1TB file-system in place, after you swapped drives, and let the parity drive track those changes as the file-system is expanded to 1.5TB, exactly as unRAID does it. 

 

The only possibility I can think of is to pre-clear the new 1.5TB disk  (this is critical to not affect parity later)

then, use "dd if=/dev/mdX of=/dev/sdZ1 bs=2048k" to copy an exact image of what is on the old disk to the new, but as a smaller 1TB file-system in the 1.5TB partition.

Then, stop the array, assign the new drive for the old, and then use the "trust-my-parity" process (as described in the wiki) to have it only do a check of the parity.  It will probably find a few differences in the superblock at the front of the new disk, but it will correct parity to be in sync within the first few seconds of it running.

 

Now, you will still only have a 1TB file-system on the 1.5TB partition...  The last step is to resize the file-system to the full size of the partition.

To to that resize, stop samba and anything else accessing the drive, un-mount the drive being upgraded, issue the appropriate command (resize_reiserfs /dev/mdX to expand the reiserfs file-system on the new disk attached to the "md" device.)  Then, re-mount /dev/mdX on /dev/diskX, and re-start samba and you should be there.

 

This is all entirely theoretical.  I've never tried it, and it might fail, but odds are pretty good it will work.  In any case, you will not be without parity protection, and you must make absolutely certain the new 1.5TB drive is not bigger than your existing 1.5TB parity drive, even by a few bytes.  and it would not hurt to run a full parity check when done.

 

This whole process will still take many hours, as copying from one drive to another will take that long... also, NO writes of new files to the drive being copied can occur during the copy to the new, otherwise, they might not be reflected in what was copied.

 

You are entirely on your own if you elect to try this... be absolutely certain what disks you are operating on, one mistake and you will overwrite your data.  Expect to see some parity sync errors when you assign the old drive for the new.  They are housekeeping bytes, for mount timestamps and status.  They will be different between the two disks being swapped.  I can't think of any way around those differences.

 

Needless to say, Lime-Technology has not approved of this, nor do they even know I theorized it could be done.    You will get no support from them if you lose your data...  (we'll all shed a collective tear, but we'll not be able to do a lot either if it blows up. )

 

Joe L.

Lol - thanks Joe.  Currently lime is only a backup.  At the point we upgrade our production servers, it will serve as the primary storage for old jobs.  I believe I will complete my 1.5TB upgrades prior to making it my only storage for these jobs. 

There was a thread that I started a month or two ago that discussed alternate drive replacement strategies to maintain parity protection. I only have BB access at the moment, but will try and post the link tonight.

I found your one on removing a drive and remaining protected - but it didn't seem to include replacing the drive.  I suppose if there was some type of drive clone - you could do this, clone the drive while the tower was off - then replace it.

I found your one on removing a drive and remaining protected - but it didn't seem to include replacing the drive.  I suppose if there was some type of drive clone - you could do this, clone the drive while the tower was off - then replace it.

True, but that did not involve upsizing a drive and keeping the data intact , all while keeping parity protection.  If you clone a drive while it is off-line, it is still the same size file-system as previous.

 

The procedure I outlined earlier was in fact a drive clone while "offline" (Or, at least while you were not writing to it)  It has just never been tried by anyone.

 

Joe L.

The procedure I outlined earlier was in fact a drive clone while "offline" (Or, at least while you were not writing to it)  It has just never been tried by anyone.

 

I guess over time I've just become shy of trying new stuff that "should" work 

 

and it might fail, but odds are pretty good it will work...we'll all shed a collective tear

I suppose worst case scenario, I lose my new drive and old drive and parity. 

 

I may go ahead and try your way & see what happens...unless I find somebody who has successfully done it a different way.

 

I suppose worst case scenario, I lose my new drive and old drive and parity. 

If you "write" to the wrong drive when you clone the disk being up-sized you could clobber your own data.  Other than that, you would still have the old drive that was un-assigned, with your old data, so it is pretty hard to clobber it too.

 

Glad you understand, but what you are looking for is a feature sometimes known as read-only snapshots.  Even then, I doubt many RAID arrays will let you up-size a disk and keep parity protection and keep the array running...

 

The risks are not really high... but you at least understand there is no built-in procedure other than going without parity protection for the time it takes to reconstruct the old data onto a new drive.

 

To be best prepared in case of an error.. I'd do this

Stop the array.

Make a copy of the entire config directory.  (before un-assigning old disk/assigning new)

Power down, swap drives, installing new, putting old aside for now. Power up.

Then, un-assign old, assign new on devices page (you may not need to if using the same disk controller slot.  unRAID will recognize you are up-sizing)

Start reconstruction onto new drive by pressing "Start" after check the checkbox below it.  Whatever you do, don't press the button labeled "restore"

 

If all goes as planned, you'll have the new disk reconstructed the next morning.

 

If something were to happen.

If a different data drive fails...   Stop array, copy BACK the old config, power down, put back in the old smaller data drive, take out the one you were upsizing. Power up.  You will now have a valid config, but with a different "failed" drive.  Proceed as with any drive failure.

This probably will not work as well as I originally thought, since in re-sizing the drive being up-sized, parity would have been written.  Can't undo those just by copying back the config folder and putting back the original disk.

If the parity disk were to fail...  Stop the array, copy back the old config folder, power down, put the old data drive back in its original slot, removing the newer 1.5TB drive, then power up.  It will probably still detect the parity drive failure... proceed as with any other drive failure.

 

If the new 1.5TB drive fails, just stop the reconstruction process, stop the array, power down, replace the 1.5TB with a new 1.5TB and proceed.  You are now back to the beginning with all drives working.

OR

Stop the array, copy back the old config folder, power down, put the old data drive back in its original slot, removing the newer 1.5TB drive, then power up.  thinking more, this second alternate if the new drive fails is probably not workable, since the parity drive saw the re-size operation. 

 

In almost every case some cases, you were "without" parity protection, but in actuality you could get back to a parity protected state simply by copying back the original config folder and putting the original drive being up-sized into place.

 

There is a lot to be said for just letting unRAID do what it was designed to do... might be easiest...  The secret is to always make a copy of your config folder when in a stopped state, before any change in the disk configuration. 

 

 

This makes sense...this is what I am going to do.  In theory I shouldn't lose anything unless two drives die at the same time.  I didn't realize something as simple as making a backup of the config dir would essentially keep the old parity.  Thank you - really appreciate it. 

 

P.S.  I don't even know exactly what restore does - but I keep seeing people saying not to do it - so I can promise I won't push that button unless I know exactly what it's going to do.

This makes sense...this is what I am going to do.  In theory I shouldn't lose anything unless two drives die at the same time.  I didn't realize something as simple as making a backup of the config dir would essentially keep the old parity.  Thank you - really appreciate it. 

 

P.S.  I don't even know exactly what restore does - but I keep seeing people saying not to do it - so I can promise I won't push that button unless I know exactly what it's going to do.

the config folder DOES NOT keep the old parity.  It keeps the old status of your array and the assigned disks serial numbers and good/bad status.  A copy can be used to fool the array into thinking you did not switch disks (and then switch it back) Basically, it is the list of the disk serial numbers and disk state (DISK_OK/DISK_BAD/DISK_NEW).

 

In fact, the first scenario I described can't work, as the parity drive will have already been modified as the file-system size was up-sized.  So... I'll need to update my prior post.  You are still somewhat at risk if a data disk dies.  You can probably recover and then use reiserfsck to fix the bitmaps on the recovered drive, but let's say it might be messy. 

 

You could, of course, clone the parity drive, and restore it, but it starts getting to where the whole array should be cloned to have any real comfort.

 

The restore button actually resets the array to a point before ANY parity calcs were made.  In effect, it erases any knowledge of prior parity calcs.  It renames the existing config/super.dat file (effectively deleting it) and forces the superblock to be rebuilt from the currently assigned, working drives.

If pressed when a failed drive is present, you could lose the data on the failed drive.  (there are very specific exceptions, but a special procedure is needed for those exceptions to have it not invalidate parity)

 

Joe L.

Ok - so it keeps the disk "configuration"...but - say I lose my parity drive during a rebuild.  I *should* be able to shut everything down, restore the config directory, put the old hard drive back in, (so I have all 14 original HD's) and put in a NEW parity drive and the server will rebuild the parity.  Likewise, if I lose disk 4 while rebuilding disk 5, I just replace disk 5 with the old disk, replace the config directory, and it will begin rebuilding disk 4.  So I will have to have TWO disks die in the coarse of the rebuild to lose data.  Am I right here?

 

Bummer - just re-read your post...

 

Ok - after I get all these upgraded and this becomes the only copy of all my data...if a drive were to die and I replaced that drive with one of equal size, would your idea work?

Ok - after I get all these upgraded and this becomes the only copy of all my data...if a drive were to die and I replaced that drive with one of equal size, would your idea work?

If not up-sizing at the same time, then yes.  It would work.

I'm a little brain-dead right now, but if he is going to have all the upgrade disks on-hand for the process, couldn't you use one of the other empty drives to keep a backup of the drive being upgraded and combine that with the procedure outlined here?  Or dd-copy your smaller parity drive, upgrade the parity drive using unraid (having the backup incase), then dd-copy one data drive to a spare drive and then upgrade in unraid as usual and repeat for each drive.  I've obviously left out the details of array stopping/starting etc.

The way that I was thinking goes something like this (if needed, post back and I can give more detailed instructions).  This procedure requires that you have one available (spare) port to use.  If you are out of ports, you could temporarily remove the cache disk assignment and use that port to perform this procedure, then reassign cache after you are done:

 

1.  Run preclear script on new disk

2.  Parity check

3.  Install new disk on NEW port

4.  Add disk to the array (now you have both the old and new disks in the array)

5.  Copy data from old (disk to replace), to new disk (parity is maintained)

6.  Zero out the old disk (use "dd" command) (makes it "invisible" from a parity perspective)

7.  Stop the array, remove old disk from array from devices page

8.  Do the "trust my parity" procedure, excluding the old disk from the array

 

This keeps parity protection throughout, but destroys the contents of the old disk in the process.  (Good if you want to sell it).

 

The other methold is not as strong from a parity perspective, but honestly this is what I do ...

 

1.  Run preclear script on new disk

2.  Parity check

3.  Exchange the disks and do the rebuild as per normal

4.  Run a READ ONLY parity check

5.  At this point, with no errors, you are fully protected again

 

If you were to have a problem and want to get back to where you were .

 

6.  Put old disk back in machine (if you had removed it) and reconnect it to a port (if you had disconnected it)

7.  Use the "trust my parity" procedure to update the array configuration (assign old disk, remove new disk)

8.  Run a parity check.

 

The parity check may find some differences early on in the drive due to size differences.  But the vast majority should be clean.  The preclear script will have zeroed out the entire new disk so that parity calcs above the size of the old disk should be unaffected.

 

Make sure that your drive temps are reasonable - in a few recent posts, users that opened their cases also lost some of their cooling sending temps through the roof.  Make sure your temps stay in the comfort zone throughout any rebuild operation.

 

 

6.  Zero out the old disk (use "dd" command) (makes it "invisible" from a parity perspective)

7.  Stop the array, remove old disk from array from devices page

 

This is the process I suggested the unRAID have built in, to wipe and remove disk from array while retaining parity protection.