Hey guys,

 

currently it's possible to add new disks to the array without affecting parity. That means I can add disks like crazy and still all data is always protected. However, as far as I understand (someone correct me if I'm wrong, please) I cannot remove disks without having to rebuild parity. That means when removing a disk there's a time period where all disks are unprotected. If a disk fails while parity is just being rebuilt the data on the failing disk is lost. Right?

 

Now what I want is a feature to safely remove a disk from the array. It could be realized by setting all sectors to zero and updating the parity accordingly. Afterwards when all sectors are zero, it should be possible to remove the disk without affecting parity. Doing it this way would mean that we could remove disks without ever losing parity protection.

 

Why is this useful? I mean who wants to *remove* disks? Well, it's useful not only for removing disks but also for replacing a smaller disk with a bigger one. We could first add the bigger disk (as usual that works without affecting parity), then we could copy the data from the old smaller disk to the new bigger disk. Then we could zero the smaller disk out and finally remove it - again without affecting parity. The whole process would work without ever losing parity protection.

 

What do you guys think?

What is the difference between "update parity" and "rebuild parity"?  They sound the same to me, thus you haven't changed anything.

 

 

Bill

What is the difference between "update parity" and "rebuild parity"?  They sound the same to me, thus you haven't changed anything.

Maybe I explained it bad. Let me try again:

 

If you want to replace a small disk with a bigger disk with unRAID 4.2 you have to totally rebuild the parity. But what happens if one of the other data disks fails during the parity rebuilding process? The content of that disk will be lost in this situation. Or am I wrong?

 

Now my wish is that there should be a new button (or something similar) with which I can "unformat" a specific disk. This unformat would clear the disk in a similar way as you write data to the disk. That means, the parity stays valid although the drive is cleared. When the clearing is done, you can shutdown the server and remove the drive and the parity is still valid without having to be rebuilt. This way parity protection would never be lost. Technically this should be easily possible.

I think your premise is wrong.  Someone else can chime in to confirm but if you just take out the old smaller drive and insert the new larger one, it will rebuild the REPLACED drive, not the PARITY drive.  Thus there is no parity rebuild so no increased risk.

 

http://www.lime-technology.com/wordpress/?page_id=47#toc-replace-a-single-disk-with-a-bigger-one

 

I actually was going a different direction with my first response, so ignore that one.

 

 

Bill

Hmmmmm... Thanks, Bill. I seem to have missed this part of the manual. I think I checked the wiki and this solution was not described there. Anyway, I'm happy to see that replacing is smaller disk with a bigger one is well covered by unRAID.

 

So my suggestion would really only improve the situation if one wants to remove one disk from the array. I am at this situation now cause I'm replacing a bunch of 500GB disks with 1TB disks and I don't really need to double the array capacity right now. So I'm reducing the number of disks in my array. But currently I don't see how I could do that without losing parity protection for a short period of time.

Technically, every time you are "down" a disk, your array is exposed.  Being "down" being defined as, your parity disk does not reflect the current configuration.

 

However, the chances of a second disk failure during the relatively short time an array rebuild takes to complete is fairly low, and not something to lose sleep over.

 

In the field, the most common cause of RAID failures is having a drive go down, and then discovering during the rebuild that one of the remaining drives in the array has a bad block... since unRAID does not do proactive data scrubbing (which would catch this pre-failure), this is actually quite an exposure.  (To unRAID's credit, it is also far cheaper than the alternatives...)

 

unRAID does have the advantage that even with a rebuild failure, all you lose is the data that was on the drives that failed, not the whole array since each drive in the array (except for the parity drive) is a complete filesystem that can be read on any Linux box.

 

SirWired

In the field, the most common cause of RAID failures is having a drive go down, and then discovering during the rebuild that one of the remaining drives in the array has a bad block... since unRAID does not do proactive data scrubbing (which would catch this pre-failure), this is actually quite an exposure.

Just out of interest: What happens with unRAID in this situation? I mean it would still be possible to recover 99% of the data (except this one bad block), but would unRAID allow that? Or would it completely refuse to make any use of the parity and abort operation at the bad block?

When you have to rebuild a drive (be it parity or a data drive), unRAID is going to read EVERY SINGLE sector on EVERY SINGLE disk and has no way to recover from a read error.  There is no time when the risk of finding a disk error is higher.  If you haven't done a parity check before replacing a disk, the chances of the finding an error while doing a rebuild is significant.

 

I like the idea of a button to remove a drive from an array.  All it would have to do is "format' a disk, just like when adding a disk to an array (setting all the sectors to binary zero), and then do a simple configuration update to remove the disk while keeping the configuration valid.  This capability would also allow you to grow a disk - here's how.  You could copy everything off the disk you want to replace, do this "remove drive" thing, add the new drive, and then copy everything back.  A better option, if you had an extra port, would be to add the new drive, copy everything to it, and then remove the old drive.  This one simple feature would enable you to do all normal maintenance tasks without losing redundancy.  The only time the redundancy would be lost is when recovering from a real failure.

http://lime-technology.com/forum/index.php?topic=991.msg9854#msg9854

 

In the thread linked above, I answered a question about removing a drive from an unRAID array, and added some thoughts about the process, and about the ideas in this topic.  I like new ideas, but I find this one somewhat problematic, for now, and requires further thought.  I may very well have mis-analyzed it though.

 

Some may not know that parity info is not related to existing files, but to the physical bits on every physical disk.  That includes the file system structures, and all written disk sectors, whether they contain current file data or past deleted data.  So even if the drive were to appear completely empty, all drive space free, it still may be full of non-zero bits across the entire disk surface, that have to be removed from the parity info.  Since unRAID uses an even parity scheme, zero bits do not affect parity, but one bits do, and require a bit flip on the parity drive.

 

RobJ is correct...

To remove a drive as you described, the unRaid software would have to completely zero a drive, not just delete the files.

 

Lets say we invent a new choice for the "Devices" page.  It is a choice labeled "erase-and-remove-drive"

 

When you choose this, you will be prompted in some way "are you sure... ALL data on the drive will be erased with no chance of un-erase"

 

Then, when you start the array, instead of mounting the disk on /mnt/diskX it will instead do the following:

  write zeros to every byte on the disk being erased.  This will also update the parity drive as it goes, so parity is correct as the disk is erased.  This writing of zeros could take several hours or more on a large drive... 

 

Then, once zeroed, the drive serial number can be deleted from the super.dat file.  At that point it is done.  the drive will show as un-assigned from that point on.

 

The point RobJ made is that until this process of writing zeros and re-building the super.dat file is complete, you are still vulnerable to another disk failing.  As he said, it will probably take noticeably longer than simply using the existing "Restore" button, since it involves reading parity in addition to all the other disks.

 

So... copy the defective drive's contents elsewhere, stop the array, un-assign it, check the checkbox under the "Restore" button and press "Restore"  Several hours later you will be protected again.  RMA the defective drive... when its replacement arrives, install it as a new drive in the currently empty slot in the array and re-assign it.

 

Joe L.

 

 

To remove a drive as you described, the unRaid software would have to completely zero a drive, not just delete the files.

Sure, that's what I suggested a new button on the web interface.

 

Then, when you start the array, instead of mounting the disk on /mnt/diskX it will instead do the following:

  write zeros to every byte on the disk being erased.  This will also update the parity drive as it goes, so parity is correct as the disk is erased.  This writing of zeros could take several hours or more on a large drive...

As does rebuilding the parity from scratch.

 

The point RobJ made is that until this process of writing zeros and re-building the super.dat file is complete, you are still vulnerable to another disk failing.

 

[...]

 

So... copy the defective drive's contents elsewhere

I think you misunderstand my suggestion. I was never talking about a defective disk in this thread. I was talking about an ok disk. So in the moment when the "zero this disk" button is pressed, the parity is valid, all drives are valid and thus the array is fully protected. It is in this situation that there's currently no way to remove the disk without losing parity protection. And in this situation a "zero this disk" button would allow removing a disk without ever losing parity protection.

Instead of going through all the trouble to zero the drive, why not just execute a manual parity check prior to the drive pull?  If the parity is good, you can pull your drive, and then re-calculate parity...  The chances of you losing data during the parity re-build is, as I have said, fairly low.

 

Even RAID arrays meant for enterprise use don't zero out drives prior to removal.  (They also do background scans...)

 

SirWired

RobJ is correct...

To remove a drive as you described, the unRaid software would have to completely zero a drive, not just delete the files.

 

Lets say we invent a new choice for the "Devices" page.  It is a choice labeled "erase-and-remove-drive"

 

When you choose this, you will be prompted in some way "are you sure... ALL data on the drive will be erased with no chance of un-erase"

 

Then, when you start the array, instead of mounting the disk on /mnt/diskX it will instead do the following:

  write zeros to every byte on the disk being erased.   This will also update the parity drive as it goes, so parity is correct as the disk is erased.  This writing of zeros could take several hours or more on a large drive...   

 

Then, once zeroed, the drive serial number can be deleted from the super.dat file.  At that point it is done.  the drive will show as un-assigned from that point on.

 

I am with you up to here.

 

The point RobJ made is that until this process of writing zeros and re-building the super.dat file is complete, you are still vulnerable to another disk failing.  As he said, it will probably take noticeably longer than simply using the existing "Restore" button, since it involves reading parity in addition to all the other disks.

 

Not sure I follow this.  A disk can fail at any moment, so in this sense you are always "vulnerable" to a disk failing.  The point is that with parity entact, you can rebuild a failed disk.

 

I agree it will take a while, but probably not as long as rebuilding parity (see below).

 

So... copy the defective drive's contents elsewhere, stop the array, un-assign it, check the checkbox under the "Restore" button and press "Restore"   Several hours later you will be protected again.  RMA the defective drive... when its replacement arrives, install it as a new drive in the currently empty slot in the array and re-assign it.

 

Joe L.

 

This will always be an option, but not sure why you'd want do.  Rebuilding parity will likely be SLOWER than the zero out, especially for arrays with lots of disks.  (Parity rebuilding will read the entirety of  each and every drive and update the entire parity disk.  Zeroing out will require reading/writing the entire parity disk and writing the entire zero out disk.  The other disks won't be touched.)

 

To make this easier to implement, the whole zero out operation could be cancelled if it doesn't complete "in one sitting".  If another drive fails, there is a power failure, or the array is stopped for any reason, it could be that the user would just have to restart the "zero out and drop" process from scratch.

 

(Sorry madshi, I ddin't see your post until I had posted this one.  I think we are saying basically the same thing).

 

 

- Brian

OK... when removing a drive that has not failed, the process of zeroing it and then re-building the super.dat file would reduce the time the array is vulnerable to a disk failure to nearly zero, as opposed to several hours while parity gets re-calculated today if we simply un-assign the drive and press the "Restore" button.

 

Zeroing would do nothing to shorten the time the array is vulnerable if trying to remove a failed drive for RMA replacement.  Best situation there would be to purchase locally a new drive and install it, letting the array rebuild it. (may not get the best price, but the array stays protected) 

 

No matter what, there is still the possibility of a power failure during the zero-out process.  That would leave you with a partially zero'd disk, and parity might need to be re-calculated anyway, since the array was not shut down cleanly.

 

 

 

Joe L.

Zeroing would do nothing to shorten the time the array is vulnerable if trying to remove a failed drive for RMA replacement.  Best situation there would be to purchase locally a new drive and install it, letting the array rebuild it. (may not get the best price, but the array stays protected) 

 

Completely agree.  This only helps when you are removing a drive not replacing a failed one.  (This removing a drive feature could be used to help upgrade a drive without unprotecting the entire array.)

 

 

I thought of a few issues that would have to be addressed:

 

1.  When you start writing binary zeros to a drive, you instantly destroy the partition table, journal area, superblock - basically ALL the reiserfs structures at the start of the disk (I have some recent first-hand experience with this).  In essense you have unformatted the disk.  Although parity might be fine, unRAID will detect that the disk as UNFORMATTED.  Not good. 

 

A potential fix is to zero out the disk in reverse that preserves the first part of the disk until just before the drive is dropped from the array.

 

2.  The kind of I/O needed to zero out the disk is probably device level I/O (e.g., /dev/md1).  The kind of I/O that unRAID is tracking may be tied to the filesystem (/mnt/disk1).  In short, if it doesn't already, unRAID would have to have a way of doing low-level I/O while maintaining parity. 

 

- Brian

 

 

Instead of going through all the trouble to zero the drive, why not just execute a manual parity check prior to the drive pull?  If the parity is good, you can pull your drive, and then re-calculate parity...  The chances of you losing data during the parity re-build is, as I have said, fairly low.

 

That is the best thing to do today.  Run a parity check BEFORE replacing a drive.  It is a dry run for the parity rebuild to follow.

 

The advantage of madshi's suggestion is that it allows a fully protected way of replacing a disk.  It will also be MUCH faster than doing a parity check followed by a parity rebuild!  (Although, as I thought about it, the time required to copy data somewhere else and then back would negate much of this advantage).

 

Even RAID arrays meant for enterprise use don't zero out drives prior to removal.  (They also do background scans...)

 

SirWired

 

RAID arrays do not have this concept of replacing a smaller disk with a larger disk.  The only time you'd look to swap out a RAID disk is if it failed.

I would love to be able to remove a drive from the array without having to actually zero out the drive itself.

From my perspective, the parity itself could be recalculated without the drive,

 

From what I saw a write to one drive involves the drive and the parity drive. The other drives are not involved.

Therefore removal only requires xor'ing out the drive you want to remove.

 

For me, I would like to be able to remove a drive to bring to another location,

use it there..

Then when I bring it back, add it back into the array.

 

I have to move massive amounts of data between home and work to carry my vmware machines.

So I don't actually want the drive destroyed. just xor'ed out of parity and removed from the array.

 

You basically have this ability now.. unRaid does re-calculate all of parity rather than xor out/in the one drive being added/removed so it is probably slower than just doing the operations on the single drive.  As you said, that would speed things up.  Either way, your array would probably lose parity protection until complete.

 

To remove a drive from the array today:

Stop the array.

Un-assign the array

Use the "Restore" button (after checking its checkbox) to forget about the un-assigned drive and then re-calculate parity on the ones remaining in the array.

 

To add it back, again stop the array, then assign the drive and once more use the "Restore" button to rebuild parity with the new set of assigned drives.

 

Although wasteful of space I can think of one way to not lose parity protection.  It is to use a pair of drives that are added/removed.  The second drive to be added will be used ad a "parity calc of the drive being added"

 

Easiest would be to implement a raid0 mirror pair of drives.  When you bring your drive home it is first mirrored and then both the drive AND its mirror copy are added to the array.  (once mirrored, since every bit exists on both drives, NO parity changes are needed... the add to the array can be instant)

 

as long as they are kept mirrored, the pair of drives can be removed from the array without any parity changes.  Of course, since they are mirrored, there is no need for them to be part of the array at all.  (you knew there was a catch  )

 

Downside is cost of a second drive... to be used for the mirror copy.  Upside is even more data safety.  At no time is the data in the array at risk.

 

Thinking a little more, you might just as easily build a two disk version of unRaid in a small portable case and bring it from home to work and back.  Parity is ALWAYS correct, in either location.

 

Joe L.

WeeboTech -

 

Right now there is NO way to be able to remove a drive from the array without losing parity until it is rebuilt.  I realize having it zero the drive may not be the slickest, but it would be pretty easy to do, and the mechansims are already in place to keep parity up to date as it does its thing. 

 

JoeL -

 

Maybe I'm extra sensitive, but the idea of hitting the restore button scares me.  The appeal of unRAID is that you are parity protected, and the second you hit the restore button you are 10 billion sector I/Os from getting parity turned back on - any one of which would get an error and compromise your data. 

 

IMHO, the only time you should HAVE to lose redundancy is if a drive actually fails (and it goes into its drive simularion mode).  There is no avoiding that.  But "normal" maintenance things like adding a drive, taking one out, and replacing a drive should all have procedures to follow to keep parity intact throughout.  Right now you can add a disk without losing parity, but you can't remove or replace a disk.

 

One question.  If you did that RAID 1 thing and kept the drives in sync and in the array (not sure how, but assuming that part is possible), then, as you say, removing the two drives at the same time would have no affect on parity.  My question is, how would you tell unRAID?  Wouldn't unRAID be" upset" that two drives were pulled and want you to hit restore?  Is there some way to fake it out and basically say - trust me, parity is in sync with these drives?

 

-Brian

 

 

WeeboTech -

 

Right now there is NO way to be able to remove a drive from the array without losing parity until it is rebuilt.  I realize having it zero the drive may not be the slickest, but it would be pretty easy to do, and the mechansims are already in place to keep parity up to date as it does its thing. 

 

JoeL -

 

Maybe I'm extra sensitive, but the idea of hitting the restore button scares me.  The appeal of unRAID is that you are parity protected, and the second you hit the restore button you are 10 billion sector I/Os from getting parity turned back on - any one of which would get an error and compromise your data.

True... you are without parity protection until it is completed.

IMHO, the only time you should HAVE to lose redundancy is if a drive actually fails (and it goes into its drive simularion mode).  There is no avoiding that.  But "normal" maintenance things like adding a drive, taking one out, and replacing a drive should all have procedures to follow to keep parity intact throughout.  Right now you can add a disk without losing parity, but you can't remove or replace a disk.

Good point.

One question.  If you did that RAID 1 thing and kept the drives in sync and in the array (not sure how, but assuming that part is possible), then, as you say, removing the two drives at the same time would have no affect on parity.  My question is, how would you tell unRAID?  Wouldn't unRAID be" upset" that two drives were pulled and want you to hit restore?  Is there some way to fake it out and basically say - trust me, parity is in sync with these drives?

-Brian

 

Brian,

Today you cannot do as I said... You cannot define a mirrored pair, nor can you add or delete them from the configuration stored the last time you used the "Restore" button.

 

I was suggesting a new feature... defining a portable, mirrored pair that could be moved from array to array.  Since they contents of the mirrored pair would be identical, and by definition have "even parity" they could be added or removed from the full array with no need to re-calculate parity.  The changes would only be to the super.dat file.

 

On the other hand, they would still need to be kept in sync with each other to be able to remove them in a single update of the super.dat file.  Also, since they are in effect a raid-0 mirror array it would be FAR simpler to simply allow the creation of a mirror pair, and use them that way without affecting any of the existing drives, and NOT add them to the super.dat file as part of the main unRaid array. 

 

They would instead, be a mirrored pair of "portable disks." designed for movement between arrays.  If it was possible to have them in a USB enclosure(s), even better.  Those could be moved back and forth from home/work as needed and never need to touch the "Restore" button at all.

 

Joe L.

The only way I can think of to not lose parity protection when re-configuring the array is to have the ability to define a second "parity disk" and have it used for the new calculations (adding or removing a disk from the array).

 

Once the new calculations are done, the parity disk can be swapped logically to use the new one and the config changed to indicate that it is the one to use going forward. (untill they are swapped once more on the next re-config)

 

Complications exist though if you attempt to write to any data disk while calculating the new parity drive contents as both the original parity and the new would need to be updated.

 

There is one other MAJOR issue... It takes 5 or so hours on my array to calculate parity.  There is NO WAY I will take a drive to work from home, add it to my array there, and then wait 5 or more hours before I can use it.    As I said, a simple portable mirrored pair would do for movement from home/work.  We just would need an easy way to add it to the shares unRaid is presenting to the LAN.

 

Joe L.

 

Joe L.

The only way I can think of to not lose parity protection when re-configuring the array is to have the ability to define a second "parity disk" and have it used for the new calculations (adding or removing a disk from the array).

 

I'm confused.  Why wouldn't the feature described earlier in this thread do this without adding any more disks.  In short ..

 

1.  You'd hit some button on the web gui to remove a disk from the array.

2.  unRAID would start writing binary zeros to the disk, starting at the end of the drive and moving towards the front.  The drive would still be in the array so parity would be updated.

3.  When the disk is completely filled by binary zeros, unRAID would then drop the drive from the configuration automatically.  (Since it is full of binary zeros, parity is correct whether it is in the array or out)

DONE

 

Very similar to how a drive is added to the array - only in reverse!

 

-Brian

The only way I can think of to not lose parity protection when re-configuring the array is to have the ability to define a second "parity disk" and have it used for the new calculations (adding or removing a disk from the array).

 

I'm confused.  Why wouldn't the feature described earlier in this thread do this without adding any more disks.  In short .. zero the drive being removed.... then mark it as removed... parity is already correct.

You are not too confused, but describing a different need.... WeeboTech wanted to be able to move a disk from home to work and back to allow ease of editing large video file.  You want a method of removing a drive from the array without caring about its contents.

 

Your suggestion of zeroing a drive works in theory, but only if you do not care about the data on the drive you are removing.... unfortunately, I do not think WeeboTech wants to take a cleared drive from home to work...  He wants to move a drive and its data AND not lose parity protection for a period of time when it is added to his array.

 

A mirrored portable pair of drives permits nearly instant addition or removal to an existing array since it would not affect parity at all.

This feature does not currently exist... but... it could be added to Tom's laundry list.

I just didn't read his post carefully. 

 

Thanks for clarifying. 

 

There is an outstanding enhancement request to be able to expose a disk that is NOT part of the array via unRAID.  Would that feature help?

1.  You'd hit some button on the web gui to remove a disk from the array.

2.  unRAID would start writing binary zeros to the disk, starting at the end of the drive and moving towards the front.  The drive would still be in the array so parity would be updated.

3.  When the disk is completely filled by binary zeros, unRAID would then drop the drive from the configuration automatically.  (Since it is full of binary zeros, parity is correct whether it is in the array or out)

 

What i have suggested elsewhere is a possibly quicker version, similar, except to skip the write to the drive being removed.  On selecting the option to remove a data drive, unRAID will loop as long as there are blocks in the data drive, reading them and updating the corresponding blocks of the parity drive.  Plus, only data blocks with binary ones require modifications to the parity drive.  Once the loop reaches the end of blocks that have ever been written to, then no accesses of the parity drive will occur, and it should finish fast.  Blocks that do require parity updating will require a single read from the data drive and a read then write to the parity drive, which should run fairly fast.  In addition, you end up with an unassigned data drive that is still untouched, any data is still intact, which may or may not be useful.  I don't see any useful reason to zero out the drive, plus it takes longer