Simple request; have two parity drives. At the least hot spare support on the SATA offering.

This has been mentioned once before, but before I say that for sure...

 

Do you mean

 

A.  Use the combined storage capacity of two drives dedicated for parity use to simulate a single larger parity drive?

 

or

 

B. Mirror the parity drive contents in a second parity drive. Both parity drives would be identical copies of each other? Either one being able to be used to restore the contents of a failed drive.

 

 

If "A" then we have already requested this feature, and I think Tom said it could be done. (may not be high on his list, but hopefully, not forgotten)  This would allow maximal use of the spare drives most of us have laying around.  I could pair up a 300G and a 250G drive to simulate a 550G parity drive and then use a new 500G drive I get on sale for media storage.  Flexibility is the key here. I would love this feature to be supported as we grow our arrays.

 

If "B" then I'll have to ask your reasons behind the request.  The parity drive is no more important than any other drive in an unRaid array, SATA or IDE.  Why do you want to mirror it?  Do you think it is the most likely to fail?

 

Joe L.

If "B" then I'll have to ask your reasons behind the request.  The parity drive is no more important than any other drive in an unRaid array, SATA or IDE.  Why do you want to mirror it?  Do you think it is the most likely to fail?

 

Joe,

 

Definitely B; and yes the parity drive is presumably on constantly thus is more prone to failure. Option A doesn't seem useful to me given that  very large drives (400gb +) are so (relatively) cheap these days. Basically,  I hope that once the SATA solution arrives there is enough bandwidth to sustain at least 16 drives in a build. With 12+ drives the chance of failure improves commesurably with every drive. I was considering setting up a RAID6, but came across Unraid which seems to be a nice option. So yes it would great to have a mirrored Parity drive (or at least a hot standby drive option) kick-in in the case of primary disk failure. Perhaps I'm paranoid, but I've seen many a RAID crap out in my day working as a DBA. I would not be looking farward to re-ripping my 890 DVDs and 1200 CDs, 200 DV tapes, if you know what I mean.

 

-alex

If "B" then I'll have to ask your reasons behind the request.  The parity drive is no more important than any other drive in an unRaid array, SATA or IDE.  Why do you want to mirror it?  Do you think it is the most likely to fail?

 

Joe,

 

Definitely B; and yes the parity drive is presumably on constantly thus is more prone to failure. Option A doesn't seem useful to me given that  very large drives (400gb +) are so (relatively) cheap these days. Basically,  I hope that once the SATA solution arrives there is enough bandwidth to sustain at least 16 drives in a build. With 12+ drives the chance of failure improves commesurably with every drive. I was considering setting up a RAID6, but came across Unraid which seems to be a nice option. So yes it would great to have a mirrored Parity drive (or at least a hot standby drive option) kick-in in the case of primary disk failure. Perhaps I'm paranoid, but I've seen many a RAID crap out in my day working as a DBA. I would not be looking farward to re-ripping my 890 DVDs and 1200 CDs, 200 DV tapes, if you know what I mean.

 

-alex

Alex,

Unless you are constantly writing to the unRaid array the parity drive will be idle.  It is only active (and spun up) when either you are writing to one of the drives in the array, or when it is reconstructing the data from a failed data drive.

 

For most of us in our homes, the most active drives will be the media drives, not the parity drive, since most of the time we are playing the movies/music we stored on the server and not putting new data on the server.  If your profile of use is different, then, perhaps the parity drive will be most active for you.  For me, it is the least active disk in the array and not even spinning most of the time.  I know I added a file to one of my disks on Friday, so that is the last time it spun up, and even then, it was only for an hour till it spun down again.  True, when initially loading your collection of DVD movies, Cds, and DV tapes the parity drive will be most active, but thereafter, it will be the least active.

 

As a professional software developer, I know of only two types of hard disks...

 

Those that have already crashed

 

and

 

Those that have not yet crashed... it is only a matter of time.

 

As I'm sure you know, the key is to replace the failed drive as soon as possible.  For that I use a shell script I created to alert me in the case of a failure of a disk in the unRaid array.    I would not have any more use for a mirrored parity drive than any other data drive.  I would rather use the space for more media storage if I had a spare 500G drive laying around. 

 

On the other hand, being able to use a few smaller drives to act as a larger parity drive lets me use a larger drive for data.  That is why I would opt for "A" and have no use for "B" at all.

 

Joe L.

Joe,

 

My use wont be substantially different from yours, less I migrate my work over from a readynas unit over to the unRAID (which would impact the parity drive heavily with continuous writes of small files to at least  one disk in the array).

 

I'm going to go for 500GB disks (hopefully x16, min. x12), so the reburning fear in me is great. This is why I'd rather have more protection than less. Perhaps you are right though, and the thing to request is hot standby support, rather than mirrored parity drives. I wonder what's involved in accomplishing the latter?

 

Regards,

Alex

Hot standby support is on "the list" for a future release.

I'll echo the Why in this. I've had my Parity drive fail in the past. Result? No loss of data - I simply lost protection of my data until the drive was replaced and Parity rebuilt. If I'd had a double failure then I'd have lost whatever data was on the second drive but *NOT* all of my data - this is why not striping is so nice.

 

Double failures are pretty rare but frequent enough that having one that cost me all of my data would make me nervous. If I was going to goto the trouble of running hot spares I'd runnormal RAID and just hope it didn't burp

A hot spare sounds like a nice feature.  It would work great with a 16 disk sata configuration.  I think with a 12 drive system it might be overkill.  What I might do is once my unraid server is full is buy an extra drive, put it in the system as parity and make a spare.  That way if the parity drive dies I have one sitting on the self I can throw in the system and not be unprotected.

A hot spare sounds like a nice feature.  It would work great with a 16 disk sata configuration.  I think with a 12 drive system it might be overkill. 

 

erikatcuse,

 

I think when we're dealing with 400gb+ drives, a spare for a 12 disk array is still a good idea, if one considers their time as valuable. At the end of the day, I would imagine, if there is an implementation for 16 drives or more it would be effective on 12 anyway.

 

Following this (= http://www.avsforum.com/avs-vb/showthread.php?t=664437&highlight=raid) thread on AVSF, I was actually thinking a nice, relatively inexpensive, configuration could be arrived in having a 4 drive 1U server in combination with this (= http://www.norcotek.com/item_detail.php?categoryid=8&modelno=DS-1220) chassis/e-SATA card.

 

-alex

 

ps, can we properly href links here?

If I was going to go so far as to have a hot spare I might as well run "real" RAID and get the performance benefits that come with it. I'd still worry about corruption of the volume but dual failures are certainly out if you've gto a hot spare. IMO multiple parity drivs is overkill but suit yourself. Personally I'd rather the development time go towards speeding R/W performance...

Resurrecting an old thread.  If work is being considered for multiple parity disks, then presumably it would be worth doing right the first time. Therefore, I propose a non-striped variant of the RAID-6 parity approach:

 

http://en.wikipedia.org/wiki/Redundant_array_of_independent_disks#RAID_6

 

Like the current unraid, instead of distributing that parity across disks, it would be stored on a single, second, parity disk.

 

This would protect the unit againt dataloss even if any two disks failed.

 

-brendan

Ponderng this - unless I've missed something how would having two Parity disks protect against a dual failure exactly? I can see a hot spare saving you from a dual failure since whenthe first drive died the spare would take over and then parity would save the second drive. But really, if you have two drives fail like that wouldn't whatever took out the other two likely take out a third as well? I cannot help but think it would be a power issue or physical problem that would take out two - either that or you're not paying any attention to the health of the system for many months on end...

 

So, hot spare I can maybe see (I won't use it) but a second parity drive just seems like asking for addiitonal latency when writing and I think we've got enough of that now...

BLKMGK,

 

Your reasoning is exactly the same as mine, as long as we reasonably keep track of the array health we should be able to deal with any single disk failure and never experience two disks failing at the same time.

 

On the other hand, my initial request,  to use the combined storage capacity of two drives dedicated for parity use to simulate a single larger parity drive is still a very good idea and does not result in wasted storage capacity.

 

I think Tom said it could be done. (may not be high on his list, but hopefully, not forgotten)  This would allow maximal use of the spare drives most of us have laying around.  I currently have a 500G parity drive and a one 500G data drive in addition to several 250G drives in my array. I also have a 300G drive in an external USB drive housing I use for backing up some PCs.

 

Let's say I find a Labor Day sale for a 750G drive for $75. (I can dream, can't I) This sale is limited to 1 drive per store, and I was the one.

 

Now... today I can add that drive to my unRaid array replacing my existing parity drive.  I can then re-deploy my old 500G parity drive as a data drive and if I have a free slot gain 500G of storage space.

 

However, if I could take my spare 300G drive out of the USB enclosure and logically combine that with the existing 500G parity drive to simulate an 800G parity drive I would be able to use the  new 750G as a data drive and gain 750Gig of space for stored media.

 

The process of logically combining the two physical parity drives as one would roughly equivalent to "striping" the parity data on two very big stripes, each physical parity drive being one huge stripe, the two stripes in combination adding up to a virtual 800G parity drive,  each of the two parity drives protecting their respective blocks of data on the media drives.

 

The parity data for the first 500G would be on the existing paarity drive and in fact would not need to be changed as we add the new 750G media drive as long as the new drive is zeroed before it is added to the array.  The new 300G parity drive would protect the top 250G of the new 750G drive.

 

This makes a lot of sense if we have a number of smaller drives and a lot of open slots (like me)  I will be watching the sales, and eventually I will add a 750G drive to my array. It would be a shame to add a single 750G drive as a parity drive and only gain 500G. (the size of my existing parity drive that could be re-used for data)

 

Now, to put thhis in perspective with the other outstanding issues.

 

1. Notification of errors/failed drives (yes, I know I wrote a script to do this, but it should be built in)

2. Performance (concurrent write/read without stutter)

3. DMA Error handling (should NOT lock up the box)

4. Error Logging.  (write logs to a second USB flash, external USB disk, floppy disk, so we can track down lockups without having to tail the syslog in a telnet window)

5. Security

6. SATA support

7. Multi-Drive virtual simulation of larger single parity drive.

 

I expect I'll be waiting a few more weeks  before item "7" makes your release notes.

 

Joe L.

 

 

Joe L.

Aaaah, now I see what you are looking for! Yeah, that makes sense to me. Given a choice I wouldn't do it since itmight mean additionaldrives spinning blah blah but when I pop for a 750Gig drive (and I've come close) yeah being able to use two other drives instead of being forced to buy TWO big drives does indeed make sense. Actually, I do recall reading your suggestion todo that awhile back but reading the last couple of entries here I guess that was sort of lost...

Aaaah, now I see what you are looking for! Yeah, that makes sense to me. Given a choice I wouldn't do it since it might mean additionaldrives spinning blah blah but when I pop for a 750Gig drive (and I've come close) yeah being able to use two other drives instead of being forced to buy TWO big drives does indeed make sense.

It would not necessarily cause more drives to be spinning.  Only if you wrote to the top 250 Gig of the new 750 Gig drive would the second parity drive be written to, and spun up.  Otherwise, the second parity drive would be sitting there idle like most the other drives.  In fact, if your usage profile is anything like mine, the parity drive is idle most of the time anyway.

Actually, I do recall reading your suggestion to do that awhile back but reading the last couple of entries here I guess that was sort of lost...

It is fairly easy to make one "virtual" parity drive out of several physical disks.  The parity drive is treated as a series of blocks of data, there is no file-system, and having X number of blocks of data on a single disk is not much different than a portion of the blocks on one disk and the remaining on another.

 

As I said, this only makes sense for us with spare slots and multiple smaller drives. In my scenario I could keep buying 750G drives until I fill all 10 slots with them.  Then, 7.5 terabytes of storage will be on-line.  I'll still be using my 500G drive and my 300G drive as a virtual 800G parity drive.  Then, when I want to upgrade at that point I'll be forced to purchase two more 750G drives, one for parity, and one for additional storage. (hopefully they will be cheaper at that point) I'll revert the config to a single parity  drive, put a 750G in the parity slot, rebuild parity, then fill the remaining open slot with another 750G. 

 

It would then contain 8.25Terabytes ... room for almost 2000 DVD images if I only store the main title... way more than I currently own, or likely to own in the next few years... On the other hand, at that point in time 1000G drives might be affordable...  Let's see, my two (current) 500G drives combined might be perfect then as a 1000G virtual parity drive.  The upgrade path continues.

 

Like I said, once Tom gets everything else higher in priority taken care of this would sure be nice to have.  Since it has taken him nearly a year to get to where wake-on-lan, power management, and  SATA spindown support is imminent, and that none of the other higher priority items have been addressed so far as I am aware, (concurrent r/w performance, UNIX and Samba security) I expect this virtual parity drive feature will not occur soon.

 

Let's just hope the imminent release goes smoothly, and that a side effect of the upgrade to the new Linux kernel will be improved performance.

 

Joe L.

Aaaah, now I see what you are looking for! Yeah, that makes sense to me. Given a choice I wouldn't do it since it might mean additionaldrives spinning blah blah but when I pop for a 750Gig drive (and I've come close) yeah being able to use two other drives instead of being forced to buy TWO big drives does indeed make sense.

It would not necessarily cause more drives to be spinning.  Only if you wrote to the top 250 Gig of the new 750 Gig drive would the second parity drive be written to, and spun up.  Otherwise, the second parity drive would be sitting there idle like most the other drives.  In fact, if your usage profile is anything like mine, the parity drive is idle most of the time anyway.

Actually, I do recall reading your suggestion to do that awhile back but reading the last couple of entries here I guess that was sort of lost...

It is fairly easy to make one "virtual" parity drive out of several physical disks.  The parity drive is treated as a series of blocks of data, there is no file-system, and having X number of blocks of data on a single disk is not much different than a portion of the blocks on one disk and the remaining on another.

 

As I said, this only makes sense for us with spare slots and multiple smaller drives. In my scenario I could keep buying 750G drives until I fill all 10 slots with them.  Then, 7.5 terabytes of storage will be on-line.  I'll still be using my 500G drive and my 300G drive as a virtual 800G parity drive.  Then, when I want to upgrade at that point I'll be forced to purchase two more 750G drives, one for parity, and one for additional storage. (hopefully they will be cheaper at that point) I'll revert the config to a single parity  drive, put a 750G in the parity slot, rebuild parity, then fill the remaining open slot with another 750G. 

 

It would then contain 8.25Terabytes ... room for almost 2000 DVD images if I only store the main title... way more than I currently own, or likely to own in the next few years... On the other hand, at that point in time 1000G drives might be affordable...  Let's see, my two (current) 500G drives combined might be perfect then as a 1000G virtual parity drive.  The upgrade path continues.

 

Like I said, once Tom gets everything else higher in priority taken care of this would sure be nice to have.  Since it has taken him nearly a year to get to where wake-on-lan, power management, and  SATA spindown support is imminent, and that none of the other higher priority items have been addressed so far as I am aware, (concurrent r/w performance, UNIX and Samba security) I expect this virtual parity drive feature will not occur soon.

 

Let's just hope the imminent release goes smoothly, and that a side effect of the upgrade to the new Linux kernel will be improved performance.

 

Joe L.

 

Bringing an old thread back to life..

 

I think I am not understanding something here.  If you use two existing disks as a combined logical parity drive (disk 1 being the original 500gb parity, and disk 2 being a 250gb data disk previously) I don't see how you gain anything by using those as parity instead of the single new 750gb drive.

 

You still have only the 500gb worth of new storage space that you would have had if you used the 750gb as parity, and changed disk 1 (the original 500gb parity) to a data role.

 

The only way this makes sense (not really?) is using your example (having an unused 300gb disk lying around), but that doesn't sound like a reasonable example to me, as it only makes sense that adding two new disks (750gb and 300gb) to the array is better than only adding one.  I don't see the need for dual-parity to realize the size increase benefits in that situation:

 

Single parity scenario (currently supported):

 

Add 750gb as new parity

Assign 500gb old parity as data

Assign unused 300gb as data

 

= 800gb of new empty space

 

VS.

 

Combined parity scenario (currently unsupported):

 

Keep 500gb parity

Assign 300gb as additional parity

Assign 750 gb as new parity

 

= 750gb of new empty space

 

This has obviously been rehashed many times by people who know what they are talking about.  Can someone give me another example that shows the benefit of combined parity?

 

-kenshin

bump - I know you are out there Joe L.!

I'm trying to poke holes in your logic as far as overall capacity, and so far, can't.

 

Biggest advantage of a concatenated parity drive would then be a single 750G volume rather than two smaller ones.  This was far more of an issue before user-shares were introduced.

 

Of course, you realize that if Tom reads this thread, he may never bump the priority of concatenated parity drives upward on his todo list  :'(

 

Joe L.

a single 750G volume rather than two smaller ones.  This was far more of an issue before user-shares were introduced.

 

I understand how a newcomer like me wouldn't appreciate that unless it were pointed out.  Thanks!

 

Of course, you realize that if Tom reads this thread, he may never bump the priority of concatenated parity drives upward on his todo list  Cry

 

Sorry!  Is there a reason you still want it with user shares now working (at least for reads)?  I would prefer that he bump user share writing; then this problem becomes even less relevant.

 

-Kenshin

a single 750G volume rather than two smaller ones.  This was far more of an issue before user-shares were introduced.

 

I understand how a newcomer like me wouldn't appreciate that unless it were pointed out.  Thanks!

 

Of course, you realize that if Tom reads this thread, he may never bump the priority of concatenated parity drives upward on his todo list  Cry

 

Sorry!  Is there a reason you still want it with user shares now working (at least for reads)?  I would prefer that he bump user share writing; then this problem becomes even less relevant.

 

-Kenshin

I too would rather see writable user shares first, before concatenated parity drives, but even higher in priority than both of those is security. I can deal with writing to disk1, disk2, etc for now, but until security is addressed, business use is very limited. Nobody would leave all their business data open to anybody on their LAN.

 

Concatenated user shares can stay where they are in priority... but I suspect they will move closer towards the tail end of the priority list than the head as larger disks get cheaper.

That makes sense.  For the last part, did you mean concatenated "parity" might get bumped down as larger drives are released?

 

Next release: 4.2-beta1

 

- writable user shares

- share level security

- better 'clearing' of new disks

 

You've been around here much longer than I have, so maybe you can guess what features will actually make it to the next build (based on other posts, previous conversations, etc.), but it seems like we might be getting both features if Tom doesn't need to push one of them off. 

 

As far as security for business use; you/others have probably already considered placing the unraid server on a separate vlan.  That won't meet the same needs as users/groups, but I would think that it could provide the level of security necessary for certain implementations (i.e. multiple app servers with multiple NIC's (one on the vlan) using unraid for app data storage/backup).  That way the servers would be accessible to the users, however the unRAID array would not be, and unRAID is a better choice than tape backup (imho).

 

What roles do you see unRAID filling in a multiuser environment?  An entry level file server?

 

-kenshin