Will there ever be support for multiple parity drives?

[bubbaQ]

Got to be careful with polls... self selected population

 

I agree that the application platform is probably more important to the installed userbase than DP.  But dual parity is a definite feature that people consider when shopping (and comparing) NAS platforms.  Even folks that don't know what it means, it is something unRAID does not have, and others do have.  UnRAID suffers in that situation because of it.  And polls don't capture those lost sales/recommendations.

[Brucey7]

Got to be careful with polls... self selected population

 

Absolutely, your poll is slewed because it is only responded to by the enthusiasts who populate this board.  To get an accurate poll, you would need to email your user base.

 

It is not true to say all the NAS players who only provide NAS are dead.  Far from it.  Before they add nice to have type features, they provide core functionality that cannot be faulted.  Not so unRAID, the core functionality is unfortunate and I am being polite.  Relying on a single parity disk is a disaster waiting to happen.  Dual and even triple parity is required to bring unRAID core functionality at least up to the norm.

 

From the outside looking in, it is my humble opinion that the development is driven by techies lacking commercial acumen.

 

Whilst virtualisation was a core pressure more than a decade ago, there are some things you just don't run on your production database hardware and that is anything else!

 

unRAID had some interesting USP's.  Successful USP's by their very nature, don't remain unique for ever.  For a long time, unRAID was the go to home storage server of choice if you wanted parity protected data, lots of disks supported, disks of different sizes, user shares, individual disk file systems and Write Seldom/Read Often applications.  Today, there are many others, but unRAID is no longer a safe repository for your data speaking relative to competing solutions.

 

The Home NAS market is growing at a CAGR > 14% (source, Yahoo Finance).  Some NAS vendors like Thecus are growing at 35% CAGR.  It could well be that Limetech's license sales are exceeding this which would vindicate their policies.....

 

I for one would be prepared to pay a premium to unlock dual parity.  Likewise, if a competitor comes along offering a USB stick of competing software and Dual Parity, plug and play to the unRAID community, I would be saying "Bye-Bye Tom, thanks for all the fish".

 

 

[binhex]

I love what LT have done with v6 and i ts a really Goff step in the right direction. But v6.1 should def include DP, not just for user peace of mind but also for commercial reasons to bring unraid up to scratch with the competition, disks are getting bigger, protection needs to be improved to allow for this trend.

[BobPhoenix]

I for one would be prepared to pay a premium to unlock dual parity.

I think this is a good idea.  Should give LT incentive to implement it in 6.1 as I would be willing to pay up myself on at least one of my servers.

[tdallen]

As someone who a) really appreciates what 6.0 is doing for the base unRAID feature set, and b) doesn't need dual parity (yet, anyway)...

 

I really think that dual parity should be the #1 priority for 6.1.  Seriously, I don't need it and I'll have to wait for my pet feature additions in 6.2 and beyond, but I think it's the right thing to do.  You are loosing some sales based on the fact that this box isn't checked yet and fighting a perception that unRAID provides less fault tolerance than competing solutions.  In the product world the value of some features are disproportionate to the number of people who will use them.  I think dual parity is such a case - it would help reinforce your credentials as a premier fault tolerant storage provider.

 

This isn't a criticism of the priority calls to date - I love what I can do with unRAID 6 and I'm glad you've given me a lot of cool, usable new features.  I really think you should consider dual parity next, though.

[garycase]

"... As someone who ... doesn't need dual parity ..."

 

An opinion you'll likely change the instant you have a disk fail during a rebuild of another failed disk.

 

 

... You are loosing some sales based on the fact that this box isn't checked yet ...

 

It's not just a box to be checked -- it's a major improvement in fault tolerance ... not only does it provide protection against a disk failing during a rebuild, but it also makes it possible to tell WHERE an error is when a parity error is detected during a parity check.

 

 

[archedraft]

 

but it also makes it possible to tell WHERE an error is when a parity error is detected during a parity check.

 

Until just now I didn't think I would use dual parity. I only have 3 data drives so the chances of 1 failing durning a rebuild is fairly slim; however, being able to know where the error is would be fantastic. That said I am loving everything about version 6 especially the virtualization aspect. I know this probably won't happen but I would love to hear unRAID's internal big picture road map for the next few years i.e. what are the major features you are planning to implement.

[tdallen]

It's not just a box to be checked -- it's a major improvement in fault tolerance ... not only does it provide protection against a disk failing during a rebuild, but it also makes it possible to tell WHERE an error is when a parity error is detected during a parity check.

 

I think you may have missed the part where I firmly advocated that dual parity should be LimeTech's #1 priority after 6.0 is released...   The point I'm trying to make is that users value dual parity even if they won't immediately implement it.  It's an expected part of a redundant storage solution, for all the reasons you mention.

 

I got the feeling from JonP that LimeTech has targeted feature additions because, to paraphrase, "that's what the customer base is asking for".  From my viewpoint - the marketplace of customers who buy redundant storage software is asking for, and does expect dual parity - even if some of them won't immediately implement it.  As someone who recently implemented unRAID, I feel like I took a risk going with a solution that doesn't have dual parity.  The thought that "that's a feature that I might need some day, and unRAID doesn't have it" is impacting purchase decisions today.  I'd love to see LimeTech address dual parity as the #1 feature addition post the 6.0 release.

[WeeboTech]

 

but it also makes it possible to tell WHERE an error is when a parity error is detected during a parity check.

 

Until just now I didn't think I would use dual parity. I only have 3 data drives so the chances of 1 failing durning a rebuild is fairly slim; however, being able to know where the error is would be fantastic.

 

It only takes one pending sector to cause a rebuild to fail. (Take it from experience).

Thankfully, those are now being monitored and exposed via gui based attributes.

[jonp]

It is not true to say all the NAS players who only provide NAS are dead.  Far from it.

 

I don't know of a single competitor that doesn't support the installation of 3rd party applications.  In fact, many of them are branching out past just support for these apps, and implementing even more tightly integrated services beyond traditional storage.

 

A true "traditional NAS" would not have support for any applications to be installed.  At best, it would be a purely headless appliance that would support a few file sharing protocols and that's it.  That is what I'm referring to as being a dead model.

 

From the outside looking in, it is my humble opinion that the development is driven by techies lacking commercial acumen.

 

I worked in the enterprise IT industry for 10 years before joining Lime Tech.  To be more specific, I worked for a national integrator / reseller of infrastructure solutions (think Cisco, NetApp, Citrix, VMWare, EMC, Microsoft, etc).  The years before that, I started a few companies in the web space.  Tom's background along with all our bios can be read on our website.  We have done our research and prioritize features that we feel will serve our needs for growth through product differentiation in the markets we compete, and in new markets we feel we can branch into as well.

 

there are some things you just don't run on your production database hardware and that is anything else!

 

While this may have been true for some time in the enterprise IT world, it certainly isn't what the consumer market has demanded and received in the past 10 years.  Also, even in the enterprise IT space, I never met a customer that put a database on a NAS.  A SAN, sure, but that's an entirely different solution and is apples to oranges when comparing to unRAID and the NAS market.

 

Furthermore, the big theme in tech nowadays is convergence and consolidation.  Cisco UCS is a prime example, Nutanix is another (and a great example of storage / compute convergence).  We're simply bringing that same next-generation thinking down the consumer market in a useful and meaningful way.

 

The Home NAS market is growing at a CAGR > 14% (source, Yahoo Finance).  Some NAS vendors like Thecus are growing at 35% CAGR.  It could well be that Limetech's license sales are exceeding this which would vindicate their policies.....

 

We have spent serious time analyzing the market of competitors in the NAS arena and aligned our development as such.  unRAID 6 has been in development for a little over a year now, and I feel that we've grown in many key areas relating to core NAS functionality including support for more filesystems, cache pooling, an upgraded web management interface with better reporting, system notifications, and recently UPS support.  We have a lot more planned all in due time.

[ljm42]

I was talking with a friend this weekend about unRAID.  He was very impressed with the ability to mix and match drive sizes and to add drives as you need them.  He was definitely interested until he learned that it could only handle a single drive loss, even when I explained that if you were unlucky enough to lose two drives at once you would only lose the data on those two drives and not the whole array.  I even mentioned that because it uses standard file systems there was a good chance of recovering the lost data.  It didn't matter, for him, a two parity system is a must.

 

For me on the other hand, I don't know that I would actually implement the second parity drive if it were available.  I only have room for 8 drives in my case and I'm not sure I want to three of them to be overhead (two parity plus cache).  I have everything backed up in Crashplan so even if I did lose two drives, nothing would actually be lost. 

 

I'm more interested in the items already identified for 6.1 and 6.2 in the roadmap (plus this one of course: http://lime-technology.com/forum/index.php?topic=34883.0 )

 

So basically, Limetech is in a tough spot  Different people want different things and they have to figure out which ones to prioritize.  No matter what they do first, somebody is going to complain.  I'll try not to complain too loudly 

[ootuoyetahi]

So basically, Limetech is in a tough spot  Different people want different things and they have to figure out which ones to prioritize.  No matter what they do first, somebody is going to complain.  I'll try not to complain too loudly 

 

If they find themselves in a tough spot its because they put themselves there. This is supposed to be a nas os and they have strayed from the course by getting so caught up in other things. The core feature of the application should be ever evolving. Once dual parity is implemented, they shouldn't forget about it for 3 years (again), they should work on improving it somehow yet again.

 

In my opinion they are missing the two biggest features of a storage solution and that is support for multiple drive failures and encryption. Without either of those they are never going to be an option for the corporate world.

[Brucey7]

@jonp, you missed the point.  Every other NAS player out there may have added support for apps, but they did it after they had their core functionality right.  That means dual disk parity as a minimum.

 

I don't want to get into a p?ss?ng contest about your background.  When I decided to invest in my first unRAID system, I flew one of the Greenleaf guys and his girlfriend to Thailand to hand deliver it.  I retired at 49 from the IT industry, I was a Fortune 2000 Global IT Director (at age 31) before founding a number of ISV's and then marketing service businesses around the IT sector, one of which I took to IPO.  If it's IT and commercial acumen, I know my onions, I've walked the walk.  You can take advice, or you can take umbrage, it's not my future at stake.

 

To coin a well used phrase, you don't put lipstick on a pig.  You need to get dual parity sorted before you choose what colour lipstick you want.  The request for DP is a consistent theme on this board.  It's not a nice to have, it's a must have.  I do applaud you for getting away from ReiserFS, that was probably the second most important thing after DP.

 

I can't help feel that you are looking at other players and trying to play "catch up" rather than look at your USP's and play to those.  unRAID is a niche product, others have their sights set on your market, I think you should protect it before thinking you are credible to go after theirs.  I also suspect the amount of development effort you are putting into these nice to have's is holding you back and might even reach a point where maintenance issues will stall your development effort.

 

I also wonder if you have considered (or even polled) what uses unRAID could be put to/has been put to, in the business sector.  I would have bought it for journaling, backup and near-line storage.  The ability to write fast without parity and then later build parity is a fantastic USP.

 

 

[WeeboTech]

I do applaud you for getting away from ReiserFS, that was probably the second most important thing after DP.

 

I disagree with this point. (and probably only this one)

 

Given where unRAID was and how ReiserFS was being maintained, this change is so important, and critical, to insuring reliable data storage. We've seen two errors slip through with ReiserFS during kernel changes.  1 where it 'could' loose and corrupt data, 2 where it did corrupt data, not just new data but old data and meta data.

 

This change had to be implemented or we would be chasing and scrutinizing ReiserFS changes and/or issues.

[jonp]

@jonp, you missed the point.  Every other NAS player out there may have added support for apps, but they did it after they had their core functionality right.  That means dual disk parity as a minimum.

 

Didn't miss the point at all.  My point is that when comparing what the other players did to differentiate themselves long ago and the way we need to differentiate ourselves in today's market, it's apples and oranges.

 

No one here has said that dual parity isn't going to happen or isn't coming.  We are simply discussing the order of events.  Some believe that dual parity is a requirement for us to have a minimum viable product in today's market whereas others do not.

[superloopy1]

Another +1 for dual parity here having recently lost a drive during a rebuild. I'm not going to bother replacing it. Will dual parity be in place before i lose the next one? I doubt it.

[ohlwiler]

Dual parity was on the first road map that Tom published  long ago (about five years?). Alas, it is my conviction that unRaid will never have it.

 

I have been especially diligent about monitoring drive health and haven't lost any data in over 9 years of usage. I've also heard very few stories of data loss where dual parity would have helped. I suppose if dual parity was really important to me, I could move on to another solution.

 

You can't always get what you want. -Mick Jagger

[Helmonder]

Just wanted to chime in here. I think dual parity is a core NAS feature supported by most competing products / technologies. I think it should be very high on the priority list.

 

I agree that users should back up data they can't afford to lose - normally unique works like family pictures, home movies, and correspondences/financial records. But arrays also contain vast amounts of "recoverable in other ways" data and the cost and diligence to back up every file is not justified to the user. So realistically, users are making decisions about what to back up and what they are willing to lose / recover in other ways, in the event of a calamity. By investing in a NAS like unRaid, users have an economic form of redundancy to avoid the time, effort and possibly cost to re-acquire and assemble their "recoverable in other ways" data for drive failure - a common and foreseeable problem. These do not protect from unforeseeable events like acts of God or fire - things likely having them looking for a new home (for which they have no backup) also. Users are not tolerant of losing data due to insufficient redundancy options in their NAS, which they bought to explicitly protect them from drive failures!

 

I fully agree, this is the way I see it.

[Frank1940]

My PERSONAL opinion is that dual parity should be an add-on option that requires purchase of a separate license to activate.  There will be a substantial development cost to incorporate dual parity into the unRAID Operating System.  This cost should be borne by those who will actually use it.  The user-side implementation will also require an considerable higher increase in hardware costs, especially for the beginning user-- That extra parity drive,  the extra hardware and perhaps even an increase in the CPU power required.  It will probably require either a server class board for more SATA connectors or a separate SATA controller card on even a relativity small servers in terms of the number of data drives.  And dual parity still does not eliminate the need for a backup in case of a catastrophic event that destroys the hardware itself.  I, personally, am not planning to implement dual parity on my servers because I have all of the difficult-to-replace data safely stored in an offsite store facility.

[ootuoyetahi]

My PERSONAL opinion is that dual parity should be an add-on option that requires purchase of a separate license to activate.  There will be a substantial development cost to incorporate dual parity into the unRAID Operating System.  This cost should be borne by those who will actually use it.  The user-side implementation will also require an considerable higher increase in hardware costs, especially for the beginning user-- That extra parity drive,  the extra hardware and perhaps even an increase in the CPU power required.  It will probably require either a server class board for more SATA connectors or a separate SATA controller card on even a relativity small servers in terms of the number of data drives.  And dual parity still does not eliminate the need for a backup in case of a catastrophic event that destroys the hardware itself.  I, personally, am not planning to implement dual parity on my servers because I have all of the difficult-to-replace data safely stored in an offsite store facility.

 

I 100% disagree with your comment, and in fact this would lose me as a customer. I can say the same thing about all of this support for plugins and docker containers and other superficial additions. I dont use it, I dont want it, why did I pay for it? unRAIDs core functionality is (well I thought when I purchased it anyway) fault tolerance and the application needs to be brought up to date with todays technology to remain relevant.

 

[Helmonder]

Guys,

 

There is not much sense in discussing the issue. Unraid is a commercial product and the people supporting it make their own decission in the route the system takes.

 

Unraid as a buy once, use for ever license model. This means that decissions on evolving the product will (naturally) for the larger part be based on what new users want, not so much on what users that have allready purchased the product want. Please note that this is not meant as disrespecting in any way, this is a commercial fact. Peoples livelyhood is based on the commercial viability of the product, therefor it is logical to make choices that draw in new users.

 

Apparently the thought is that virtualisation is in higher demand than higher safety, this is exactly what jonp is saying, its a priority issue.

 

If there is something better for your personal use case, then start using that..

 

For me personally: I have no need at al for virtualisation... I need my setup to be rockstable, therefor I have vmware running with approx 10 virtual machines in there and unraid on a seperate box. Rocksolid. Ofcourse that means having two boxes, paying twice as much for power and hardware, that is a decission I have made, others make different ones.

 

Unraid still is (as far as I have found) a product that has NO competitor... I have not found ANY product that brings me:

 

- ease of expansion (just bang in any drive and it works)

- recovery (drive loss only looses you the drive that breaks, the rest works)

- drives able to be read on other platforms

- parity protection for loss of one drive

 

I would love double parity but as far as I know there is no product delivering double parity combined with the above features... FreeNAS and the likes are as far as I know not flexible with regard to expansion..

 

I will spend two lines on the commercial side, just meant for limetech and jonp to put in their mind when they decide on this stuff:

 

The statement is that parity is no replacement for a backup, that is partially true, but NOT completely. It is highly dependent on what kind of scenario of data loss you are thinking of.

 

Unraid is in a niche market delivering cheap flexible storage, I think the amount of users choosing such a a "cheap flexible" product will not duplicate that system times two for reasons of backup.. I think the majority of our users will backup their family photos and documents to one of the online drive offers (onedrive, dropbox, google drive, etc). 90% of the used storage will most probably not be backupped..

IF you take that as a given the dual parity would deliver a lot with respect to data safety.

[ootuoyetahi]

Guys,

 

There is not much sense in discussing the issue. Unraid is a commercial product and the people supporting it make their own decission in the route the system takes.

 

You're probably right but if nothing else the discussion will tell let them know that their polls with 141 responses dont tell the whole story.

 

Unraid is in a niche market delivering cheap flexible storage, I think the amount of users choosing such a a "cheap flexible" product will not duplicate that system times two for reasons of backup.. I think the majority of our users will backup their family photos and documents to one of the online drive offers (onedrive, dropbox, google drive, etc). 90% of the used storage will most probably not be backupped..

IF you take that as a given the dual parity would deliver a lot with respect to data safety.

 

It is a niche market but things like encryption and multiple drive parity they would potentially allow them to break into the commercial world.

[tdallen]

I think that Dual Parity is a feature that the vast majority of Pro License users would implement, and one that many Plus license users would implement as their array grows.  I don't see it as relevant to the majority of Basic License users.

 

I'd be willing to upgrade my Plus License to a Pro License if that's what it takes to get Dual Parity.  I'm not a fan of it being a separate product, though.

[Brucey7]

I'm a few years out of date, but last I heard was more than 10 drives you need dual parity as a minimum (albeit that was comparing RAID's 5 & 6).  I agonised over the decision to buy unRAID for quite a while and nearly didn't because of this single shortcoming.

 

As an aside, I would prefer an option for a rebuild not to fail on a single URE, but instead for me to firkle the resulting disk afterwards.  If you're storing media, a few errors here and there often makes almost no difference.  Even better if the files where the corruption has occurred could be identified.

 

It's all well and good to say you need a proper backup instead, if anyone has a sensible, inexpensive option to backup well over 100TB of media, I'm all ears.

[gundamguy]

I'm a few years out of date, but last I heard was more than 10 drives you need dual parity as a minimum (albeit that was comparing RAID's 5 & 6).  I agonised over the decision to buy unRAID for quite a while and nearly didn't because of this single shortcoming.

 

As an aside, I would prefer an option for a rebuild not to fail on a single URE, but instead for me to firkle the resulting disk afterwards.  If you're storing media, a few errors here and there often makes almost no difference.  Even better if the files where the corruption has occurred could be identified.

 

It's all well and good to say you need a proper backup instead, if anyone has a sensible, inexpensive option to backup well over 100TB of media, I'm all ears.

 

I agree that dual parity is a great feature that unRAID should try to implement ASAP.

 

But I do want to say that 10 drives in RAID 5  is a bit different then 10 drives in unRAID in the sense that should you get two drive failures in RAID 5 you lose all of the data on all 10 drives. Where as with unRAID if you get two drive failures you retain the data on the other 8 drives. I mean that's not great, but it's a bit better.

 

Also isn't RAID 6 "dual parity" by definition?