unRAID Project Roadmap Announcements


Recommended Posts

I think that this announcement covers most of my wish list for unraid.  There are still a few items though:

 

1) Dual Parity

2) Hot Spares

3) cache_dirs in the GUI

 

And then things that are less important for the core product, but I'd still be rather happy to see:

 

1) Reprotect - When you have a failed drive, write the files that are on the failed drive and in the now unprotected Parity drive into spare space on the remaining data drives.  Remove the failed drive from the array.

2) Rebalance - When you have many full drives and one empty drive, move files around to spread the data evenly across all of your drives.  This should also allow the correction of high-water split levels after a reorganization.

 

I'm not looking for release plans or anything.  Just want to know if they're on the horizon.

 

Thanks,

 

-Ben

 

Link to comment
  • Replies 413
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

I think that this announcement covers most of my wish list for unraid.  There are still a few items though:

 

1) Dual Parity

2) Hot Spares

3) cache_dirs in the GUI

 

And then things that are less important for the core product, but I'd still be rather happy to see:

 

1) Reprotect - When you have a failed drive, write the files that are on the failed drive and in the now unprotected Parity drive into spare space on the remaining data drives.  Remove the failed drive from the array.

2) Rebalance - When you have many full drives and one empty drive, move files around to spread the data evenly across all of your drives.  This should also allow the correction of high-water split levels after a reorganization.

 

I'm not looking for release plans or anything.  Just want to know if they're on the horizon.

 

Thanks,

 

-Ben

 

Holy cow yes! And those last two things would be amazing! I think I'd prefer those over your first 3.

Link to comment

I think that this announcement covers most of my wish list for unraid.  There are still a few items though:

 

1) Dual Parity

2) Hot Spares

3) cache_dirs in the GUI

 

And then things that are less important for the core product, but I'd still be rather happy to see:

 

1) Reprotect - When you have a failed drive, write the files that are on the failed drive and in the now unprotected Parity drive into spare space on the remaining data drives.  Remove the failed drive from the array.

2) Rebalance - When you have many full drives and one empty drive, move files around to spread the data evenly across all of your drives.  This should also allow the correction of high-water split levels after a reorganization.

 

I'm not looking for release plans or anything.  Just want to know if they're on the horizon.

 

Thanks,

 

-Ben

 

Holy cow yes! And those last two things would be amazing! I think I'd prefer those over your first 3.

The last 2 plus cache-dirs buit in would be amazing additions that Id be extremely happy to see. Another thing that Id love to see is some memory/cpu allocations on the GUI. It would be great to not have to telnet into the system in order to see how much ram and cpu % I'm using.

Link to comment

I don't like the hot spare concept. See THIS POST.

 

In summary, I'd rather unRAID do as little as possible and leave it to me (the human) to figure out the right next step. So much of the time a red ball is due to some cause unrelated to a failing disk, and automatically trying to rebuild could easily make things worse.

Link to comment

I don't like the hot spare concept. See THIS POST.

 

In summary, I'd rather unRAID do as little as possible and leave it to me (the human) to figure out the right next step. So much of the time a red ball is due to some cause unrelated to a failing disk, and automatically trying to rebuild could easily make things worse.

I think reprotect is a bad idea for the same reason. That is what rebuilding is for. If for some reason you intend to not rebuild and instead shrink the array, then maybe it makes sense to move data. I would not object to a general-purpose file manager in the webGUI that would allow the user to do the move in this case, and that would also handle rebalance. Midnight Commander can handle this pretty well as is.
Link to comment

I think a double parity would be a better way to go than a hot spare.  It would be nice if you could modify the array dynamically, such as swap another disk in manually similar to a hot swap, but allow the user to control the action.  Also allow expansion of the array without taking the array offline.

 

I would be hesitant in allowing the system to move files around, even from the command line you can run into issues, such as open files or handling file access during the copy/move operation.  It would have be handled at a very low level.

Link to comment

I don't like the hot spare concept. See THIS POST.

 

In summary, I'd rather unRAID do as little as possible and leave it to me (the human) to figure out the right next step. So much of the time a red ball is due to some cause unrelated to a failing disk, and automatically trying to rebuild could easily make things worse.

I think reprotect is a bad idea for the same reason. That is what rebuilding is for. If for some reason you intend to not rebuild and instead shrink the array, then maybe it makes sense to move data. I would not object to a general-purpose file manager in the webGUI that would allow the user to do the move in this case, and that would also handle rebalance. Midnight Commander can handle this pretty well as is.

 

For my information, are either of you arguing that these - as features - shouldn't be available?

 

I'm actually about to build a many (23ish) drive flash backup system around unraid, and email notification is currently my big hangup.  I am the sole systems administrator for my company, and as such I'm not available 24 hours a day and 365 days a year as much as I'd like to be.

 

I personally would love to have reprotect or hot spare available, and configurable.  If I'm out of town for a 3 day weekend, I'd rather have reprotect or rebuild happen in my absence than wait for my own personal intervention. 

 

The opposite holds true for my home system.  If it goes bad while the house sitter is watching the cats and she can't watch anything on it, she can use youtube.

Link to comment

I think a double parity would be a better way to go than a hot spare.  It would be nice if you could modify the array dynamically, such as swap another disk in manually similar to a hot swap, but allow the user to control the action.  Also allow expansion of the array without taking the array offline.

 

My argument would be to allow as many (configurable) forms of protection as are reasonable.  I'd like to have the option of putting in 20 data drives, 2 parity drives, and a hot spare.  I'd also like the options of immediately failing over, or sending an email and setting a timer to fail over in x hours if an admin hasn't addressed the problem.  I'd like to never have to tell my boss that I've lost data.

 

I would be hesitant in allowing the system to move files around, even from the command line you can run into issues, such as open files or handling file access during the copy/move operation.  It would have be handled at a very low level.

 

Actually, I'd handle it at a very high level.  Even user level.  I could write a script that would crawl a (non-existent) drive and look for unused space on other drives to move files to.  It's one of those things that isn't super difficult, but would be made much better with a higher level of systems internals knowledge.  For instance, I have no idea as to how I'd approach split-levels in a script to do a reprotect.

Link to comment

I don't like the hot spare concept. See THIS POST.

 

In summary, I'd rather unRAID do as little as possible and leave it to me (the human) to figure out the right next step. So much of the time a red ball is due to some cause unrelated to a failing disk, and automatically trying to rebuild could easily make things worse.

I think reprotect is a bad idea for the same reason. That is what rebuilding is for. If for some reason you intend to not rebuild and instead shrink the array, then maybe it makes sense to move data. I would not object to a general-purpose file manager in the webGUI that would allow the user to do the move in this case, and that would also handle rebalance. Midnight Commander can handle this pretty well as is.

 

For my information, are either of you arguing that these - as features - shouldn't be available?

 

I'm actually about to build a many (23ish) drive flash backup system around unraid, and email notification is currently my big hangup.  I am the sole systems administrator for my company, and as such I'm not available 24 hours a day and 365 days a year as much as I'd like to be.

 

I personally would love to have reprotect or hot spare available, and configurable.  If I'm out of town for a 3 day weekend, I'd rather have reprotect or rebuild happen in my absence than wait for my own personal intervention. 

 

The opposite holds true for my home system.  If it goes bad while the house sitter is watching the cats and she can't watch anything on it, she can use youtube.

 

Ballsy. As much as I love UnRAID I would not personally be deploying it into a corporate environment. Any product that only has support by proxy is not business ready.

Link to comment

Ballsy. As much as I love UnRAID I would not personally be deploying it into a corporate environment. Any product that only has support by proxy is not business ready.

 

You can get email support any time.  We don't even ask if you are a "customer" (yet).  Depending on the issue, if it makes sense, we use other methods, eg., google hangouts or phone.

Link to comment

I don't like the hot spare concept. See THIS POST.

 

In summary, I'd rather unRAID do as little as possible and leave it to me (the human) to figure out the right next step. So much of the time a red ball is due to some cause unrelated to a failing disk, and automatically trying to rebuild could easily make things worse.

I think reprotect is a bad idea for the same reason. That is what rebuilding is for. If for some reason you intend to not rebuild and instead shrink the array, then maybe it makes sense to move data. I would not object to a general-purpose file manager in the webGUI that would allow the user to do the move in this case, and that would also handle rebalance. Midnight Commander can handle this pretty well as is.

 

For my information, are either of you arguing that these - as features - shouldn't be available?

 

I'm actually about to build a many (23ish) drive flash backup system around unraid, and email notification is currently my big hangup.  I am the sole systems administrator for my company, and as such I'm not available 24 hours a day and 365 days a year as much as I'd like to be.

 

I personally would love to have reprotect or hot spare available, and configurable.  If I'm out of town for a 3 day weekend, I'd rather have reprotect or rebuild happen in my absence than wait for my own personal intervention. 

 

The opposite holds true for my home system.  If it goes bad while the house sitter is watching the cats and she can't watch anything on it, she can use youtube.

 

Ballsy. As much as I love UnRAID I would not personally be deploying it into a corporate environment. Any product that only has support by proxy is not business ready.

I have a weird set of requirements.  Basically, it was do a synology using parts that we already have.

Link to comment

Ballsy. As much as I love UnRAID I would not personally be deploying it into a corporate environment. Any product that only has support by proxy is not business ready.

 

You can get email support any time.  We don't even ask if you are a "customer" (yet).  Depending on the issue, if it makes sense, we use other methods, eg., google hangouts or phone.

 

Good to hear there are (growing) options. I can understand how you'd be interested in moving into the SMB space, or even some mid-market. However, most companies are going to want proper phone support (at least in my experience). Forum/email support is fine for non-critical issues, but in an outage situation at 2am you usually don't want to be sitting in front of Outlook clicking Send/Receive repeatedly waiting for a response with your boss over your shoulder. :)

 

 

Link to comment

I don't like the hot spare concept. See THIS POST.

 

In summary, I'd rather unRAID do as little as possible and leave it to me (the human) to figure out the right next step. So much of the time a red ball is due to some cause unrelated to a failing disk, and automatically trying to rebuild could easily make things worse.

I think reprotect is a bad idea for the same reason. That is what rebuilding is for. If for some reason you intend to not rebuild and instead shrink the array, then maybe it makes sense to move data. I would not object to a general-purpose file manager in the webGUI that would allow the user to do the move in this case, and that would also handle rebalance. Midnight Commander can handle this pretty well as is.

 

For my information, are either of you arguing that these - as features - shouldn't be available?

 

I'm actually about to build a many (23ish) drive flash backup system around unraid, and email notification is currently my big hangup.  I am the sole systems administrator for my company, and as such I'm not available 24 hours a day and 365 days a year as much as I'd like to be.

 

I personally would love to have reprotect or hot spare available, and configurable.  If I'm out of town for a 3 day weekend, I'd rather have reprotect or rebuild happen in my absence than wait for my own personal intervention. 

 

The opposite holds true for my home system.  If it goes bad while the house sitter is watching the cats and she can't watch anything on it, she can use youtube.

Don't get me wrong, I am totally for have all features available for use.  I just found for most home systems, most people wouldn't wast a drive for a hot spare since it would get used very often, and when it was would leave a 10+ hour time of no protection. 

 

For a business environment, I would definitely want hot sparing as well as want multiple parity drives.  Of course that being said, my company runs EMC, which is a pretty big name in storage, and we have had 2 outages using Raid 6 with 2 drive failures on the same array when the dial home didn't function correctly, so notification is definitely a key requirement...

 

A friend of mine was a big Synology guy, since checking out my unraid setup, is dismantling it and scavenging the drives out for a unRaid setup.. I know several people at work that use Drobo and Rocket Raid and have had device failures and lost some data...

Link to comment

Don't get me wrong, I am totally for have all features available for use.  I just found for most home systems, most people wouldn't wast a drive for a hot spare since it would get used very often, and when it was would leave a 10+ hour time of no protection. 

 

For a business environment, I would definitely want hot sparing as well as want multiple parity drives.  Of course that being said, my company runs EMC, which is a pretty big name in storage, and we have had 2 outages using Raid 6 with 2 drive failures on the same array when the dial home didn't function correctly, so notification is definitely a key requirement...

 

A friend of mine was a big Synology guy, since checking out my unraid setup, is dismantling it and scavenging the drives out for a unRaid setup.. I know several people at work that use Drobo and Rocket Raid and have had device failures and lost some data...

 

Most of the time I build boxes that are just a bunch of drives in a raid 6, and nothing much on top of that.  In this case, there was a demand to be able to have it manageable without a shell by the support department.  I have a several tonnes of hardware (yes, literally, not figuratively), so it would have been silly to buy a system when I had a 2 year old 24 bay system that had been retired from it's previous purpose, along with a sledload of SSDs that were deemed to be wrong for the product, but perfectly good otherwise.

 

That said, I need email notifications.  If I have a drive fail, I need to know to log into the web interface, shut the array, take one of the spare drives in the system, and rebuild with it ASAP.  I can recover the 'bad' drive later.  I would be happier if that part would happen automatically if I didn't address in 2 hours, but that's more wishlist than anything else.

Link to comment

Two things to ponder:

 

I think quite a few users will already have a spare sitting there for emergency repair or future expansion. It might not be the norm but it wont be uncommon. I do.. always have.

 

Arguing that feature x>y or vice versa is pointless when x+y is always great than x or y.

 

Sure we can debate which would better to have first but it is better to let the devs work on anything than waste time debating or reviewing here.

 

 

 

 

Link to comment
I think quite a few users will already have a spare sitting there for emergency repair or future expansion. It might not be the norm but it wont be uncommon. I do.. always have.

I would say it should be a requirement if you can't turn your server OFF until you can get a drive delivered and run some preclears on it with additional down time if you have to RMA the drive because it is DOA.
Link to comment

I think quite a few users will already have a spare sitting there for emergency repair or future expansion. It might not be the norm but it wont be uncommon. I do.. always have.

Since Best Buy is only about 5 minutes away, I usually just run down and pickup one up if I start seeing errors and replace it.  I am usually to gready for space to let one site idle... :)

 

Arguing that feature x>y or vice versa is pointless when x+y is always great than x or y.

 

Sure we can debate which would better to have first but it is better to let the devs work on anything than waste time debating or reviewing here.

I was arguing about what features should be included or not, just a discussion on how others users use unRaid. 

 

This is probably not the correct thread for that discussions, I apologize...

 

Link to comment

Don't apologize its all good. My choice of the work "Argue" was wrong. Debate is always good.

 

For what it is worth I used to do a similar thing myself with buy on demand but I had one DOA that caused me grief so now i preclear in advance and store.

 

Each to their own the more flexibility the better.

Link to comment

24 drive machines have always seemed risky to many of us, with the current 1 array model, and only one parity drive.  So you can understand the concern with the idea that a hot spare will be the answer to any disabled drive issue, especially with so many drives to abort the automatic rebuild.  I suspect many of us have had a rebuild failure at some point, not often, but common enough with our average size arrays to be concerned about greatly increasing the drive count.  I wish we knew the numbers on how risky it is to rebuild a drive in a 24 drive array, but the failure rate is certainly more than 0% and almost surely higher than 1%.  Is it as much as 20% or 30%?  Probably not, but we don't know.

 

Dual parity will help a lot, but it is going to come with a performance hit.  How much we don't know yet, but I predict some of the enthusiasm for the advantages of dual parity will wear off quickly, once the performance hit is known.  I do hope it's small, and that ways to overlap the increased I/O are found, to minimize the hit.

 

Long ago, I suggested a model for multiple arrays, and I'd like to push it a bit again.  If UnRAID supported multiple arrays, of differing types, it would be possible to run 3 arrays in that 24 bay machine, each with its own parity drive and 6 or 7 data drives.  Rebuilds would be much safer, and a little quicker, because any drive failure is isolated to a single smaller array, won't affect the rest of the drives.  Parity checks would be a little quicker, fewer drives per array.  Multiple drive failures can still happen, but would be less likely to cause data loss.  Multiple arrays give you multiple parity drives with some (but not all) of the advantages of dual parity, but without the performance loss.  Multiple arrays will certainly require significant development hours, but once done, opens a variety of advantages.

Link to comment

Two things to ponder:

 

I think quite a few users will already have a spare sitting there for emergency repair or future expansion. It might not be the norm but it wont be uncommon. I do.. always have.

 

Something else to ponder - if you have a drive that's supposed to be a spare, powered up and running in RAID 6 type arrangement, you are eating into it's life and opening it to any power spikes that might take out a working data disk.

 

So if 2x parity drives are supposed to protect you against simultaneous drive failures - well the highest probability of that happening is a power spike - in which case haven't gained much and could lose the spare as well.

 

By all means have a cold spare ready and waiting - but if you do want to use it at all I'd suggest it's better off as an offline backup of key files - or better still and offline backup of key files inside a firesafe.

Link to comment

24 drive machines have always seemed risky to many of us, with the current 1 array model, and only one parity drive.  So you can understand the concern with the idea that a hot spare will be the answer to any disabled drive issue, especially with so many drives to abort the automatic rebuild.  I suspect many of us have had a rebuild failure at some point, not often, but common enough with our average size arrays to be concerned about greatly increasing the drive count.  I wish we knew the numbers on how risky it is to rebuild a drive in a 24 drive array, but the failure rate is certainly more than 0% and almost surely higher than 1%.  Is it as much as 20% or 30%?  Probably not, but we don't know.

 

Dual parity will help a lot, but it is going to come with a performance hit.  How much we don't know yet, but I predict some of the enthusiasm for the advantages of dual parity will wear off quickly, once the performance hit is known.  I do hope it's small, and that ways to overlap the increased I/O are found, to minimize the hit.

 

Long ago, I suggested a model for multiple arrays, and I'd like to push it a bit again.  If UnRAID supported multiple arrays, of differing types, it would be possible to run 3 arrays in that 24 bay machine, each with its own parity drive and 6 or 7 data drives.  Rebuilds would be much safer, and a little quicker, because any drive failure is isolated to a single smaller array, won't affect the rest of the drives.  Parity checks would be a little quicker, fewer drives per array.  Multiple drive failures can still happen, but would be less likely to cause data loss.  Multiple arrays give you multiple parity drives with some (but not all) of the advantages of dual parity, but without the performance loss.  Multiple arrays will certainly require significant development hours, but once done, opens a variety of advantages.

 

I think once you get up to that many drives in a single system, you are probably stretching things a bit on even just the system capabilities on most of the commodity hardware.

 

I like the idea of multiple arrays.  What would be nice is to have multiple protection groups each with a parity disk, and then still have them seen as a single filesystem.

Link to comment

Dual parity will help a lot, but it is going to come with a performance hit.  How much we don't know yet, but I predict some of the enthusiasm for the advantages of dual parity will wear off quickly, once the performance hit is known.  I do hope it's small, and that ways to overlap the increased I/O are found, to minimize the hit.

 

Long ago, I suggested a model for multiple arrays, and I'd like to push it a bit again.  If UnRAID supported multiple arrays, of differing types, it would be possible to run 3 arrays in that 24 bay machine, each with its own parity drive and 6 or 7 data drives.  Rebuilds would be much safer, and a little quicker, because any drive failure is isolated to a single smaller array, won't affect the rest of the drives.  Parity checks would be a little quicker, fewer drives per array.  Multiple drive failures can still happen, but would be less likely to cause data loss.  Multiple arrays give you multiple parity drives with some (but not all) of the advantages of dual parity, but without the performance loss.  Multiple arrays will certainly require significant development hours, but once done, opens a variety of advantages.

 

Certainly any performance hit for dual parity will not bother me personally,  I only ever write to cache.

 

I am also very interested in the ability to run multiple arrays with their own parity in the same box with user shares spanning them. 

Link to comment

 

Certainly any performance hit for dual parity will not bother me personally,  I only ever write to cache.

 

I am also very interested in the ability to run multiple arrays with their own parity in the same box with user shares spanning them.

 

I would also be interested in like a master unraid to access the filesystems of all slave unraids ... in or out of the same box

aka one user share for all boxes...

Link to comment

Well the reason I keep a spare is not because of the time it takes for me to go to the store and buy a new drive. The reason is the time it takes to preclear that drive. I just precleared a 3tb and each cycle took 53 hours. Multiply that by 3 for 3 cycles and you're looking at more than a week of downtime. That's why I always keep a precleared spare in the array and it gets spun down

 

Link to comment

I probably should do that.  No the best practice or safest, but I figure the rebuild is going to write to every sector of the drive, so that should somewhat test the drive and I monitor the syslog and smart report during the build incase I see any errors happen, I can abort the rebuild.  When I was running raid controllers, there was no real way to test the drives other than do a rebuild, so I just kept doing that.  It has worked fine for the past couple of drive failures under Unraid, but I am sure once I get burnt I will switch.. :)

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.