First off... you guys rock.  I'm currently floating around the ocean on a Navy Ship and internet is crawling, so thanks to Tom for activating my account for this post!  I'm amazed at community support and I plan on getting my hand dirty here in the next few weeks.

I'm preparing to build an unRAID box and was explaining the benefits of it to a friend of mine.  Based on some of the questions he asked, I think I've developed an idea that would use fairly simple scripting and yet provide unRAID with an AMAZING new feature: Self-Heal

 

First, I've seen some confusion on definitions, so here are the ones I will use in this post. If they catch on elsewhere, all the better!

 

+RESET Parity - Recalculates Parity (Maybe just label the button “Recalculate” so its clear). This is the “This is how it's going to stay” button.

+REBUILD Array - (restore) - replace missing drive and “rebuild” the array with that drive

+HEAL the array - explanation below

 

A few more for discussion purposes:

+SimDrive - Simulated Drive or Parity Driven Drive (since the data is available and you can even write to the “missing drive” or failed drive, this is my reference to it.

+SimData - Simulated or Synthesized missing data that is calculated on the fly via Parity

 

The HEAL feature:

If the amount of data on the lost drive is less than the available free space on the remaining drives, the system can “heal.” The main reason this works is because of the fact that even with a failed drive, you can access the data (which is synthesized or parity driven) and treat it like it still exists.  The steps are as follows (though step 2 isn't currently available)

 

1)Detects failed drive (now synthesized data or SynthData)

2)Looks for Hot/Warm Spare and if available it Rebuilds resulting in normal array.  If not found, continue to step three.

3)Copies the SimData to the free space of the remaining drives.

4)Deletes synthesized data from SimDrive (maybe just shorten to “mv” command to combine these two steps)

5)Removes Failed drive from the array

6)Reset parity

 

Now you are back to a “Normal Array” configuration minus the capacity of your failed drive.  If you replace the drive, it would be like any normal “expansion” process.

 

Example assuming same sized drives:

3 Drives Total (2D,1P) - 45% (you have 55% worth of free space on working drive and 45% worth of data  on failed drive equaling 90% capacity used once data is copied over)

 

Assuming same sized drives, xT=Total Drives, xP=parity drive, =data drives, and Percent=how full the array can be and still heal.  The short is this: the more drives you have, the fuller the array can be and still heal.  The formula is (100/nD)*(nD-1)=%full (where nD is number of data drives). So you could also say (100/(n-1))*(n-2)=%full where n=total drives. Here's how it works out.

 

Highlights of a few sizes:

Drives Percent

3T (2D+1P) 50.0%

4T (3D+1P) 66.6%

5T (4D+1P) 75.0%

6T (5D+1P) 80.0%

11T (10D+1P) 90

20T (19D+1P) 94.7

…

101T (100D+1P) 99%

 

Now, if that works, and it should since it's just using basic command line programs like cp or mv, you can extend it to Multi Heal.

 

MULTI-HEAL: One drive fails and the system heals. Then another drive fails and the system heals. This continues as long as space is available.  Allowable percentages are according this formula:  (100/nD)*(nD-f)=%full (where nD=number of data drives; f=number of failed drives) or  (100/(n-1))*(n-1-f)=%full (where n=number of total drives). Assuming all drives are the same size, here are some examples:

Drives Failures %Full

3T (2D+1P) 1 50.0%

4T (3D+1P) 1 66.6%

4T (3D+1P) 2 33.3%

5T (4D+1P) 1 75%

5T (4D+1P) 2 50%

5T (4D+1P) 3 25%

6T (5D+1P) 1 80.0%

6T (5D+1P) 2 60.0%

6T (5D+1P) 3 40.0%

6T (5D+1P) 4 20.0%

20T (19D+1P) 1 99%

20T (19D+1P) 3 84.2%

20T (19D+1P) 5 73.7%

20T (19D+1P) 10 47.3%

20T (19D+1P) 13 31.6%

20T (19D+1P) 15 21%

20T (19D+1P) 18 5.3%

 

Bottom Lines:

- You can have (-1) drives fail when your array is at (100/xD)% full.

- Every drive added allows for another failure, albeit at a lower fill percentage.

 

Other Applications:

 

SMALLER DRIVE REPLACEMENT: Heal can help if you have to replace a drive with a smaller.  Let's say you lose a 2TB drive that is 75% full (1.5TB) and your array is so full it can't fully HEAL (maybe only 1GB of free space).  All you have is a spare 1GB laying around so you insert it, add to the array, and then HEAL. Presto! You just replaced a drive in the array with a smaller drive, which is allowed so long as the %Free+Drive>=SimData

 

PARTIAL HEAL:  If the amount of data is more than free space, it moves as much as possible.  Remaining data can be sent to the cache drive or possibly to a shared folder on the network (preconfigured).  Once it's moved, Reset/Calucate Parity and you've got a normal array with the only unprotected data on the cache drive (it would automatically move off as soon as you add a drive like normal) or on the network (but hey, at least you have ONE copy - or set it to move it to more than one location!!!)

 

DATA LOCATION INTEGRITY: If all of the SimData can be moved to one other single drive, it will.  Example: you have 500GB of SimData and one of your 2TB drives is only half full... since there is enough room, it will move it all to the one drive in an effort to keep the information organized in a single folder (maybe even relabel the folder though this might cause problems with links - the main drawback).  The data is covered under parity as written to the drive and it's all in one neat folder, assuming available space)

 

CONFIGURATION:

Have three simple options:

1)Heal Now - Manual

2)Auto Heal - begins heal as soon as failed drive is detected - send alert

3)Delayed Heal - Send alert - if no intervention in ___ Hours, then Auto Heal

 

You would also need to set up the emergency network folder (Amazon S3, anyone?) in a config file.

Email config needs to be in a file, also.

 

SO..... that's what I came up with last night in a couple of hours.  Seems simple enough for most of the implementation (we're talking cp, mv, etc. for most of the work).  In fact, someone could manually move the data and shrink their array, so unless people point out some serious flaw, I may take the time to write a howto.  But this is all new to me and still theory so I'm hoping some heavy hitters will chime in.

 

Comments, please!

KermitJr

Dude, that would be awesome but since I'm not a rocket scientist nor a Navy Captain for that matter, I'll wait to hear from some of the Linux gurus that hang around here before I get exited.

 

Take care man and come back home safe.

I'm definitely no Rocket Scientist (or Captain for that matter!!!) but the theory seems to be doable manually now, and if it can be done manually, it's scriptable.  This doesn't seem too hard for basics, either.

 

Too bad I'm not a programmer, either! 

 

Cheers,

KermitJr

That sound interresting.

I would like to see something like that.

It definitely could help to protect the data before you can buy a new HD.

 

As you said : if it can be done manualy, it can be scripted.

That sound interresting.

I would like to see something like that.

It definitely could help to protect the data before you can buy a new HD.

 

As you said : if it can be done manualy, it can be scripted.

I'll be the first to applaud improved protection of our data, but....

in all the assistance I've helped with unRAID users, there have been absolutely no situations where recovery could have been scripted. 

 

There are just too many variables... in fact, in some situations the absolute worst thing to do is risk ANY writes to the array as to change parity.

 

Yes, you can copy critical files from a failed "virtual disk" to a physical disk, but if they are really critical, you should have "backups" elsewhere anyway.

Any attempt to rebuild parity without the failed drive leaves you without parity protection for hours.  In some cases moving all the data off of a failed drive and then rebuilding parity without that drive in the array can work.  I've done that myself once, when I was unable to replace a "failed" drive...  this might be the best solution if you are unable to replace a drive in a reasonable amount of time ( a few days)

 

In my case, it was not a drive that failed, but several drives... and it was not the disks themselves, but the "y" power splitter that fed them that was defective.  I copied all the data off the failing drives and rebuilt parity without them until I could isolate the true cause. 

 

If you can script the recovery process... many will love you... Personally... I think it is too complex to script, with far too many variables to consider to constantly protect the data.  The best we can do is an early alert to any failure, and when implemented, a "raid6" like checksum to allow recovery from multiple failed devices.

 

Until then, the best we can do is keeping really critical data on multiple disks... and keeping backups... elsewhere...

 

Good luck...  and get home safely... 

 

Joe L.

I'm in agreement with Joe. I would not want this healing process to be automated unless there was a hot spare feature in unRAID.

 

Parts of this can be scripted. Such as moving data from the virtual simulated drive to other drives.

If the userfs could detect that one of the md devices is being virtualized, then a scripted rsync --remove-sent-files could be used to rsync and remove the files from the virtual drive.

In fact this is what I did the other day when a drive failed.

After that I made a decision to recalculate parity on the array without the failed drive.

The issue with this healing process by moving current data is moving the current data to the array via some selection mechanism which leaves out the virtual drive. I think the usershare filesystem would need some improvements to assist with this.

 

 

I still feel there are more features needed at the MD device level before automated healing could be done.

The healing as mentioned can be an application.

The removal or addition of drives to the parity protected array could use some refinement.

I.E. the ability to add or remove a single drive to the parity drive would make additions and deletions to the array so much faster.

I see many statements about the time required to create parity.

 

In my mind that time would drive drastically if a process could read the current parity and fold in the new drive.

It's already been proven that if you write 0's to a single drive, you fold out (XOR out) the parity for that drive and then the parity can be relied upon with a specific command.

 

Yes, you can copy critical files from a failed "virtual disk" to a physical disk, but if they are really critical, you should have "backups" elsewhere anyway.

I agree completely... backups are a must!

Any attempt to rebuild parity without the failed drive leaves you without parity protection for hours. 

Exactly.  But if your drive is failed... you're sorta without parity protection since you're only one drive away from failure on both drives. (the original lost drive and then the second drive assuming it's not the parity drive).  So some action in this case is better than no action (yes, I know that could be argued against!)

In some cases moving all the data off of a failed drive and then rebuilding parity without that drive in the array can work.  I've done that myself once, when I was unable to replace a "failed" drive...   this might be the best solution if you are unable to replace a drive in a reasonable amount of time (a few days).

This was more along the lines of what I was thinking... when you can't replace the drive right away (like you're waiting for a delivery -  or maybe you can simply do permanently with one less drive) which is a likely scenario for a typical home user (movie collections, etc.). 

If you can script the recovery process... many will love you... Personally... I think it is too complex to script, with far too many variables to consider to constantly protect the data.  The best we can do is an early alert to any failure, and when implemented, a "raid6" like checksum to allow recovery from multiple failed devices.

That's why I had mentioned a basic one to start with (even if it simply copied the SimData to the other disks in the array immediately). So  1)Detect Failure => 2)Copy data to empty drive space.  Even then, if you lose parity, you still have all the data.

 

I also think it's vital that you have the "auto" process be an option since many might not like it.

Good luck...  and get home safely...   

Thanks.  Any day now!

KermitJr

 

very interesting idea; looking forward to updates

Worthy goal but I have to agree with Joe and WeeboTech. Knowing when a drive fails is tricky. Not like a light bulb. And when it appears to happen, the real cause could be a bad cable, bad memory stick, or other HW problem. Your auto-recovery attempt could make things worse. Given the value and size of the data on a big array, even a small risk of data loss is too high. If you read back through the forums over the past year or two and the varied and complex diagnostic and recovery techniques employed in real-world failures, you'd understand better where we "old timers" are coming from.

 

If you need a simple example of how this could make things worse here is one. Suppose a cooling fan in the computer fails causing an always active drive's temp to rise and triggering what looks like a drive failure to your program. Your copying and parity build actions will spin up all the drives and raise the temps even higher, perhaps inducung more drive problems and data loss. I believe unRAID needs to inform the user and let him (or her) figure out how to recover.

That's why I had mentioned a basic one to start with (even if it simply copied the SimData to the other disks in the array immediately). So  1)Detect Failure => 2)Copy data to empty drive space.  Even then, if you lose parity, you still have all the data.

It is not that easy.... or as clean cut.

 

First, there is no such thing as "immediately" copy data to another disk  The reason is simple, to copy data it must be read from all the remaining disks and re-constructed.    On my array it could easily take 16 hours or more.  With most arrays today having 1TB drives, "immediately" is more like 6 to 10 hours, or more.

 

Drives are marked as "bad" if they are unable to be written to...  The process of copying the "simulated" data of the first failed drive to another could result in a "write" failure, and have the second drive marked as failed... losing the contents of both.

 

If you have copied the data from a "simulated" drive to a working drive it does not prevent you from losing that data...  not unless you drop parity protection and then re-calculate parity without the failed drive.  That process of re-computing parity will take many hours, during that process you have no parity protection from any subsequent data drive failure.  To see this, let's say you have a 3 drive array... 1 parity, 2 data.    1 data drive fails, and using parity you copy its simulated data to the remaining data drive.  Now, the second data drive fails... before you could replace and rebuild the first data drive...  Guess what... all your data is gone.  It did not matter that it was duplicated.

 

The addition of "raid6" with a second layer of parity protection will be able to recover from a 2 drive failure.  It comes with the cost of even slower "writes" to the array... but your data is more secure...  (You can choose... performance vs. data safety)  Many will love it when it is eventually added to unRAID, others will never use it as it will slow things down even more. 

 

As I said earlier... your goals are great... but in any implementation, it is as likely to cause more damage than not.  When an array is having problems, unlike corporate servers, many home users can simply shut down until a replacement disk can be purchased.   

 

Joe L.

First, RAID6 is a better, standardized, well documented way of protecting against 2 simultaneous failures.  That's more important.

 

Second, unRAID does need better overall handling of failures -- i.e. notice (lots of notice) and better recovery (i.e. auto or semi-auto recovery, hot spare, converting to a read-only mode, etc.)

 

I'd like to see RAID6 FIRST. 

 

Then a hot-spare concept, that is a drive fails, unRAID makes that "virtual" drive (now failed and being simulated from parity) read-only while the contents of the "physical" failed drive are copied to the hot spare, using parity to reconstruct what can't be read, and then either wait for the operator to approve the swap, or optionally auto-swap the hot spare into the array while swapping the failed drive out, and then enabling writing on the replacement drive in the array.

 

I do NOT like rebuilding a complete drive from parity, when possibly only one spot is bad. If you get the dreaded second drive failure during the rebuild process, you are hosed.

Good points all.

 

as I mentioned, and added a new thread in the features forum.

having user shares smart enough to avoid writing to a disabled disk provides a step to saving your data in failed drive situation.

I.E. moving data from the virtualized drive to other real drives.

 

Sort of like the mover job does with the cache drive, only the user share avoids using the missing drive automatically.

 

I personally would not want the movement of data to occur on it's own.

A button or command would do it.

 

I'll mention it again, having a spare drive on hand for this kind of scenario is important if your data is important.

I have a special external SATA/USB trayless box specifically for this use. it's saved me a few times.

 

 

First, RAID6 is a better, standardized, well documented way of protecting against 2 simultaneous failures.  That's more important.

 

Second, unRAID does need better overall handling of failures -- i.e. notice (lots of notice) and better recovery (i.e. auto or semi-auto recovery, hot spare, converting to a read-only mode, etc.)

 

Agreed. I think we can add some notification jobs. I've been playing with ssmtp and mail with some good success.

As far as hot spare. I like it, Read only.. I think this should only be at the usershare level when it uses the fill (split) algorithms.

I would not want the filesystem itself to be readonly, Just changing the allocation of files with usershares so files can be moved easier.

I think this is a good and do-able compromise for the future.

 

I'd like to see RAID6 FIRST. 

 

Then a hot-spare concept....

 

Agreed++;

 

Writing to an array with a failed disk is risky.... I'd like to see a way to OPTIONALLY prevent it.

 

Alternatively (and better) is the ability to take an array that has had multiple drive failures offline, and do some sort of "offline recovery" that would not disable a disk when there is an error, but keep going... so if one sector is bad on disk 7, it is rebuilt from parity from the others, and then if a different sector is bad on disk 2, it could be rebuilt since that sector on disk 7 is still OK.

 

So this needs to all be done offline:

- Do a scan of the entire system, to ID the bad sectors (and possible the file names involved too).

- You end up with n bad drives.

- Then you put n new drives in the system of proper size

- Unraid rebuilds the n failed drives

- Failed drives are swapped for their rebuilt siblings.

 

Then array can be brought online.

 

If the same sector is bad on 2 or more drives, the scan can ID the file involved.... so the user can at least know what was unrecoverable.

Because this is an offline scan, it can be a stand-alone program, and doesn't have to involve the inRAID core.

Writing to an array with a failed disk is risky.... I'd like to see a way to OPTIONALLY prevent it.

 

This is feasible manually with a mount -o rw /dev/mdx /mnt/diskx

What we probably need is a front end feature.

 

 

Alternatively (and better).....

 

It sounds like a good idea, however given the amount of time to develop this type of feature and/or tool, I feel the investment might be better towards RAID6.

 

 

Yes, time would be much better spent on RAID6.

 

But the off-line recovery could be developed by someone other than Tom.... it does not have to be part of the unRAID kernel.  While it would have to read and modify the unRAID config on the flash, and possibly update the superblock, it would NOT have to be part of the unRIAD kernel as long as it was being run on an off-line array.