A little background:

 

S.M.A.R.T. enabled drives lie a little about their capacity.  Instead of making all of their sectors available to you for data, they keep some sectors in "reserve".  If a bad sector is detected, S.M.A.R.T. has the ability to take it out of service and substitute (remap) one of its good "reserve" sectors to its exact sector number.  The OS is none the wiser.  Because of this, S.M.A.R.T. drives should never have "bad sectors" at the OS level.  The remapping allows them to make the OS think it is a 100% error free disk, whilte the drive keeps its known bad sectors inaccessible.

 

Spinrite is a data recovery / disk maintenance tool.  It will read an entire disk looking for bad sectors, and if it finds one, it will try very hard to read the bad sector.  Once its done its best, it tells S.M.A.R.T. to remap the sector, and then it writes the sector contents (it's best guess) back to the disk.  (That's an over simplication, but basically that's what it does).  I heartily recommend this tool (or a similar one) to unRAID users.  (My guess is that a "recertified drive" is one that has had a few sectores remapped!)

 

Feature Request:

 

When unRAID hits a read error, it uses the redundancy of the array to reconstruct the sector and increments an error count.  My suggestion is that it should also tell S.M.A.R.T. to remap the bad sector, and write the reconstructed data back to the remapped sector.  In this way unRAID would actually CORRECT errors, and not just notice them.  unRAID has a huge advantage over Spinrite - it can reconstuct the sector.  Sprinrite has a hard time trying to read bad sectors (and it usually gets a few bytes wrong, a disaster if its in a .ZIP file).

 

Similarly, when unRAID hits a write error, instead of taking the entire disk offline - it should tell S.M.A.R.T. to remap a sector and then retry the write operation.

 

Certainly logic would have to get added to detect true drive failures, so if an excessive number of remappings are done in a row, it should give up and take the drive down.

 

If this feature were implemented, unRAID would become a self-correcting machine.  Errors would just get corrected, instead of the current method having to realize a drive has been taken offline, remove the drive, run Spinrite on the drive (or replace the drive), and then have unRAID rebuild the drive contents (risking a read error during the intensive rebuild process).

 

My experience with many years as a hobbyist and IT professional, disks usually don't fail unless you drop them or abuse them.  It is MUCH more common for a few bad sectors to develop over time.  I have 3 Tivos, and have abused 6 300G drives for over 3 years with continuous high temperature operation (Tivo CONSTANTLY writres 2 video/audio streams, and Tivo cases are brutally hot and have no good options to add cooling).  Even these drives haven't failed - but they have had a few sectors remapped with Spinrite.

 

Tom, if you're reading this, please consider adding this feature.  unRAID would become the first S.M.A.R.T. Aware File System.

 

Cheers!

 

- Brian

 

i had no idea SMART worked that way  and i think it would be a great feature for unraid

that said, i dont think this will be an easy one to implement (ill leave that up to the true linux guys)

 

you got my thumbs up

I had read somewhere that programs themselves do not re-assign sectors, but rather the hardware does.

What I read was that by writing all zeros to a sector the S.M.A.R.T firmware will reassign sectors on a failed write.

 

So perhaps a few writes of a zero sector during a failure may work enough to trigger a re-map.

 

SMART drives do this on their own.  If unRAID sees an error reported by the disk, then the disk's built-in error correction has already failed to correct the problem... or it took so long it timed out (which is where TLER drives come in).  It that case, I'd want the drive failed, so I can pull it, diagnose/test it, and reformat/reinstall it.

 

Remember, all the data is intact in the Reiser FS ... it is not striped.  So if the drive has a noncatostraphic failure as you describe, you can pull the data off.

 

 

I have never seen a S.M.A.R.T. drive remap a bad sector on its own.  The only way I know to do it is run Spinrite.  I believe (someone correct me if this isn't correct) but if a WRITE fails, S.M.A.R.T. may remap the sector automatically - but writes don't usually fail.  It's the READS that fail - and I don't think there is auto remapping for reads.

 

I did a stupid thing and wound up hitting the big red switch on the array in a hurry.  When I powered back up, one of my disks had a sector error, which interfered with recovering the data on the disk.  It would not just ignore the error and go on.  I ran Spinrite and (in its own sweet time) it remapped the bad sector.  The reiserfsck recovery tool was then able to run.

 

I think that it would be a useful feature for unRAID to perform some Spinrite-like type functionality in real time when it encounters disk errors, and have a dramatic affect on array reliability and availability.

 

Brian

SMART is all about predictive action.  SMART drives will reallocate a sector when read error thresholds are exceeded.  Look at any SMART drive and read the "Raw Read Error Rate" and see it does monitor read errors and any technical white paper on SMART will tell you that reallocation will occur in anticipation of failure predicted by read errors.. 

Drives have done dynamic sector re-mapping for many years... All SMART does is provide an interface so the host can find out how many have been remapped.

 

SirWired

A few years ago I lost a very small file that had some very important information in it.  I kept getting a read error.  I tried to copy the file to another disk for several hours - over and over, attempt after attempt, hoping it would read just once - it got error after error.  That's when I bought Spinrite.  It found the error, did its best to read the sector, invoked the drive's remapping, and allowed me to recover most of the contents of the file - only a few bytes were messed up, which wasn't a big deal. 

 

I continue to have very occasional bad sectors crop up on disks from time to time.  They do not take care of themselves - and they shouldn't (see below).  Spinrite has an interface where it reports whether it has found bad sectors.  Depending on the disk format, it will even tell you what file the error occurred in - so at least you know what file might be corrupt.

 

If drives did the remapping automatically - suddenly files that used to be fine would have corrupted data with no hint as to why.  Obvously the drive hardware has no way to tell the end-user!  Also, tools like SpinRite WOULDN"T WORK, because the physical sector they'd be trying to read would have been remapped and therefore be hidden by the drive.  Drives don't and shouldn't take a sector out of service based on a read error.  That should be left to array managers (like RAID / unRAID control systems), who are able to remap the data ACCURATELY.

 

I read about the TLER drive feature based on bubbaQ's post .  It seems that some enterprise drives actually take so long trying to recover from an error that RAID arrays give up.  Thinking the drive has failed it takes it out of service.  TLER drives limit the duration of error recovery attempts and report errors back to the RAID array.  RAID arrays have error recovery features that can then be employed to prevent having to take the drive out of service!  I'm not sure what those features are - but would not be surprised if it is this type of sector remapping I'm advocating for unRAID!

 

 

 

 

Just because you see errors and sectors going bad does not mean SMART is not remapping sectors for you.  If the drive fails to get the data and have it pass ECC, then you will get an error.  That is an example of SMART not predicting that error and remapping the sector proactively. 

 

If you get errors that are corrected by ECC or retries, you will NOT get a report of that error, but SMART looks at the pattern, and will, if it meets the algorithm's thresholds, remap the sector and rewrte the data to the remapped sector.  This does happen, and is completely transparent to you.

 

Take a drive and print out the remapped sector list.  Then use it hard for a couple of years and look again.  You will have more remapped sectors, without ever having an error reported to you or running Spinrite or other utility.

 

Please do your research and don't just rely on your own anecdotal experience.

Please do your research and don't just rely on your own anecdotal experience.

 

If I followed your advice I'd never post anything!

 

 

Posting our experiences is fine, no matter how little they represent a statistically significant sample, but I agree that we all need to be careful that we don't position such experiences as universal truth.

 

 

Bill

Ouch.  That SMARTS. 

 

I really wasn't trying to say that drives don't do what they are supposed to do relative to sector remapping, just that despite their best efforts read errors can and do still occur. 

 

My main point is that unRAID is in a unique position to help the drive to an even better job.  The drive has no way to reconstruct a bad sector - but unRAID does.  If unRAID helped out when read errors occurred, it could instruct the drive to remap the bad sector and write the reconstructed contents to that sector.  A drive could never do this on its own.  IMHO, implementing this feature would help unRAID arrays stay substantially healthier and require less user intervention than the current method.

 

Sorry if I didn't do as good a job as I should have separating fact from conjecture.  Hope no offense was taken.  None was intended.

 

- Brian

bjp999,

 

As Bubba said, the drive is often remapping bad sectors for you; they have done this even before SMART existed.  (I believe the drive in my 286 did this...)  Luckily for you, the error-correcting codes in the drive are pretty darn close to magic, and you never notice most of those errors.  However, the frequency of these errors is an indication of impending doom, and SMART exists so you, the user, have a way of monitoring the errors which you ordinarily would never see.

 

The functionality you are looking for is called "background scan", and it is implemented in most enterprise disk arrays.  They periodically sweep through every last sector on the array, and look for bad data.  If the data can be recovered by the drive alone, it is caught right then and there (as opposed to later on, when it might be worse), the sector is remapped by the drive, and the PFA (Predictive Failure Analysis) counts in the drive go up.

 

If the data cannot be recovered by the drive, then it does indeed do parity reconstruction right then, and write the correct data to a forced remap.

 

This sort of function would be for the folks that write the md driver, as opposed to Tom.  As best I can tell, unRAID takes the base md driver, and does some kind of evil "wrapping" to it to give you the functionality we all know and love.

 

SirWired

Thanks sirwired.  You and bubbaq know a lot about this - and  I am learning more and more as we dialog.

 

If I understand right, a drive is performing low level analysis on sectors it encounters and is able to predict sector failures before they happen.  When one is predicted, the drvie is able to remap the sector while the marginal sector is still readable - allowing it to 100% accurately remap the sector and avoiding a future failure.

 

But if it doesn't predict a certain failure, and a bad sector is encountered, then the build-in remapper can't do anything about it because it's too late.  If the algorithm allowed the sector to be remapped AFTER the failure, it could not ensure data accuracy, and would risk invisibly corrupting a piece of data in some random file.  Rather than that, the failed sector is allowed to continue to be "exposed" and reported as a bad sector.  Although not a great way, it is a way for the drive to tell someone it has a problem.  This signal allows a RAID or unRAID control system (or a tool like Spinrite) to see there was an error and take some corrective action.  In the case of RAID/unRAID it would increment an error counter and reconstruct the failed sector from the other drives in the array.  In the case of Spinrite, it can try to read the bad sector and then use the drives S.M.A.R.T. services to remap the sector directly.

 

Some RAID control systems run a "background scan", causing each and every sector on the disk to be read and therefore subjected to Predictive Failure Analysis (PFA).  If this isn't done, a marginal but seldomly accessed sector will get weaker and weaker and NEVER be detected.  Then, when you go to access that sector two years later, it will fail and you're in trouble.

 

Is that basically right?

 

Now unRAID doesn't do background scans, but via the parity check feature allows you do manually do the equivalent.  So running a parity check does more than just check to see if Tom is calculating parity correctly.  It causes each sector to run through the drive's PFA and automatic remapping features.  If this is right, it should be recommended TO EVERYONE, whether they use unRAID or not, that they periodically do a full disk scan to help keep their drives healthy.

 

And if ignorant people like me knew to do this, then perhaps the incidence of failed sectors as drives age would be cut substantially.

 

Thanks again!

 

Brian

If I understand right, a drive is performing low level analysis on sectors it encounters and is able to predict sector failures before they happen.  When one is predicted, the drvie is able to remap the sector while the marginal sector is still readable - allowing it to 100% accurately remap the sector and avoiding a future failure.

 

Correct.

 

But if it doesn't predict a certain failure, and a bad sector is encountered, then the build-in remapper can't do anything about it because it's too late.

 

Correct.  And this indicates a MUCH more serious type of failure as it is a sector that went very bad, very fast, with no prior warning.  That's why a RAID system in this situation usually fails the drive, rather than keep using it.  This drive needs some SERIOUS testing and diagnostics.  I don't want unRAID transparently remapping it and going along it's merry way.

I did some more reading, and thought I would post another piece of the S.M.A.R.T. puzzle.

 

As already described, if the drive detects a sector error when trying to do a WRITE, it will immediately remap the sector.  But if it detects a READ error, it will not.  But it will not forget that it got that read error either.  The very next time someone tries to WRITE to that sector, it will instantly remap the sector and THEN do the write.  Makes perfect sense from the drive's perspective.

 

Pretty S.M.A.R.T. 

Sector remapping is deceptively simple.  When there is a read error, the drive remembers the sector is bad.  When there is a subsequent write to that sector, the sector is immediately remapped and then the write occurs.  That's it.  Not so complicated or mysterious.

 

Adjusted Feature Request:

 

Feature request 1:

If parity is valid, when unRAID hits a read error, it already rebuilds the sector using the other drives and parity.  If, in addition, it would simply write that corrected sector back to the bad sector # on the offending drive, it would give the drive the opportunity to correct the error.  It could be done outside the file system - a direct I/O to the drive - parity logic does not need to "see" this event.

 

Feature request 2:

If parity is not intact, unRAID could try to determine whether the bad sector was NOT already in use by a file.  If it wasn't, it could rewrite the sector with any value without causing problems.

 

Feature request 3:

If the bad sector was part of a file, things get stickier.  If unRAID were to do something here, there should be a setting.  One option would be to do nothing - like it does today.  Another option would be for unRAID to write the sector containing a specific string repeated over and over (e.g., "REMAPPED SECTOR.  REMAPPED SECTOR.  REMAPPED SECTOR. ...") to the offending sector.  This would remap the sector and give the user the ability to search for the offending file. unRAID could put the filename in the syslog (or better yet into a new file on the boot device) to make it easier to find as well (if it can figure out the filename).

 

Feature 1 seems like a perfect solution, analagous to what RAID arrays do.  I think it should be implemented.

 

Feature 2 might be hard, since unRAID would have to figure out whether a specific sector is part of a file or not.  If that is easy, though, this would be a nice feature with no risk.

 

Feature 3 is imperfect.  If Feature 2 is hard to implement, feature 3 could be done on ANY read error on a drive.  If it was a part of a file, it would be found through a detailed text search, if it was part of slack space, no harm.  Again, if implemented a setting should control whether this is attempted.  Many users (me included) might prefer to use a data recovery tool.

 

It would be appropriate to continue to report the error count to the user - but you may now need 2 counts.  # of corrected errors and # of uncorrected errors.

 

[Wait a sec, let me get my kevlar on.]

 

Bubbaq / SirWired, does this sound better?

 

-Brian

Nope.  A hard error -- fail the drive.

unRAID doesn't fail a drive under these circumstances, it just increments an error count.  In your world, a couple of bad sectors and you're array is in pieces all over the floor.  Is that good?

 

I'd rather the array were more resilient, correcting errors that it can correct 100% accurately with its redundancy, and keeping me informed.  If I start to see a pattern (e.g., a drive having several recoverable errors in a week), I want to be able to decide when to pull the drive.  I don't want to suddenly be thrust into an emergency situation having to buy an expensive disk, at retail, so that I can keep my data safe.  I don't want to have to physically remove the disk and put it into another computer for correction.  I only want unRAID to force me to exchange a drive when the drive is truly failing - not just a sector or two among tens of billions has gone bad.

 

One thing I have learned is that running full disk scans (reading and writing) is a good maintenance procedure to let the drives perform their predictive remapping.  This is especially important on “stagnant” drives, that do not have frequent read/write activity.  Thanks to you and SirWired for the excellent information.  I hope that others have found it as useful as I have!

 

unRAID doesn't fail a drive under these circumstances, it just increments an error count.

 

Yes, it does... and it should.

 

To correct some possible misconceptions, unRAID is implemented chiefly in the MD driver.  It is not just a wrapper or hook.  It is a massive rewrite of MD.c

 

You also need to realize what it would take to implement this pet project of yours, and how many people actually care about it.  There are several hundred suggestions on the plate for features to ad to unRAID that many people want, and those widely requested features should be done long before this pet project.  With the limited time and resources available for unRAID enhancements, this one is requested by only 1 person.

 

If it is so important to you, you have all the source code for the driver... you implement it.  What you want would have to be a hook modded into the unRAID MD driver.  The unRAID source code (md.c and unraid.c) is in the distribution, so have at it.  Any expert in C code should be able to do it.

 

Sector remapping is deceptively simple.  When there is a read error, the drive remembers the sector is bad.  When there is a subsequent write to that sector, the sector is immediately remapped and then the write occurs.  That's it.  Not so complicated or mysterious.

 

That's nice to say... can you back it up?  Show me the source code if it is so simple.  The concept is simple.  The source code is not so simple.

 

Of course, you'll have to get a license for the code from Spinrite... they don't just give that away.  Or if you want to roll your own, the SMART association likely has patents/copyrights on their stuff, so you'll likely need a license of some sort from them, such as to get their docs or to use their commend set in order to implement this wiz-bang feature of yours.

 

I don't mean to be a hard ass, but I am a software developer and have written several million lines of source code in 14 different languages.  I have had to deal with "pet" feature requests from users for many years.  You have to balance the bang for the buck... and in this case, the buck is the time and resources needed to do it (a lot) versus the bang of a transparent feature that would rarely get used, that is only requested by one person (very little).  You also can't just trow a feature like this into the pot and distribute it.  There has to be QC testing on a LOT of different hardware combinations.  Low-level driver programming is a BITCH for things like memory leaks.

This thread is about a suggested new feature.  I think I presented some compelling reasons why it is a good idea.  You decided not to respond to those reasons, and instead went into some tangential technical issues, and made it a little personal.  These have nothing to do with the idea.

 

I admit that my initial suggestion was based on an ignorant view of what SMART is and how it works.  I think my refinement is very different.  Please read it one more time, as some of what you say imply that you are mixing the old suggestion with the new one.  (But note that the need to license Spinrite technology is NOT a part of either.).

 

The feature I’ve presented (certainly feature #1) is in line with what RAID manufacturers are doing.  The whole reason behind TLER drives (thanks for mentioned) is AVOIDING taking a drive out of service when drive errors are detected, while still maintaining the array’s integrity.  This is the goal for them, and I think should be the goal for unRAID as well.

 

Whether this is hard or easy, high priority or low priority, easy to debug or hard to debug, our professional credentials - it is all irrelevant to whether it is a good idea.

 

What this comes down to, IMHO, is a difference of opinion about the fundamental goal of redundant arrays.  One the one extreme is the desire to keep the array integral for as long as possible by proactively taking corrective actions, and on the other side is the desire to eject marginal drives as soon as they are detected.  We may just have to agree to disagree about where unRAID should be along that continuum.  Ultimately Tom will decide where HE wants his product to be.

 

I’m the new guy here and not trying to piss you off.  I’ve seen many posts where you are helping folks on a daily basis.  You clearly know unRAID and unix very well.  Sorry if I hit a nerve with something I said - it was not intentional.

 

-Brian

 

If builders build buildings the way programmers wrote programs, then the first woodpecker that cam along would destroy civilization. (Weinberg's Law)

 

OMG - It already does this:

 

From the online manual:

 

"When a read operation fails the system will return the reconstructed data of the failed block. The system then tries to write the reconstructed data back to the failing disk. If this write operation fails then the hard disk will be disabled."

 

You need to understand how drive remapping works to understand that the "write of the reconstructed data back to the failing disk" will perform a sector remap!  If it fails, that's a very bad thing, and the drive should be failed.

 

This was my recommended feature #1.  PERFECT!

 

Tom, if you are reading, could you confirm that the "write back to failed disk" logic would be applied during a parity check?  If it does, a by-product of a full parity check would be remapping of all previously read bad sectors using reconstructed data, instead of having to wait until the sector is enountered.

 

Tom, if you are reading, could you confirm that the "write back to failed disk" logic would be applied during a parity check?

 

Yes. 

You didn't piss me off and I hope the feeling is mutual.  I'm not not a warm fuzzy -- I'm an engineer with autism and we don't come less socialized than that.

 

I'm glad you are satisfied with the status quo.  I still stand by the fact that there are a lot of other features and enhancements that are more important and appropriate than interfacing directly with the SMART interface for remapping.  Drives are optimized black boxes... leave them alone.

Thanks Tom!  Please consider this feature request officially withdrawn.  Now that I understand it better, the current functionality is great!

 

Bubbaq -

 

I am not upset in any way.  You are a huge asset here, and I was doing my best to plead my case WITHOUT ruffling your feathers.  Next debate I want to be on YOUR team!

 

I have sent you a PM.