TLER (Time Limited Error Recovery)

[bubbaQ]

I have seen some problems discussed elsewhere concerning RAID issues using desktop drives rather than raid-specific drives.  The problem is with error recovery typically done by desktop drives when they hit a bad spot, that causes them to timeout and be dropped from the array as failed.  (Anyone who experienced a failing desktop drives knows that your apps can hang for 15-20 seconds while the disk retries a bad spot.)

 

Raid-specific drives have features to limit the retries so the drive will give up more quickly on a bad spot (such as Error Recovery Control (ERC) on Seagate drives , Time-Limited Error Recovery (TLER) on WD drives, and Command Completion Time Limit (CCTL), used by Samsung and Hitachi).

 

Does unRAID's customizations to the MD driver account for this?  If not, should it?

[limetech]

Account for it in what manner?

 

 

[bubbaQ]

What timeout does unRAID/MD use to flag a disk as failed? 

 

A shorter timeout typical of RAID arrays is 8 seconds, while a typical desktop system would be 15 seconds or more.  If a typical RAID system timeout of 8 seconds is employed, and there are desktop disks in the array with extended error correction found in typical desktop drives, the drive could be doing error recovery for longer than the 8 second RAID timeout, and then be flagged as failed by the array.

[limetech]

The unRAID driver does not add any additional timeout or retries - all such error recovery is done by the lower-level drivers.  All recovery actions taken by the unRAID driver are as a result of 'fatal' error being reported by the lower-level driver:

 

When a disk-read has failed: unRAID driver will attempt to read parity plus all the other drives in order to 'reconstruct' the missing data.  Once this is accomplished, this data is sent to the requestor, and then the driver attempts to write this data to the disk which reported the failed read.  If this 're-write' also fails, then see next paragraph.

 

When a disk-write has failed: unRAID driver will immediately 'disable' the drive.  If the array has valid parity then the written data will be retained (via parity logic) and 'success' is reported; otherwise the request failure is reported.

 

Note that in read error recovery which involves reading all the 'other' disks, there could be a significant delay if one or more of the other disks are spun-down.  This of course is unavoidable.

 

Here's my take on TLER: it's just non-sense   I suspect it was added at the request of some very large hard drive customers who found it easier to strong-arm the hard drive vendor, rather than fix their firmware, at the ultimate request of uninformed end-users.  Look at this way: if a hard drive is taking longer than 8 sec to recover from an error, you probably want to have it disabled.

 

Also, I think TLER might have some historical roots.  Long time ago several high density (for it's time) drives would periodically execute an internal "recalibration", where the heads would seek off to some special cylinder for the purpose of thermal recalibration.  If an i/o request came to the drive during this period, there indeed would be a substantial delay which didn't necessarily mean the disk was bad.  I don't believe any hard drive built in the last decade or more does such nonsense anymore.

 

The other historical root would be with video applications.  I can remember SMD hard drives which had a programmable mode where all retries were eliminated.  This was because the data was streaming off the drives at the video rate & it was deemed less of a problem seeing a small glitch in a line or two of a video frame vs. massive frame-loss jittering which would occur if the video stream was interrupted due to disk retries.

[Billped]

Welcome back, Tom.

 

 

 

Cheers,

Bill

[bubbaQ]

I remember Tcal headaches too.... believe me.  But I don't consider TLER as complete nonsense though. 

 

if a hard drive is taking longer than 8 sec to recover from an error, you probably want to have it disabled.

 

Not necessarily.  There are two philosophies on a disk read error... one says the drive is bad, the other says the sector is bad but the drive is still good.  Modern drives typically take the "sector is bad" approach and flag the sector as bad and reallocate it.  As you know, every drive has bad sectors, and new bad sectors develop over time without the whole drive being tossed into the trash.  Error handling/reporting/predicting features such as S.M.A.R.T. come into play in analyzing the history of such errors and reallocations.

 

A drive can develop a bad sector, the normal recovery and reallocation may take more than 8 seconds (say 10 seconds), and then the drive can continue to perform fine for years.  But if the drive itself is flagged as failed the RAID during that 10 seconds when it should not be.

 

The unRAID driver does not add any additional timeout or retries - all such error recovery is done by the lower-level drivers.  All recovery actions taken by the unRAID driver are as a result of 'fatal' error being reported by the lower-level driver.

 

That explains part of it.  Thank you.  I guess my question is what is the low level driver timeout for reporting a failure?.... typical RAID timeout of 8 seconds, or typical desktop drive timeout of 15 seconds?

 

This is another illustration of the benefit of unRAID over traditional RAID .  Traditional RAID should use RAID specific drives with TLER, while unRAID would be better off with a longer timeout to allow for drive error recovery to reallocate a single bad sector (15 seconds or more) and has no need for RAID specific drives or TLER.

 

When a disk-read has failed: unRAID driver will attempt to read parity plus all the other drives in order to 'reconstruct' the missing data.  Once this is accomplished, this data is sent to the requestor, and then the driver attempts to write this data to the disk which reported the failed read.  If this 're-write' also fails, then see next paragraph.

 

This would work perfectly if there is sufficient time for the disk to reallocate the bad sector (probably will be true since at least one disk likely needs to be spun up) ... since once the sector is reallocated by the drive's on-board algorithms, the drive is still fully functional and should not be disabled.

 

[whoarethey]

Hello, I am a new user and am considering using unraid with SAMSUNG EcoGreen F2 HD154UI 1.5TB drives

just wondering if there is a "verdict" on CCTL (on or off) before I make any decisions

[purko]

Hello, I am a new user and am considering using unraid with SAMSUNG EcoGreen F2 HD154UI 1.5TB drives

 

And you had to start off by digging up a thread from 3 years ago?

 

Trust me, whichever disk you pick it will have nothing to do with what disks used to be 3 years ago.

 

 

[whoarethey]

Sorry about bringing up an old thread.

Figured this is kinda related, and you guys provided a nice description.

People seem to be making a big deal out of not being able to enable TLER in new WD Green drives. Don't know if this is related to CCTL

I do not have much technical expertise and any advice on this topic would be appreciated.

 

Thanks

[vca]

I asked about TLER and its effect on unRAID in this message:

 

http://lime-technology.com/forum/index.php?topic=4396.0

 

but no one replied.

 

Seems that modern Western Digital Green desktop drives can take up to 2 minutes trying to resolve a failing block read - which is quite a bit longer than the 15 seconds people are talking about.

 

So is this sort of thing going to result in the drive being marked bad or is the driver stack prepared to wait patiently until the read command returns?

 

Regards,

 

Stephen

 

[Joe L.]

I asked about TLER and its effect on unRAID in this message:

 

http://lime-technology.com/forum/index.php?topic=4396.0

 

but no one replied.

 

Seems that modern Western Digital Green desktop drives can take up to 2 minutes trying to resolve a failing block read - which is quite a bit longer than the 15 seconds people are talking about.

 

So is this sort of thing going to result in the drive being marked bad or is the driver stack prepared to wait patiently until the read command returns?

 

Regards,

 

Stephen

 

I honestly don't know... I've seen disks wait.. when deadlocked for resources, but it would depend on if the drive returned an error.  I don't think I remember seeing any time-out logic in the "md" device, but I can't say what is in the SATA disk controller drivers, I've never looked there.

 

Joe L.

[bubbaQ]

Tom explained this previously.

 

1) The time-out will be seen as an error by unRAID, 2) all the other disks will be spun up and the data reconstructed from parity, and 3) the data data re-written to the data disk.  If the re-write fails, unRAID takes the drive off line.

 

The question is how long the low-level disk driver and controller will wait for the disk before they give up.... for that you need to look in the kernel, the driver, and your controller's firmware.