Question about running multiple pre-clear passes


Recommended Posts

Joe L.

Badblocks writes a whole disk of patterns, then reads it back and compares what it expects.

 

It does what your DD does, but with 4 patterns.

 

it also can do a non-destructive write mode test which uses random patterns, reads a bunch of blocks, writes a pattern, reads it back. compares, then re-writes the previously read blocks.

 

Reminds me of spinrite. although on a 2TB drive it's taken almost a week to complete. I'm so sorry I did this test. But it's a test on an existing filesystem and will exercise it's usability for testing questionable drives.

Judging by how slow this is going, I doubt it's just exercising the cache.

 

The only update badblocks needs is some form of status that shows transfer speed so we can see how fast we're going like the DD does.

 

I suppose you could update the DD test to write/read specific patterns per pass. But then you would need some type of program to constantly feed DD other patterns like reading /dev/zero does.

 

I could write a C program to constantly feed blocks of the pattern, but as I mentioned previously,  badblocks can keep track of what blocks are bad and provide output. This output can be provided back into badblocks for further write testing to possibly cause a remap. Or it can be used for telling mkreiserfs to skip these blocks.  

 

The only downside of ANY badblocks is in parity handling .

I.E. Full parity operations will not bypass the bad blocks.

The way unRAID works, if you have any physical defects on any of your array drives, you are at jeopardy of failure.

 

I'm planning a monthly regimen now to do smartctl -t short and a badblocks read test on every drive once a month.

Since it's such a lengthy process I'll do one drive a day based on the day of month.

 

disk1=1, disk2=2, disk3=3.. I'll move the monthly parity check to the 28th of the month for now.

 

I have confidence in the badblocks program as I have a few questionable drives and it discovered bad spots on them.

Link to comment
  • Replies 53
  • Created
  • Last Reply

Top Posters In This Topic

sorry to jump in at the end of this conversation but does that mean that your preclear script could suffer from the same fate Joe L.

or does it force write and reads to every physical sector ?

 

no, it cannot suffer from the same fate, because I read the entire disk first, then write the entire disk, then read the entire disk.   The small 64meg (at best) internal buffer is long since re-used by the time I go to write, or re-read any given sector to verify what was written.

 

The problem only occurs if you write, and immediately re-read the same sectors without other activity between that would use the same buffers.

Link to comment
  • 2 weeks later...

I'll just pop on here and say that 2 passes with preclear is not enough. I had a dead drive that I RMA'd to Seagate for replacement. The replacement drive passed 2 preclear cycles last week. I went to add the disk to my array today and I got an error from my controller. Moved the drive around, same error. Pulled it out and tried it in a "toaster" docking station, drive won't even spin up, just buzzes occasionally. Needless to say, not too happy. Seagate is replacing the drive again and paying for shipping both ways. Very good customer service.

 

I think i might go for a 2 sets of 2, or 3, cycles on a drive, with some kind of "intermission" between the sets, maybe a few days between them? Any sense in doing that (aside from my obvious recent experience)? Or just do a long workout (5 or 6 cycles)?

Link to comment

use badblocks.

 

in destructive write mode it does what a preclear does, but with 4 passes with 4 different patterns.

0xaa = 01010101

0x55 = 10101010

0xff  = 11111111

0x00 = 00000000

 

If you do the -o /tmp/badblocks.out it will log what bad blocks it found. Then you will know if the drive is safe to use or not.

 

There is also the non destructive read/write/read/write mode. This one really works out the drive and it's slow. > 120 hours for a 2TB drive.

 

It really triggered allot of sectors to show up as marginal.

I did this first, (it uses random patterns). Then I did the 4 pass destructive write mode

 

The 4 pass destructive write takes about a day on a 2tb drive.

 

 

 

Link to comment

use badblocks.

...

There is also the non destructive read/write/read/write mode. This one really works out the drive and it's slow. > 120 hours for a 2TB drive.

 

It really triggered allot of sectors to show up as marginal.

...

Marginal? Sounds interesting. I'm curious ... how is this marginality indicated?

 

Thanks.

 

--UhClem

 

Link to comment

sooo... Weebo are you and Joe gonna do some sort of best practice for new hdd's on the wiki

up to now i just run the preclear script x3 and then put the drive into my array.

I plan on adding an option to the preclear script to use badblocks for the initial "read" phase. 

I am not convinced that anything that writes and then immediately re-reads a sector is not "fooled" by the internal cache of the hard-disk. 

 

If each of the "passes" of badblocks is large enough o bypass he internal cache on the hard-disk, then it will do as expected.

 

I need to read more about how badblocks deals with cache memory in the disks.  (something that was not common when it was originally written)

 

Instead of preclear 3x, consider badblocks in destructive write mode followed by preclear 1x.

 

Joe L.

Link to comment

sooo... Weebo are you and Joe gonna do some sort of best practice for new hdd's on the wiki

up to now i just run the preclear script x3 and then put the drive into my array.

I plan on adding an option to the preclear script to use badblocks for the initial "read" phase.   

I am not convinced that anything that writes and then immediately re-reads a sector is not "fooled" by the internal cache of the hard-disk.   

 

If each of the "passes" of badblocks is large enough o bypass he internal cache on the hard-disk, then it will do as expected.

 

I need to read more about how badblocks deals with cache memory in the disks.  (something that was not common when it was originally written)

 

Instead of preclear 3x, consider badblocks in destructive write mode followed by preclear 1x.

 

Joe L.

 

I keep telling you that badblocks in destructive write mode does what the DD does.

It writes the whole drive, with a pattern, then turns around and re-reads the whole drive.

It keeps track of problem areas and can write them out to a file.

If that file has any badblocks in it, the disk is not a candidate for unRAID.

From my experience if there are pending sectors also, the disk is not a candidate for unRAID as it will cause a problem in the future when a recovery is occurring and every sector of every drive is read to reconstruct a failed drive.

 

I can say this with confidence, a non destructive read/write/read/write, detected weak sectors in my drive as the pending sectors went up.

The destructive write seems to have caused them to be reassigned.

 

It's the non destructive read/write/read/write that you may not trust, which is understandable. But for me, revealed problems.. Just takes almost 140 hours on a 2TB drive.

 

I know that it opens the device in direct mode but I'm not sure what that means. I did not see where it does anything to the drive itself, nor does DD. Also, The number of sectors read/written in non destructive mode is configurable.

 

As far as adding badblocks to the initial read phase, that is not useful enough. It needs to be in the final read phase. 

With the -o outputlog parameter. If this output log has any bad blocks, then the user needs to be warned.

Link to comment

use badblocks.

...

There is also the non destructive read/write/read/write mode. This one really works out the drive and it's slow. > 120 hours for a 2TB drive.

 

It really triggered allot of sectors to show up as marginal.

...

Marginal? Sounds interesting. I'm curious ... how is this marginality indicated?

 

Thanks.

 

--UhClem

 

 

1. If there is output in the badblocks log.

2. If the number of pending sectors is > 0 or increases.

3. If reallocated sectors increases significantly.

 

What I've come to learn is any pending sectors are a time bomb for any recover effort in the future.

unRAID needs to read every sector of every drive to recover a drive. So if there are problem spots in your array. This could cause a multiple drive failure in the future.

 

It really is prudent to do smart tests and badblocks in readonly mode tests on a regular basis.

The monthly parity check is also a good test, at least one of these three tests needs to occur on a scheduled basis. Parity check being the easiest.

Link to comment

I keep telling you that badblocks in destructive write mode does what the DD does.

It writes the whole drive, with a pattern, then turns around and re-reads the whole drive.

It keeps track of problem areas and can write them out to a file.

If that file has any badblocks in it, the disk is not a candidate for unRAID.

Somehow, I missed that fact in all your posts.  Writing the entire drive eliminates the internal cache from being used instead of the physical platters.  If it leaves all zeros as its final patten, then it could be used for almost the entire process.

It's the non destructive read/write/read/write that you may not trust, which is understandable. But for me, revealed problems.. Just takes almost 140 hours on a 2TB drive.

Yes, do need to learn more.  If the number of blocks read is greater than the internal cache, or if the cache is flushed/bypassed, then it would also work as we would like.

I know that it opens the device in direct mode but I'm not sure what that means.

It bypasses the linux buffer cache.
I did not see where it does anything to the drive itself, nor does DD. Also, The number of sectors read/written in non destructive mode is configurable.

 

As far as adding badblocks to the initial read phase, that is not useful enough. It needs to be in the final read phase. 

With the -o outputlog parameter. If this output log has any bad blocks, then the user needs to be warned.

And how is that different than alerting the user if there are sectors pending re-allocation?

 

Joe L.

 

Link to comment

 

 

I keep telling you that badblocks in destructive write mode does what the DD does.

It writes the whole drive, with a pattern, then turns around and re-reads the whole drive.

It keeps track of problem areas and can write them out to a file.

If that file has any badblocks in it, the disk is not a candidate for unRAID.

 

Somehow, I missed that fact in all your posts.   Writing the entire drive eliminates the internal cache from being used instead of the physical platters.   If it leaves all zeros as its final patten, then it could be used for almost the entire process.

 

This is part of what I've been trying to say. The only caveat is, the 4 pass test takes a very long time.

The positive to badblocks is you have immediate feedback in the latest version when there are badblocks on the disk.

 

 

 

As far as adding badblocks to the initial read phase, that is not useful enough. It needs to be in the final read phase. 

With the -o outputlog parameter. If this output log has any bad blocks, then the user needs to be warned.

 

And how is that different than alerting the user if there are sectors pending re-allocation?

Joe L.

 

It's almost the same, but anything showing up in badblocks output log means that it actually made it to the kernel/application layer.

 

My point is, running badblocks as the initial read is not useful enough because it does not test what is needed on the far side of of the preclear. I.E. if you have badblocks or pending sectors at the end of the process the drive is not usable for unRAID.

 

I propose consideration to replace DD, or add an option to use badblocks in destructive write mode instead of the DD test.

 

Link to comment

 

 

I keep telling you that badblocks in destructive write mode does what the DD does.

It writes the whole drive, with a pattern, then turns around and re-reads the whole drive.

It keeps track of problem areas and can write them out to a file.

If that file has any badblocks in it, the disk is not a candidate for unRAID.

 

Somehow, I missed that fact in all your posts.   Writing the entire drive eliminates the internal cache from being used instead of the physical platters.   If it leaves all zeros as its final patten, then it could be used for almost the entire process.

 

This is part of what I've been trying to say. The only caveat is, the 4 pass test takes a very long time.

The positive to badblocks is you have immediate feedback in the latest version when there are badblocks on the disk.

 

 

 

As far as adding badblocks to the initial read phase, that is not useful enough. It needs to be in the final read phase. 

With the -o outputlog parameter. If this output log has any bad blocks, then the user needs to be warned.

 

And how is that different than alerting the user if there are sectors pending re-allocation?

Joe L.

 

It's almost the same, but anything showing up in badblocks output log means that it actually made it to the kernel/application layer.

 

My point is, running badblocks as the initial read is not useful enough because it does not test what is needed on the far side of of the preclear. I.E. if you have badblocks or pending sectors at the end of the process the drive is not usable for unRAID.

 

I propose consideration to replace DD, or add an option to use badblocks in destructive write mode instead of the DD test.

 

And that is (sort-of) what I said when I wrote:

If it leaves all zeros as its final patten' date=' then it could be used for almost the entire process.[/quote']

 

I went searching for more on badblocks vs. cache and found other discussions, but no firm answers:

http://www.linuxquestions.org/questions/linux-hardware-18/badblocks-vs-disk-cache-198556/

 

Joe L.

 

 

 

Link to comment

I went searching for more on badblocks vs. cache and found other discussions, but no firm answers:

http://www.linuxquestions.org/questions/linux-hardware-18/badblocks-vs-disk-cache-198556/

Joe L.

 

I think this is only an issue in non destructive read/write mode.

In that case I would calculate the number of sectors to work with to be 2x cache.

 

So if the current largest cache is 64 MB, We could set values to work with by [-b block_size] * [-c blocks_at_once]

 

destructive write works on the whole drive or parts of a drive of you specific starting block and count.

 

 

Link to comment

use badblocks.

...

There is also the non destructive read/write/read/write mode. This one really works out the drive and it's slow. > 120 hours for a 2TB drive.

 

It really triggered allot of sectors to show up as marginal.

...

Marginal? Sounds interesting. I'm curious ... how is this marginality indicated?

 

Thanks.

 

--UhClem

 

 

1. If there is output in the badblocks log.

2. If the number of pending sectors is > 0 or increases.

3. If reallocated sectors increases significantly.

 

What I've come to learn is any pending sectors are a time bomb for any recover effort in the future.

...

Thanks for the reply. Maybe I took your initial statement too literally.

It really triggered allot of sectors to show up as marginal.

Which suggested that badblocks was detecting marginal sectors. And that is what caught my attention, and prompted my query.

 

[is it possible that your #1 above is where badblocks indicates my notion of sector marginality? I've never run it, but a careful read of the man page did not indicate any such reporting.]

 

In my opinion, a Current Pending Sector is pretty much on its death bed, but no death certificate has been written (Reallocated). It just could survive (and then it will tell all its neighbors (the entire cylinder) about the bright white light it saw :)).

 

I was expecting badblocks to warn about sectors that were actually marginal--you know, like high cholesterol and diabetes, maybe even early cancer.

 

"I'll let you be in my dream if I can be in yours." -- Bob Dylan said that.

"Just because a sector isn't bad doesn't mean it's good." -- I said that.

 

-- UhClem

 

 

 

 

Link to comment

[is it possible that your #1 above is where badblocks indicates my notion of sector marginality? I've never run it, but a careful read of the man page did not indicate any such reporting.]

 

In my opinion, a Current Pending Sector is pretty much on its death bed, but no death certificate has been written (Reallocated). It just could survive (and then it will tell all its neighbors (the entire cylinder) about the bright white light it saw ).

 

I was expecting badblocks to warn about sectors that were actually marginal--you know, like high cholesterol and diabetes, maybe even early cancer.

 

That was pretty funny!!!! bright light. LOL!

 

Here's some food for thought. If badblocks comes back with any errors in the badblocks output file. Those sectors are currently questionable and probably unusable.

 

I looked in the code. Basically if what was written did not come back the same way. It's flagged as an error.

 

So if pending sectors is "deathbed" scenario, then any badblocks are the same status.

They need to be re-written (Which sometimes refreshes them) and they are then revived or they need to be marked offline uncorrectable or reallocated.

 

I can state this with confidence.

I did a non destructive read/write test. it took many days. It logged 151 errors.

More sectors then before popped up as pending.

I did a smart short test. It failed on an LBA Read error.

I did a 4 pass destructive write test.

It passed with no errors.

I did a smart short test.

it passed and got past the bad sectors.

no bad blocks showed up this time.

 

Link to comment

Log output for discussion.

 

First Diff of smart stats readonly test

root@atlas /home/rcotrone/log #diff -u scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart1 scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart2 
--- scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart1    2011-11-30 19:49:16.000000000 -0500
+++ scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart2    2011-12-01 15:06:32.000000000 -0500
@@ -10,7 +10,7 @@
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
-Local Time is:    Thu Dec  1 00:49:16 2011 Local time zone must be set--see zic m
+Local Time is:    Thu Dec  1 20:06:33 2011 Local time zone must be set--see zic m
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

@@ -58,15 +58,15 @@
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       449
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
-  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       7178
+  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       7197
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       90
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       51
-193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       6262
-194 Temperature_Celsius     0x0022   112   096   000    Old_age   Always       -       40
+193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       6362
+194 Temperature_Celsius     0x0022   113   096   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
-197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       47
+197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       49
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       34
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

 

Second test after non destructive read/write test. 151 errors logged.

root@atlas /home/rcotrone/log #diff -u scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart2 scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart3
--- scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart2    2011-12-01 15:06:32.000000000 -0500
+++ scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart3    2011-12-13 06:44:27.000000000 -0500
@@ -10,7 +10,7 @@
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
-Local Time is:    Thu Dec  1 20:06:33 2011 Local time zone must be set--see zic m
+Local Time is:    Tue Dec 13 11:49:43 2011 Local time zone must be set--see zic m
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

@@ -54,20 +54,20 @@
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
-  3 Spin_Up_Time            0x0027   158   150   021    Pre-fail  Always       -       9083
-  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       449
+  3 Spin_Up_Time            0x0027   158   150   021    Pre-fail  Always       -       9100
+  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       450
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
-  9 Power_On_Hours          0x0032   091   091   000    Old_age   Always       -       7197
+  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7475
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
- 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       90
+ 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       91
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       51
-193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       6362
-194 Temperature_Celsius     0x0022   113   096   000    Old_age   Always       -       39
+193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       7432
+194 Temperature_Celsius     0x0022   112   096   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
-197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       49
-198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       34
+197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       316
+198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       42
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

 

Next Test. Smart Short test

root@atlas /home/rcotrone/log #diff -u scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart3 scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart4
--- scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart3    2011-12-13 06:44:27.000000000 -0500
+++ scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart4    2011-12-13 07:55:39.000000000 -0500
@@ -10,7 +10,7 @@
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
-Local Time is:    Tue Dec 13 11:49:43 2011 Local time zone must be set--see zic m
+Local Time is:    Tue Dec 13 13:00:55 2011 Local time zone must be set--see zic m
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

@@ -21,9 +21,8 @@
Offline data collection status:  (0x84)        Offline data collection activity
                                       was suspended by an interrupting command from host.
                                       Auto Offline Data Collection: Enabled.
-Self-test execution status:      (   0)        The previous self-test routine completed
-                                       without error or no self-test has ever 
-                                       been run.
+Self-test execution status:      ( 121)        The previous self-test completed having
+                                       the read element of the test failed.
Total time to complete Offline 
data collection:                (41160) seconds.
Offline data collection
@@ -58,7 +57,7 @@
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       450
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
-  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7475
+  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7477
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       91
@@ -66,7 +65,7 @@
193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       7432
194 Temperature_Celsius     0x0022   112   096   000    Old_age   Always       -       40
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
-197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       316
+197 Current_Pending_Sector  0x0032   199   199   000    Old_age   Always       -       328
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       42
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0
@@ -76,22 +75,23 @@

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
-# 1  Short offline       Completed: read failure       90%      7115         318245470
-# 2  Extended offline    Completed without error       00%      7107         -
-# 3  Short offline       Completed without error       00%      7100         -
-# 4  Short offline       Completed: read failure       90%      7060         1110673314
-# 5  Extended offline    Completed: read failure       90%      7054         1110673314
-# 6  Short offline       Completed: read failure       90%      7054         1110673314
-# 7  Short offline       Completed: read failure       90%      7053         1110673314
-# 8  Short offline       Completed: read failure       90%      4718         1110673314
+# 1  Short offline       Completed: read failure       90%      7475         9372545
+# 2  Short offline       Completed: read failure       90%      7115         318245470
+# 3  Extended offline    Completed without error       00%      7107         -
+# 4  Short offline       Completed without error       00%      7100         -
+# 5  Short offline       Completed: read failure       90%      7060         1110673314
+# 6  Extended offline    Completed: read failure       90%      7054         1110673314
+# 7  Short offline       Completed: read failure       90%      7054         1110673314
+# 8  Short offline       Completed: read failure       90%      7053         1110673314
# 9  Short offline       Completed: read failure       90%      4718         1110673314
-#10  Short offline       Completed: read failure       90%      4039         1110673314
-#11  Short offline       Completed: read failure       90%      4028         1110673314
+#10  Short offline       Completed: read failure       90%      4718         1110673314
+#11  Short offline       Completed: read failure       90%      4039         1110673314
#12  Short offline       Completed: read failure       90%      4028         1110673314
-#13  Extended offline    Interrupted (host reset)      10%       250         -
-#14  Short offline       Completed without error       00%       181         -
-#15  Extended offline    Completed without error       00%       157         -
-#16  Short offline       Completed without error       00%       150         -
+#13  Short offline       Completed: read failure       90%      4028         1110673314
+#14  Extended offline    Interrupted (host reset)      10%       250         -
+#15  Short offline       Completed without error       00%       181         -
+#16  Extended offline    Completed without error       00%       157         -
+#17  Short offline       Completed without error       00%       150         -

 

Destructive 4 pass write/read test.

root@atlas /home/rcotrone/log #diff -u scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart4 scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart5
--- scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart4    2011-12-13 07:55:39.000000000 -0500
+++ scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart5    2011-12-15 13:01:46.000000000 -0500
@@ -10,7 +10,7 @@
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
-Local Time is:    Tue Dec 13 13:00:55 2011 Local time zone must be set--see zic m
+Local Time is:    Thu Dec 15 18:01:45 2011 Local time zone must be set--see zic m
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

@@ -57,15 +57,15 @@
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       450
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
-  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7477
+  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7530
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       91
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       51
-193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       7432
-194 Temperature_Celsius     0x0022   112   096   000    Old_age   Always       -       40
+193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       7437
+194 Temperature_Celsius     0x0022   113   096   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
-197 Current_Pending_Sector  0x0032   199   199   000    Old_age   Always       -       328
+197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       42
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

 

last smart short test

root@atlas /home/rcotrone/log #diff -u scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart5 scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart6
--- scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart5    2011-12-15 13:01:46.000000000 -0500
+++ scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644.smart6    2011-12-15 13:16:13.000000000 -0500
@@ -10,7 +10,7 @@
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  Exact ATA specification draft version not indicated
-Local Time is:    Thu Dec 15 18:01:45 2011 Local time zone must be set--see zic m
+Local Time is:    Thu Dec 15 18:16:12 2011 Local time zone must be set--see zic m
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

@@ -18,11 +18,12 @@
SMART overall-health self-assessment test result: PASSED

General SMART Values:
-Offline data collection status:  (0x84)        Offline data collection activity
-                                       was suspended by an interrupting command from host.
+Offline data collection status:  (0x85)        Offline data collection activity
+                                       was aborted by an interrupting command from host.
                                       Auto Offline Data Collection: Enabled.
-Self-test execution status:      ( 121)        The previous self-test completed having
-                                       the read element of the test failed.
+Self-test execution status:      (   0)        The previous self-test routine completed
+                                       without error or no self-test has ever 
+                                       been run.
Total time to complete Offline 
data collection:                (41160) seconds.
Offline data collection
@@ -62,8 +63,8 @@
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       91
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       51
-193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       7437
-194 Temperature_Celsius     0x0022   113   096   000    Old_age   Always       -       39
+193 Load_Cycle_Count        0x0032   198   198   000    Old_age   Always       -       7441
+194 Temperature_Celsius     0x0022   116   096   000    Old_age   Always       -       36
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       42
@@ -75,23 +76,24 @@

There's allot of diff data snipped out here.  but what is significant is. 

-# 1  Short offline       Completed: read failure       90%      7475         9372545
+# 1  Short offline       Completed without error       00%      7530         -
+# 2  Short offline       Completed: read failure       90%      7475         9372545[/font]

 

I'm doing one final read test to see if anything shows up, I'll report or edit this post with the data.

 

Link to comment

In my opinion, a Current Pending Sector is pretty much on its death bed, but no death certificate has been written (Reallocated). It just could survive (and then it will tell all its neighbors (the entire cylinder) about the bright white light it saw ).

 

That was pretty funny!!!! bright light. LOL!

Glad you liked it. (I've been a 'net jokester since the first days of Usenet [early 80s], and it's nice to hear the audience laugh [on this medium].)

Here's some food for thought. If badblocks comes back with any errors in the badblocks output file. Those sectors are currently questionable and probably unusable.

 

I looked in the code. Basically if what was written did not come back the same way. It's flagged as an error.

Thanks for checking. (This geriatric hacker should have done it himself, but I couldn't reach my walker.) That's reasonable.

So if pending sectors is "deathbed" scenario, then any badblocks are the same status.

They need to be re-written (Which sometimes refreshes them) and they are then revived or they need to be marked offline uncorrectable or reallocated.

 

I can state this with confidence.

I did a non destructive read/write test. it took many days. It logged 151 errors.

More sectors then before popped up as pending.

I did a smart short test. It failed on an LBA Read error.

I did a 4 pass destructive write test.

It passed with no errors.

I did a smart short test.

it passed and got past the bad sectors.

no bad blocks showed up this time.

 

That is very interesting! Thanks for relating it. [Can I assume that this was all done on your box with the ECC memory, and that the ECC protection was actually engaged?]

 

And thanks for the follow-on post with all the gory details (kudos on good record-keeping). Surgery can be a messy business. I'm not sure I would consider that patient cured; maybe just temporary remission. To be honest, I suspect congenital platter surface cancer.

 

Did you return it to the general population (your array) or send it to the hospice? Maybe a convalescent home for close observation [then RMA (aka DNR)].

 

--UhClem

 

Link to comment

LOL, you're killing me.

 

As for the test machine, It's not on an ECC supported host. So Yes there could be issues.

However I think the smart tests go along with the application findings.

 

Read test

-197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       47

+197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       49

198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       34

 

non destructive read test

-197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       49

-198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       34

+197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       316

+198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       42

 

Short smart test

-197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       316

+197 Current_Pending_Sector  0x0032   199   199   000    Old_age   Always       -       328

198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       42

 

Destructive write/read test.

-197 Current_Pending_Sector  0x0032   199   199   000    Old_age   Always       -       328

+197 Current_Pending_Sector  0x0032   200   199   000    Old_age   Always       -       0

198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       42

 

These tests encompass both the machine and the internal hard drive firmware, So if a problem existed. it was an issue with the surface.

 

What may have happened with this drive is when the machine was powered down at some point. the heads were not totally parked and data may have been scrambled. if the sectors were really bad, they probably would have stayed bad.

 

My only concern is that the non destructive read/write/read/write test did not refresh the sectors.

I may retry this on another questionable drive and instead of using the built in random test pattern, use 0x00 to see if that forces a rewrite of the sector.

Link to comment

6 hours later

 

no more read errors.

 

root@atlas /home/rcotrone/log #cat readtest.log

root@unraid /tmp #badblocks-1.42 -vs -o /tmp/badblocks.out /dev/disk/by-id/scsi-SATA_WDC_WD20EADS-00_WD-WCAVY1816644

Checking blocks 0 to 1953514583

Checking for bad blocks (read-only test): done                                               

Pass completed, 0 bad blocks found. (0/0/0 errors)

 

smart reports show no difference.

 

I'm probably gong to run this drive through another non destructive read/write test to see what it finds.

But for now it looks ok, although the smart reports show the failed LBA's and that has me nervous enough to RMA the drive.

 

Link to comment

My take is that there are two separate classes of errors instigated by the single procedure (the badblocks run). First, there's the Read-vs-Write mismatches, which I would attribute mainly to lack of ECC memory (more on this below). Second are the Current_Pending_Sector errors (ie, ramp-up of the count). Those are, by definition, in the jurisdiction of the drive itself, most likely (in this scenario) platter surface related. A third potential source of errors, which you didn't receive, would have been CRC errors (from smart), and would have been attributable to cable/connection problems.

 

The hard drive's prime directive is to guarantee that the 512 bytes it returns at the SATA connector for a READ request is exactly the same as it received at the SATA connector during the most recent (prior) WRITE request, for a specific LBA (sector#). It uses very sophisticated ECC (much hairier than ECC for RAM) along with encoding trickery [see RLL] (and other stuff I don't understand) to accomplish this. Also, that same ECC will tell it when it can't make the guarantee, and then it returns an error, but only after numerous retries (10-20+).

 

Between the drive's SATA connector and the controller/HBA, CRC is employed to assure integrity (or return an error; which should elicit a retry (by the host); which will usually succeed). Hmmm, does the drive/smart only log the CRC errors that it catches (during a WRITE request)?

 

Thus, those data-mismatch errors reported by badblocks should have no relationship to the Current_Pending_Sector increases. They can be attributed to glitches that occur between the application's user-space memory and the controller (on the way out) and between the controller and the ultimate comparison-check in the application code (on the way back in). Of those 151 errors reported that you mentioned, how many were mismatch errors vs. hard i/o errors? It appears that this corresponded to an increase in the Current_Panding_Sector from 49 to 316.

 

One thing we don't know is the precise criteria the drive firmware uses for adding, and possibly removing, Current_Pending_Sector status (separate from escalating to Reallocation). Ie, the exact chain of events and conditions which took place for your drive. Questions I have (but can't expect you to have the data to answer [but, surprise me!]) are:

1) Did Current_Pending_Sector (CPS), after each Write-then-ReadAndTest pass, steadily increase, or did the Write clear the vast majority (by merit of the Write succeeding, or, if not, forcing a Reallocation)? (It doesn't appear that you anticipated the potential use for this; I only realized it after I finished my reply.)

2) How many CPS bumps had a corresponding (Uncorrectable)Error returned to host?

2a) Did badblocks actually report any I/O errors?

2b) Did syslog evidence any I/O errors?

3) Were the LBA's for these CPS bumps highly localized (bunched, or consecutive)?

 

Regarding your surmisal about an abnormal power-down being a cause ... maybe, and that's why I asked about the bunching of LBA#s for the CPS guys. In general, power instability (within the drive) can apparently produce what now appears to be your situation: sector(s) are written weakly/insufficiently, and subsequent Read has problem, causing CPS bump; but the next following Write of that LBA (with proper power) succeeds, and CPS gets decremented and that LBA comes off the watch-list. I imagine there can also be areas of (permanently) marginal surface, such that failure is inconsistent--sometimes, it succeeds within the firmware's retry count, sometimes not.

 

It could be enlightening if you did have a record of some of these problematic LBAs. Then, a badblocks test over a very limited range would be quick, regardless of how many passes (though you would want to heed your 2XCacheMB concern). And, rather than multi-pass, better would be multiple single-pass runs with logged smart's interspersed.

 

Another puzzle: with all the read/write testing this drive has endured, shouldn't the "failed LBAs" you mention (end of last post) be represented in a non-zero Reallocated_Sector_Count? From your diff's, it would seem that is still 0.

 

Personally, I'm looking forward to one of my own drives becoming "interesting" (like yours, or some variation). I've got some ideas, and some low-level code written to help test them. I just need my own lab rat.

 

--UhClem

 

 

Link to comment

Interesting take on this.

 

I'm going to disregard the ECC memory issue.

This test ran for almost a week in various stages. The 151 errors logged by Badblocks was within a small subset and syslog reported the drives having issues at that time. So I'm going to lean towards the drive having issues, timing out the system getting the error feedback and possibly the badblocks getting a short read or error and logging it as a read error.

 

There were corresponding syslog entries during the bunched errors.

 

Furthermore this drive was taken out of service by unRAID a number of months back for having issues returning valid data.

That machine has ecc memory. There are no CRC errors logged in the smart reports within the drive.

 

I did notice Current pending sectors decreased, no reallocations, but also an increase in offline uncorrectable sectors.

So in that case I suppose the sectors were marked permanently bad without requiring reallocation.

 

So in answering the questions

> 1) Did Current_Pending_Sector (CPS), after each Write-then-ReadAndTest pass, steadily increase, or did the Write clear the vast majority (by merit of the Write succeeding, or, if not, forcing a Reallocation)?

 

CPS increased during the non destructive read/write test. All localized.

 

> 2) How many CPS bumps had a corresponding (Uncorrectable)Error returned to host?

I can't tell, they happened all at once and I did not check the syslog for number, only saw the ata errors right around the time of the badblock errors.

 

>  2a) Did badblocks actually report any I/O errors?

Not I/O errors but read errors, syslog entries showed ata read errors.

 

> 2b) Did syslog evidence any I/O errors?

Yes, during non destrictive read/write.

 

> 3) Were the LBA's for these CPS bumps highly localized (bunched, or consecutive)?

This I cannot tell, the smart report shows some, badblocks reporteed 151 read error tests in a big bunch.

But the LBA errors in the smart report do not seem to be too consecutive.

 

 

I'll probably do a long test on the drive next to see what it shows in smart.

I still do not trust the drive, so I'm going to RMA.

Link to comment

Thanks very much for providing that info. Your tests, and results, with a real live specimen, have really caught my interest.

 

Given the extra details you've provided, I am also inclined to retract my memory parity error suggestion. But, I'm going to need to take a look at badblocks' code. My concern is with those "short reads" that you made mention of. (If we're on the same wavelength here) This would be where the entire read() call's byte count request was not fully satisfied, because the (hard, kernel-logged) read_error occurred on a sector within (but not at the start of) that request. I'm wondering if, somehow, badblocks is using "too much" data during the subsequent comparison test. Or, and this part really really concerns me, is the drive violating its prime directive (Never return data whose integrity it can't 100.01% vouch for) ??

 

Well, I looked at the code, in routine do_read(), and it does look fine.  I'm puzzled and/or worried.

 

Just thought of something ... I don't have any WD drives. Is this specific drive of yours a 4KB sector model?

 

Thanks again. (I've got some testing code that handles a similar situation, but haven't had the opportunity to test its analogous error-handling.)

 

Shall we call this "Lab rat by proxy"?

 

--UhClem

 

Link to comment

Well, I looked at the code, in routine do_read(), and it does look fine.  I'm puzzled and/or worried.

 

Just thought of something ... I don't have any WD drives. Is this specific drive of yours a 4KB sector model?

 

Thanks again. (I've got some testing code that handles a similar situation, but haven't had the opportunity to test its analogous error-handling.)

 

Shall we call this "Lab rat by proxy"?

 

--UhClem

 

 

This particular WD drive is a WDEADS 2tb. It was one of the last ones before the 4K advanced format drives took over. I bought them because I planned for use on an XP machine. I just so happen to run out of space and used it in unRAID.

 

As far as the do_read. It might do the correct thing, but then again, the kernel did report errors during the ATA communication. So the drive may have taken to long to respond or was reset.

 

I had a Seagate drive recently, just go off line at a bad sector. (read my thread on ddrescue). the only way to recover the data finally was to read the drive backwards (which worked).

 

I just got hold of a bunch of drives, maybe I'll have s'more to test with.

Link to comment

Is this specific drive of yours a 4KB sector model?

This particular WD drive is a WDEADS 2tb. It was one of the last ones before the 4K advanced format drives took over.

Drilling even deeper ...

Before I asked, I had checked on the WD20EADS model (based on the smart output you provided) but I found this Technical Bulletin from Netgear/ReadyNAS on the subject (excerpted below). And, I could find nothing relevant from WDC.

 

Maybe I wasn't specific enough (with the lone word specific, above), or maybe I'm not quite brave enough to take you literally on your use of the word It, above. [Your test sample has the "-00" suffix, but I couldn't find anything to confirm whether more characters of precision were needed.]

 

I'm not trying to bust your chops here :), just trying to gather all relevant facts so I can best analyze this very critical exceptional handling scenario. Thanks.

 

==excerpt from ReadyNAS/Netgear Technical Bulletin==

In the meantime, please note that WD has released the 4K sector drives under the same base models as the 512-byte sector drives.  For instance, the WD20EADS drive are available in both 512-byte and 4K sectors.  You will need to look carefully at the notes in the compatibility list where we specifically state the full model number, include the suffix after the dash “-”.  In the case of WD20EADS, we will put in the notes that the model that passed qual is WD20EADS-00R6B0.  This is the 512-byte model.  Whenever a suffix note is mentioned, refrain from using drives with a different suffix.  We advise that if you cannot verify the suffix from the disk retailer, either avoid purchasing the drive(s) from there or ensure that the retailer has a quick return or exchange policy.

===

 

--UhClem

 

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.