March 3, 201511 yr I have a WD20EARS drive that has nearly 27,000 hours on it. I first noticed problems when i was constantly getting a single error when performing parity checks. I noticed this drive had a pending sector, and took action to swap it out. I purchased a new WD Red drive (3TB) and got the array up and running smoothly since, without issues. Once i was comfortable that the array was stable (~month of use after drive swaps, a couple parity checks came back clean) i began poking around this drive. It's run through 2 full pre-clear cycles and the pending sector did NOT change. Current smart report: SMART status Info for /dev/sdc smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (AF) Device Model: WDC WD20EARS-00MVWB0 Serial Number: WD-WCAZAA019830 LU WWN Device Id: 5 0014ee 2b0f2b497 Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Tue Mar 3 16:27:06 2015 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 245) Self-test routine in progress... 50% of test remaining. Total time to complete Offline data collection: (38460) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 371) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 167 163 021 Pre-fail Always - 6650 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2735 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 2 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 064 064 000 Old_age Always - 26734 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 170 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 103 193 Load_Cycle_Count 0x0032 193 193 000 Old_age Always - 22691 194 Temperature_Celsius 0x0022 117 110 000 Old_age Always - 33 196 Reallocated_Event_Count 0x0032 198 198 000 Old_age Always - 2 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Astute readers will see that the drive is currently undergoing a long smart test. I still feel this drive is good if I can solve this pesky pending sector, until then it will remain outside of my array.
March 4, 201511 yr hit it with badblocks -svw, that will write more than zeros. takes about twice as long as preclear.
March 4, 201511 yr If the pending sector goes away without a reallocated sector or reallocated event, then you cannot trust the pending sector has gone. http://en.wikipedia.org/wiki/S.M.A.R.T. Count of "unstable" sectors (waiting to be remapped, because of unrecoverable read errors). If an unstable sector is subsequently read successfully, the sector is remapped and this value is decreased. Read errors on a sector will not remap the sector immediately (since the correct value cannot be read and so the value to remap is not known, and also it might become readable later); instead, the drive firmware remembers that the sector needs to be remapped, and will remap it the next time it's written.[36] However some drives will not immediately remap such sectors when written; instead the drive will first attempt to write to the problem sector and if the write operation is successful then the sector will be marked good (in this case, the "Reallocation Event Count" (0xC4) will not be increased). This is a serious shortcoming, for if such a drive contains marginal sectors that consistently fail only after some time has passed following a successful write operation, then the drive will never remap these problem sectors. If you can find the bad block (badblocks reports it). You can pass that information to mkfs which will then map it and refrain from it's use. However, you can probably never trust this as an array drive. Maybe a cache drive or backup drive.
March 8, 201511 yr Author badblocks has finished, unfortunately nothing found!! root@Tower:~# badblocks -svw /dev/sdc Checking for bad blocks in read-write mode From block 0 to 1953514583 Testing with pattern 0xaa: done Reading and comparing: done Testing with pattern 0x55: done Reading and comparing: done Testing with pattern 0xff: done Reading and comparing: done Testing with pattern 0x00: done Reading and comparing: done Pass completed, 0 bad blocks found. Now reallocated event and sector counts are 3, but pending is still at 1.
March 8, 201511 yr badblocks has finished, unfortunately nothing found!! ... Now reallocated event and sector counts are 3, but pending is still at 1. It means there is a weak sector that keeps getting put back into service. From what I read that's a firmware issue with the pending sector logic (as stated on Wikipedia). I.E. it's not foolproof. I would look in the syslog for ATA messages and retries at the kernel level. I wouldn't use this as an array drive, maybe a cache drive. There's another tool I've used in the past called scrub which writes on the filesystem completely. I've used this on a live system and it reallocated the pending sectors. Yet in this case, badblocks has passed through the drive 8 times. 4 writes, 4 reads, the pending sector should have been re-allocated. You could try another 4 pass badblocks and check again, if it's not reallocated after that I would not use the drive. There's no way to know where the weak sectors are. I suppose you could try to repair it with spinrite which is a commercial application. I'm not sure it's worth it. If it were my drive, I might try spinrite for the pure academic learning purpose. I'm still waiting for my own drive with a pending sector that badblocks captures to try a dd operation to force reallocation, but who knows when that will happen.
March 8, 201511 yr Author Thanks Weebo, I'll definately avoid using this as an array drive. While it was in service it was causing a single parity error each time it ran. 0 in 2 or 3 checks since replacement, so that mystery was solved. The syslog is clean in terms of ATA or retries, it was even during the parity checks. I'm going to have to chalk this up to "ghosts in the machines". This drive will become something else, but nothing to do with unRAID.
March 8, 201511 yr Thanks Weebo, I'll definately avoid using this as an array drive. While it was in service it was causing a single parity error each time it ran. 0 in 2 or 3 checks since replacement, so that mystery was solved. I'm going to have to chalk this up to "ghosts in the machines". This drive will become something else, but nothing to do with unRAID. This means the data being read or recalculated with error correction is(was) incorrect. It's not a ghost. It's a weak sector that isn't returning what the parity check or vica versa. One of the two is wrong, parity or this sector. I think this is why we need dual parity or some kind of bitrot detection/repair.
March 8, 201511 yr Actually, a sector was reallocated. Post the new smart -a output. The pending may be the same, or another.
March 8, 201511 yr Thanks Weebo, I'll definately avoid using this as an array drive. While it was in service it was causing a single parity error each time it ran. 0 in 2 or 3 checks since replacement, so that mystery was solved. I'm going to have to chalk this up to "ghosts in the machines". This drive will become something else, but nothing to do with unRAID. This means the data being read or recalculated with error correction is(was) incorrect. It's not a ghost. It's a weak sector that isn't returning what the parity check or vica versa. One of the two is wrong, parity or this sector. I think this is why we need dual parity or some kind of bitrot detection/repair. btrfs has scrubbing for bitrot.
March 8, 201511 yr I think this is why we need dual parity or some kind of bitrot detection/repair. btrfs has scrubbing for bitrot. From what I was told, it only works if the file system is mounted a certain way and it does not actually repair it unless the drive is part of a fault tolerant pool. Please correct me if I'm wrong.
March 8, 201511 yr Author Actually, a sector was reallocated. Post the new smart -a output. The pending may be the same, or another. smartctl -a -d ata /dev/sdc (--) smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (AF) Device Model: WDC WD20EARS-00MVWB0 Serial Number: WD-WCAZAA019830 LU WWN Device Id: 5 0014ee 2b0f2b497 Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Sun Mar 8 12:59:31 2015 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (38460) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 371) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 167 163 021 Pre-fail Always - 6650 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2735 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 3 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 064 064 000 Old_age Always - 26848 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 170 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 103 193 Load_Cycle_Count 0x0032 193 193 000 Old_age Always - 22788 194 Temperature_Celsius 0x0022 125 110 000 Old_age Always - 25 196 Reallocated_Event_Count 0x0032 197 197 000 Old_age Always - 3 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 26738 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
March 8, 201511 yr I think this is why we need dual parity or some kind of bitrot detection/repair. btrfs has scrubbing for bitrot. From what I was told, it only works if the file system is mounted a certain way and it does not actually repair it unless the drive is part of a fault tolerant pool. Please correct me if I'm wrong. No special mounting required, but repair does require a source of correct data.
April 23, 201511 yr Author So an update to this. The drive was pre-cleared numerous times and always passed with no changes to the reallocated count or pending count. I wanted to start playing with Dockers, so used it as my cache drive (config files backed up). Shortly afterwards the reallocated count began to rise (currently at 6) and pending remains at 1. Like I mentioned, there is nothing else on this drive so if it craps out with stuff on it, no big deal. I have several other 2.5" drives that can jump in to replace it if need be.
April 23, 201511 yr As I presume you know, a pending sector indicates that the S.M.A.R.T. system has recognized that the sector has issues, but has been unable to reallocate it because it can't get a successful read of the existing data to do the reallocation. If you WRITE to the sector, it SHOULD result in a reallocation, since the existing data doesn't matter at that point; but I believe the system actually attempts writing to the sector first, before reallocating it ... and if the write succeeds it doesn't do the reallocation. However, if the write status is good; but when you attempt to subsequently read the sector it has errors, you can end up in a loop, whereby a write temporarily clears the "pending" status; but the next read results in it being reset to pending. Unfortunately, there's no way to force the drive to simply reallocate the sector. ... but if you're in this kind of a loop, it's a good sign that the disk media is bad in that area -- and it's likely that nearby sectors will see a migration of the same problem. That likely explains why your reallocated count is growing. Personally, I'd relegate that drive to use for backups [where you'll likely write the data, validate it, and store the drive away]; or just toss it. Get a new, reliable drive for your Dockers !! [if not, be sure you're backing them all up frequently to another drive or to your array.]
April 23, 201511 yr Since the drive passes a long test, I doubt this is actually a media problem. More likely a mechanical or power issue, even firmware.
April 23, 201511 yr Author Gary, dockers configuration is backed up nightly to the array to a non cached share. The image can be rebuilt at ease.
April 30, 201511 yr Author The drive is now up to 13 reallocated sectors with pending staying at 1. I see a SSD in my near future... Damn dockers and VMs, making me spend more money. First ram (2GB to 8GB), now vt-d and a SSD are needed. I wonder what the wife needs to buy . . . . .
May 6, 201511 yr Author Another update, the reallocated count is now up to 18, pending still at 1. Otherwise the drive is working fine, its hosting my dockers / VMs and other app data. Backups happen nightly, LOL.
May 6, 201511 yr Author Question for the group. Would you rather spend $139 on a 3TB WD Red and use it to replace a perfectly fine 2TB drive, to use that 2TB drive to replace this hunk a crapola...OR.. spend $139 on a 250GB SSD to replace the 2TB hunk of crapola???
May 6, 201511 yr Question for the group. Would you rather spend $139 on a 3TB WD Red and use it to replace a perfectly fine 2TB drive, to use that 2TB drive to replace this hunk a crapola...OR.. spend $139 on a 250GB SSD to replace the 2TB hunk of crapola??? Do you need capacity or speed? {Obviously you don't need the $139 }
May 6, 201511 yr Author My array has plenty of capacity for the next year at least. I do not find that a regular hard drive being used as my cache drive to be a performance bottle neck. decisions decisions!!!!
May 6, 201511 yr My array has plenty of capacity for the next year at least. I do not find that a regular hard drive being used as my cache drive to be a performance bottle neck. decisions decisions!!!! Spinners can generally keep up with the network anyway. But what about using cache for dockers and VMs? Not sure you would see a bottleneck there either, but that seems to be why many are choosing SSDs.
May 6, 201511 yr Author I'm liking the idea of adding a red drive where a green drive currently exists in my array and reusing that green drive for cache/apps.
Archived
This topic is now archived and is closed to further replies.