September 5, 201213 yr I had a parity drive failure in mid August. I RMAed the drive, replaced it and then recalculated parity from my data drives. All of this happened without any issues. On 9/1/12 unmenu ran its scheduled monthly parity check. I was getting hourly email notifications and noticed it was taking a long time...at one point I noticed that the parity check was going at 20 kb/s when I normally see 40,000 kb/s. At the time I was getting 20kb/s I remotely connected to my unraid server and pulled up the main user interface. At that time all drives were green, however I noticed disk5 had errors listed. I was on vacation so I did not have direct access to the server. Utimately the parity check finished however when it finished the parity drive was disabled and has the red dot next to it. Attached is a screenshot. I have a syslog however it is too large to attach. However, here is a link to the zipped syslog in my public dropbox folder: https://dl.dropbox.com/u/10262611/syslog-2012-09-05.zip Thanks, Dan
September 5, 201213 yr Author For the Parity I get this: Smartctl open device: /dev/sda failed: No such device Disk 7 Smart Report: I get this (the /dev/sba: Smartctl open device: /dev/sdb failed: No such device I tried to browse disk7 and I can't get any files to open??? Disk 5 Smart Report: Statistics for /dev/sdc WDC_WD10EARS-00Y_WD-WCAV56190116 smartctl -a -d ata /dev/sdc smartctl 5.39.1 2010-01-28 r3054 [i486-slackware-linux-gnu] (local build) Copyright © 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: WDC WD10EARS-00Y5B1 Serial Number: WD-WCAV56190116 Firmware Version: 80.00A80 User Capacity: 1,000,204,886,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Wed Sep 5 18:06:04 2012 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: FAILED! Drive failure expected in less than 24 hours. SAVE ALL DATA. See vendor-specific Attribute list for failed Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 73) The previous self-test completed having a test element that failed and the test element that failed is not known. Total time to complete Offline data collection: (18960) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 219) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3031) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 197 197 051 Pre-fail Always - 64859 3 Spin_Up_Time 0x0027 126 121 021 Pre-fail Always - 6666 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2248 5 Reallocated_Sector_Ct 0x0033 097 097 140 Pre-fail Always FAILING_NOW 818 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 19348 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 66 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 26 193 Load_Cycle_Count 0x0032 151 151 000 Old_age Always - 148296 194 Temperature_Celsius 0x0022 117 114 000 Old_age Always - 30 196 Reallocated_Event_Count 0x0032 001 001 000 Old_age Always - 783 197 Current_Pending_Sector 0x0032 193 193 000 Old_age Always - 1235 198 Offline_Uncorrectable 0x0030 200 198 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 032 000 Old_age Offline - 19 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed: unknown failure 90% 19348 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
September 5, 201213 yr Author Based on the above results, I am at a loss as to how to proceed. Not sure if I should power cycle my unraid box? One interesting note....the parity drive, disk 5 and disk 7 are all on pcie monoprice sata cards...the one with the sil3132 chipset. In the past it seems like the drives that have failed have always been connected to one of these cards.
September 6, 201213 yr If you search through the forum a believe a user was having a problem with the sil chip-set causing him to have some kind of silent corruption. Looks like disk 5 need to be replaced: SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 197 197 051 Pre-fail Always - 64859 3 Spin_Up_Time 0x0027 126 121 021 Pre-fail Always - 6666 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2248 5 Reallocated_Sector_Ct 0x0033 097 097 140 Pre-fail Always FAILING_NOW 818 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 19348 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 66 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 26 193 Load_Cycle_Count 0x0032 151 151 000 Old_age Always - 148296 194 Temperature_Celsius 0x0022 117 114 000 Old_age Always - 30 196 Reallocated_Event_Count 0x0032 001 001 000 Old_age Always - 783 197 Current_Pending_Sector 0x0032 193 193 000 Old_age Always - 1235 198 Offline_Uncorrectable 0x0030 200 198 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 032 000 Old_age Offline - 19
September 6, 201213 yr Author So, how should I proceed. The parity drive is disabled...and I wonder if it is still good. Do I reboot the server and see if I can get smart reports?
September 7, 201213 yr Shutdown the server check all cables and power connections and attempt a restart.
September 7, 201213 yr Author I shutdown checked cables and restarted. The weird thing is that the parity drive has a blue ball and says new parity disk installed. I assume that I need to preserve whatever is on my parity drive so I am pretty sure I don't check the box about I'm sure I want to do this as it will recreate new parity. What I assume I should do is unassign the parity drive and then see If I can start the array to gain access to the data drives. What do you think? Dan
September 7, 201213 yr I shutdown checked cables and restarted. The weird thing is that the parity drive has a blue ball and says new parity disk installed. I assume that I need to preserve whatever is on my parity drive so I am pretty sure I don't check the box about I'm sure I want to do this as it will recreate new parity. What I assume I should do is unassign the parity drive and then see If I can start the array to gain access to the data drives. What do you think? Dan but DO NOT write to the data disks, that would invalidate parity, even with it un-assigned, as it would no longer be in sync with the data. You do need to deal with the failing disk, and for that to be re-constructed properly, you'll want valid parity. On the 4.X series you can force the array to think parity is good and disk5 is bad with some command line commands. I'm not as sure you can do the same on the 5.0 series... Check with lime-technologies before trying, as you only get one shot. Joe L.
September 8, 201213 yr Author I have unassigned the parity drive and started the array. It looks like I am able to access all drives. Is there a way to check and see if all of my data is OK? Would a reiserfs check do any good? I have spot checked several files and everything looks good. I am also able to run smart reports now on the two drives that I could not before - they look fine to me, but I am no expert in reading these reports. Here is the parity smart report: SMART status Info for /dev/sda smartctl 5.39.1 2010-01-28 r3054 [i486-slackware-linux-gnu] (local build) Copyright © 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Device Model: WDC WD20EARX-008FB0 Serial Number: WD-WCAZAF142676 Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Sat Sep 8 08:57:28 2012 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (33960) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x30b5) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 187 187 021 Pre-fail Always - 5633 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 7 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 382 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 7 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 6 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 2368 194 Temperature_Celsius 0x0022 119 115 000 Old_age Always - 31 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 382 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Here is the disk 7 smart report: Statistics for /dev/sdb ST3320620AS_5QF04PL8 smartctl -a -d ata /dev/sdb smartctl 5.39.1 2010-01-28 r3054 [i486-slackware-linux-gnu] (local build) Copyright © 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.10 family Device Model: ST3320620AS Serial Number: 5QF04PL8 Firmware Version: 3.AAC User Capacity: 320,072,933,376 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Sat Sep 8 09:01:41 2012 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 430) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 115) minutes. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 097 090 006 Pre-fail Always - 201310988 3 Spin_Up_Time 0x0003 095 090 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 098 098 020 Old_age Always - 2281 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 071 060 030 Pre-fail Always - 258533023349 9 Power_On_Hours 0x0032 078 078 000 Old_age Always - 19632 10 Spin_Retry_Count 0x0013 100 099 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 476 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 065 044 045 Old_age Always In_the_past 35 (Lifetime Min/Max 34/36) 194 Temperature_Celsius 0x0022 035 056 000 Old_age Always - 35 (0 16 0 0) 195 Hardware_ECC_Recovered 0x001a 063 052 000 Old_age Always - 36373330 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0 202 Data_Address_Mark_Errs 0x0032 100 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 19632 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
September 9, 201213 yr Author Unraid Version 4.7. I am confident my issue is with the two monoprice sil3132 sata cards. I have had 4 drive failures in the last couple of months and they all have been connected to these controllers.....so I need to replace these two cards. Any recommendations: Supermicro AOC-SASLP 8mv I think is the obvious choice for 4.7. I wonder how close we are to 5.0 final because if we were close maybe I should get a flashed m1050 card to get 6gb/s support. My concern is make the transition to 5.0 while I am in the middle of these drive issues. Any suggestions? Dan
September 9, 201213 yr Use the "trust my parity procedure". It works on version 4.7. You can rebuild the failing disk.
September 9, 201213 yr The m1015's are stable in the latest beta 5.0-rc6-r8168-test2. I have two and they have been running fine.
September 11, 201213 yr Author Current Status: I have ordered the Supermicro AOC saslp 8mv. Once I receive it this is what I think I should do (please confirm or add suggestions). 1. Take Screenshot of main tower page for device assignments and backup data from failing disk5. 2. Powerdown unraid, pull the two SIL3132 cards and install the new Supermicro AOC SASLP 8MV (Question should my parity drive be connected to one of the motherboard sata ports or the supermicro add on card for best performance?) 3. Boot Unraid and assign disks Now I am confused...I assume I have to go through the trust my array procedure: http://lime-technology.com/wiki/index.php/Make_unRAID_Trust_the_Parity_Drive,_Avoid_Rebuilding_Parity_Unnecessarily however this procedure says that you must keep the parity drive in the same slot...which I am confused about since I have to move it off the board it is installed on...data drives can be in different slots. Also it says to only do that procedure if you do not have any missing disks, which I will since Disk 5 is bad. Any suggestions would be greatly appreciated. Dan
September 11, 201213 yr this procedure says that you must keep the parity drive in the same slot...which I am confused about since I have to move it off the board it is installed on...data drives can be in different slots. Also it says to only do that procedure if you do not have any missing disks, which I will since Disk 5 is bad.Physical slot isn't what they are saying, it's the disk list in unraid that matters. As far as rebuilding disk 5, that's what you use the command to tell unraid that parity is good and disk 5 is bad before you start the array.
Archived
This topic is now archived and is closed to further replies.