March 3, 201016 yr I recently updated from 4.5 to 4.5.1 because i was having problems with windows sayign there were "too many files open". However, after upgrading to 4.5.1 the errors seemed to become more frequent. usually when they happened, all of the shares on the unraid server appeared empty in windows, and i woudl have to start and stop the array in order for it to work again. well this mornign it happened again and after restarting the array, I noticed that one of my discs has a RED icon next to it. So, DO I immediately assume that the drive is dead and buy a new one or are there certain checks i shoudl perform before doign that? whats the best course of action?
March 3, 201016 yr Author oh and if it helps, when I click on the disc with the red icon (disk1), for "File system type:" it says: "partition 1 does not exist"
March 3, 201016 yr whats the best course of action? Post a syslog. Do NOT press the button labeled as "Restore" unless you do not want the old contents of the failed drive. Pressing it will forget the drive ever existed and immediately invalidate parity causing a new parity calc to start when you next Start the array. Only press the button labeled "Start" if you replace the drive with another. If the disk did not mount properly, it would not have a file-system, but there could be other corruption if you did not shot the array down cleanly. Odds are you will need to perform a file-system check on the drive BUT we will not know until you post a copy of the syslog. Instructions for grabbing a copy of the syslog are in the wiki under "troubleshooting" Attach it to your next post. Joe L.
March 3, 201016 yr Author thanks for the reply - one additional note is that although its just disk 1 that has the red dot, the array will not start. I click the start button and a few seconds later refresh the page and the array is still stopped and says "Configuration Valid". luckily even though the array is stopped and none of the share are accessible, the flash share is accessible and I was able to get the syslog and attach it here. syslog.txt
March 3, 201016 yr The disk identified as /dev/sde is not responding to the disk controller properly. It could be a defective disk, or, it could be a bad SATA cable, or a cable getting noise induced in it because it is adjacent to another noisy cable, or bad port on the disk controller. Most likely will be the cable... but the fact that it sees the disk means whatever it is is not completely broken. Apparently, the disk just won't identify itself. Mar 3 08:05:44 StorageTower kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Mar 3 08:05:44 StorageTower kernel: ata5.00: failed to read native max address (err_mask=0x1) Mar 3 08:05:44 StorageTower kernel: ata5.00: HPA support seems broken, skipping HPA handling Mar 3 08:05:44 StorageTower kernel: ata5.00: ATA-7: WDC WD5000KS-00MNB0, 07.02E07, max UDMA/133 Mar 3 08:05:44 StorageTower kernel: ata5.00: 976773168 sectors, multi 0: LBA48 NCQ (depth 31/32) Mar 3 08:05:44 StorageTower kernel: ata5.00: configured for UDMA/133 (device error ignored) Mar 3 08:05:44 StorageTower kernel: scsi 5:0:0:0: Direct-Access ATA WDC WD5000KS-00M 07.0 PQ: 0 ANSI: 5 Mar 3 08:05:44 StorageTower kernel: sd 5:0:0:0: [sde] 976773168 512-byte logical blocks: (500 GB/465 GiB) Mar 3 08:05:44 StorageTower kernel: sd 5:0:0:0: [sde] Write Protect is off Mar 3 08:05:44 StorageTower kernel: sd 5:0:0:0: [sde] Mode Sense: 00 3a 00 00 Mar 3 08:05:44 StorageTower kernel: sd 5:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA Mar 3 08:05:44 StorageTower kernel: sde: Mar 3 08:05:44 StorageTower kernel: ata5: failed to read log page 10h (errno=-5) Mar 3 08:05:44 StorageTower kernel: ata5.00: exception Emask 0x1 SAct 0x1 SErr 0x0 action 0x0 Mar 3 08:05:44 StorageTower kernel: ata5.00: irq_stat 0x40000008 Mar 3 08:05:44 StorageTower kernel: ata5.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag 0 ncq 4096 in Mar 3 08:05:44 StorageTower kernel: res 40/00:04:00:00:00/00:00:00:00:00/40 Emask 0x1 (device error) Mar 3 08:05:44 StorageTower kernel: ata5.00: status: { DRDY } Mar 3 08:05:44 StorageTower kernel: ata5.00: both IDENTIFYs aborted, assuming NODEV Mar 3 08:05:44 StorageTower kernel: ata5.00: revalidation failed (errno=-2) Mar 3 08:05:44 StorageTower kernel: ata5: hard resetting link Mar 3 08:05:44 StorageTower kernel: ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Mar 3 08:05:44 StorageTower kernel: ata5.00: configured for UDMA/133 (device error ignored) Mar 3 08:05:44 StorageTower kernel: ata5: EH complete Mar 3 08:05:44 StorageTower kernel: ata5: failed to read log page 10h (errno=-5) Mar 3 08:05:44 StorageTower kernel: ata5.00: NCQ disabled due to excessive errors Mar 3 08:05:44 StorageTower kernel: ata5.00: exception Emask 0x1 SAct 0x1 SErr 0x0 action 0x0 Mar 3 08:05:44 StorageTower kernel: ata5.00: irq_stat 0x40000008 Mar 3 08:05:44 StorageTower kernel: ata5.00: cmd 60/08:00:00:00:00/00:00:00:00:00/40 tag 0 ncq 4096 in Mar 3 08:05:44 StorageTower kernel: res 40/00:04:00:00:00/00:00:00:00:00/40 Emask 0x1 (device error) Mar 3 08:05:44 StorageTower kernel: ata5.00: status: { DRDY } Mar 3 08:05:44 StorageTower kernel: ata5.00: both IDENTIFYs aborted, assuming NODEV Mar 3 08:05:44 StorageTower kernel: ata5.00: revalidation failed (errno=-2) Mar 3 08:05:44 StorageTower kernel: ata5: hard resetting link You can get the status of the drive by typing smartctl -a -d ata /dev/sde somehow, I don't think it will respond. If it does, you can try a short test with: smartctl -d ata -t short /dev/sde Wait a few minutes then get a second status report using the first command. If you do resolve the cable issue (if it is a cable that is loose or bad) you will need to un-assign the disk, start the array, then re-assign the disk and start it once more. (It will then think it is a new replacement drive and begin the process of re-constructing the contents onto it.) You will press the "Start" button to re-start the array. As I said, do not press the button labeled as "Restore" as it has nothing to do with restoring data. Pressing it now will cause your array to forget the failed disk ever existed and it will set a new disk configuration without it (and immediately invalidate parity)
March 3, 201016 yr Author interesting - unfortunatley i am not home with the server right now but tonight when i get home I will inspect the cable. I ran the two commands you mentioned... smartctl -a -d ata /dev/sde It did return a bunch of info that mean a whole lot to me... so then I ran the second command smartctl -d ata -t short /dev/sde and this one said to wait two minutes for the test to complete and returned me to the command prompt - so what am i supposed to do after those two minutes have passed to see the results? here is the "transcript" from my telnet session so you can see the results of the commands: StorageTower login: root Linux 2.6.31.12-unRAID. root@StorageTower:~# smartctl -a -d ata /dev/sde smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle n Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Second Generation Serial ATA family Device Model: WDC WD5000KS-00MNB0 Serial Number: WD-WCANU1237754 Firmware Version: 07.02E07 User Capacity: 500,107,862,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Wed Mar 3 10:47:54 2010 GMT+5 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: FAILED! Drive failure expected in less than 24 hours. SAVE ALL DATA. See vendor-specific Attribute list for failed Attributes. General SMART Values: Offline data collection status: (0x85) Offline data collection activity was aborted by an interrupting command f rom host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 241) Self-test routine in progress... 10% of test remaining. Total time to complete Offline data collection: (14400) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 179) minutes. Conveyance self-test routine recommended polling time: ( 6) minutes. SCT capabilities: (0x103f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0003 217 216 021 Pre-fail Always - 6125 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2231 5 Reallocated_Sector_Ct 0x0033 117 117 140 Pre-fail Always FAILI NG_NOW 664 7 Seek_Error_Rate 0x000f 193 192 051 Pre-fail Always - 135 9 Power_On_Hours 0x0032 065 065 000 Old_age Always - 26177 10 Spin_Retry_Count 0x0013 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 051 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 175 194 Temperature_Celsius 0x0022 253 253 000 Old_age Always - 44 196 Reallocated_Event_Count 0x0032 194 194 000 Old_age Always - 6 197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 165 200 Multi_Zone_Error_Rate 0x0009 200 200 051 Pre-fail Offline - 0 SMART Error Log Version: 1 ATA Error Count: 251 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 251 occurred at disk power-on lifetime: 26177 hours (1090 days + 17 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 00 00 00 00 00 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- e3 00 00 00 00 00 00 00 44d+19:19:26.983 IDLE ec 00 00 00 00 00 00 00 44d+19:19:26.896 IDENTIFY DEVICE ef 03 46 00 00 00 00 00 44d+19:19:26.896 SET FEATURES [set transfer mode] ec 00 00 00 00 00 00 00 44d+19:19:26.895 IDENTIFY DEVICE c8 00 08 00 00 00 00 00 44d+19:19:26.895 READ DMA Error 250 occurred at disk power-on lifetime: 26177 hours (1090 days + 17 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 46 00 00 00 a0 Device Fault; Error: ABRT Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- ef 03 46 00 00 00 00 00 44d+19:19:26.896 SET FEATURES [set transfer mode] ec 00 00 00 00 00 00 00 44d+19:19:26.895 IDENTIFY DEVICE c8 00 08 00 00 00 00 00 44d+19:19:26.895 READ DMA ec 00 00 00 00 00 00 00 44d+19:19:26.894 IDENTIFY DEVICE ef 03 46 00 00 00 00 00 44d+19:19:26.894 SET FEATURES [set transfer mode] Error 249 occurred at disk power-on lifetime: 26177 hours (1090 days + 17 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 08 00 00 00 e0 Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 00 00 00 00 00 44d+19:19:26.895 READ DMA ec 00 00 00 00 00 00 00 44d+19:19:26.894 IDENTIFY DEVICE ef 03 46 00 00 00 00 00 44d+19:19:26.894 SET FEATURES [set transfer mode] ec 00 00 00 00 00 00 00 44d+19:19:26.892 IDENTIFY DEVICE c8 00 08 00 00 00 00 00 44d+19:19:26.892 READ DMA Error 248 occurred at disk power-on lifetime: 26177 hours (1090 days + 17 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 46 00 00 00 a0 Device Fault; Error: ABRT Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- ef 03 46 00 00 00 00 00 44d+19:19:26.894 SET FEATURES [set transfer mode] ec 00 00 00 00 00 00 00 44d+19:19:26.892 IDENTIFY DEVICE c8 00 08 00 00 00 00 00 44d+19:19:26.892 READ DMA ec 00 00 00 00 00 00 00 44d+19:19:26.891 IDENTIFY DEVICE ef 03 46 00 00 00 00 00 44d+19:19:26.891 SET FEATURES [set transfer mode] Error 247 occurred at disk power-on lifetime: 26177 hours (1090 days + 17 hours) When the command that caused the error occurred, the device was active or idle . After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 61 08 00 00 00 e0 Device Fault; Error: ABRT 8 sectors at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 00 00 00 00 00 44d+19:19:26.892 READ DMA ec 00 00 00 00 00 00 00 44d+19:19:26.891 IDENTIFY DEVICE ef 03 46 00 00 00 00 00 44d+19:19:26.891 SET FEATURES [set transfer mode] ec 00 00 00 00 00 00 00 44d+19:19:26.890 IDENTIFY DEVICE c8 00 08 00 00 00 00 00 44d+19:19:26.890 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA _of_first_error # 1 Short offline Completed without error 00% 6657 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@StorageTower:~# smartctl -d ata -t short /dev/sde smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION === Sending command: "Execute SMART Short self-test routine immediately in off-line mode". Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful. Testing has begun. Please wait 2 minutes for test to complete. Test will complete after Wed Mar 3 10:51:26 2010 Use smartctl -X to abort test. root@StorageTower:~#
March 3, 201016 yr interesting - unfortunatley i am not home with the server right now but tonight when i get home I will inspect the cable. I ran the two commands you mentioned... smartctl -a -d ata /dev/sde It did return a bunch of info that mean a whole lot to me... You cut off the right-most side of the report, but I can see this line which says it all: 5 Reallocated_Sector_Ct 0x0033 117 117 140 Pre-fail Always FAILI Your drive has failed. It has used up its entire amount of spare sectors and re-allocated all it can. so then I ran the second command smartctl -d ata -t short /dev/sde and this one said to wait two minutes for the test to complete and returned me to the command prompt - so what am i supposed to do after those two minutes have passed to see the results? I guess you missed where I said just to get a second status report. Wait a few minutes then get a second status report using the first command. But that does not matter since you also did not read the top of the first status report where it says: SMART overall-health self-assessment test result: FAILED! Drive failure expected in less than 24 hours. SAVE ALL DATA. Now, it always says drive failure expected in 24 hours or less... Trust me, the drive has already failed. Now is the time to go to the local store and purchase a replacement drive as large (or larger) than the failed drive, but not larger than the parity drive. No time to seek out a sale price. Then, Stop the array, power down, replace the failed drive, power up and press the "Start" button to begin the reconstruction of the new drive. As I said earlier, do not even think of pressing the button labeled "Restore" as it does not restore data, it resets your disk configuration to a point before you had calculated parity. It immediately invalidates parity and would eliminate your ability to re-construct the old contents of the drive. Be very careful when replacing the drive so you do not dislodge any other SATA cables. Don't want to complicate the failure by having two or more drives fail at the same time. Joe L. .
March 3, 201016 yr Author ok gotcha - and yes sorry i did miss the part where you said to run another status report. sorry about that... I will run out at lunch to wal mart or somethgin and get a new drive and replace it when i get home tonight. thank you so much for your assistance!!
Archived
This topic is now archived and is closed to further replies.