October 19, 201015 yr Hi, I came home from work today and went to look at my server and noticed a dreaded red-ball in the web interface with 2 Errors reported. Immediately went to the syslog and there is 600kb of the following line: Oct 19 17:52:22 TheBurrow kernel: mdcmd (544027): spindown 6 Oct 19 17:52:22 TheBurrow kernel: md: disk6: ATA_OP_STANDBYNOW1 ioctl error: -5 Appears the rest of the log has been flushed out by all repeats of that operation (spindowns?) apart from the very first line which reads Oct 19 04:40:01 TheBurrow syslogd 1.4.1: restart. Dont know if the error actually occurred at 4:40am or if that is just how far back the error log goes. I did watch something streamed from that disc to my PS3 last night at approximately midnight though and that seemed to work fine. I then tried to run a SMART report on that drive and it came back reporting: root@TheBurrow:~# smartctl -d ata -tlong /dev/sda smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle n Home page is http://smartmontools.sourceforge.net/ Smartctl: Device Read Identity Failed (not an ATA/ATAPI device) A mandatory SMART command failed: exiting. To continue, add one or more '-T perm So it appears that the system cant even see disk 6 (sda) anymore? I havent rebooted the machine yet or done anything apart from the syslog and SMART attempt. Thought i'd come here first for pointers on where to go from here. Any help would be greatly appreciated at this point in time System specs are as follows: Asus P5Q-Deluxe - AHCI mode enabled E6400 (currently underclocked to 1.6ghz) 2gb G.Skill Ram 2xAdapatec 1430SA cards - BIOS disabled. Seasonic M12-700W PSU Lian Li PC-A17B 3xNorco SS-500 5in3 hotswap modules Lexar JumpDrive Firefly 4gb Unraid 4.4.2
October 19, 201015 yr Hi, I came home from work today and went to look at my server and noticed a dreaded red-ball in the web interface with 2 Errors reported. Try rebooting. If drive will still not respond to a smart report request then stop the array, power down, and make confirm the connectors on the cables to the disk are still in place and not loose. If you then still find the disk un-responsive to a smart report you've had your first disk failure. (or you have a bad cable or far less likely, a bad port on the disk controller) It is almost impossible to predict when it happened, since the ability of the array to "simulate" the failed drive to play your movie makes it impossible to use that to determine when it failed. You might have been playing the movie last night from the simulated disk, re-constructed from parity and the other data disks. Whatever you do, DO NOT press a button labeled as "restore" (if it exists on your server version) and DO NOT initialize a new disk configuration or you'll lose the ability to re-construct the failed disk onto its replacement. To replace the disk, just stop the array power down, replace the drive, power up and press "Start" to begin the process of re-constructing the old contents onto the replacement drive..
October 19, 201015 yr Author Joe, Thanks for the feedback. I knew I could trust in your knowledge to point me in the right direction. I rebooted the machine and its come back up to the main unraid web interface still with the red ball as expected. I've putty'd into the server and its letting me do a SMART check this time so looks like the drive is responding to the server now. I've set it to do a smart -tlong and will report back tomorrow morning with the smart report.
October 19, 201015 yr Hello, Odd, I had exactly the same problem with a 1Tb Hitachi, almost the same time. And only 2 errors. The only difference is the continuous error that I was receiving, can't recall what it was, but it was just spamming the syslog. It was red on the drive interface. I tried smartctl, it acted like it couldn't even see the drive, I didn't even get an error, just the smartctl "banner", and the command prompt. I rebooted, still red. There's a procedure somewhere here I followed that fixed it. Might need verification from one of the gurus, but it's along these lines Reboot If the array started (mine didn't), stop the array On the devices page, unassign the disk from the array Start the array Immediately stop the array On the devices page, reassign the disk back to the array in the same drive assignment as before Restart the array (on mine, this immediately kicked off a parity check/rebuild) Once I ran through this procedure and the parity check finished, it was happy. Here's the procedure in the wiki. It's not what I followed, but it's roughly the same. I follow a procedure that Joe L. had posted to another drive failure post, and not the one in his link from the wiki. I just can't find it now http://lime-technology.com/wiki/index.php?title=FAQ#How_do_I_re-enable_a_failed_disk.3F
October 19, 201015 yr Author Thanks for those tips ajeffco. I'd say thats where ill be going next. Here is my SMART report from the drive. smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: SAMSUNG HD103SI Serial Number: S1VSJ9CS910232 Firmware Version: 1AG01118 User Capacity: 1,000,204,886,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 3b Local Time is: Wed Oct 20 07:21:49 2010 EST ==> WARNING: May need -F samsung or -F samsung2 enabled; see manual for details. SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 22) The self-test routine was aborted by the host. Total time to complete Offline data collection: (12959) seconds. Offline data collection capabilities: (0x7f) SMART execute Offline immediate. Auto Offline data collection on/off support. Abort Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 217) minutes. Conveyance self-test routine recommended polling time: ( 23) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 074 074 011 Pre-fail Always - 8680 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 184 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 253 253 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10662 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 1671 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 4 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 082 071 000 Old_age Always - 18 (Lifetime Min/Max 16/23) 194 Temperature_Celsius 0x0022 076 069 000 Old_age Always - 24 (Lifetime Min/Max 11/24) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 368311298 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Aborted by host 60% 1663 - # 2 Short offline Aborted by host 30% 1662 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. The line below seems to indicate a large number of Hardware errors. Does the disc look ok to use still or should I be heading out to get a replacement? 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 368311298
October 19, 201015 yr Thanks for those tips ajeffco. I'd say thats where ill be going next. Here is my SMART report from the drive. smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: SAMSUNG HD103SI Serial Number: S1VSJ9CS910232 Firmware Version: 1AG01118 User Capacity: 1,000,204,886,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 3b Local Time is: Wed Oct 20 07:21:49 2010 EST ==> WARNING: May need -F samsung or -F samsung2 enabled; see manual for details. SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 22) The self-test routine was aborted by the host. Total time to complete Offline data collection: (12959) seconds. Offline data collection capabilities: (0x7f) SMART execute Offline immediate. Auto Offline data collection on/off support. Abort Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 217) minutes. Conveyance self-test routine recommended polling time: ( 23) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 074 074 011 Pre-fail Always - 8680 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 184 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 253 253 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10662 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 1671 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 4 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 082 071 000 Old_age Always - 18 (Lifetime Min/Max 16/23) 194 Temperature_Celsius 0x0022 076 069 000 Old_age Always - 24 (Lifetime Min/Max 11/24) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 368311298 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Aborted by host 60% 1663 - # 2 Short offline Aborted by host 30% 1662 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. The line below seems to indicate a large number of Hardware errors. Does the disc look ok to use still or should I be heading out to get a replacement? 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 368311298 The raw values cannot be interpreted by anyone but the manufacturer. You really need to look at the column called "VALUE" and the column called "THRESH". If the VALUE is equal or below the THRESH, that is bad. Otherwise, you are within the parameters as set by the manufacturer. In the case of the attribute you specified above, the VALUE is 100 and the THRESH is 0. So the VALUE is way above the THRESH and you are ok. The only values whose raw values actually mean something are the Reallocated Sector Ct, Reallocated Event Count, and Current Pending Sectors. If you see these values at anything other than 0, there are some steps you need to take to verify that the drive is not failing. (Frequently when these values start to increase, they keep right on increasing until the drive fails). You can also look at the raw value of the termperate attributes, as well as some of the age-related counts (spin up time, start stop count, power on hours, etc.). Brian
October 19, 201015 yr Actually it shows NO errors at all. The "raw" number has meaning only to the manufacturer. The normalized value is still the original 100, unchanged from when the disk was first initialized. Its failure threshold is 000, and it is nowhere near that. In fact, it has not budged... Nothing looks wrong with the drive at this time... but... did you check for a loose cable? Remember, a "write" to the drive failed... if the cable happens to be maiking connection now, great, but trust me, it will fail again if you do not get it seated properly and it is loose. I'd Stop the array Un-assign the drive on the devices page so it will forget its model/serial number Power down Re-seat the connectors to the drive Power up. Start the array with it un-assigned Stop the array Re-assign the drive (now that it forgot the model/serial number of the drive it will accept it as its own replacement) Start the array by pressing "Start" Allow the re-construction of the disk to complete. Perform a full parity check (Just to make sure you can read what you just wrote) Joe L.
October 19, 201015 yr Author Actually it shows NO errors at all. The normalized value is still the original 100, unchanged from when the disk was first initialized. Its failure threshold is 000, and it is nowhere near that. In fact, it has not budged... Nothing looks wrong with the drive at this time... but... did you check for a loose cable? Remember, a "write" to the drive failed... if the cable happens to be maiking connection now, great, but trust me, it will fail again if you do not get it seated properly and it is loose. Thanks for the feedback guys. Glad to see I haven't (apparently) had another drive go turkey on me. No I didnt break it down yet Joe. I'll be doing that tonight when I get home from work and making sure all the cables etc are ok and reseating it in the hotswap module. Ill then follow your process listed above and get back to the forum as to whether Unraid rebuilds and passes parity check.
October 19, 201015 yr Ah yea, I forgot to mention the cables I had just moved these drives from one mobo to another case/mobo. I replaced the cable completely to make sure it wasn't part of the problem.
October 20, 201015 yr Author I'd Stop the array Un-assign the drive on the devices page so it will forget its model/serial number Power down Re-seat the connectors to the drive Power up. Start the array with it un-assigned Stop the array Re-assign the drive (now that it forgot the model/serial number of the drive it will accept it as its own replacement) Start the array by pressing "Start" Allow the re-construction of the disk to complete. Perform a full parity check (Just to make sure you can read what you just wrote) OK I followed the above procedure overnight after breaking down my server and checking all the cabling and reseating the drive.. I went to bed while it was still rebuilding and it appeared to complete that ok according to the web interface. When I woke up this morning at 7am I refreshed the web interface and was greeted by a green ball on the drive and figured everything was ok. I then attempted to access the rebuilt disk i saw it spin up all the other drives instead of just that drive and realised it was going to read from parity again. The web interface immediately changed to a red ball on that drive. I've attached a copy of the syslog (syslog1) which thankfully has not wrapped around with the error like it did before so hopefully someone can pick up what is starting the error. Tried to run SMART on the disc and of course it returned the not a valid device error where it thinks there is no drive of that ID. It looks like it finished the rebuild at 3:30 this morning and then spun down the disks an hour later (as expected) at 4:30 and then at 5:06 something happened that caused an error. The only plugin I have running on my server is the PS3MediaServer plugin so I'm assuming it tried to access the disk for its "staying alive" thing it does and it failed to wake the drive from spindown or something? Oct 21 03:31:40 TheBurrow kernel: md: sync done. time=20903sec rate=46728K/sec Oct 21 03:31:41 TheBurrow kernel: md: recovery thread sync completion status: 0 Oct 21 04:31:50 TheBurrow kernel: mdcmd (2487): spindown 0 Oct 21 04:31:51 TheBurrow kernel: mdcmd (2488): spindown 1 Oct 21 04:31:52 TheBurrow kernel: mdcmd (2489): spindown 2 Oct 21 04:31:53 TheBurrow kernel: mdcmd (2490): spindown 3 Oct 21 04:31:54 TheBurrow kernel: mdcmd (2491): spindown 4 Oct 21 04:31:55 TheBurrow kernel: mdcmd (2492): spindown 5 Oct 21 04:31:56 TheBurrow kernel: mdcmd (2493): spindown 6 Oct 21 04:31:57 TheBurrow kernel: mdcmd (2494): spindown 7 Oct 21 04:31:58 TheBurrow kernel: mdcmd (2495): spindown 8 Oct 21 04:31:58 TheBurrow kernel: mdcmd (2496): spindown 9 Oct 21 05:06:40 TheBurrow kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x100000 action 0x6 frozen Oct 21 05:06:40 TheBurrow kernel: ata1: edma_err_cause=00000020 pp_flags=00000000, SError=00100000 Oct 21 05:06:40 TheBurrow kernel: ata1: SError: { Dispar } Oct 21 05:06:40 TheBurrow kernel: ata1: hard resetting link Oct 21 05:06:40 TheBurrow kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Oct 21 05:06:40 TheBurrow kernel: ata1.00: configured for UDMA/133 Oct 21 05:06:40 TheBurrow kernel: ata1: EH complete So it appeared the drive worked right up until it was spundown and then disappeared from the server again. I've rebooted the server again and the drive is now coming up as not installed and isnt listed in the devices page as a disk to add. But when I booted my system the drive was reported by the 1430SA as being connected, healthy and available but when I check the /dev/disk/by-id# directory the drive isnt listed in there. Ive taken and attached a second syslog (syslog2) after reboot to see if there is anything in there that is showing up why the drive isnt showing up in the unraid web-interface. I'm now at a loss as to whether its the drive failing (to just wake up or otherwise), a cable gone bad, my 1430SA controller playing up or my norco SS-500 hot-swap bay playing up. syslog.zip syslog2.zip
Archived
This topic is now archived and is closed to further replies.