August 10, 201114 yr Version: Unraid 4.7 Syslog is attached I was moving some files from one spot on my server to another. Mid-move, it stopped responding and became unavailable. I attempted to shutdown -r now and it failed to shutdown. I hard rebooted the machine and it booted up but the array did not fully mount, only the cache drive. Interface is responding very very slowly. Unmenu, also responding very slowly, shows parity check in progress, going at something like 32k/sec and would take 132 days to complete. Attempted to stop parity check through unmenu, and appeared to have worked. Attempted to stop array through unmenu, but there was no change in the regular web interface. Ended up hard resetting again. Same situation the second time around. Found several errors in the syslog. Looks like maybe I have a bad parity drive? I'm having a hard time discerning the lines, but these at the end appear to suggest a drive failure. Aug 9 18:20:15 Tower kernel: ata9.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 (Errors) Aug 9 18:20:15 Tower kernel: ata9.00: BMDMA stat 0x4 (Drive related) Aug 9 18:20:15 Tower kernel: ata9.00: failed command: READ DMA (Minor Issues) Aug 9 18:20:15 Tower kernel: ata9.00: cmd c8/00:08:c7:25:b1/00:00:00:00:00/e8 tag 0 dma 4096 in (Drive related) Aug 9 18:20:15 Tower kernel: res 51/40:00:c8:25:b1/00:00:00:00:00/e8 Emask 0x9 (media error) (Errors) Aug 9 18:20:15 Tower kernel: ata9.00: status: { DRDY ERR } (Drive related) Aug 9 18:20:15 Tower kernel: ata9.00: error: { UNC } (Errors) Aug 9 18:20:20 Tower kernel: ata9.00: configured for UDMA/133 (Drive related) Aug 9 18:20:20 Tower kernel: sd 9:0:0:0: [sdh] Unhandled sense code (Drive related) Aug 9 18:20:20 Tower kernel: sd 9:0:0:0: [sdh] Result: hostbyte=0x00 driverbyte=0x08 (System) Aug 9 18:20:20 Tower kernel: sd 9:0:0:0: [sdh] Sense Key : 0x3 [current] [descriptor] (Drive related) Aug 9 18:20:20 Tower kernel: Descriptor sense data with sense descriptors (in hex): Aug 9 18:20:20 Tower kernel: 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 Aug 9 18:20:20 Tower kernel: 08 b1 25 c8 Aug 9 18:20:20 Tower kernel: sd 9:0:0:0: [sdh] ASC=0x11 ASCQ=0x4 (Drive related) Aug 9 18:20:20 Tower kernel: sd 9:0:0:0: [sdh] CDB: cdb[0]=0x28: 28 00 08 b1 25 c7 00 00 08 00 (Drive related) Aug 9 18:20:20 Tower kernel: end_request: I/O error, dev sdh, sector 145827272 (Errors) Aug 9 18:20:20 Tower kernel: ata9: EH complete (Drive related) Aug 9 18:20:20 Tower kernel: md: disk0 read error (Errors) Aug 9 18:20:20 Tower kernel: handle_stripe read error: 145827208/0, count: 1 (Errors) Thanks for the help! syslog.txt
August 10, 201114 yr Apparently you interrupted the replay of the system journals. (prior to their being mounted.) You must be patient. That process can cake as long as 15 minutes or more, especially if there is also a parity check in progress accessing the same disks. During that process, they will show as "mounting" If you had left them alone, they would have probably eventually mounted. Now, the bulk of the errors are media errors. (un-readable sectors on the disk) /dev/sdh Joe L.
August 10, 201114 yr Author I have left it running since it booted up again (now 1 hr 20 minutes). Number of errors has gone up significantly. How long should I let it run? I do have access to telnet, or the console directly. Finally got unmenu to come back. Also attached updated syslog. syslog-2011-08-09.zip
August 10, 201114 yr Get a "smart" report from the parity drive, /dev/sdh It will give a better picture of what is happening with it. You can use the disk-management page in unMENU, or the "sm" link on the disk in MyMain.
August 10, 201114 yr Author I was unable to get there via either other method, so I ran it from telnet. These are the results from the smart report on /dev/sdh root@Tower:~# smartctl -a -d ata /dev/sdh smartctl 5.39.1 2010-01-28 r3054 [i486-slackware-linux-gnu] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green family Device Model: WDC WD20EADS-32S2B0 Serial Number: WD-WCAVY1342243 Firmware Version: 01.00A01 User Capacity: 2,000,398,934,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Tue Aug 9 20:04:41 2011 CDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (39060) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 186 186 051 Pre-fail Always - 90207 3 Spin_Up_Time 0x0027 143 141 021 Pre-fail Always - 9833 4 Start_Stop_Count 0x0032 098 098 000 Old_age Always - 2960 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 087 087 000 Old_age Always - 9570 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 49 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 18 193 Load_Cycle_Count 0x0032 182 182 000 Old_age Always - 56430 194 Temperature_Celsius 0x0022 106 071 000 Old_age Always - 46 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 196 196 000 Old_age Always - 1463 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 199 199 000 Old_age Offline - 386 SMART Error Log Version: 1 ATA Error Count: 3731 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 3731 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 08 5f b8 d8 e8 Error: UNC 8 sectors at LBA = 0x08d8b85f = 148420703 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 5f b8 d8 e8 08 01:52:06.391 READ DMA ec 00 00 00 00 00 a0 08 01:52:03.416 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 01:52:03.416 SET FEATURES [set transfer mode] Error 3730 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 08 5f b8 d8 e8 Error: UNC 8 sectors at LBA = 0x08d8b85f = 148420703 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 5f b8 d8 e8 08 01:51:58.543 READ DMA ec 00 00 00 00 00 a0 08 01:51:56.112 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 01:51:56.112 SET FEATURES [set transfer mode] Error 3729 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 08 5f b8 d8 e8 Error: UNC 8 sectors at LBA = 0x08d8b85f = 148420703 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 5f b8 d8 e8 08 01:51:51.250 READ DMA ec 00 00 00 00 00 a0 08 01:51:48.817 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 01:51:48.817 SET FEATURES [set transfer mode] Error 3728 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 08 5f b8 d8 e8 Error: UNC 8 sectors at LBA = 0x08d8b85f = 148420703 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 5f b8 d8 e8 08 01:51:44.002 READ DMA ec 00 00 00 00 00 a0 08 01:51:41.571 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 01:51:41.571 SET FEATURES [set transfer mode] Error 3727 occurred at disk power-on lifetime: 9570 hours (398 days + 18 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 08 5f b8 d8 e8 Error: UNC 8 sectors at LBA = 0x08d8b85f = 148420703 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 5f b8 d8 e8 08 01:51:36.702 READ DMA ec 00 00 00 00 00 a0 08 01:51:33.768 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 01:51:33.768 SET FEATURES [set transfer mode] SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 6102 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@Tower:~#
August 10, 201114 yr Author Ok so I will order a new WD20EARS. Until it gets here how can I access the files?
August 10, 201114 yr Author Overnight the remaining drives mounted. This is the current status. Should I wait a week for the parity check to finish? New drive will be here before then. syslog-2011-08-10.zip
August 10, 201114 yr Overnight the remaining drives mounted. This is the current status. Should I wait a week for the parity check to finish? New drive will be here before then. No... Stop the parity check, stop the array, power down, and replace the parity disk. You will take the chances that the replacement is good, so do an initial parity calc, then a subsequent parity check.
August 11, 201114 yr Author I ordered another drive, but for future reference, is there an easy way to get the server back up and running with a 'full' array without replacing it? I have enough free space. What I am wondering is can I pick a new parity drive from my existing drives, have unraid move the contents to other disks and begin the parity calculation? I am guessing I have to do the moving of the existing contents manually.
August 11, 201114 yr I ordered another drive, but for future reference, is there an easy way to get the server back up and running with a 'full' array without replacing it? I have enough free space. What I am wondering is can I pick a new parity drive from my existing drives, have unraid move the contents to other disks and begin the parity calculation? I am guessing I have to do the moving of the existing contents manually. Yes, you can move all the files off of a drive, then stop the array assign that drive as parity set a new disk configuration by logging in and typing initconfig then start the array and let it build parity based on the new disk configuration.
Archived
This topic is now archived and is closed to further replies.