December 26, 20169 yr Am now diagnosing.. Nevery had any issues.. Changed no hardware, Smart Values were fine.. - I will now bring down the array, save log and reboot and try to reattach drives. - Two drives have red X, disk 3 and disk 9. - They cannot be re-attached after bringing down the array. - I will now reboot the server and try to re-attach again. - Diagnostics is attached It seems to have started here: ec 26 01:01:07 Tower kernel: sas: Enter sas_scsi_recover_host busy: 36 failed: 36 Dec 26 01:01:07 Tower kernel: sas: trying to find task 0xffff88015885ab00 Dec 26 01:01:07 Tower kernel: sas: sas_scsi_find_task: aborting task 0xffff88015885ab00 Dec 26 01:01:07 Tower kernel: sas: sas_scsi_find_task: task 0xffff88015885ab00 is aborted Dec 26 01:01:07 Tower kernel: sas: sas_eh_handle_sas_errors: task 0xffff88015885ab00 is aborted Dec 26 01:01:07 Tower kernel: sas: trying to find task 0xffff88076feae000 - Smart values in the diagnostics show an error getting the smart command. I checked the SMART values manually something like two or three weeks ago, all were clean. Since I have two failed drives at the same time, and also no apparant wear and tear on the drives I think a controller failure is most likely.. I have not worked physically on the system for several weeks so a loose cable or something also does not seem likely. tower-diagnostics-20161226-0720.zip
December 26, 20169 yr Author Array has restarted succesfully, disks still have red X (obviously) and array also auto-started. Now bringing down array again to see if I can add drives.. I am expecting I cannot.. I will start with bringing down all dockers and turning off auto-start, this to attempt and get the array as quiet as possible.. - All dockers are now down and will no longer auto-start - All VM's are now down and will no longer auto-start - New dianostics attached (this is the one after the reboot) - I will now power down the system fully and restart it, to see if a cold boot makes a difference, if it does now I will open up the case and check insides. - System is now restarting. Marvell controllers are now restarting. I get a "not ready or unknown device type"On port 4 on controller 1. So ONE physical failure on the controllers, not two.. - Restarted system still shows to disabled disks, they still cannot be re-attached. System appears to not see them at all. tower-diagnostics-20161226-0732_1.zip
December 26, 20169 yr Author Opened up the system and checked cabling.. Nothing obvious... restarted again.. still no disk3 and disk9... Also no "unknown" disks.. System appears to really not see the drives.. Both drives are in two different drive cages. I moved both failed drives to other drive cages.. Fingers crossed.. -> Just restarted, both drives are still marked X, which is logical.. After stopping the array I still cannot get the drives re-attached.. Even now they are in a different drive cage.. I will do one more attempt to swap them with known working locations.. If the issue then moves to different drives I have an issue with sata and/or connections, if the issue remains with the disks the issue is disk based.
December 26, 20169 yr Author Ok... I placed both drives in a location that another drive was in that was not giving me a red X.. After restarting the system (powerdown) the disk3 and disk9 drives remain the unavailabe ones, so the error moved with the drives to the new locaiton. After stopping the array I still cannot re-add the drives.. I will now work form the assumption that both drives have fully failed at the same time.. I still think this is extremely unlikely but I also cannot think of another solution.. Any insights are appreciated... Since I have no new disks available and its christmas I have now started a MOVE from the failed disks towards correct working array disks. I have enough internal space available to accomodate for the data. The data is now beiing moved from the emulated disks towards a good disk. Scary moments since I am now without parity protection against new failures... Two drives out... God am I happy I set up dual parity.. Ofcourse I also still have my full system backup.. So data-loss should not be an issue in any case but it is going to cost me a sh*tload of work to get this fixed... Aargh..
December 26, 20169 yr Author Mmm... Just did some catching up wrt re-enabling the drive.. Did I do it correctly ? Just read read the manual: https://lime-technology.com/wiki/index.php/Troubleshooting#Re-enable_the_drive It states I need to unassign the disk first.. I want to be absolutely sure I understand this correctly.. Should I really unassign the disk, then restart the array, then stop the array and then re-assign it ? Will parity remain emulating the failed drive after I unassign it ?
December 26, 20169 yr Community Expert Both disks came online after rebooting, disk3 has a pending sector and should be replaced but disk9 looks OK, looks like disk3 brought the other one down. So rebuild disk9 using the same disk, replace disk3 (or leave disable while you don't have a spare).
December 26, 20169 yr Community Expert To rebuild disk 9: -stop the array -unassign disk9 (select no device) -might as well unassign disk 3 -start the array -stop the array -reassign disk9 -leave disk3 unassigned -start array to begin rebuild
December 26, 20169 yr Author Thanks Johny ! Any chance or idea on how it is possible that disk9 brought the other one down ? Should I try and preclear it to see if it comes back alive ? -> Disk9 is now rebuilding. -> In the meantime If will continue copying off the data of disk3 to other parts of the array. I wil probably just remove disk3 and then do a parity rebuild.. I have the spare space.
December 26, 20169 yr Community Expert Driver crashed one of the SASLP, looking at the logs a little more carefully it's possible (even probable) that disk3 is also fine and the pending sector is a false positive, so this is what I recommend: Rebuild disk9, do an extended SMART test on disk3, if it passes also rebuild to the same disk. PS. are you sure array auto-started with both disks disable? it shouldn't. in this case it should, because it was reboot with emulated disks, not missing disks.
December 26, 20169 yr Author The extended test failed after 3 seconds: smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.8.12-unRAID] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Red Device Model: WDC WD30EFRX-68AX9N0 Serial Number: WD-WMC1T0076253 LU WWN Device Id: 5 0014ee 058ae6776 Firmware Version: 80.00A80 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Mon Dec 26 11:43:15 2016 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: (40680) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 408) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x70bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 1 3 Spin_Up_Time 0x0027 179 175 021 Pre-fail Always - 6025 4 Start_Stop_Count 0x0032 091 091 000 Old_age Always - 9488 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 048 048 000 Old_age Always - 38476 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 192 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 109 193 Load_Cycle_Count 0x0032 197 197 000 Old_age Always - 9378 194 Temperature_Celsius 0x0022 121 103 000 Old_age Always - 29 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 38476 29384330 # 2 Short offline Completed: read failure 90% 38475 29384334 # 3 Short offline Completed without error 00% 400 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. It seems to fail during the self-test.
December 26, 20169 yr Community Expert OK, the pending sector is real, in that case you can replace it or try a couple of preclears to see if it clears.
December 26, 20169 yr Author I walked thru my Smart values again to assess age of my drives, the failed drive was 4 1/2 years old, I guess it is not out of band that it is failing.. I did some checks on my other drives and I find that I have 4 that are more then 3 1/2 years old.. 4 and 3 TB drives.. I just ordered an 8TB Seagate desktop drive (It will replace my current 6TB WD RED parity drive) and a new 8TB WD RED. With the 6TB reused parity drive that gives me 14TB of new space, I will then phase out the 4 older drives (also 14TB together).
December 29, 20169 yr Author The rebuild on itself for disk9 completed this night. The move of all data off of disk3 als completed this morning. I have now done a new config and both parity drives are now rebuilding. The new disks should arrive today or tomorrow. I will give them a 3-pass preclear before usign them in the array (this is also why I had to do the new config, I would not want to run without parity protection for over a week.. Man am I happy with the dual parity... !
December 30, 20169 yr Author The disks arrived yesterday and have been acclimating to the inhouse temperature since then, parity rebuild has now completed and I will start preclear on the two new 8TB drives. Thanks for the support. Issue closed.
Archived
This topic is now archived and is closed to further replies.