November 17, 201510 yr Author Found my backup USB key (Hurray!) and after tweaking with the bios settings provided here http://lime-technology.com/unraid-server-installation/ I finally was able to boot the unraid server! Now back to upgrading the RAM and fixing that damn drive... More to come soon..
November 18, 201510 yr Author OK, upgraded memory, server is up and running, the broken key backup didn't work, so I upgraded the backup key running version 4.7 to 5.0.6, reassigned all the drives (thankfully I have saved in an xls the the old configuration), told the system parity is valid so no parity recalculation. I am not going to run any upgrade utils yet as I already have a message from unraid that there is a REISERFS error on md9 (vs-5150 search_by_key: invalid format found in block 119867091. Fsck?). I will try again to run reiserfsck on disk9.
November 18, 201510 yr Author Here is the reiserfsck report: =========================== root@Tower:~# reiserfsck --check /dev/md9 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md9 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Wed Nov 18 23:29:24 2015 ########### Replaying journal: Trans replayed: mountid 296, transid 61998, desc 6185, len 1, commit 6187, next trans offset 6170 Trans replayed: mountid 296, transid 61999, desc 6188, len 1, commit 6190, next trans offset 6173 Replaying journal: Done. Reiserfs journal '/dev/md9' in blocks [18..8211]: 2 transactions replayed Checking internal tree.. \/ 1 (of 5|/ 2 (of 160|/ 94 (of 141-block 27328513: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (27328513), whole subtree is skipped / 3 (of 160\block 27328555: The level of the node (0) is not correct, (2) expected the problem in the internal node occured (27328555), whole subtree is skipped / 2 (of 5|block 92008419: The level of the node (0) is not correct, (3) expected the problem in the internal node occured (92008419), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 3 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Wed Nov 18 23:31:38 2015 ########### What next?
November 19, 201510 yr Author Thanks dgaschk, running the command now. I still have the original md9 hard disk that was disabled, perhaps the data I will lose is still available on the drive? Any suggestions on the best way to retrieve that?
November 19, 201510 yr Author There is a comprehensive (long) list of files, with all the actions performed. Bottom line these were the last entries Objects without names 240 Empty lost dirs removed 32 Dirs linked to /lost+found: 22 Pass 4 - finished done 1, 0 /sec Deleted unreachable items 6630 I am going to start the unraid server now, but I think this is terrible news. I am going to put back the old drive in and performe a REISERFS check on it too and see if maybe it was parity that was corrupted, and it created a corrupted rebuild of the drive.
November 19, 201510 yr Author Mmh that now won't work. It won't start the array as it says that The replacement disk must be as big or bigger than the original. The rebuilt disk is 2TB, the original is 500Gig. I am stumped now... What to do?
November 19, 201510 yr Mmh that now won't work. It won't start the array as it says that The replacement disk must be as big or bigger than the original. The rebuilt disk is 2TB, the original is 500Gig. I am stumped now... What to do? Have you got a way to plug in the drive external to the array. If necessary this can be via a SATA->USB adapter. That way you can run reiserfsck against the disk without disturbing the disks currently in the array.
November 19, 201510 yr Author No I don't. But I can back up the usb key, wipe it, and install a fresh version of unraid, re-link all of the drives, and declare parity valid (no parity check). Then go to maintenance mode and check the drive..
November 20, 201510 yr Author So I did what I set out to do, put back the old drive, and run reiserfsck --check /dev/md9 The result was: No corruptions found There are on the filesystem: Leaves 112664 Internal nodes 744 Directories 652 Other files 3812 Data block pointers 113435927 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Fri Nov 20 15:47:00 2015 ########### So what gives? Was the parity corrupted why the drive did not build properly when I replaced it? I will run the command on all the drives tomorrow and then rebuild parity, then wipe the new drive and start moving files to it and reduce the number of drives in the system.
November 21, 201510 yr Author It would seem my parity was fuxored. Synch errors 21433. So I can try to replace drive 9 again, and see if the server rebuilds it properly... Round 2, here we go
November 24, 201510 yr Author Replaced drive 9, rebuilt perfectly this time. Moved data from three other drives to consolidate the server. I am seeing some errors (384 on drive 11 and 54 on drive 5) reported by the tower webgui main window. Should I look into it and how? Suggestions welcome and thanks in advance.
November 24, 201510 yr Should I look into it and how? Suggestions welcome and thanks in advance. Yes. I would start by getting a SMART report of those two drives.
November 26, 201510 yr Author One of the drives I removed, alongside with two of the others I consolidated on a 2TB drive, upgraded to v6.1.4 and now recalculating parity. Smart report on the remaining drive with errors coming tonight, once parity is recalculated.
November 26, 201510 yr Author Now that I have upgraded to v6, the main screen shows errors for disk10 (134). Smart report attached WDC_WD20EADS-00S2B0_WD-WCAVY1112779-20151126-1222.txt
November 26, 201510 yr Author I set up the notifications, and I also got these notifications about the drives unRAID Disk 3 SMART health [5]: 26-11-2015 12:34 Warning [TOWER] - reallocated sector ct is 2 ST3500641AS_3PM02LCB (sdd) unRAID Disk 4 SMART health [5]: 26-11-2015 12:34 Warning [TOWER] - reallocated sector ct is 12 ST3500641AS_3PM0K0YE (sdc) unRAID Disk 8 SMART health [188]: 26-11-2015 12:34 Warning [TOWER] - command timeout is 2 ST4000DM000-1F2168_Z3008G1J (sdl) unRAID Disk 9 SMART health [197]: 26-11-2015 12:34 Warning [TOWER] - current pending sector is 1 WDC_WD20EARS-00J2GB0_WD-WCAYY0067586 (sdh) unRAID Disk 10 SMART health [198]: 26-11-2015 12:34 Warning [TOWER] - offline uncorrectable is 5 WDC_WD20EADS-00S2B0_WD-WCAVY1112779 (sdb) unRAID Disk 10 SMART health [197]: 26-11-2015 12:34 Warning [TOWER] - current pending sector is 12 WDC_WD20EADS-00S2B0_WD-WCAVY1112779 (sdb) unRAID array errors: 26-11-2015 12:34 Warning [TOWER] - array has errors Array has 1 disk with read errors I noticed that there is a dashboard too, and the smart status there is consistent with what I see in the pop up alerts. What gives?
November 26, 201510 yr The ones to worry about are disk 9 and 10 because of the pending sectors, you should replace both ASAP. Since there are 2 disks with possible errors, the rebuild could fail or some files get corrupted, I would replace disk 10 first, keep the old disk in case it fails so you can try to copy the data.
November 26, 201510 yr Also keep an eye on disks 3 and 4, if the reallocated sector count increases you might want to consider replacing them. With V6 notifications it's easy to keep track, and prevents surprises like the one you just had, much less likely to have multiple problem disks at the same time.
November 27, 201510 yr Author The ones to worry about are disk 9 and 10 because of the pending sectors, you should replace both ASAP. Since there are 2 disks with possible errors, the rebuild could fail or some files get corrupted, I would replace disk 10 first, keep the old disk in case it fails so you can try to copy the data. Both drives are out of manifacturer warranty. Is 1 and 12 pending sectors really bad for those drives? What if the value remains the same every time I ran the smart test? They are both WD 2TB drives, I could buy a 4TB drive and rebuild the disk with 12 bad sectors, and then transfer the data from the other drive and consolidate them... I am planning to consolidate the other drives too, just not right now.
November 27, 201510 yr Disk 10 has bad sectors for sure and that is why you’ve seen errors in the WebGUI, disk 9 could be ok, I’ve seen before disks with a single pending sector that are ok, but you should at least run a SMART extend test on disk 9 and if it fails should be replace soon, disk 10 should be replaced now. The problem with having a disk with bad sectors is that if a different disk fails you won’t be able to do a correct rebuild, as Unraid needs to successfully read all other disks + parity for a rebuild. Note that if disk 9 also has bad sectors, you may not be able to correctly rebuild disk 10, one or more files could get corrupted, this is why it’s important to have backups or at least checksum files to know what files got corrupted.
November 28, 201510 yr Author Extended smart test on disk9 passed below smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.1.13-unRAID] (local build) Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (AF) Device Model: WDC WD20EARS-00J2GB0 Serial Number: WD-WCAYY0067586 LU WWN Device Id: 5 0014ee 204805b65 Firmware Version: 80.00A80 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Sat Nov 28 00:15:21 2015 PST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (39600) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 451) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3031) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 63 3 Spin_Up_Time 0x0027 191 164 021 Pre-fail Always - 7425 4 Start_Stop_Count 0x0032 095 095 000 Old_age Always - 5952 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 038 038 000 Old_age Always - 45349 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 196 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 77 193 Load_Cycle_Count 0x0032 068 068 000 Old_age Always - 398530 194 Temperature_Celsius 0x0022 117 098 000 Old_age Always - 35 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 45344 - # 2 Extended offline Completed without error 00% 45333 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
November 28, 201510 yr Author But I have just received bad news on drive 2.. smartctl 6.2 2013-07-26 r3841 [x86_64-linux-4.1.13-unRAID] (local build) Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.10 Device Model: ST3500630AS Serial Number: 5QG0A04X Firmware Version: 3.AAE User Capacity: 500,107,862,016 bytes [500 GB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA/ATAPI-7 (minor revision not indicated) Local Time is: Sat Nov 28 00:40:17 2015 PST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: FAILED! Drive failure expected in less than 24 hours. SAVE ALL DATA. See vendor-specific Attribute list for failed Attributes. General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 73) The previous self-test completed having a test element that failed and the test element that failed is not known. Total time to complete Offline data collection: ( 430) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 163) minutes. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 107 080 006 Pre-fail Always - 115196435 3 Spin_Up_Time 0x0003 094 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 093 093 020 Old_age Always - 7203 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 056 045 030 Pre-fail Always - 3071217287865 9 Power_On_Hours 0x0032 069 069 000 Old_age Always - 28011 10 Spin_Retry_Count 0x0013 092 040 097 Pre-fail Always FAILING_NOW 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 323 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 061 048 045 Old_age Always - 39 (Min/Max 36/42) 194 Temperature_Celsius 0x0022 039 052 000 Old_age Always - 39 (0 7 0 0 0) 195 Hardware_ECC_Recovered 0x001a 060 055 000 Old_age Always - 76007460 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0 202 Data_Address_Mark_Errs 0x0032 100 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: unknown failure 90% 14408 843638155 # 2 Short offline Completed: unknown failure 90% 14102 238316519 # 3 Short offline Completed: unknown failure 90% 14094 686677484 # 4 Extended offline Completed: unknown failure 90% 14094 148646063 SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
November 28, 201510 yr Since you have issues with other disks I would keep disk 9 for now, keep monitoring smart attributes, if pending_sectores increase replace it ASAP. As for disk 2, and as far as I know, Spin_Retry_Count means the disk could fail at any moment but until it fails data should be OK, I would first replace disk 12, since it has bad sectors and then disk 2.
November 29, 201510 yr Author The saga continues. I addressed the risk of disk2 by transferring all of the files to drive 8, the newest (and the only with real available space more than 5% of the disk). So far so good. Replaced disk10 with a new 4TB drive. Rebuild begins. Disk 9 reported to have 169 Million read errors. Apparently the rebuild is successful. How do I know that it was really successful? I try to access smart records for disk9 to see if it is a cabling issue, or a disk issue. It is spun down and won't spun back up. Poweroff and restart. All the drive allocations have disappeared. Reallocate drives and parity starts checking. Parity check is interrupted hours later, now disk1 has a red cross next to it, it is in error state (640) and AM OFFICIALLY LOSING THE WILL TO LIVE. Interestingly disk9 now shows ZERO read errors. I cannot believe I am having all these drive issues all of a sudden. I am seriously thinking of moving back to v5.
November 30, 201510 yr Author Just purchased 8 locking SATA cables to connect the 8 motherboard ports to the drives, and then will work to consolidate the other drives. Will reseat all the current SATA cables (again) tonight, remap the drives and start parity check again. Hopefully disk1 does not have serious issues and the problem is only due to poor cable seating. If it is due to serious issues, will I be able to rebuild it as I was recalculating parity?
Archived
This topic is now archived and is closed to further replies.