spinbot Posted March 23, 2016 Share Posted March 23, 2016 Basics: unRAID Server Pro version: 5.0.5 Background: One of my drives appeared to have some issues as it was automatically removed from the array. I had 3 other drives "spare" on my shelf, so I first tried to replace it was my former parity 4TB drive that I believed to have no issues, I just ended up replacing it as a new 4TB drive i put in was a few bits larger (HPA maybe). I didn't preclear the former 4TB parity drive, which may turn out to have been a problem. I did start the "rebuild" process after I changed the drive assigned from the faulty 1.5TB to the 4TB and let the process run for a many hours. When I checked back in on it, 4 of my other drives were showing errors now, despite the fact they shouldn't even be needed for this process. On top of that, the rebuild went from having "hours" remaining to "weeks". Clearly something was wrong so I stopped the process, shut the machine down and went through every connector to ensure they were all snug. Given the 4 drives showing errors were all off the same break-out cable, I figured it must have gotten loose on the SATA card. I turn the machine back on and the only drive with issues was the one that I was trying to rebuild. As I now have my doubts on the reliability of that former 4TB parity drive, I opted to hook up a new, precleared/passed 2TB drive I had instead for the rebuild. Here is where the problem stands: When I change the settings to select the 2TB drive, it thinks the drive that is replacing is the 4TB drive. The actual drive it is replacing is a 1.5TB drive - the 4TB is the one I canceled the rebuild on. Question: How do I get my parity to rebuild the 1.5TB of data on the 2TB drive when server thinks I should have a 4TB in that slot? I don't believe the data is lost as the parity drive shouldn't have changed what's stored on it. Any direction would be appreciated as this is not the typical error I have dealt with before when replacing drives. Thanks in advance! Quote Link to comment
JorgeB Posted March 23, 2016 Share Posted March 23, 2016 How do I get my parity to rebuild the 1.5TB of data on the 2TB drive when server thinks I should have a 4TB in that slot? You can't, since you started rebuilding with a 4TB disk the filesystem was expanded to that capacity, rebuild was canceled but it's not possible now to swap to a smaller drive. If the old disk was OK you could do a new config, but if it's bad you can only use a 4TB or bigger (depending on parity size). Quote Link to comment
spinbot Posted March 23, 2016 Author Share Posted March 23, 2016 At this point, would I be best off preclearing the drive that I had to abort the rebuild on, assuming it was okay and the rebuild issue was related to some other factor? My only other available drive is larger than the parity drive, so that would present a new challenge having to replace the parity drive first before being able to rebuilt the other drive. If I preclear the 4TB that the array now things is in the slot and then try to rebuild to it, I am now starting to question if the parity drive has the necessary info to rebuild the 1.5TB of lost data or did it get changed when I tried to rebuild it my first attempt (thus its expectation I should be putting in a 4TB for the rebuild) EDIT: What if the drive that started this process is actually still okay? It still connected to the array and I am able to run smart status and hdparm on it. Although I wouldn't be able to set this drive back to the space the server thinks it should be (disk9), would I be able to add it like a new disk (disk12) and rebuild the parity? I realize that if the drive really is bad then I will have messed up my parity and never be able to restore it. The issue with this process would be that I doubt the software will let me leave "Disk9" as "no device" and then rebuild the parity. Quote Link to comment
JorgeB Posted March 23, 2016 Share Posted March 23, 2016 The parity can rebuild all your 1.5TB of data as long you use a 4TB or larger disk. If you think the 4TB disk is OK and the previous issue was the cable you can try to rebuild again, it there's any error you can also abort again. Quote Link to comment
JorgeB Posted March 23, 2016 Share Posted March 23, 2016 EDIT: What if the drive that started this process is actually still okay? It still connected to the array and I am able to run smart status and hdparm on it. Although I wouldn't be able to set this drive back to the space the server thinks it should be (disk9), would I be able to add it like a new disk (disk12) and rebuild the parity? I realize that if the drive really is bad then I will have messed up my parity and never be able to restore it. The issue with this process would be that I doubt the software will let me leave "Disk9" as "no device" and then rebuild the parity. If the 1.5TB disk is OK you could do a new config, reassign all your disks and sync parity, but this is more risky, if the disk is really bad you'll be left with a bad disk and no parity. Quote Link to comment
spinbot Posted March 23, 2016 Author Share Posted March 23, 2016 Thanks for the info. I think I will start with hooking the 4TB drive I had issues with again, but before trying to rebuild the data to it, I am going to do some testing on it. I'll run at a preclear cycle and check some of the drive info before and after the test to see if error rates have risen or not. This will take a few days to perform, given the size of the drive! I'll be back, hopefully with good news Quote Link to comment
spinbot Posted March 23, 2016 Author Share Posted March 23, 2016 Without doing anything but powering down the server, hooking up the 4TB drive that I previously cancelled the rebuild on, the drive automatically returned to rebuilding itself. I guess I will let this ride out as long as the estimated time remains reasonable (800 minutes) and the "error" count doesn't increase. Assuming it completes this time, I would think my next step would be to run a parity check without correcting errors. If it passes, then maybe run a full parity check again and do some testing of the drive, just as an additional reliability check. Quote Link to comment
spinbot Posted March 24, 2016 Author Share Posted March 24, 2016 Things have become more "challenging" now and the potential for data loss has risen I believe. The rebuild of the former 1.5TB drive on to 4TB drive completed, however 3 negative observations: 1. Despite the array showing up and running okay, when I check disk9's contents it doesn't have 1.5TB of data. It actually has under 400GB. 2. My parity status is listed as: Parity is Valid:. Last parity check 16885 days ago . Parity updated 172293584 times to address sync errors (that means my last parity check was done before I was born) 3. During the rebuild of disk9 , for some reason my device status shows disk10 as having the same number of write to it as disk9 had , however it has a huge error rate: writes: 18842983 errors: 154394030 Why the array was trying to write to disk10 during the rebuild of disk9 makes no sense to me. As I type this, disk10 was just kicked out of the array, which surely is from the error counts, however prior to its dropping I did check the files and like disk9, it was well below the 1.5TB it should have had. I'm starting to get that drowning feeling as things go from concerning to very concerning (as I am fully aware 2 drives going down is unrecoverable). I defer back to those of higher skill and seek your advice as to what might be happening and the best course of action moving forward. Thanks for any advice/assistance you can provide in advance. Boat anchor is not the option as too many hundreds of hours went in to building this and transferring DVD's to digital form! I went to download and attach my syslog, but I don't have anything in it, which is another suspicious matter. I checked through both UnMenu as well as the Main UnRaid GUI interface. Quote Link to comment
JorgeB Posted March 24, 2016 Share Posted March 24, 2016 It's difficult to give advice without diagnostics, since you're still on v5 see here how to get SMART reports: http://lime-technology.com/wiki/index.php?title=Troubleshooting#Obtaining_a_SMART_report If possible also include one from the first failed disk. Quote Link to comment
spinbot Posted March 24, 2016 Author Share Posted March 24, 2016 this is the drive that started the problems some time ago. As I had spare drives, I initially tried replacing it was a 4TB drive but it ended up being a few bit bigger than my parity drive so I ended up replacing the parity instead. I then used the old parity drive to replace the drive I am posting here: SMART status Info for /dev/sdi smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (AF) Device Model: WDC WD15EARS-00J2GB0 Serial Number: WD-WCAYY0273566 LU WWN Device Id: 5 0014ee 2afdf0659 Firmware Version: 80.00A80 User Capacity: 1,500,301,910,016 bytes [1.50 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Thu Mar 24 11:02:23 2016 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (34560) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 394) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3031) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 220 154 021 Pre-fail Always - 5966 4 Start_Stop_Count 0x0032 096 096 000 Old_age Always - 4206 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 045 045 000 Old_age Always - 40258 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 121 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 44 193 Load_Cycle_Count 0x0032 193 193 000 Old_age Always - 23910 194 Temperature_Celsius 0x0022 126 102 000 Old_age Always - 26 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 40236 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Here is the 4TB drive I tried to rebuild the above on to: Statistics for /dev/sdf WDC_WD40EFRX-68WT0N0_WD-WCC4E1333752 smartctl -a -d ata /dev/sdf smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WDC WD40EFRX-68WT0N0 Serial Number: WD-WCC4E1333752 LU WWN Device Id: 5 0014ee 209efb4a8 Firmware Version: 80.00A80 User Capacity: 4,000,785,948,160 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Thu Mar 24 11:06:05 2016 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (52080) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 521) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x703d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 180 178 021 Pre-fail Always - 7983 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 186 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 2666 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 15 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 3 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 381 194 Temperature_Celsius 0x0022 127 104 000 Old_age Always - 25 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Here is the "disk10" that had all the errors during the rebuild of disk9 (above): SMART status Info for /dev/sdo smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.11 Device Model: ST31500341AS Serial Number: 9VS2L1L4 LU WWN Device Id: 5 000c50 015af8095 Firmware Version: CC1H User Capacity: 1,500,301,910,016 bytes [1.50 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Thu Mar 24 11:09:23 2016 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 609) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 299) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x103f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 118 099 006 Pre-fail Always - 185620484 3 Spin_Up_Time 0x0003 100 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 096 096 020 Old_age Always - 4420 5 Reallocated_Sector_Ct 0x0033 099 099 036 Pre-fail Always - 53 7 Seek_Error_Rate 0x000f 074 060 030 Pre-fail Always - 25605074 9 Power_On_Hours 0x0032 043 043 000 Old_age Always - 50207 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 8 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 146 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 1 189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 270 190 Airflow_Temperature_Cel 0x0022 071 046 045 Old_age Always - 29 (Min/Max 28/38) 194 Temperature_Celsius 0x0022 029 054 000 Old_age Always - 29 (0 7 0 0 0) 195 Hardware_ECC_Recovered 0x001a 040 030 000 Old_age Always - 185620484 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 3 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 271673861352864 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2254842753 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 3816245401 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 38352 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Quote Link to comment
JorgeB Posted March 24, 2016 Share Posted March 24, 2016 this is the drive that started the problems some time ago. As I had spare drives, I initially tried replacing it was a 4TB drive but it ended up being a few bit bigger than my parity drive so I ended up replacing the parity instead. I then used the old parity drive to replace the drive I am posting here: SMART status Info for /dev/sdi 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 The first disk that failed appears to have issues, good news is that if the rebuild is not successful, and with some luck, you should be able to recover most data from it. Both other disks look fine, there are some CRC_UDMA errors on disk10, this usually indicates bad sata cable, but very few errors to be an issue, and nothing that would explain disappearing data, but disk 10 showing errors during the rebuild can explain the missing data on disk9, those parity sync errors are also troubling, a rebuild should not cause sync errors. That many disks showing errors and dropping out, would point to cable/controller/power supply issue, at this point, and if you have access to another server I would use it, do a new config to create an array with all disks except the old disk9, check that disk10 shows all data and do a parity sync, and then try to copy all possible data from old disk9 to the new array. If you can't use another server try at least a different power supply and controller(s). Quote Link to comment
spinbot Posted March 24, 2016 Author Share Posted March 24, 2016 Only "extra" items I have are single SATA cables (not breakout cables) and hard drives. What information in the first Smart Report I posted brings you to the conclusion it was a problem? I've got to try and figure this out with the limited options I have with the hope of not having to go to the basement and dig out DVD's and re-rip hundreds of them. If disk9's original 1.5TB HDD, the original problem, is actually still functioning, is it too late to put it back in to the array now that the server thinks a 4TB drive replaced it? I can also check, or just replace the sata cable on disk10. If its on a breakout cable, then I would have to order one online. If this makes any difference, I did make a backup of my Flash Drive before these drive swaps started (ie. taken on Mar.21). Quote Link to comment
JorgeB Posted March 25, 2016 Share Posted March 25, 2016 Original disk9 has one pending sector, that usually means bad sector(s), a single pending sector can be a false positive, but since the disk failed it's probably not, you can confirm by doing an extended SMART test, grab a new SMART report, a failure will look like this: Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 6681 2850171655 If it passes it's ok but if it fails with a read error it's really bad and shouldn't be used in the array, but you should still be able to copy most of it's data if needed. The issue here is that you're having multiple errors and it's difficult to do a successful rebuild like that. Quote Link to comment
spinbot Posted March 27, 2016 Author Share Posted March 27, 2016 The idea was that if the original disk9 (WDC_WD15EARS-00_WD-WCAYY0273566) 1.5TB drive I put in was truly fine and lets just say it was a loose cable or one tiny thing that triggered its stoppage, then is their a way to return that to array in the location the 4TB I tried to rebuild in its place? If it can go back in, then maybe the disk10 (ST31500341AS_9VS2L1L4) that somehow came up with errors during the previous drives repair and then somehow lost 2/3 of the information could be restored? Here's another odd observation: when i view the drives not in the protected array (using UnMenu's Disk Management Section), the two drives mentioned above are listed as 1TB in size, yet both are actually 1.5TB, as you can see from their model/serial numbers). Model/Serial Temp Size Device Mounted File System WDC_WD15EARS-00_WD-WCAYY0273566 26°C 1T /dev/sdi partition (1,465,138,552 blocks): /dev/sdi1 reiserfs ST31500341AS_9VS2L1L4 28°C 1T /dev/sdn partition (1,465,138,552 blocks): /dev/sdn1 reiserfs ------------ I am running the long smart test on the original "disk9" right now, using the option in UnMenu which appears to run the command line: smartctl -t long -d ata /dev/sdi 2>&1 I don't believe I've run this test before, so I am not sure how long it will take before results are presented. I will post them once they appear, or in the morning should it take a while (I'm -5:00GMT - Eastern Time Zone = 00:34 Mar.27 currently) Quote Link to comment
JorgeB Posted March 27, 2016 Share Posted March 27, 2016 The idea was that if the original disk9 (WDC_WD15EARS-00_WD-WCAYY0273566) 1.5TB drive I put in was truly fine and lets just say it was a loose cable or one tiny thing that triggered its stoppage, then is their a way to return that to array in the location the 4TB I tried to rebuild in its place? If it can go back in, then maybe the disk10 (ST31500341AS_9VS2L1L4) that somehow came up with errors during the previous drives repair and then somehow lost 2/3 of the information could be restored? If old disk9 passes the extended SMART test you can try to do a new config, this may also help with disk10 missing data. -take a screenshot of current array -go to tools and click new config -reassign all disks, using old disk9, don't assign parity for now, leave it empty -start array and check if all data is there, especially disks 9 and 10 -if all looks fine, stop array, assign parity and start array to begin parity sync if disk10 is still not showing all data don't assign parity, you can try and see how the emulated disk looks like, but the many sync errors you mentioned above are troubling for this option. Here's another odd observation: when i view the drives not in the protected array (using UnMenu's Disk Management Section), the two drives mentioned above are listed as 1TB in size, yet both are actually 1.5TB, as you can see from their model/serial numbers). Model/Serial Temp Size Device Mounted File System WDC_WD15EARS-00_WD-WCAYY0273566 26°C 1T /dev/sdi partition (1,465,138,552 blocks): /dev/sdi1 reiserfs ST31500341AS_9VS2L1L4 28°C 1T /dev/sdn partition (1,465,138,552 blocks): /dev/sdn1 reiserfs ------------ The disk sizes are wrong but the partition sizes are correct, I haven't used Unmenu in so long that I don't remember if it's normal, but would think not, you should consider upgrading to v6 after solving this, many new features like system notifications, SMART warnings, etc and it's harder to support v5 because most users don't use it anymore. Quote Link to comment
spinbot Posted March 27, 2016 Author Share Posted March 27, 2016 I do agree I need to upgrade to version 6. Historically I was upgrading the software each time I need support for larger hard drives, however I will make an effort to stay more current for various reasons you mentioned. I bailed out of UnMenu as I waited the 6 hours and I thought it would display the results of the long smart test, but instead when I reloaded the page it seemed to just start it all over again. This time, I used the command line unmenu showed " smartctl -t long -d ata /dev/sdi 2>&1 " it was running. Testing has begun. Please wait 394 minutes for test to complete. Test will complete after Sun Mar 27 16:57:24 2016 I just have to figure out how to display the results. I was looking through the Wiki and I believe one of these gives me results: smartctl -a -A /dev/sdi or smartctl -a -d ata /dev/sdi I'll give them a try after the estimated time of completion. Quote Link to comment
spinbot Posted March 28, 2016 Author Share Posted March 28, 2016 So the original drive with issues seems to look good after 3 smart tests (one was aborted by my mistake). I see nothing that worries me, beyond the fact it fell out of the array initially for reasons I am unsure of. EDIT: I'm starting to think temperature may be the culprit in all my issues. Despite me having three 4 in 3 drive cages in the server, each with its own intake fan in front, the drives in the middle were hitting higher temperatures than those in the top and bottom. I checked the fan and its running. I cleaned out all the filters, as they were dusty. I've been running a window fan right in front of the server during all the current issues and the drive temps have been perfect. With respect to the two drives in question, disk10's Short Smart Report is showing the drive's worst as 54C: 194 Temperature_Celsius 0x0022 030 054 000 Old_age Always - 30 (0 7 0 0 0) Disk 9's Long Smart Report must be faulty, as it says the temperature worst is 102C yet the current temp is 123C: 194 Temperature_Celsius 0x0022 123 102 000 Old_age Always - 29 Disk 9 Long Smart Report: smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright © 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (AF) Device Model: WDC WD15EARS-00J2GB0 Serial Number: WD-WCAYY0273566 LU WWN Device Id: 5 0014ee 2afdf0659 Firmware Version: 80.00A80 User Capacity: 1,500,301,910,016 bytes [1.50 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Sun Mar 27 21:08:11 2016 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (34560) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 394) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3031) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 220 154 021 Pre-fail Always - 5958 4 Start_Stop_Count 0x0032 096 096 000 Old_age Always - 4207 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 045 045 000 Old_age Always - 40340 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 122 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 44 193 Load_Cycle_Count 0x0032 193 193 000 Old_age Always - 23923 194 Temperature_Celsius 0x0022 123 102 000 Old_age Always - 29 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 40334 - # 2 Extended offline Aborted by host 90% 40329 - # 3 Extended offline Completed without error 00% 40324 - # 4 Short offline Completed without error 00% 40236 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Quote Link to comment
JorgeB Posted March 28, 2016 Share Posted March 28, 2016 With respect to the two drives in question, disk10's Short Smart Report is showing the drive's worst as 54C: 194 Temperature_Celsius 0x0022 030 054 000 Old_age Always - 30 (0 7 0 0 0) Above what I like but not enough to damage, usually it's >60C. Disk 9's Long Smart Report must be faulty, as it says the temperature worst is 102C yet the current temp is 123C: 194 Temperature_Celsius 0x0022 123 102 000 Old_age Always - 29 This is normal for some disks, it means worst temp was 50C (123-102+29), hot but also ok. I consider critical HDD temp anything above 55C but do try to keep them <45C. Disk 9 passed the extended test, so you can try the new config. Quote Link to comment
spinbot Posted March 28, 2016 Author Share Posted March 28, 2016 I usually worry about sustained temps over 50C, but being conservation on this is better than liberal! The drive with the unrealistic temperature, I'm guessing its reporting in Fahrenheit, based on your math you converted with. Darn Americans, if only the world could get them to change to Metric and end confusion like this! lol I'm just running the long Smart Test on disk10. I'll have the results of it by morning. If all looks "safe", I will attempt the steps you suggested earlier. Regardless of my outcome, I would like to thank you now for the assistance, guidance, hand holding and reassurances. They all prove invaluable when I working in an area outside my comfort zone, especially being a person with OCD tendencies. We hate to "try" something unless we know the outcome ahead of time. Risk taking is not one of my best qualities, thanks to 3/4 of my life being plagued with health issues and trying to avoid any further problems. Life's a bitch, then you date one, then you find Ms. Wright! (ya, my girlfriends last name is "Wright", so how can I go wrong with that!) Quote Link to comment
spinbot Posted March 29, 2016 Author Share Posted March 29, 2016 These steps worked: -take a screenshot of current array -go to tools and click new config -reassign all disks, using old disk9, don't assign parity for now, leave it empty -start array and check if all data is there, especially disks 9 and 10 -if all looks fine, stop array, assign parity and start array to begin parity sync Both the original disk9 and disk10 had all their data on them, so I am now just waiting for the Parity to build. Only one small hiccup in that I lost power during the first attempt and since I had the server on my "bench" cleaning and pulling drives, I didn't have it in its UPS. At the moment, its done syncing all my data drives (the largest was 2TB) and is just zero'ing out the 2TB left on the parity drive. In 3 hours, it should be golden! I will take the advice from earlier and upgrade to whatever version of 6 that is either official or stable. I'm pretty sure I can find a thread on that process. Then, I will have to replace my parity drive (thanks to HPA crap) as my one spare 4TB data drive is a few thousand bits larger. Then, I can integrate the smaller 4TB in to the array to replace my smallest drive, then repeat the process for two other drives I have that are bigger than those in the server now. Weeks of fun! Thanks for the help, this issue is officially resolved now! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.