actionfreak Posted January 11, 2012 Share Posted January 11, 2012 I have a Norco 4220 running 4.5.6. The server has been running well for about a year. Then one day it wouldn't boot up. I was getting an error that looked like this: Initializing controller 1 ping up group 1 detecting port 7...............warning time out error pd not ready or unknown device type detecting port 6 error pd not ready or unknown device type detecting port 5 error pd not ready or unknown device type something wrong with the adapter, press any key to escape bios! After troubleshooting I found that it didn't give this error when disk9 (12th slot counting from left to right, top to bottom) was removed. I replaced disk9 with another hard drive I have (replaced WD20EADS with WD Black) and let Unraid rebuild the data. The next day disk6 (slot 8 counting left to right, top to bottom. I replaced the WD20EARS with a WD20EADS and attempted to let Unraid rebuild the data. The main page started showing massive errors on disk9 and the system becomes unresponsive. The syslog scrolls errors like this: Jan 10 20:01:49 Tower kernel: md: disk9 read error Jan 10 20:01:49 Tower kernel: handle_stripe read error: 29790528/9, count: 1 Jan 10 20:01:49 Tower kernel: md: disk9 read error Jan 10 20:01:49 Tower kernel: handle_stripe read error: 29790536/9, count: 1 Jan 10 20:01:49 Tower kernel: md: disk9 read error Jan 10 20:01:49 Tower kernel: handle_stripe read error: 29790544/9, count: 1 Jan 10 20:01:49 Tower kernel: md: disk9 read error Jan 10 20:01:49 Tower kernel: handle_stripe read error: 29790552/9, count: 1 Jan 10 20:01:49 Tower kernel: md: disk9 read error Jan 10 20:01:49 Tower kernel: handle_stripe read error: 29790560/9, count: 1 Jan 10 20:01:49 Tower kernel: md: disk9 read error The syslog is attached. Any ideas on what I can do now? syslog_01102012.txt Quote Link to comment
dgaschk Posted January 11, 2012 Share Posted January 11, 2012 Disk 9 needs to be replaced and/or rebuilt. Post a SMART report for disk 9. Quote Link to comment
actionfreak Posted January 12, 2012 Author Share Posted January 12, 2012 I let the server run last night and all day today without touching it. The main page shows that disk 6 is up now and disk 9 is failed. When I access disk 6 or disk 9 using windows explorer, I see a lot of errors for the drive in the syslog. As mentioned earlier, both of these drives have already been replaced once. I pasted the errors below as well as the smart reports for disk 6 and for disk 9. Another syslog is attached. Jan 11 17:58:36 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 17237 does not match to the expected one 1 Jan 11 17:58:36 Tower kernel: REISERFS error (device md6): vs-5150 search_by_key: invalid format found in block 191844562. Fsck? Jan 11 17:58:36 Tower kernel: REISERFS error (device md6): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 458 0x0 SD] Jan 11 17:58:36 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 17237 does not match to the expected one 1 Jan 11 17:58:36 Tower kernel: REISERFS error (device md6): vs-5150 search_by_key: invalid format found in block 191844562. Fsck? Jan 11 17:58:36 Tower kernel: REISERFS error (device md6): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 458 0x0 SD] Jan 11 18:00:13 Tower kernel: REISERFS error (device md9): vs-5150 search_by_key: invalid format found in block 192757109. Fsck? Jan 11 18:00:13 Tower kernel: REISERFS error (device md9): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 208 0x0 SD] Jan 11 18:00:13 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 53598 does not match to the expected one 2 Jan 11 18:00:13 Tower kernel: REISERFS error (device md9): vs-5150 search_by_key: invalid format found in block 192071364. Fsck? Jan 11 18:00:13 Tower kernel: REISERFS error (device md9): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [4 178 0x0 SD] DISK9 =~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2012.01.11 17:55:34 =~=~=~=~=~=~=~=~=~=~=~= smartctl -a -d ata /dev/sda f smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD2001FASS-00W2B0 Serial Number: WD-WMAY00444354 Firmware Version: 01.00101 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Wed Jan 11 17:56:14 2012 MST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (29460) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3037) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 31 3 Spin_Up_Time 0x0027 178 160 021 Pre-fail Always - 13075 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 77 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 092 092 000 Old_age Always - 6362 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 73 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 36 193 Load_Cycle_Count 0x0032 199 199 000 Old_age Always - 4958 194 Temperature_Celsius 0x0022 125 110 000 Old_age Always - 27 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 87 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 67 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 190 190 000 Old_age Offline - 2158 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 0 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. DISK6 root@Tower:~# smartctl -a -d ata /dev/sdf b smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD20EADS-00S2B0 Serial Number: WD-WCAVY4219344 Firmware Version: 01.00A01 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Wed Jan 11 17:56:48 2012 MST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (42780) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 162 150 021 Pre-fail Always - 8866 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 121 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 6731 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 28 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 25 193 Load_Cycle_Count 0x0032 163 163 000 Old_age Always - 113056 194 Temperature_Celsius 0x0022 126 098 000 Old_age Always - 26 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. tower_syslog_01112012-1.txt Quote Link to comment
Rajahal Posted January 12, 2012 Share Posted January 12, 2012 The good news is that disks 6 and 9 appear to be healthy (based on their SMART reports). The bad news is that their file systems are corrupted. Instructions on how to fix the corrupted file systems can be found here. You'll need to do one drive at a time, and reiserfsck will take a long time to run on each one. unRAID itself doesn't actually care about the file system errors. If you are worried about another disk failing during the long reiserfsck procedure (which would likely result in data loss), then you can replace disk 9 with yet another disk and let unRAID rebuilt it before doing the reiserfsck steps. In this case, unRAID would rebuild the corrupted file system. You could then run the reiserfsck procedure after the data-rebuild is complete. Quote Link to comment
actionfreak Posted January 13, 2012 Author Share Posted January 13, 2012 Thanks. It says that three corruptions can only be fixed with --rebuild-tree. What is the exact format of the command that I should use? root@Tower:~# reiserfsck --check /dev/md9 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md9 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Thu Jan 12 19:28:45 2012 ########### Replaying journal: Done. Reiserfs journal '/dev/md9' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Checking internal tree.. \/ 5 (of 146\/ 70 (of 86/block 191561729: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (191561729),/ 6 (of 146-/ 1 (of 85\block 191561746: The level of the node (56965) is not correct, (1) expected the problem in the internal node occured (191561746),/ 7 (of 146|block 191667184: The level of the node (10444) is not correct, (2) expected the problem in the internal node occured (191667184), whole subtrfinished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 3 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Thu Jan 12 19:35:16 2012 ########### root@Tower:~# Quote Link to comment
Rajahal Posted January 13, 2012 Share Posted January 13, 2012 reiserfsck --rebuild-tree /dev/md9 Quote Link to comment
dgaschk Posted January 13, 2012 Share Posted January 13, 2012 See here: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems Quote Link to comment
actionfreak Posted January 14, 2012 Author Share Posted January 14, 2012 The reiserfsck procedure completed on disk9. I can access the folders without the same errors. I do still have the red ball next to the disk though in the main page though. How do I fix that? Should I go ahead and run the reiserfsck procedure on disk6 now? Here is the new smart report for disk9. root@Tower:~# smartctl -a -d ata /dev/sdf smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Alle n Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD2001FASS-00W2B0 Serial Number: WD-WMAY00444354 Firmware Version: 01.00101 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Fri Jan 13 17:46:40 2012 MST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (29460) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off supp ort. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3037) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_ FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 31 3 Spin_Up_Time 0x0027 169 160 021 Pre-fail Always - 13550 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 86 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 092 092 000 Old_age Always - 6398 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 77 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 36 193 Load_Cycle_Count 0x0032 198 198 000 Old_age Always - 6239 194 Temperature_Celsius 0x0022 125 110 000 Old_age Always - 27 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 87 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 67 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 190 190 000 Old_age Offline - 2158 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA _of_first_error # 1 Short offline Completed without error 00% 0 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@Tower:~# Quote Link to comment
dgaschk Posted January 14, 2012 Share Posted January 14, 2012 The disk with a red ball needs to be rebuilt. 1. Stop the array and un-assign the disk. 2. start the array and then stop the array. (This causes unRAID to "forget" about the disk) 3. Assign the disk and start the array. The rebuild will begin. You may need to run reiserfsck again. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.