August 14, 200916 yr This is not a "simple" drive failure, I've been replacing 5 older IDE drives with a pair of new 1TB SATA drives and ran into some grief. Here's what I've done so far: 1. I successfully parity checked and then shut down the array, removed one 300G drive and replaced it with a 1TB drive 2. When the array was restarted the removed disk was reconstructed on the new disk and the file system expanded, all without incident (this was disk4 in the array) 3. I then copied two other 320G disks (disk2 and disk3 in the array) into subdirectories on the new disk (disk4) using commands: cp -prv /mnt/disk2 /mnt/disk4/disk2 cp -prv /mnt/disk3 /mnt/disk4/disk3 this worked ok, appart from being quite slow as having four IDE drives in the system reduces write speeds to about 10MB/sec 4. I then did a full file compare to make sure all was copied correctly 5. I then removed the two 320G IDE disks from the system and replaced the SATA data cables on two drives with new short cables (10 inch long with clips). On restart the IDE disks were marked as missing (as would be expected) so I went with the "restore" button and parity started to rebuild. The parity rebuild was much faster than before (going from 24MB/s to 40MB/s). 6. When I next checked on the system (a few hours after the parity rebuild should have finished) I found the system pretty much dead, I was even unable to login from the console, after typing my password it gave me a message about being unable to log the error as it was out of memory. 7. I rebooted the system and a after a short while it started having a problem with disk8 (a 750G SATA drive) and disabled it. This was one of the drives I had changed the cable on. I have saved the log file for this session as syslog-2009-08-13-1.txt. 8. I shut down the system and returned the original SATA cables and made sure all the other connectors were seated. 9. On restart it still shows disk8 with a red ball and unmenu shows disk8 as "DISK_DSBL". I have saved the logfile as: syslog-2009-08-13-2.txt. 10. It shows the parity disk with a green ball and unmenu reports "Parity is Valid:. Last parity check < 1 day ago with no sync errors." 11. I can see the contents of disk8, and they appear ok, but I'm guessing these are being simulated with parity? 12. What I'm worried about is did the error happen after the parity was rebuilt or might it have been part way though the parity build that took place after I removed the two 320G drives and before I found the system unresponsive (hence the parity might be incomplete). Is there a time stamp somewhere that I can find out when the parity build was completed (unmenu only tells me it was less than a day ago)? 13. the unmenu smart report does not show anything bad on disk8, but I have not done a short or long smart test yet. 14. the smart output for disk8 is: Statistics for /dev/sdc ST3750640AS_3QD1FMMC smartctl version 5.38 [i486-slackware-linux-gnu] Copyright © 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.10 family Device Model: ST3750640AS Serial Number: 3QD1FMMC Firmware Version: 3.AAE User Capacity: 750,156,374,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Thu Aug 13 19:49:00 2009 MDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 430) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 202) minutes. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 117 095 006 Pre-fail Always - 164059081 3 Spin_Up_Time 0x0003 095 093 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 393 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 39705648 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1162 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 097 097 020 Old_age Always - 3697 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 064 047 045 Old_age Always - 36 (Lifetime Min/Max 31/36) 194 Temperature_Celsius 0x0022 036 053 000 Old_age Always - 36 (0 25 0 0) 195 Hardware_ECC_Recovered 0x001a 093 057 000 Old_age Always - 26160059 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0 202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 946 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. ----------- Regards, Stephen
August 14, 200916 yr Data cables, in my experience, can be persnickety. I've had problems that were "fixed" by simply unplugging and replugging in a cable. For this reason, I am hesitant to unplug or replace a cable on a working drive. When I do, I am prepared for some type of mischief like you have had. unRAID will take a drive out of service if it becomes unresponsive or reports write errors. A bad, loose, or otherwise bad cable connection is the most common cause. But once taken out of service, even if the condition is fixed, that drive will NOT be put back in service without user involvement. You are right that unRAID is simulating the failed disk, but since you are unsure that the parity build was successful, the simulation may be imperfect. In other words, I would not rely on it. Here is what I would do .... 1. Make a backup copy of the "config" folder on your flash 2. Press the restore button and remove the parity disk from the array (leaving all of the data disks assigned, including disk8). (Be especially careful that the parity slot is unassigned. Assigning the wrong disk to the parity slot is deadly!) 3. Start the array. (with no parity disk in place, no parity build will occur) 4. Ensure that you can access each disk in the array. Perform some more extensive testing of disk8. Check the syslog and make sure you are not seeing errors occur. Also make sure that ALL of the disk8 data is present. Do not write to ANY disk in the array except disk8! 5. After you are confident that disk8 is working properly, stop the array, add the parity drive to the array, and restart the array. Parity will build. If disk8 is still a problem even after trying a couple different cables and SATA ports, you'd be able to, using the backup config folder, get back to the same point you are now and proceed to try and rebuild disk8 onto a new disk. If seems very unlikely this would be necessary. (This is the reason not to write to the other array disks while checking it out). Good luck!
August 14, 200916 yr Author Data cables, in my experience, can be persnickety. I've had problems that were "fixed" by simply unplugging and replugging in a cable. For this reason, I am hesitant to unplug or replace a cable on a working drive. When I do, I am prepared for some type of mischief like you have had. unRAID will take a drive out of service if it becomes unresponsive or reports write errors. A bad, loose, or otherwise bad cable connection is the most common cause. But once taken out of service, even if the condition is fixed, that drive will NOT be put back in service without user involvement. You are right that unRAID is simulating the failed disk, but since you are unsure that the parity build was successful, the simulation may be imperfect. In other words, I would not rely on it. The other disks in the array were accessible and appeared ok. I was able to run a compare of SHA1 digests for the bulk of the data on disk8 against my backups (which have an SHA1 digest for each file) and they all compared correctly except for one file (which I can fix by just restoring it) so I replaced the 750G drive with a new drive, checked all the cables again and started a rebuild. Once the rebuild is done can I put the 750G drive back into the unRAID server, but not assign it to the array, and then test it (run the smartctl tests etc.)?
August 15, 200916 yr Author I've got the array up and running and have observed an error with the file system on disk8, so found the wiki guide to this: http://lime-technology.com/wiki/index.php/Check_Disk_Filesystems And ran the reiserfsck on /dev/md8 to see what it had to say: reiserfsck --check started at Sat Aug 15 16:50:07 2009 ########### Replaying journal.. Reiserfs journal '/dev/md8' in blocks [18..8211]: 0 transactions replayed Checking internal tree../ 8 (of 8)/ 29 (of 112)/ 15 (of 170)block 65804479: T he level of the node (27973) is not correct, (1) expected the problem in the internal node occured (65804479), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 1 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Sat Aug 15 16:57:20 2009 ########### So as there are some warnings about the use of the --rebuild-tree option that is suggested, what would be the right thing to do at this time? Regards, Stephen
August 15, 200916 yr This happened because the initial parity build did not complete. Therefore - after some point - the parity data was garbage and garbage was rebuilt onto the disk8 replacement disk. I think you can allow reiserfsck to do its thing and clean up the disk. You've already reported (I think) that all of your data is complete except for one file that you fixed. It could be that reiserfsck will "recover" some garbage that you'll have to delete. Hopefully, in the end, you'll have a filesystem that is valid and all of your data will be intact. It will just take a while for reiserfsck to complete. If you had followed my instructions you would have avoided this. But I don't think it will be a huge deal to fix. But only time will tell.
August 16, 200916 yr Author This happened because the initial parity build did not complete. Therefore - after some point - the parity data was garbage and garbage was rebuilt onto the disk8 replacement disk. That's what I was a bit worried about. Though the unRAID and unmenu both showed that parity had been rebuilt... I think you can allow reiserfsck to do its thing and clean up the disk. You've already reported (I think) that all of your data is complete except for one file that you fixed. It could be that reiserfsck will "recover" some garbage that you'll have to delete. Hopefully, in the end, you'll have a filesystem that is valid and all of your data will be intact. It will just take a while for reiserfsck to complete. If you had followed my instructions you would have avoided this. But I don't think it will be a huge deal to fix. But only time will tell. Our timing was off... Since I still have the original 750G drive unchanged I hooked it up, left it out of the array, and ran reiserfsck on it. That completed without issue, though it did replay something from its journal file that the disk8 which had been rebuilt from parity did not. Still, as the disk8 had been used for a day it may already have done this. Anyway, just to be really sure I'm going to do the rebuild-tree on the "disk8" and then run a file compare between disk8 and the old 750G drive. Maybe I missed something the first time round. If I do find that disk8 has corruption that I missed I could then remove it from the array, put the old 750G drive back in its place and hit the "restore" button to forget the whole thing. Right? An interesting walk through the dark side of unRAID, but not something you could do with a conventional RAID5 setup! Thanks, Stephen
Archived
This topic is now archived and is closed to further replies.