June 3, 200818 yr So I awoke this morning to a failed disk.. only... I can't really say it failed, but for some reason went offline. I believe because the spin up failed. Thus in turn it is disabled from the front page on web interface. I rebooted a few times and cannot seem to get the drive back online. I tested the drive with smartctl and it does respond. so things seem a bit weird at the current point. I have two thoughts. 1. Put in a spare drive and rebuild onto it. 2. check the filesystem by mounting directly then "re-store". Since I have not had a drive fail, but then actually exist without really being failed I'm a lil perplexed. I believe the drive is ATA4. (I know about drive ATA6 being bad. I have to run diags on it. I believe it came to me bad and that happen to be one of the spares).
June 3, 200818 yr So I awoke this morning to a failed disk.. only... I can't really say it failed, but for some reason went offline. I believe because the spin up failed. Thus in turn it is disabled from the front page on web interface. I rebooted a few times and cannot seem to get the drive back online. I tested the drive with smartctl and it does respond. so things seem a bit weird at the current point. I have two thoughts. 1. Put in a spare drive and rebuild onto it. 2. check the filesystem by mounting directly then "re-store". Since I have not had a drive fail, but then actually exist without really being failed I'm a lil perplexed. I believe the drive is ATA4. (I know about drive ATA6 being bad. I have to run diags on it. I believe it came to me bad and that happen to be one of the spares). Your disk has failed... If the RED light is on in the management page, it cannot be written to. DO NOT USE THE "RESTORE" BUTTON unless you do not want the contents of that disk. It will remove the disk AND ITS DATA from the array, and immediately rebuild parity without it. You will be able to both read and write to the failed disk. This is normal and is exactly why you have an unRAID array. The reads are actually being performed on all your other disks and the software is reconstructing what it would have read from the failed disk if it could have been read. In the same way, "writes" to it are being made to the parity drive. To your software, the array is letting you read and write files. At this time, you are not protected against a second simultaneous disk failure. With that in mind, you will want to replace the failed drive as soon as possible. So... when you have the replacement drive in hand, stop the array, power down, replace the drive, power back up, on the main management page it should have noticed the disk change. It will have a checkbox (I think) just under the "Start" button to begin the process of rebuilding the data from the failed drive onto the new drive. Do not use the "Restore" button. It will permanently erase the failed drive from your server... and all the data that was on it too. It is only used to save a new "Initial Configuration" when you are removing a drive and do not intend to ever replace it with another in the same slot. The replacement drive you purchase must be as large, or larger than the one that failed, and no larger than the parity drive. If the only drive you can get is larger than the parity drive, then it must be assigned as parity, and the old parity drive re-assigned as the new replacement for your data drive. This is called a "parity-swap" operation. There is one possibility you might check... You might check to ensure the power and data cables on the failed drive are inserted securely. If you think that was the case, let us know and we'll point you to the series of steps to get unRAID forget about the fact that the drive had failed when it tried to write to it. Joe L.
June 3, 200818 yr Author Here's the interesting thing.. The disk itself is not actually bad. I ram a smartctl -tlong on it this morning. It finished a lil while ago without any hiccups. Gonna have to figure this one out. In any case I won't use the re-store. I'll take a spare and rebuild onto it. Then take this drive out of the array then compare the two to see what the deal is. root@Atlas:/boot# smartctl -a /dev/sdd smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD10EACS-00ZJB0 Serial Number: WD-WCASJ1352706 Firmware Version: 01.01B01 User Capacity: 1,000,204,886,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Tue Jun 3 15:22:14 2008 GMT+4 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (26400) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0003 191 175 021 Pre-fail Always - 7441 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 140 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x000e 200 200 051 Old_age Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 978 10 Spin_Retry_Count 0x0012 100 100 051 Old_age Always - 0 11 Calibration_Retry_Count 0x0012 100 100 051 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 91 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 164 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 1595 194 Temperature_Celsius 0x0022 111 106 000 Old_age Always - 41 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 051 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 977 - # 2 Short offline Completed without error 00% 969 - # 3 Short offline Completed without error 00% 406 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
June 3, 200818 yr Here's the interesting thing.. The disk itself is not actually bad. I ram a smartctl -tlong on it this morning. It finished a lil while ago without any hiccups. Gonna have to figure this one out. In any case I won't use the re-store. I'll take a spare and rebuild onto it. Then take this drive out of the array then compare the two to see what the deal is. root@Atlas:/boot# smartctl -a /dev/sdd smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: WDC WD10EACS-00ZJB0 Serial Number: WD-WCASJ1352706 Firmware Version: 01.01B01 User Capacity: 1,000,204,886,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Tue Jun 3 15:22:14 2008 GMT+4 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (26400) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0003 191 175 021 Pre-fail Always - 7441 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 140 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x000e 200 200 051 Old_age Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 978 10 Spin_Retry_Count 0x0012 100 100 051 Old_age Always - 0 11 Calibration_Retry_Count 0x0012 100 100 051 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 91 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 164 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 1595 194 Temperature_Celsius 0x0022 111 106 000 Old_age Always - 41 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 051 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 977 - # 2 Short offline Completed without error 00% 969 - # 3 Short offline Completed without error 00% 406 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Perhaps the "long" smartctl test interfered with writing to the disk at some point? Or, a loose cable? (The drive statistics would not change, but it sure would be tough to "write" to it) Joe L.
June 3, 200818 yr Author It's something else. I did not issue the smartctl test until 6:30 in the morn. Apparently the drive had issues around 22:00. I was asleep then.. I may have to review spin up timers to see what the deal is.
June 3, 200818 yr You do know a lot more then me when it comes to this kind of stuff... but is it possible the disk is bad but SMART isn't reporting the problem? From my understanding SMART works most of the time... but even if SMART says there isn't a problem the disk could still have a problem with it. Also, from my understanding, a disk taken offline from unRAID only happens when an error occurs while writing to the disk. That means that running checks while reading from the disk may not give you any errors if it's a write related error. In this case, I'd add a new drive to the array and have the new drive rebuilt with your parity information. Then you can DD the old drive and check smart again to see what happens. In this case, the entire disk will be written to so maybe SMART will report an error if there is one.
June 4, 200818 yr I don't believe there's any thing wrong with sdd on ATA4. It looks to me that there is a problem with the ATA4 connection to it though. The SATA link went down 3 or 4 times, with considerable trouble communicating with it. It was re-assigned to sdh (instead of the original sdd) 3 times, as if it was a hot swap connection. Is it possible that the cable has vibrated a little loose? Or if it is in a hot swap slot, perhaps it is not making a tight connection? Perhaps this disk needs to be handcuffed to its bus seat! I've had similar situations in the past with my bad nForce board, where a whole disk controller would quit on me, and the 1 or 2 drives attached would of course drop out of the array, and be disabled by unRAID. I also had a drive that if allowed to spin down, could not spin up, and was therefore disabled. In each case, all of the drives and parity were perfect, but unRAID had to rebuild something, either the parity drive or a data drive, even though what was being rebuilt was identical to what was already there. I would have liked to request an enhancement to the intelligence of the parity sync algorithm. Let a parity sync always begin in parity check mode, and proceed according to whether it is finding errors or not. If correct, stay in check mode. If it finds more than 4 errors in a row, then switch to sync mode. Then pause periodically while in sync mode to do a check, and if correct, then switch back to check mode.
June 4, 200818 yr Author This makes sense! I'll open'er up later. So far did a reiserfsck and it wants to replay 80 transactiions. I'll update more as I find out. Could very well be a slightly loose connection. Apparently from the logs it happened over the course of a day or so. Have to get my syslog checking program in place so it emails me these errors as they occur.
June 4, 200818 yr unRAID looks at every serious disk error as a failed disk - and takes its preprogrammed response (take a disk out of service, or bring down the array if multiple disks are affected). These decisions are FINAL. There is no way to tell unRAID - "that wasn't a drive failure - I fixed it - make the configuration valid again" without rebuilding parity from scratch. It MAY be possible to keep a backup super.dat file from the last time you did a configuration change and restore that file to "trick" unRAID into believing it never had an error. I have NOT heard of people doing this and would NOT do this without Tom or some knowledgeable forum members saying that would work. I just don't know what having a non-up-to-date super.dat file would do. The good news is that when one of these screwy things happens, the chances that you've had a REAL disk failure are low. So you can recover by putting things right and rebuilding parity. WeeboTech, this is likely what you need to do. I think you already know that. I do believe, however, that Tom should consider providing features or publishing technical information so that knowledgeable users could recover from this type of situation. I could see a button called "Make Configuration Valid" that would do exactly that and then initiate a full parity check. I can see little downside to that feature - worst thing that could happen is that the parity check would find a ton of errors, and in effect, rebuild parity. Another idea is to add a button that says "Use Last Known Good Configuration" that would work similarly. A way to "undo" pressing the restore button would also be highly desirable! These sorts of "trust the human" features are, IMO, lacking. Another random thought is that I would prefer that unRAID take the array down when (it thinks) a drive fails, rather than have unRAID go into a "simulate the failed drive" mode. "Simulate failed disk" could be a checkbox that you'd have to check to bring up the array in this mode. If properly implemented, this model would allow a person to fix whatever problem caused the disk to (appear to) fail, and then unRAID would start the array just like any dirty shutdown. In addition to letting the human fix the problem, it would also draw the human's attention to the failure rather quickly!
June 4, 200818 yr Author I don't think this particular case was an actual drive failure, but some failure to communicate with the drive did occur. It seems it happened a number of times. In this case, I put in another drive, reassigned disk 3 to the new drive and rebuilt. Later on I'm going to assign disk 3 back to the suspect drive. This should for a regeneration of disk 3 back onto that drive and we'll see how it goes from there. As of now, I have a backup of disk 3 onto this new drive so I'm OK. I would not have fiddled with the super.dat to bring the drive back online. I just thought it was odd how the drive was off line and I found no way to have the disk be rebuilt back onto the same drive. The one thing that I've always disliked about linux is the dynamic device assignments. Sometimes sdc is one drive and other times it's another. It seems every time I add a drive, the parity drive is always off because it gets shifted. annoys the heck out of me.
June 4, 200818 yr I would not have fiddled with the super.dat to bring the drive back online. I just thought it was odd how the drive was off line and I found no way to have the disk be rebuilt back onto the same drive. Saving a copy of the super.dat, after the array is cleanly stopped, will let you copy it back in an event like this to be back online with a minimum of fuss. I've done it many times. Just don't do it when the array is running, since the "clean stop" bit is not set and it will force a complete parity check when you use it to come back online. Unfortunately, there is no "easy" way to set the bit. (although I recently noticed a new command argument in the source code that was not there previously, to do exactly that) This is the comment in the source code /* hacks */ /* This is for start_array() to tell it which slot starts out 'invalid'. Normally * this is always 0 (parity slot), but can be set to another slot number to make that * disk start out 'invalid' upon start. Can also be set high (eg, 99) so that no * disk starts out 'invalid'. These are for cases where, eg, super block is lost * and user knows the correct array config and wants to prevent parity rebuild, or * wants to initiate data reconstruct. */ static int invalidslot = 0; and this would seem to allow setting it. if (!strcmp("invalidslot", name)) invalidslot = token ? value : 0; I don't know the exact syntax, but it seems as if you could use the mdcmd interface to /proc to set the invalidslot to 99 and rebuild a bad superblock. The one thing that I've always disliked about linux is the dynamic device assignments. Sometimes sdc is one drive and other times it's another. It seems every time I add a drive, the parity drive is always off because it gets shifted. annoys the heck out of me. Actually, we really don't care which dynamic device is assigned. The problem is that unRAID assigns devices (/dev/sdb) to logical slots in the array instead of assigning the model/serial-number of a physical drive to a logical slot in the array. If it assigned a specific physical model/serial to parity, then I could care less if it is /dev/sda, or hda, or sdj. It could move it to a different dynamic device and keep the same parity drive in the parity slot. /boot/config/disk.cfg currently has the "device" Id of assigned disks. It should instead have the drive model/serial number of the assigned disks. If it did, the dynamic assignment would not affect anything. Joe L.
June 4, 200818 yr Author The one thing that I've always disliked about linux is the dynamic device assignments. Sometimes sdc is one drive and other times it's another. It seems every time I add a drive, the parity drive is always off because it gets shifted. annoys the heck out of me. Actually, we really don't care which dynamic device is assigned. The problem is that unRAID assigns devices (/dev/sdb) to logical slots in the array instead of assigning the model/serial-number of a physical drive to a logical slot in the array. If it assigned a specific physical model/serial to parity, then I could care less if it is /dev/sda, or hda, or sdj. It could move it to a different dynamic device and keep the same parity drive in the parity slot. /boot/config/disk.cfg currently has the "device" Id of assigned disks. It should instead have the drive model/serial number of the assigned disks. If it did, the dynamic assignment would not affect anything. Joe L. I was thinking about this last night. I agree, if it used labels or serial numbers of the drives it would remove reliance on device position on the PCI bus. As far as super.dat, I plan to write a tool to export the superblock as text and then allow re-importing it. I've been wondering if it's possible to assign the devices to a different partition. This way on my smaller machine which uses all 500GB drives, I could use a 1/2 a 1TB drive for parity and the other half for cache.
June 9, 200818 yr Actually, we really don't care which dynamic device is assigned. The problem is that unRAID assigns devices (/dev/sdb) to logical slots in the array instead of assigning the model/serial-number of a physical drive to a logical slot in the array. If it assigned a specific physical model/serial to parity, then I could care less if it is /dev/sda, or hda, or sdj. It could move it to a different dynamic device and keep the same parity drive in the parity slot. /boot/config/disk.cfg currently has the "device" Id of assigned disks. It should instead have the drive model/serial number of the assigned disks. If it did, the dynamic assignment would not affect anything. The purpose of the disk.cfg file is to identify disk slots, not assigned hard drives. A slot is a physical non-changing location where you can plug in a hard drive. When we refer to a disk, e.g., "disk3", we really mean "the disk in slot 3". This is important because if we say disk3 has been disabled, we want a user to be able to positively identify which disk of the array is in fact "disk3". Hence, the disk.cfg file records the PCI-bus ID of each disk "port". This ID information does not change unless the hardware configuration changes (ie, motherboard or disk controllers). When a customer purchases a server, e.g., the MD-1500, we include the proper 'disk.cfg' file for that hardware configuration so that the disk numbering matches a diagram we also include. The super.dat file records the disk model/serial number which is currently assigned to each slot - it does not record the linux device identifier (hda, hdb, sda, sdb, etc). As WeeboTech has noticed, the linux device identifier can change from boot-up to boot-up depending on what is installed and even on the timing of drivers being loaded. Fortunately the linux 'udev' subsystem provides sufficient information to generate a mapping between currently assigned device identifiers and PCI ID's. When the unRAID server gets started, we first have figured out this mapping, then converted the device identifers to <major,minor> pairs for each slot, and that is what you see being passed to the unRAID driver.
Archived
This topic is now archived and is closed to further replies.