Well I just encountered what seems to be my first drive failure and hopefully not two drive failures. 

I was looking at some files in a folder and it seemed that there were multiple files missing.  I have been doing some house cleaning but was sure I did not delete the files that are missing, so I immediately suspected there was an issue with UnRAID.  After bringing up the GUI, I noticed that there was one drive showing as Red.  Not sure what red meant, so I went to wiki and see that red is "the disk is disabled". 

I did not disable the drive, so I am guessing that UnRAID disabled it due to a problem with the drive.  I stopped the array and then went back to the disk status page and now have a drive labeled as missing and a drive showing / labeled as not installed.  I have not done anything beyond that.  If it was a one drive issue I might of felt comfortable enough to move forward with doing a power down and a drive swap, but seeing I am getting an error on two drives, I am a little more hesitant to move forward without getting some input.

 

Just a small run down,  on my system:

UnRAID version: 4.3.2

parity is a 750gig seagate

disk1 is a 500gig seagate which is the one showing as Missing

disk 2, 3, 4, and cache seem to be ok

disk5 is a 750gig seagate showing as not installed

 

I do not have a spare drive under 1tb (I just put another UnRAID system together that has (5) 1tb drives that are still empty so I could use them) so I would have to do a "swap-disable" (see wiki section "replace failed disk" http://lime-technology.com/wiki/index.php?title=UnRAID_Manual#Replace_a_failed_disk

 

I have printed out the Main tab and device tab of the GUI so I will not lose the drive placement during this repair job. (the disk's S/N show up on both pages for disk1 but only show up on the devices page for disk5.

 

Can anyone help me through the process of getting UnRAID back up and running with no data loss.  Thanks for any help anyone can give.

 

The drive that is listed as "missing" probably could not be un-mounted when you stopped the array since you had "cd" to it and it was your current directory.   Since it could not be un-mounted, it could not be mounted again.   unRAID mis-reports it as "missing"

 

Before you do anything else, save a copy of your syslog.

 

After logging in via telnet, type:

cp /var/log/syslog /boot/syslog_09092008.txt

chmod a-x /boot/syslog_09092008.txt

 

To get your server back in sync, with all the drives mounted, all you need to do is reboot.  So, go back to the web-interface, and press reboot. 

 

At that point, it will probably come back with the one drive still "red" and marked as bad.

 

Let us know how that works.   Hopefully, that will fix the missing drive.

 

Then,  you can go through a drive swap... or, if lucky, it is just a loose cable on a drive.   If it is, you must un-assign the drive, reboot, then re-assign the drive to the same slot to get it to be recognized as a new drive.

 

Whatever you do, DO NOT press the "Restore" button.  It does not restore data, it only removes the super.dat file that holds your array configuration and parity status.   It then forces a complete rebuild of parity from the working and assigned drives... forgetting about any defective or missing drives.... wiping any trace of  hope of doing a rebuild of your data on a replacement drive.

 

Use only the "Start" button when replacing a defective drive if you expect it to be filled with the prior defective disk's  data.

 

Joe L.

Here is the syslog.  The explanation of the missing drive makes sense.  Let me know if you notice any other issues with my syslog.

Thanks for your help.

I rebooted UnRAID and disk1 looks fine now.  Now all I have is disk 5 that is showing as having an error.  I will do a swap swap-disable and should be back up and running.

I commend you for capturing the syslog *before* you rebooted!  It shows a lot of disk errors, but most of them are just follow-on, errors caused by the disabling of both Disk 1 and 5.  The onset of issues begins around 1:38pm with a "exception Emask" error on Disk 5 of the frozen and (timeout) variety.  The system tried over and over for the next 70 seconds to regain contact with the drive, but was unsuccessful.  At that time, it disabled the ata5 channel, which included Disk 5 *and* Disk 1 (ata5.01 and ata5.00), most unfortunate.  These drives were using the ata_piix driver, which I suspect may be an IDE emulation driver, and perhaps emulated IDE channels only too well.  You apparently have an Intel board with ICH7 disk controller, and a later ICH controller would probably allow you to use AHCI, or a native SATA mode.  I may be wrong here, just speculating.

 

Any way, you can ignore all of the errors that follow that, after both drives had been disabled by the system.

 

I'm not surprised that Disk 1 was fine on reboot, and would not have been surprised if Disk 5 was fine too, but apparently not?  Would you be able to post another syslog, following the reboot, so we can see what the kernel sees when it identifies and tries to setup Disk 5?  That particular sequence of errors is often not fatal, and the drive often boots later and is readable.  Do make sure you power off, not just reboot.  I can't guarantee anything here though, as there is no indication as to what went wrong all of a sudden.  It could be the drive, or it could have been a problem with the cable, or disk controller, or a failure to spin up.

 

I'm hoping to add a 'Sanitize' option to the syslog plugin, so that users can obtain a copy of their syslogs with registration names and duped files 'sanitized' or stripped.  That will make it a lot less work for users such as yourself.  I suppose I'll also add an option to sanitize local IP's and disk serial numbers, although I don't see a danger with them.  I can't think of any harmful way they could be used.

Swap-Disable procedure which completed this am:

1. Captured Syslog.txt and posted to forum as per Joe's request

      -See http://lime-technology.com/?page_id=33 for getting to the proper screen to capture log

      -See http://lime-technology.com/wiki/index.php?title=Troubleshooting#Capturing_your_syslog (section is called "Capturing your syslog") Joe gives instruction on this above

      -To view the log I usually Change the Export settings for the Flash Drive to Export Read/Write on the Shares tab of the GUI http://tower/shares.htm Then I go to my network places and navigate to \\Tower\flash

            If you can not see the flash share then you may need to add UPnP User Interface in windows go to your control panel and click add remove programs / Add Remove Windows components (on left bar) / Click on

            Network  Services and in the bottom right click on Details / then check the box UPnP User Interface.

            (there are other methods for viewing the syslog.txt file)

      -removed any private info in syslog prior to post

2. Printed "disk devices" assignments as shown on the "Devices" tab of the GUI located at http://tower/devices.htm (I just pulled the page up and printed the screen which is needed info for drive swaps and moving drives around in the on the mother board)

**** (possible additional step) Should I have "backup 'config/super.dat' and 'config/disk.cfg' files" to my workstation??? http://lime-technology.com/wiki/index.php?title=Replacing_a_Data_Drive

3. Powered Down

4. maybe should of powered back up and capture another syslog as per Rob's request but did not get request until after step 5.  (sorry Rob... I know to do this next time.)

5. Located the physical location of the parity drive (good drive and disk5 (failed drive) being careful to match the serial number on the print out in step 2.

    pulled disk5 (failed drive)

    pulled 750gig parity drive and placed parity drive in slot where 750gig disk5 was

    inserted new 1tb drive in to slot where old parity drive was (1tb was my only warm spare / on-call drive I had)

6. Powered up UnRAID (came up fine.... the Parity Drive and disk5 indicator light on http://tower/main.htm were showing as blue (all others were green)

7. Skipped this step

    I did not have to re-map the parity drive or disk5 on http://tower/devices.htm of the gui (seeing I Jockeyed the new drives into the slots as per step 5 unraid did the matching)

    if you did not do this then you will need to go through assigning the drives as needed on http://tower/devices.htm of the GUI

7. I did not do a screen capture for this step but UnRAID had a message saying it was ready to copy all the data from the old parity to the new parity drive... with some type of message saying "I am sure" with a check box.

    I checked the box and hit "Start" and UnRAID started mirroring/copying the old parity on to the new parity. Once Unraid was done with the copy it did a rebuild on disk5 (no user steps between parity copy and rebuild...)

8. By morning The parity copy was done, disk5 was rebuilt, and parity was valid.

9. to simplify matters (I learned my lesson) I will avoid all the steps above by make sure my warm spares/ on-call drives are bigger than any disk in the arry, but not bigger than my parity disk. (If I do purchase a drive bigger than the parity disk) I will go ahead at that time and swap out the parity.

 

I did have a stressful moment during the copy/rebuild ( Swap-Disable procedure )... We had a storm roll through in the middle of the night, during this process.  I got up and frantically looked for a post on shutting down the array during a swap-disable or even during a parity copy and/or disk rebuild.  I found one post of interest http://lime-technology.com/forum/index.php?topic=578.0  (thanks Joe and Tom).  It seems it is possible to stop the array and power down, if one were to run into this situation (I am working on having the server on a UPS but I am not sure I would buy a big enough UPS to allow for an all night Swap-disable) a ups and generator would be ideal but not everyone is going to have this, so it is nice to know that you can stop the array and power down during these situations.  I am not 100% that this is an option seeing by the time I found the post and the storm got to the point where I felt I need to interrupt the process, stop and power down... the array was done with the swap-disable and parity was valid, so I just stopped the array powered down and went back to bed.

 

I am looking for comments and corrections on the above.  I will make the necessary adds/edits so that this post will one day possibly help someone else.  I definitely need some help with step 7 seeing I did not take careful notes or take a screen shot of what one will be confronted with during the start of a swap-disable

 

Rob... I attached a syslog but I think it was after the parity was valid.

 

Rob,

I meant to thank you for your translation of the syslog.... thanks

 

As you mentions a Sanitize option would really be nice.  Removing of my IP info was overkill and as you mentioned I am not sure how one could use this info (as you can see I did not go to such drastic measures on the second syslog posted, seeing it is not adding any level of security).  I am not sure anyone could do anything with the serial numbers either.

 

My board is a ASUS P5LV2-VM.  Are you suggesting a change to the bios to help performance?  I have a majority of the drives (5 of them) in a 5 in 3 bay SATA backplane ( DS-3151SSBK ). The tow drives that I was jockeying around were both in the backplane.  All but one of the drives in the backplane are connected straight to the motherboards SATA connections the other is connected to a IDE card with some type of SATA support, which is probably the cache disk or parity (I am open for configuration changes to boost performance, which I feel is a little on the slow side.  I have added all the performance suggestions a week or so ago which are listed on the wiki site).

 

Let me know if you need any more info.  I will gladly post a list of system components, but will need to shut down the array to take a look inside, or hunt down the receipts.

 

One last thing....

As Joe has said,"(success stories are less likely to have somebody make a post to say it worked. People with problems often post to say something did not work as expected.)."

 

IT WORKED!!!!

 

I wanted to be sure to comment on what a great product this is.  I am a very happy customer and Love UnRAID.  And the above is just one more success story for UnRAID to add to its resume.

 

As Crankbearing has said, "great product - great service - great support = Very Happy Customer"

I am working on having the server on a UPS but I am not sure I would buy a big enough UPS to allow for an all night Swap-disable

Glad you are back up and running.

 

Trust me, my UPS has an estimated run time of about 10 minutes with all the disks spinning in my array.  I *know* my parity checks take a bit longer than 10 minutes, so odds are very very high power from the UPS would not last to rebuild a drive through an extended power outage.

 

Joe L.

Rob... I attached a syslog but I think it was after the parity was valid.

 

Yep, and no original Disk 5  (serial ending in A6T).  It does show a successful swap-disable, perhaps the first I've seen in a syslog.

 

The initial Disk 5 error detected looks to me to be identical to the error sequence returned by a failure to spin up.  Recently, one or more users had compatibility problems with certain drives and the spin down/spin up commands, and the error sequence looks the same.  In your case, I don't believe it has anything to do with spin up or down, but that the disk was down and could not be contacted.  That could have been a communications failure, from an electrical spike, or cable issue, or ...  But the drive itself is probably OK, won't know for sure until you test it, check a SMART report and do a reiserfsck on it.

 

Some troubling things here...

 

There were absolutely NO issues with Disk 1, yet it was disabled along with Disk 5, by a kernel module.  Neither the reiserfs or unRAID modules were aware of the disabling of both drives, and continued to try to issue commands to them, with resultant errors.  Later, unRAID tries to spin all of the drives up, but was unsuccessful with either of these 2 drives.  unRAID decides to mark Disk 5 as disabled, and Disk 1 as Missing.  Missing is better, because on the next boot, it will be 'found' again, whereas Disabled will not be allowed to Start even if it is actually OK (I believe).

 

I don't see how you could have proceeded differently, based on what you observed.  And based on your report of what you observed, I don't see how Joe could have advised differently.  Yet there may not have been anything wrong with the system, apart from a temporary lost contact with one drive.  I think it is very likely that, if you had rebooted the system, and performed the Make unRAID Trust the Parity Drive, Avoid Rebuilding Parity Unnecessarily procedure, that everything would have been back to normal very quickly.  There would have been some transactions replayed in the syslog, and a few parity errors at the beginning of the parity check, but that is all.  If someone had checked the syslog, unless they had experience with this kind of issue, they would have been even more alarmed, at all of the errors reported, and the type of errors reported.

 

The reason I mentioned the 'IDE emulation' mode is that the setup of the SATA drives and the way the failure was handled is very similar to the way that IDE drives are handled, when 2 drives are on the same IDE cable.  In your case, the 4 SATA ports were assigned scsi5 and scsi6, 2 drives each (ata5.00 and ata5.01, ata6.00 and ata6.01), which seems very similar to the way IDE drives are assigned.  Normally, if the drives are in a SATA mode, I see scsi5, scsi6, scsi7, and scsi8, which correspond to ata5.00, ata6.00, ata7.00, and ata8.00, all on different 'channels'.  Also, when Disk 5 could not be accessed, the kernel failed both drives on ata5, very similar to the way when one drive on an IDE channel fails, it can 'take down' the other drive too.

 

Check the BIOS menus, and see if there is a better SATA mode for the onboard drives.  Look for a native or enhanced SATA mode, best of all would be AHCI mode.  Your parity drive is connected to an onboard port, your best choice.  An AHCI mode may provide slightly better simultaneous access performance.  I don't think your ICH7 controller fully supports all of the features of the AHCI mode though, but that's a minor nit.  I think it is possible that if the drives had been in a more native SATA mode, that Disk 1 might not have been affected by the Disk 5 troubles (unless the problem is actually with the ICH7 chipset, or its driver).

 

For the original Disk 5 (serial ...A6T), you can re-install it for testing, but do NOT assign it in unRAID.  When booted, you should be able to obtain a SMART report and reiserfsck it, perhaps even mount it like unRAID does and access the files on it.  I would expect to see some transactions replayed on its first mounting.  You cannot add it to unRAID, unless you either clear the drive, or rebuild parity.

Rob,

Thanks for the reply.  I will do some investigative work on Disk 5 to see what happened but it will have to be after I get back from a trip that I am going on.  I also will check the bios to see what my options are for SATA modes.