Hello.

 

Last night I noticed one of my drives (disk5) had a red ball next to it.  First thing I tried to do was capture the syslog before doing anything else...  unfortunately I copied the wrong syslog file from /boot/logs instead of /var/log.

 

Everything seemed to go down hill from there. 

 

unMENU was responding to my browser requests but when I tried to view the status of my server at //Titan I got a fail-to-load-page message in my browser. 

 

I then shut the server down and tried rebooting and it took a long time to come back up.  Once it finally booted the same thing happened as before:  unMENU was responding to my browser requests but when I tried to view the status of my server at //Titan I got a fail-to-load-page message in my browser. 

 

I knew there was an issue with disk5 since just a few minutes earlier I noticed the red ball on disk5 so I decided to shut the server down and unplug disk5 from the array.  Once I did this, unRAID booted normally and I was then able to access my server at //Titan and now I got what I normally would have expected:  The array came up, disk5 was disabled and probably needs to be replaced...  Fine.

 

My concern is how the server behaved when this (bad?) disk was attached to the array:  none of the other disks mounted and the entire array seemed to be in some weird hung/non-functional state when the bad data disk was attached.  I thought the other disks in the array should mount normally EXCEPT for this failed disk but that wasn't the case.

 

Anyway, if anyone would care to look at my log files I would greatly appreciate any thoughts or advice.

 

I've also attached a screen shot of the unMENU main screen with and without the suspected bad disk attached.  As you can see, none of the other disks in the array mount when the bad disk is attached.  Once I remove the power cable from the suspected bad disk, the array comes up.

 

Thanks for your time.

logs.zip

After a fruitless shop around town for a reasonably priced 2TB drive I finally ordered a replacement from Newegg.  I know I should have already had a spare drive on hand but I was a bad little monkey and never got around to doing this. 

 

The replacement drive will be here today.  I should know if this was in fact the problem (a failed disk) once I get the new drive pre-cleared and added to the array.

 

I'm still curious as to how/why a failed data disk could cause the entire array to become unusable.  Has this happened to anyone else before?

 

Thanks,

 

Seven

 

 

 

 

Just as an experiment I added an old 320GB drive I had to the array in place of the suspected failed Seagate 1.5TB drive.  The system booted normally and I could access my server at //Titan.

 

All of the other array disks came up normally and were recognized by unRAID - although the array did not mount (because the replacement disk needs to be equal to or greater than the size of the failed disk) but that's OK.

 

At this point I was just curious if the system would boot normally when another disk was plugged into that controller spot... and it did.  

 

So this leads me to believe it's definitely a bad disk that I'm dealing with here.

 

I'll post again once I get the replacement drive installed. 

 

I've also attached another syslog with the 320GB test disk installed in place of the suspected failed disk.

syslog-2010-10-26.txt

 

- I received the replacement disk today (WD20EARS 2TB)

- I removed the bad disk from the array case

- Installed a jumper over pins 7/8 on the new WD20EARS drive

- Installed the new drive into the array

- preclear_disk.sh started on new disk 25-minutes ago, now at 8% on the pre-read

 

Just for kicks I tried mounting the failed disk using my external SATA dock on my Windows PC...  it's dead for sure.  Luckily the disk is just under 3-years old so I'll be RMA'ing this bad boy.

 

 

 

Well the preclear_disk is now at 85% and 22-hours into the process.  After reading more about preclear it seems preclearing wasn't absolutely mandatory when replacing a failed disk...  but I guess it couldn't hurt as long as I don't lose another disk in the array while I preclear and rebuild the new disk.  I'm keeping my fingers crossed.

 

This is all still a learning process for me.  It's a lot of information to take in! 

-Preclear completed successfully on my WD20EARS drive.  Total run time = 24:47:21

-Added disk to array to replace the failed disk

-Parity + data disk rebuild successful, run time approx. 7-hours.

 

I have attached the final syslog file as well as the s.m.a.r.t. reports that were created by the preclear process.

 

This was a good learning experience for me.  I need to do a better job monitoring my unraid server health.  I'm going to be researching what others are doing to monitor their disks and I will always have a precleared spare disk on hand in case of future disk failures.

 

I'm still curious why my entire array fell over because of a single failed data drive...   but maybe it's a peculiarity with my motherboard or the built-in disk controllers or something.

 

 

Final_Logs.zip

I've seen cases where a motherboard won't boot at all while a bad drive is connected to it, or it will fail mid-boot (I assume it hangs when scanning all the drives, or something like that).  So my guess is that the motherboard was freaking out about the bad disk, not unRAID.  Just a guess, though.

 

It is true that you don't have to preclear a drive when replacing a bad one, and depending on how much you trust your other drives it might be a good idea not to (since that's an extra 24 hours of running without protection).  However, if you have the luxury of preclearing a replacement drive, it is always a good idea since a replacement drive failing on you during a data rebuild is the stuff of nightmares.  A few things you could do to protect your array during the preclear process are:

 

1) Unplug all your array drives, so that only the replacement drive being precleared is active.  Your array will be offline, of course, but at least your data will be safe.  Save a screenshot of the web page before doing this in case you hook the drives back up to the wrong slots.

 

2) Preclear the drive on a different machine (it could be running unRAID Basic), and keep your main server powered off.

 

3) Have your replacement drive precleared in advance, ready to step in to save the day (this would make your replacement drive a warm or cold spare, depending on if you keep it inside or outside the server).

 

Obviously option 3 is the best, but it requires foresight.  Personally, I rely on options 1 or 2, depending on whether or not I have a spare server set up (I generally do, since I'm often building them for others).  I'm too cheap to have a warm or cold spare sitting around

Obviously option 3 is the best, but it requires foresight.  Personally, I rely on options 1 or 2, depending on whether or not I have a spare server set up (I generally do, since I'm often building them for others).  I'm too cheap to have a warm or cold spare sitting around

 

Rajahal,  thanks for the advice.  I will definitely be keeping a spare disk precleared and ready to "save the day" in the future.  To me a $100 disk that I keep available as an emergency spare is much cheaper than the possibility of losing my data should another disk in the array fail while I was waiting for a replacement to arrive in the mail.

 

Thanks again for you help!