Hi folks,

 

I've been running my unraid server for a few months now. It's been working great. However, just today i received a notification that one or more drives had failed. I went to the main page of unraid and found a red dot beside my parity drive (DISK_DSBL status). The errors column had a count of 2 and the tempurature for that disk was showing as 0 degrees. I also noticed that one of my data drives was showing a 0 degree temperature and it had 14000 error count but it still shows a green dot. I'm worried that i've somehow had two drives fail at the same time. I've stopped the array. I know i should reseat my SATA connections for those drives so i will do that when i get home. One thing im wondering is i recently installed a SuperMicro 5-in-3 drive cage. Maybe it is malfunctioning and causing my drives to look as though they're not working properly. I'm running unRAID v4.7 final. I've attached my syslog dump.

 

Can anyone comment on the best course of action?

 

Thanx for any and all help. Much appreciated.

 

-Dave

 

syslog-2011-02-11.zip

Okay. New developments... i tried to do some 'smart' tests (via smartctl) on the drives i was concerned about (ie. parity & disk1). For whatever reason, i was getting a 'Device Read Identity Failed' message. After searching the forums for this i saw a thread where they said they rebooted and were then able to run the smart checks. I decided to reboot. The reboot went fine. Now on the unRAID desktop, im seeing a blue ball beside my parity drive (unRAID thinks it's a new drive) and the data disk that i was concerned about still has a green dot. I ran short smart tests on them and both came back as 'PASSED' without any errors in the report. I'm going to run full smart checks on these drives as well as run the western digital diagnostic tool (drives are WD20EARS) before i bring the array back online.

 

Just curious if anyone else has seen anything like this before ?

 

-DaViS

Yet more information:

 

I checked out the smart diagnostics under unMenu 'myMain' tab and found some problems. The 'disk1' data drive is showing with the following errors:

 

» current_pending_sector=1

» offline_uncorrectable=1

» multi_zone_error_rate=1

 

The 'current_pending_sector' error is critical. Therefore the drive is not in good shape. That's fine though. I can replace it under warranty.

 

My concern is that the parity drive is now showing as a new drive. This means if i do get a replacement, then i wont be able to rebuild the data for disk1. I'm thinking my best option might be to copy the data from disk1 manually (cuz i think this drive is still readable, ie. it hasn't died completely). Then replace it and rebuild parity. Then re-copy the data over to the unRAID server.

 

My parity drive is showing with this error:

 

multi_zone_error_rate=3

 

This is not critical but wondering if this may have caused the issue where my parity drive is no longer recognized.

 

Has anyone seen this behavior ? Would really appreciate some responses. Thanx.

 

-DaViS

Yet more information:

 

I checked out the smart diagnostics under unMenu 'myMain' tab and found some problems. The 'disk1' data drive is showing with the following errors:

 

» current_pending_sector=1

» offline_uncorrectable=1

» multi_zone_error_rate=1

 

The 'current_pending_sector' error is critical. Therefore the drive is not in good shape. That's fine though. I can replace it under warranty.

 

My concern is that the parity drive is now showing as a new drive. This means if i do get a replacement, then i wont be able to rebuild the data for disk1. I'm thinking my best option might be to copy the data from disk1 manually (cuz i think this drive is still readable, ie. it hasn't died completely). Then replace it and rebuild parity. Then re-copy the data over to the unRAID server.

 

My parity drive is showing with this error:

 

multi_zone_error_rate=3

 

This is not critical but wondering if this may have caused the issue where my parity drive is no longer recognized.

 

Has anyone seen this behavior ? Would really appreciate some responses. Thanx.

 

-DaViS

If your parity drive is disabled it is because a "write" to it failed.  it is a simple as that, all it takes is 1 write that fails to ANY disk in the array and it will be disabled.

 

It might have failed because the disk itself is bad, or because the disk controller is bad, or because the cable to the disk is loose, or the power cable is loose, or a splitter you used to feed the disk is loose or intermittent,  or the power supply is marginal and unable to supply everything calling for power at the time the write occurred.

 

It might be the disk has taken itself off-line and will not respond again until power cycled. (EARS drives seem to act this way)

 

Disks typically have a reserve pool of several thousand spare sectors.  The disk manufacturer  will probably not consider one sector pending re-allocation as a failed disk.  (let us know if you inquire and they consider the disk a candidate for an RMA)

 

To get a failed disk back in service you must first check to see if it responds to commands at all.  A "short" is probably quickest.  Request one, then wait 5 minutes and get another status report.  If it is successful you can then get unRAID to re-construct the failed drive back onto itself.  Remember, a "wirte" to it failed, so it does not have the correct contents and it must be reconstructed.

 

To reconstruct it yo must first get unRAID to forget its model/serial number.

To do that

Stop the array

Un-assign the disk on the devices page

Go back to the main page and Start the array with the drive un-assigned  (This causes unRAID to forget the model/serial number)

Stop the array once more

re-assign the drive.

Go back to the main page ans press "Start" once more.  The drive will re-construct onto itself.

 

Joe L.

Thanx for the reply Joe. L.

 

When you say the disk will reconstruct onto itself, do you mean it will do this using parity information? If that is the case, this will not work for me because unraid thinks my parity drive is new.

 

I've already requested a 'short' SMART test and it comes back as passed for both the parity drive and disk1, even though unMenu shows that pending re-allocation error under smart view for disk1.

 

Thanx for the reply Joe. L.

 

When you say the disk will reconstruct onto itself, do you mean it will do this using parity information? If that is the case, this will not work for me because unraid thinks my parity drive is new.

 

I've already requested a 'short' SMART test and it comes back as passed for both the parity drive and disk1, even though unMenu shows that pending re-allocation error under smart view for disk1.

 

If unRAID thinks it is new, it indicated you started the array once when it was un-responsive.  (or un-assigned, or un-plugged)

 

When you start the array parity will be , in effect, initially calculated and written to the parity disk.  (essentially writing exactly what is thee already, but with the data that failed to be written as well, as it would have been written to the data disks.)

 

I'd stop the array, power down, re-seat the connectors to the parity disk, both data and power, power up, and let unRAID calculate parity.

 

If you've not written anything to the array since it went off line you can try the "trust my parity" procedure as described in the wiki.

Do let it run the parity check when the array starts.

 

Joe L.

As soon as i ran into these issues, i stopped the array and have NOT started it since. I haven't changed any device assignments either. This is why it seems strange that parity disk is recognized as new. I have rebooted a couple times. After the first reboot, the parity drive was showing as new. I'm going to try power cycling the pc and reseating all connectors, as you said. I have the problem drives in a 5-in-3 drive cage but will remove them and hook up directly to the mobo to eliminate that being a possible cause.

 

So parity has not been touched since the failure. My worry is that it is corrupted. If i force unraid to trust it, im afraid it may rebuild corrupted data on disk1 due to parity itself being corrupted from write failure.

 

Almost seems like my best option is just to copy the data off disk1 manually (ie. on another pc). Then force unraid to forget about the parity drive and disk1 drive. Then re-assign them as new drives and have parity rebuilt. At that point i can copy the data back to the unRAID server. Does this not sound like the safest solution ?

I would trust disk 1 more than the parity drive.  (And it is why you are correct, the "trust" process has its own issues)

 

Yes, the drive cages can have intermittent connections.  All it would take is one start of the array with the disk not responding for unRAID to forget its model/serial number and treat it as "new" when a connection could subsequently be made.

 

I say start the array, let unRAID rebuild the parity drive.  It is the least "risky" of all.

 

Joe L.

Sounds good. I'll check some connections and then start the array to build new parity. Thanx for all the help Joe L. I'll let ya know how i make out with this.

 

-DaViS

I've done some more testing and i think i've found the source of the problem. After re-seating the sata/power connections, i ran a party check, as you suggested Joe L. The parity check completed with 87 errors. So, in order to try and find the source of the issue, i removed the drive that was showing errors from my somewhat new SuperMicro 5-in-3 drive cage. I hooked it up to the same mobo sata slot directly using a different sata cable. Then i ran the parity check again. This time the parity check completed without errors. I was also able to copy files to the problem drive without errors... which was not possible before. So i now know that the problem is either with the drive cage, or with the sata cable... not the HD. I'm going to put the drive back into the drive cage tonite but with the newer sata cable. After running a parity check with this configuration, it should tell me whether the problem is the drive cage or not. If i get errors, i know for sure it's the drive cage. If no errors, then i know for sure it was the older sata cable. I'll post back with my results.

 

-DaViS

I've been a bit delayed on getting back to this. Was busy at the hospital taking care of our new baby boy, born feb. 14.

 

It appears my previous assumptions here are not correct. I had originally thought there was either an issue with the sata cable or the drive cage. After running multiple parity checks with/without drive cage etc, there were no errors. However, there are some new issues i've discovered... I've noticed that now that my server is back online, there are some files/directories missing. If i telnet into the unraid server and list for those directories, i can see the files/subdirs are there, but for some of them im getting a 'permission denied' message. I'm not able to write to these folders/files through the shares either. I read some other posts in the forum where it was suggested to run a reiserfsck check on the drive hosting the problem files. I've done so and received some errors suggesting to run a 'rebuild-tree' operation on the drive. On the instruction guide for the reiserfsck check, it says not to do this unless directed to by unraid support crew. Syslog and reiserfsck are attached. Can someone tell me what i should do here? Thanx very much.

 

-DaViS

 

reiserfsck.txt

syslog-2011-02-17.txt

You will need to follow the steps outlined here in the wiki:

http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

 

The next step for you is to run with the

--rebuild-tree

option.

It seems that running the reiserfsck check with --rebuild-tree option has fixed the issues i was having with disk1. I have no idea what got my drive into this state. Maybe it is related to the errors that i had on the drive (ie. current_pending_sector error) when i first ran into problems. The drive cage and sata cable were fine in the end. Anyway, things are back to working now. Let's hope they stay that way. Will mark this thread as [sOLVED]. Huge thanx to Joe L for all the help on this!