Invalid parity and data drive down

[ridge]

I have a problem, and I think I got myself into more of a mess trying to solve it.

 

I had what looked like a bad drive. So - given that I had a spare sitting in my array - I stopped the array and switched the old for the new drive, then brought it back online to start rebuilding the data from parity. Except it didn't work, because either the second drive was also bad, or I have a problem elsewhere. This is where it gets a little hazy as it was very late and I was tired, but I put back the old drive, and did an initconfig to reset everything. And then started the array to rebuild the parity drive.

 

The problem is, that didn't work. It got about 50% of the way through, and then quit after about 20 million (literally) parity errors that it was correcting. The attached screenshot shows you what I'm facing, after doing an initconfig and resetting the drives to their original place.

 

I've uploaded my syslog here as it's much too large to attach:

https://dl.dropboxusercontent.com/u/15336/syslog-2013-05-28.txt.zip

 

What's my next step? FYI, I have two new drives arriving in the next day or so. 2 4TB drives, one as a parity replacement, the other a new data drive.

 

Edited to add:

Running version 5.0-beta14. I'm gunshy about upgrading until things are more stable, to be honest.

 

Smart report for the drive is:

Statistics for /dev/sde Hitachi_HDS5C3020ALA632_ML2220F31LGNUE
smartctl -a -d ata /dev/sde
smartctl 5.40 2010-10-16 r3189 [i486-slackware-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)

A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

[garycase]

Well ... the "initconfig" and attempt to rebuild parity HOSED your old parity, so there's no way to recover any failed disk.

 

Your best option at this point is to simply start rebuilding the array from scratch.  First thing I'd do is rebuild the USB Flash drive with RC12a.  Then I'd add UnMenu.  And then I'd look at the SMART status and run a Short SMART Test on each drive BEFORE you add it to the array.

 

THEN ... when all the good drives have been added, add the parity drive and do a parity sync.  When that finishes (many hours), do a parity check (another long process).    When that's done, you're up and running on RC12a.  Any bad drives you'll simply have to replace and copy the data from your backups.

 

[ridge]

Thanks Gary. I suspected as much, to be honest.

 

After a reboot, I was able to run a SMART status on the drive that failed. It's attached.

smart.txt

[ridge]

After reading the upgrade instructions I'm even more gunshy about upgrading:

 

Only upgrade a server that has all disks enabled with valid parity: no missing disk(s), no disabled disk, etc. Verify everything is working correctly with your current release before proceeding.

 

Is upgrading to rc12a really the best thing to do, given the invalid parity, and disabled disk? Or does it not matter because it looks like I'm going to be rebuilding the array anyway?

[garycase]

After reading the upgrade instructions I'm even more gunshy about upgrading:

 

Only upgrade a server that has all disks enabled with valid parity: no missing disk(s), no disabled disk, etc. Verify everything is working correctly with your current release before proceeding.

 

Is upgrading to rc12a really the best thing to do, given the invalid parity, and disabled disk? Or does it not matter because it looks like I'm going to be rebuilding the array anyway?

 

The simple fact is you've already wiped out your parity drive, so you have to start from scratch ... and given that, you should use the most current version.