No clue, but it's impossible to correct parity errors during a rebuild, and if it's writing to parity it's making it worse.

 

Attach your syslog or look for lines like:

 

tower kernel: md: correcting parity, sector=139198600

I have attached diagnostics to this post, I went through the log, not all the way to the end but there are clear errors reading disks 13 and 19, 12 is the one rebuilding. So if I have two disks reporting read errors while trying to rebuild a third, I guess its fair to say that its going to be near impossible for 12 to rebuild properly? I know Gary is recommending a lengthy file compare but I've not got to disks reporting read errors and possibly questionable parity data, what's my next move? I'm more concerned about my parity data not being any good then that data on the drives since its a second copy of data anyway. When drive 12 is finished rebuilding, should I replace drives 13 and 19 one at a time or what?

tower-diagnostics-20160404-0826.zip

I don't see any corrections done to parity, that's how it should be, probably some false info on the GUI, but like already mentioned before, the rebuilt disk will have some corrupted files.

... I'm more concerned about my parity data not being any good then that data on the drives since its a second copy of data anyway.

 

While the result may be "bad" data that's parity-protected, the only way to be certain parity is good is to do a parity check (standard correcting check unless you want to spend the time for a non-correcting check just to see how much needs to be updated);  or to simply do a complete new parity sync.

 

 

... When drive 12 is finished rebuilding, should I replace drives 13 and 19 one at a time or what?

 

You could do that, although with uncertain parity there's no guarantee the rebuilds would actually be correct.  Again, without checksums or "known good" data to compare with, this is really an "iffy" proposition.    At some point I'd definitely run compares between your main server and the backup files to see which ones are different [Clearly this will take a LONG time to do -- but it's the only way to really know for sure which backup files are truly backups and which ones are corrupted.].

 

Right but what do I do about the two failing drives? How do I replace them? If I replace them one at a time a do a resync with each, but run the risk of further corrupting my parity data, then what is the best way to go forward? Is there an easy answer?

Right but what do I do about the two failing drives? How do I replace them? If I replace them one at a time a do a resync with each, but run the risk of further corrupting my parity data, then what is the best way to go forward? Is there an easy answer?

 

What I would be tempted to do is to get a listing of all of the files on any and all drives that are failing.  Do a non-correcting parity check on the other server.  (You want to be sure you would not have any trouble if you have to restore the files from there if that turns out to be necessary.)  Then order enough new drives to replace them.  Then run put them through enough preclear cycles to get a minimum of 100 hours on each drive.  (This will weed out any drive that is going to be a victim of infant morality.)  Remove those old drives and set them aside.  Install the new drives and add them to the array.  Now copy back the files on the lists that you prepared.  You could try to see what you can read from those old drives or simply copy the files back from the other server.

 

Now you can look at those old drives and see what you can recover from them. 

Thanks Frank, I get what your saying, I guess I am not sure how to go about replacing the two drives that are giving read errors. I have replacement drives that have been pre cleared and ready to go, but when I replace the drives giving read errors, am I doing a resync or ? I need to know how I am going to replace the drives, what are the steps or gotchas?

I would do a new config, replacing all bad disks with new ones, doing a parity sync, then copy all missing data from the old disks or backups.

I would do a new config, replacing all bad disks with new ones, doing a parity sync, then copy all missing data from the old disks or backups.

 

Agree -- that's the safe way to do this.  In fact, I'd be tempted to do the same with the drive you just rebuilt, since that's a suspect rebuild.    i.e. just reformat that drive so it's empty;  then replace the other two failed drives with new ones; do a New Config; and then start copying files from the main server to the backup.    Either from a list of the files that were on the replaced drives; or simply using a sync utility that will copy all of the files that aren't already on the backup server.

 

Thanks guys, that is what I'll do then, what a pain in the @$$ oh well, $h17 happens I guess.