The only way corruption like this should be possible is if the failing device returns bad read data, meaning a read command returned with 'success' status but the data it returned was actually bad.  This of course would be a serious firmware issue with the device.

 

There is a way to simulate a write error.  At any time you can type this command:

 

mdcmd set wrerror <N>

 

where N is the disk number: 0 => parity, 1 => disk1, etc.

 

After typing this command, the effect is that the very next write command (and only that write command) on that disk will be tagged as "failed" and the code that disables the device will kick in.

 

On a failed write that kicks a disk, do you spin up all the disks to compute the updated parity block? Or are you depending on the preread from the now failed disk?

Tom can confirm it, but if I understand what he said earlier, he depends on the pre-read IF the status is "success".    So if the data is actually good, then the parity update will be good as well.    But if the data isn't really good, the parity update will be incorrect ... and thus the issue you've seen.

 

I've tried to think of other scenarios that would cause the corruption issue you've noted, but virtually everything I can think of requires that some device has returned an incorrect status.  i.e. if the pre-read status is "good", but the write status is bad (thus failing the disk), it shouldn't matter => parity would already be correct, so any subsequent emulation should be just fine.

 

But clearly this didn't happen in the case you had, so something else has to be going on.  I won't call it an "edge case" again    ... but it's clearly something relatively rare and difficult to reproduce (or Tom would have seen it before and probably already have it fixed).

 

I'll play around a bit with my test server -- disconnect the SATA cable for a drive in the middle of an ongoing write and see what happens.

 

 

 

On a failed write that kicks a disk, do you spin up all the disks to compute the updated parity block? Or are you depending on the preread from the now failed disk?

If the pre-read did not fail then of course it is used to regenerate parity.  The pre-reads of P and D occur in parallel, and the writes to P and D occur in parallel.