- Had a drive (drive 13, a 4TB Seagate NAS) tell me it was about to die - end to end SMART was "FAILING NOW"

- So before I left town for a couple of days, I stopped the array, assigned a warm space to the slot, and kicked off a rebuild

- Came home to find that whilst drive 13 was being rebuilt, a different drive (drive 3) died

  - This is bad - I don't have dual parity

- Attached is the notifications:

 

- Clicking about, everything from drive 13 -looks- to have been rebuilt okay

- Clicking about, everything on the simulated-with-parity drive 3 -looks- to be intact

 

- What should I do next?  I had stopped my crashplan docker before - I'm not starting it back up, to make sure if any of these files are corrupted, they don't upload over the good copies in the cloud

- Is there anyway to verify the files on drive3/drive13 to see if any got corrupted?

 

 

You should attach the diagnostics.

 

The rebuild finish but disk13 has almost certainly some or a lot of corrupt files, the same thing will happen if next you rebuild disk1.

 

If you have checksums check all files on disk13, if you don't and the old disk13 is not completely dead use a byte level file compare utility to compare files on both disks.

You should attach the diagnostics

https://gist.github.com/JustinAiken/d4fc4fb7886de1a00da6a4bef0b90390

 

There's:

- The syslog from just before I kicked off the rebuild, until the log quit (The other one was 0 bytes)

- Misc other little files

 

The rebuild finish but disk13 has almost certainly some or a lot of corrupt files

 

Was afraid of that..

 

If you have checksums check all files on disk13

 

I don't - any recommendations for mass-checksumming in the future?

 

You don't and the old disk13 is not completely dead use a byte level file compare utility to compare files on both disks

 

Unfortunately, I ran a preclear on it at the same time as the rebuild - wanted to 0 it out before I RMA'd it off..

 

Without checksums or the old disk, only thing you can do is go file by file or restore from backups if available.

 

I use this for checksums:

 

Dynamix File Integrity plugin

... Unfortunately, I ran a preclear on it at the same time as the rebuild - wanted to 0 it out before I RMA'd it off..

 

Ouch!  As you've clearly learned, NEVER do anything to a failed drive until you've successfully rebuilt it on a new replacement.

 

... This is bad - I don't have dual parity

 

Ouch again !!  The primary purpose of dual parity is to protect against exactly this case => and the more drives you have, the higher the likelihood that it'll happen.  With 18 drives I'd have added a 2nd parity drive as soon as 6.2 hit the RC phase => ESPECIALLY if you don't have a complete set of backups.

 

At this point, as Johnnie has already noted, there's really nothing you can do except compare the files on the two failed drives to your backups -- and if you don't have backups you're simply out of luck => essentially all of the data on the two failed drives is suspect, and you have no way to validate what's good or bad.

 

Note that even with a complete set of checksums, you couldn't recover any of the bad data => you'd simply be able to tell which files were good and bad.    There's simply no substitute for backups.  [Which you should also keep checksums for, so you can validate that they're still good as well]

 

 

About the failed disk 3, and if you don't have backups, this is what I would do:

 

-Rebuild it to a new disk, this will again result on some or many corrupt files

-After the rebuild copy all the data you can from the old disk, all files copied without any error should be ok and used to replace the files on the rebuilt disk

-Files you can't copy from the old disk can be OK or not on the rebuilt disk, but you at least reduced the number of possible corrupted files, and the number of files to check for corruption

I use this for checksums: Dynamix File Integrity plugin

 

Thanks! After I get this mess resolved, I'll start using that.

 

The primary purpose of dual parity is to protect against exactly this case => and the more drives you have, the higher the likelihood that it'll happen.  With 18 drives I'd have added a 2nd parity drive as soon as 6.2 hit the RC phase

 

Haha... yeah.  I'd updated to the 6.20RC's once they came out because I was excited for dual parity, I just hadn't gotten around to setting it up yet

Learned my lesson - new 8TB ordered and on it's way!

 

At this point, as Johnnie has already noted, there's really nothing you can do except compare the files on the two failed drives to your backups -- and if you don't have backups you're simply out of luck => essentially all of the data on the two failed drives is suspect, and you have no way to validate what's good or bad.

 

Yeah... For the files I do have a backup of on these drives, I think I'll just nuke them and restore them from Crashplan (family pics, other important things are backed up!) 

 

For files I don't... guess I need to scan through lots of movies to see if they're corrupt or not!! 

 

-Rebuild it to a new disk, this will again result on some or many corrupt files

-After the rebuild copy all the data you can from the old disk, all files copied without any error should be ok and used to replace the files on the rebuilt disk

-Files you can't copy from the old disk can be OK or not on the rebuilt disk, but you at least reduced the number of possible corrupted files, and the number of files to check for corruption

 

Excellent plan - as soon as my advance RMA from Seagate comes, I'll preclear and do exactly that - thanks for the advice!