Failed drive report during replacement of another drive

[talmania]

I'm resigned to the fact that I've probably lost data here but wanted input---

 

I recently replaced my parity drive (running parity check before and after).  Then precleared the old (maybe 2 years old) parity drive.  Everything checked out perfectly so I swapped it in for drive 13.  During the rebuild at approximately 50% complete, I get a notification that drive 9 has failed.  I wake up this AM and sure enough the user shares have dropped off, disk 9 shows millions of errors but its online and the rebuild is about 72% complete and still running.

 

My plan is to let the rebuild complete and then do some troubleshooting (limited) with drive 9.  Not sure what else to check except a reboot and perhaps reseating the drive.  What I'm really after is does there exist any ability to tell/determine what files were specifically on drive 9?  I have all important shares backed up but as 90% plus of my array is media, what I really want to know is what files I lost on that particular drive.  Is there anyway of telling this?  I use Plex so perhaps there might be an option there but am unsure.

 

Thanks for any and all advice!

[itimpi]

A rebuild cannot complete successfully if another drive dropped offline or is playing up (unless you have dual parity).

Edited August 1, 2017 by itimpi

[Frank1940]

Do you still have the old drive 13 intact?  And is it good as far as you know?  Are your using any kind of a front insert/quick change cages to hold your drives?

[talmania]

Thanks for the feedback---yeah I just had a eureka moment I hope!  I do have the old drive 13 and it is in perfect condition (besides being 5+ years old).  Can I cancel the rebuild and go backwards and reassign the original drive 13 to it's previous place?

[Frank1940]

I think you can but wait until someone who has actually done it jumps in with a list of instructions.

 

Is there any possibility that you loosen one of the SATA connectors on drive 9 when you made one ow swapouts?  (It may still be good also!)

[talmania]

Possible but unlikely--they are hot swaps and the drive was present with no issues when I started the rebuild and monitored it for the first 20 percent or so.  Thanks Frank!

[Frank1940]

Post up your diagnostic file.  To the real Gurus, It might contain some clues as to what happened.

[talmania]

Thanks--can't believe I forgot them in the initial post!  Thanks again Frank--hopefully someone can provide some insight into the appropriate direction to take next.  Thanks again!

deed-diagnostics-20170801-0928.zip

[JorgeB]

Disk9 dropped offline, so there's no SMART report, it's on a SASLP, so it may not be a disk problem, you should cancel the rebuild, reboot, grab and post new diags.

[talmania]

Thanks johnnie!  I've stopped the rebuild and am trying to take the array offline and getting a repeated "REISERFS error (device md9) resierfs_read_locked_inode: I/o failure occurred trying to find stat data of [4 1463 0x0 SD]  from the console of the server.

 

Gui just says unmounting disk shares...retry unmounting disk shares.  

 

Hard power off I'm assuming?

 

[JorgeB]

That's normal since the disk dropped offline and you have another invalid disk.

 

Reboot by force if needed and grab new diags, you can try typing reboot on the console

Edited August 1, 2017 by johnnie.black

[talmania]

Gotcha--here you go!  Noticed disk9 shows no errors now and I can actually see the smart report (shows 7 current pending sectors).  Thanks so much for your help!  It's incredibly appreciated!

 

Edit:  And the parity rebuild started on Disk13 on boot.

 

 

deed-diagnostics-20170801-0955.zip

Edited August 1, 2017 by talmania
more info

[talmania]

And now Disk9 is throwing errors in the gui and can no longer read smart data.

[JorgeB]

Disk9 is really bad, this would be a good situation for dual parity, is disk13 and the array data untouched since the upgrade?

[talmania]

Just now, johnnie.black said:

Disk9 is really bad, this would be a good situation for dual parity, is disk13 and the array data untouched since the upgrade?

 

Yep completely---parity check was run and completed before the swap for disk13 which I still have available.  Is the best course of action to stop rebuild, shutdown, replace disk13 with the old one then unassign disk9 and replace it instead?  I feel like that's the correct move but I don't know if unraid will allow the old disk13 back in it's place...

[JorgeB]

You can try something, ideally with a same size replacement, do you have a 2TB spare?

[talmania]

I probably do actually---wouldn't be precleared and would be a previous pull from the same array.  Also have 5 other brand new 8TB Red's waiting for install--disk9 was on the list for replacement as it's one of the oldest in the array.  Not opposed to going out and buying a 2TB if that's required.

[JorgeB]

If possible use a 2TB disk since this is how this procedure was tested and used so far, but if you don't have one I can modified and it still should work.

 

Shutdown, swap disk 13 with original disk, use a new/spare 2tb disk for disk9.

 

-Tools -> New Config -> Retain All -> Apply
-assign missing disks (9 and 13)
-check both "parity is already valid" and "maintenance mode" before starting the array
-start the array (disk9 can show up as unmountable or with other data depending on what's there, ignore for now*)
-stop array, unassign disk9
-start array, check emulated disk 9 mounts and contents look correct
-stop array, reassign disk9
-start array to begin rebuild

 

*edit: this was true previously but new procedure now first starts the array in maintenance mode so there's no mounting.

Edited August 1, 2017 by johnnie.black

[talmania]

Thanks Johnnie--if the 2TB gives me better odds then I'll pick a new one up.  Will do so, follow the above steps and report back.

[JorgeB]

If the old disk is still detected this should work the same, but I never tested to confirm:

 

Shutdown, swap disk 13 with original disk, use old 2tb disk for disk9.

 

-Tools -> New Config -> Retain All -> Apply
-assign missing disk (13)
-check both "parity is already valid" and "maintenance mode" before starting the array
-start the array
-stop array, unassign disk9
-start array, check emulated disk 9 mounts and contents look correct

-shutdown, replace old disk9 with a new larger one
-assign new disk9
-start array to begin rebuild

Edited August 1, 2017 by johnnie.black

[talmania]

Ok thanks--looks like to me the difference is in utilizing the old current 2TB drive in this process. It's still being recognized (at least at last boot it was)--in your professional opinion is the new 2TB disk 9 option the safest route to potentially salvaging my data?

 

[JorgeB]

The second option should work as well, I just can't say 100% because it's untested, because of that, yes the 1st option would be safer.

[talmania]

2 hours ago, johnnie.black said:

If possible use a 2TB disk since this is how this procedure was tested and used so far, but if you don't have one I can modified and it still should work.

 

Shutdown, swap disk 13 with original disk, use a new/spare 2tb disk for disk9.

 

-Tools -> New Config -> Retain All -> Apply
-assign missing disks (9 and 13)
-check both "parity is already valid" and "maintenance mode" before starting the array
-start the array (disk9 can show up as unmountable or with other data depending on what's there, ignore for now*)
-stop array, unassign disk9
-start array, check emulated disk 9 mounts and contents look correct
-stop array, reassign disk9
-start array to begin rebuild

 

*edit: this was true previously but new procedure now first starts the array in maintenance mode so there's no mounting.

 

Ok i'm back--I got a new 2TB red drive and performed the steps quoted above---I stopped array and unassigned disk9.  Then I started the array with only the option checked to "add a drive as soon as possible"--not the one that started the array but didn't mount the disks.  

 

My drive9 now shows as "unmountable" and I can browse the server and see the other shares INCLUDING the disk shares but there is no disk share for "disk9".

 

Should I still proceed?  Thanks!

[JorgeB]

Post new diagnostics

[talmania]

Attached--thanks again!

deed-diagnostics-20170801-1354.zip