Drive Failure - now 2nd 'Unmountable: not mounted' after reboot.

[JorgeB]

Current release only lets you do the test if the disk is set to never spin down.

[trurl]

1 hour ago, MooTheKow said:

is this that setting?

That setting is for all disks. There is also a setting for each disk on its page that overrides that global setting for a specific disk.

[MooTheKow]

So - still haven't fully run that extended self-test.  Keeps ending up with 'Interrupted (host reset)'.

 

I don't know if this explains it/is related - but I do keep getting email notifications from Unraid while it runs that read like:

 

Event: Unraid device dev1 SMART health [5]
Subject: Warning [KOWUNRAID] - reallocated sector ct is 456
Description: ST8000DM004-2CX188_WCT0VALG (dev1)
Importance: warning

 

and

 

Event: Unraid device dev1 SMART health [5]
Subject: Warning [KOWUNRAID] - reallocated sector ct is 488
Description: ST8000DM004-2CX188_WCT0VALG (dev1)
Importance: warning

 

 

[trurl]

Looks like it is getting worse. Might be time to try cloning both disks and see what can be recovered from the clones.

 

7 hours ago, MooTheKow said:

Currently I have a couple 4TB external drives and a new 14TB internal drive.    if I need to pick up additional internal drives I can/will.

 

I'm pretty sure you can't clone to smaller disks, and externals would probably also complicate things. Maybe you could clone to that 14TB but that is larger than parity and also would make things more complicated.

 

@JorgeB is the expert on cloning so let's see what he thinks.

[JorgeB]

You can clone to a larger disk, but that disk you not be mountable in the array, also it would be larger than parity, but you could still mount it with UD, another option would be doing a parity swap then use old parity for the clone.

[MooTheKow]

Thanks @JorgeB

 

Ok - I think I'm going to go the parity swap route (that was originally my plan for the 14 TB drive anyway).

 

So - I follow those instructions to swap the parity drive.  Do you happen to have a link describing the procedure for cloning the drive? 

 

Also - the drive to be cloned is not currently part of the array.  Once I've cloned its contents onto the old parity drive - how do I go about adding it that clone back to the array?

 

Also - just for clarification - I should be able to do the parity swap without hosing things up even though I am currently down 2 drives in my array?   In theory -- then after cloning the defective disk and re-adding it to the array I would be able to then rebuild the other broken drive (onto another new drive) and end up with minimal data loss?

 

Am I missing any steps or how this may play out?

 

Also - if I use the 'mount' on that drive in the 'unassigned devices' - just to read files off of it before cloning - will that have potential to negatively affect anything?

Edited December 24, 2022 by MooTheKow

[trurl]

2 hours ago, MooTheKow said:

I should be able to do the parity swap without hosing things up even though I am currently down 2 drives in my array? 

Not going to work with single parity. You would have to New Config all disks back into the array with Parity Valid, then try it.

 

I'm not sure parity is going to do you any good at this point anyway since you have 2 unusable disks and they are unlikely to be in sync anyway.

 

You might New Config without either disk, rebuild parity to that 14TB. Then what's left of your array would at least be protected again. Then you could get spares to try to clone the other disks to see if anything can be recovered from them.

[trurl]

On second thought, it should allow parity swap because only one of the disks is disabled. The other is just unusable. Probably the parity copy step of parity swap would work, then the rebuild of the disabled disk would fail due to the other bad disk. I don't know where you would go from there. Maybe cloning that other disk, then New Config clone back into the array with Parity Valid, then you could do the unassign/reassign dance to get the other disk rebuilt. And there would probably need to be some filesystem repair along the way with both disks. Would take days, some frustration, and almost guaranteed to be some loss.

[trurl]

On 12/23/2022 at 8:49 AM, MooTheKow said:

at this point most data loss would just fall into the 'huge inconvenience' category instead of the 'now i need to cry that it is lost forever' category.

  

Maybe not worth the trouble then.

 

Or you could do this

14 minutes ago, trurl said:

New Config without either disk, rebuild parity to that 14TB. Then what's left of your array would at least be protected again. Then you could get spares to try to clone the other disks to see if anything can be recovered from them.

and then take your time going to as much trouble as seems fit.

[MooTheKow]

Ok - so had started the parity copy -- and that's going and will take a while but is going.

 

I've had very little activity for the most part writing to the array since my 8tb disk was removed from the array.    If I was able to clone that to a new 8TB drive (my old parity drive) - can I then re-add that to the array somehow and do a rebuild for the 5TB drive that has failed?  I know the parity wouldn't be completely valid -- but would it at least be able to partially recover what was lost with random corruption due to whatever was changed while that disk was offline?  Or does it not work that way (i..e has to be an all-or-nothing thing)?

 

Also - two other questions:

1) Can I attempt to mount that failing 8TB drive from unassigned disk without problems?

2) Is there a resource somewhere detailing steps to clone a drive?

 

thanks again for all the time you've taken to answer my questions and attempt to help me.  Especially given the time of year it is really appreciated - you guys are awesome.  I appreciate the help you're given even though the troubles I'm experiencing are largely my own fault for not paying close enough attention to what was going on (though to be honest - when one drive failed - i ordered a replacement and I the other didn't actually fail until I turned the power off to attach my new replacement drive 🙂 - but still my own doing)

Edited December 24, 2022 by MooTheKow

[trurl]

1 hour ago, MooTheKow said:

If I was able to clone that to a new 8TB drive (my old parity drive) - can I then re-add that to the array somehow and do a rebuild for the 5TB drive that has failed?

That and other approaches are possible. Since both are unmountable, filesystem repair will also be required. It can get pretty time-consuming, complicated, and no way to predict how much success there will be. And you won't be able to use your server until it all gets fixed.

 

Or you can New Config without those disks to get your server going again and work to recover data from them or their clones later. That way will make it impossible to rebuild either from parity of course, but no way to predict how much parity rebuild would recover anyway.

 

 

[MooTheKow]

Ok - not sure what the deal is or if this changes things -- but I am currently able to mount that 8TB drive in the 'unassigned devices' area and access the files on it (first thing I've done is create a read-only windows share on it -- then did a  "dir /b /s /od > c:\temp\filelist.txt" so i at least have a list of everything I may be losing so I can maybe sort through it and know what's gone as a worst-case scenario :-))

 

I'm not banking on this drive fully working or not eventually going completely dead on me, but since I can get to it here -- is there a way for me to re-add it to the array somehow?   Or do I really want to just try copying the contents off it off onto a 'new config' that doesn't include this drive in the array?   Assuming doing this essentially means that my 5tb drive (previously disk 3 -- the one that it recognizes but calls 'unmountable' ) is just toast and I won't be getting any data off it?

 

Any good recommendations for how to copy files from that drive to the new array if I go that route?  I'm more familiar with windows and its tools than linux, but reasonably comfortable working command lines :-).   I assume I'd want some sort of tool that'd be able to handle errors it hits while attempting to copy files without completely bailing out on the copy (historically I've used an old microsoft tool called 'RichCopy' since it is multi-threaded - but open to other suggestions)

 

 

 

[trurl]

Did you complete the parity copy step of the parity swap?

 

2 hours ago, MooTheKow said:

is there a way for me to re-add it to the array somehow?

Yes.

 

Post new diagnostics so I can see where we are now.

[MooTheKow]

Parity is still copying, won’t be done tonight so I’ll post tomorrow.

[MooTheKow]

Ok, copying of the parity drive just finished, array is still stopped - here are the diagnostics:

 

kowunraid-diagnostics-20221225-1009.zip

[trurl]

OK, let me summarize where I think we are just so we are all on the same page and nobody is confused about how we got here.

 

Parity swap has finished copying parity to a new disk, former parity is assigned as disk1 and waiting to be rebuilt.

 

But, disk3 has failed extended SMART self-test.

 

Original disk1 won't complete self-test for some reason, but it doesn't actually report that it failed self-test. It does have some SMART attributes that suggest problems, and of course, it is the disk that was disabled.

 

Looks like it was struggling, but syslog seems to indicate Unassigned Devices successfully mounted original disk1. Can you see data on that Unassigned Device?

 

 

[MooTheKow]

Yea. Summary sounds accurate and I can see all the data on the mounted original disk 1.  I have got a directory listing of everything on the drive - and have tried copying a couple 1GB files off of it onto my laptop.

Edited December 25, 2022 by MooTheKow

[trurl]

OK, do you have somewhere off the array you can copy that original disk1 data?

[MooTheKow]

The only drive I have big enough to hold it would be that 8TB former parity drive.  I have two external 4 TB drives I could copy things to though.

[trurl]

OK, since original disk1 mounts as Unassigned Device, let's just go straight to getting original disk1 back into the array, disable disk3 instead by unassigning it, and see if emulated disk3 mounts.

 

Follow these instructions very carefully. If you have questions or problems, let us know. Never tried this in the middle of parity swap so not entirely sure it will work exactly as described.

 

Go to Tools - New Config, Retain All, Apply

 

Make sure newly copied parity disk is still assigned as parity. Assign original disk1 as disk1, assign original parity as disk3, make sure all other drives are assigned as before.

 

DO NOT start the array yet.

 

Post a screenshot of Main - Array Devices and another screenshot of Main - Array Operation.

[MooTheKow]

Ok - this what you're looking for?    New 14TB drive that I copied parity to is assigned to parity.

 

Disk 1 (ending in VALG) -- the original disk 1 -- is assigned back as disk 1.

 

Disk 3 is the original parity.

 

 

[trurl]

OK, that looks correct.

 

In Main - Array Operation, check BOTH "Maintenance Mode" AND "Parity is already valid checkboxes", then Start the array. That will accept those assigned drives into the array without altering them in any way.

 

Then, Stop the array, Unassign disk3 (no device), Start the array in Normal (not Maintenance) mode.

 

That will disable/emulate disk3 so it can be rebuilt from the other disks. Probably emulated disk3 will not mount, but hopefully original disk1 will mount since it mounted OK as an Unassigned Device. We can try to deal with unmountable emulated disk3 before rebuilding.

 

Then post a screenshot of Main - Array Devices and new diagnostics.

[MooTheKow]

Well crud -- now it is showing disk 1 as unmountable.

 

 

kowunraid-diagnostics-20221225-1311.zip

[trurl]

Click on Disk3 to get to its page, set its filesystem to xfs, then check filesystem on emulated disk3. Be sure to capture the output so you can post it.

[trurl]

What is that other Unassigned Device sdc?