Hoping someone can point me in the right direction here.  What is the best way to get back up and running after two drives have presumably failed?  I'm assuming I will lose some data, but just don't know the best process to lose the "least" amount.   

 

Here is the scenario...

- Disk two (2) took itself out of the array (Disabled but emulated).  Not sure exactly why.

- I ordered a new drive to replace Disk 2

- Disk four (4) was then showing errors and eventually disabled itself - (before I was able to get the replacement disk.)

- The array is now offline

- I replaced all the SATA cables with brand new cables - no change in any state (prob too late)

- Disk 2 "appear" healthy

- Disk 4 isn't showing up anymore after several reboots

 

It appears that Disk 2 maybe be OK, but since it pulled itself from the array, I can't use it to rebuild data on Disk 4 which is completely MIA.

 

Thanks in advance!

 

square-diagnostics-20241215-2039.zip

5 minutes ago, shfaxx said:

replaced all the SATA cables

Are you sure all disks are getting good power?

 

6 minutes ago, shfaxx said:

Disk four (4) was then showing errors

What sort of errors? Do you have any SMART report or other diagnostics from when that was happening?

Power - I "assume" so, but you know what that does.   The power connector is split and drive 4 is the first in the chain.   I will try and another cable just to verify it's not this and report back.

 

Logs - I don't have any other logs other than what I posted, unless they are hiding somewhere.

 

** UPDATE **

- I swapped power connectors between two pairs of disks and there was no change.

- I swapped the SATA port between Disk 4 and my Cache drive and I now see ALL my drives and the array started.    

   -  Disk 2 is still "disabled" in and emulated state.

 

Ugh, so now how do I determine what the actual problem is?    I'm attaching a new diagnostic, hopefully that will help?

 

** UPDATE 2 **

- Disk 4 does appear to be showing some errors.    How can I "quickly" get disk 2 back into the array and then replace disk 4?

 

square-diagnostics-20241215-2151.zip

Edited December 16, 20241 yr by shfaxx

@trurl, I stopped the array for the time being, as all my dockers started to fire up and pound on the drives.   

 

I'm "assuming" i should try and rebuild Disk 2 over itself?  That would get it back into the array and then physically replace disk 4 with the spare I have???

 

Just would like some confirmation that this is the correct "safest" route forward:

 - rebuilding ontop proces...  https://docs.unraid.net/unraid-os/manual/storage-management/#rebuilding-a-drive-onto-itself

 

tnx

Edited December 16, 20241 yr by shfaxx

Disk4 has pending sectors, run an extended SMART test.

@JorgeB - Ran an extended test, complete with "Errors occurred - Check SMART report", log attached.

 

ST12000NM0007-2A1101_ZJV1CZ3P-20241216-0612.txt

That's bad news, did you try replacing or swapping cables for disk4 or is it really dead?

@JorgeB all SATA cables were replaced with brand new cables for all drives.   I have a new drive here to replace it, but since Disk 2 is disabled I can't rebuild disk 4.    I'm able to mount Disk 4 and read from it, so it's functional to some extent.   It's def making some not so normal noises...  

11 hours ago, shfaxx said:

rebuild Disk 2 over itself?

Don't rebuild disk2 over itself when disk4 is showing real problems. Maybe try rebuilding it to a new disk and keep the original as is.

 

 

See if you can recover the data from the old disk4, and also old disk2, there are some pending sectors, but possibly most data can be copied, another option is to clone it with ddrescue.

@JorgeB / @trurl -  Since Disk 4 is the one with real reported physical issues, I'm hesitant to beat that drive and try to clone it.    So, should I attempt to clone Disk 2 to a fresh disk and concentrate on getting that back into the Array? 

 

OR, should I try and get disk 4 cloned since it appear to have more physical issues? 

 

For either, can I just clone to a new drive using ddrescue?    I see the docs on ddrescue, and I can surely give this a go:  https://docs.unraid.net/unraid-os/troubleshooting/data-recovery/.   Assuming the clone works, do I simply swap the drive?  Will UNRAID be happy to see some new drive for the disk?  If I do this for disk 2, will it bring it back online and then allow me to replace / rebuild disk 4 afterward?

 

thanks!

Before attempting to clone it I would just try to copy the data from the disks somewhere else, if it fails you can still try ddrescue.

@JorgeB, if I'm able to get the data off the drive(s), via basic copy, how would I then get that data / drive(s) integrated back into the array?

 

My other issue is, it's not a "little" data...

Edited December 16, 20241 yr by shfaxx

1 hour ago, shfaxx said:

how would I then get that data / drive(s) integrated back into the array?

Tools - New Config - then resync party 

@JorgeB, Ahh ok cool.  I will purchase another drive and give this a go on Disk 2 in the mean time.

 

Any suggestions on how to copy the data?   Wouldn't I just want to use ddrescue or is there something better to do a data copy?

 

tnx

You can just mount the disks with UD plugin and use your favorite tool.

@JorgeB / @trurl, would love your opinion on this....

 

I'm thinking if I can successfully clone Disk 2 with ddrescue to my new drive, then re-insert the "new" cloned Disk 2 back into the array, that would bring the array back online.   Then, replace disk 4 with another new drive and rebuild disk 4 from parity.

 

Is this a viable option to proceed with, or are there issues in the logs which would indicate otherwise?

 

thanks!

Since physical disk2 was disabled first, it is out-of-sync with the rest of the array. So, physical disk2 (or its clone) can't be counted on to rebuild anything.

 

You can New Config with clones of both disks and hope for the best.

 

Do you have another copy of anything important and irreplaceable?

@trurl, Ahhh, ok yeah, that's kinda of what I was afraid of for Disk 2 - It was pulled out of the array a short time before I realized..  ugh.     Will start copies for both drives and keep my fingers crossed.      As for copies, yes for super critical stuff, I have on another external drive and in the cloud. 

 

Just wasn't sure of the cleanest way to get back up online with the least amount of loss.   Just going to be a pain to find what was missing, and a long process to try and copy these disks....   

 

What's the recommended copy tool folks are using?   I'm no unix guru, I see MC (midnight commander), anything else easier / better?

 

thanks!

Dynamix File Manager plugin 

@trurl, sigh...  looks like I need to upgrade unraid to >6.10.x.   Any risk of doing that in the midst of all my other issues.

MC is OK. I used it on Unraid for years. 

@trurl sorry to keep pestering.   

 

While I wait for my new drives..   if I have space on the other drives in my array (1 and 3), I assume I can just attempt to copy data from the two failed disks (2 and 4) to the other good drives, correct?   

 

I also assume I can just mount and copy from the root of one disk to the root of another and it will merge the data / folders...    Then when everything is copied off and I add the new drives, I'll re-do the parity and can re-balance the data, etc.  correct?

 

tnx

According to those diagnostics you posted, disk2 is disabled. It will not be used when you try to read data from it. Instead, its contents will be emulated by reading all other disks in the array and getting its data from the parity calculation. So, reading disk2 will actually involve reading disk4 and all other disks.

 

Might be simpler to just make a new array with the good disks and the new disks, and then try to copy whatever you can from the original disks 2, 4 by mounting them as Unassigned Devices.