Single Parity, Dual Disk failures, Rebuild Possible?

[bdcmrt07]

Hi all, a couple of weeks ago I started hearing clicking sounds from my server but in hindsight didn't react quickly enough. When i logged into my tower I had a flood of disk error notifications  Specifically, Disk 3 was disabled and contents emulated. When i look at the S.M.A.R.T summary it shows "FAILING NOW" and indicates that bad sectors have been reallocated for this disk3. I thought the clicking sound was this "disk3" hard drive.  A couple of days ago i login again now to see my logs flooded with errors for Disk 4. My disks were all spun down (disk 3 still disabled but contents emulated), and a red x next to disk 3 and 4. The logs shows disk 4 errors but i cannot pull any S.M.A.R.T information or acces the disk at all. I stopped the array and shut down the server to look at the hardware.  When i open up my tower i confirmed the clicking is actually coming from disk 4. This disk4 i have confirmed now is dead. No matter what i do Unraid will not recognize this disk 4 and will not allow me to start the array. It just says "missing" next to disk 4. It doesn't appear in the drop down menu for me to select even. In terms of what caused both drives to fail i will assume it's a power surge issue as i am not connected to a power surge protector and we have had outages this summer. 

 

So now i would love your help in how i can move forward. I have 1 parity drive and 5 data drives (now only 3 good ones apparently). How can i replace these disks and rebuild the array. Is the data on Disk 3 (disabled but contents emulated) still recoverable? I've cut my losses for disk4. My thinking is that I replace the dead drive (disk 4) and see if i can rebuild the array with what's remaining on the other disks. I am scared though as disk 3 is already disabled with bad sectors so i'm not even sure if it will let me. I feel like i've lost 2 drives worth of data here. Would really appreciate your help and guidance on how to proceed with this!

 

Current Disk Setup attached for your consideration. 

 

[trurl]

Attach Diagnostics to your NEXT post in this thread 

[bdcmrt07]

Before i shut down the system i was able to capture the attached System diagnostics and SMART log. The S.M.A.R.T report is for DISK3 which is currently "disabled/contents emulated".. I was not able to get anything for disk4. Thanks!

tower-diagnostics-20220928-1952.zip tower-smart-20220926-1319.zip

[JorgeB]

IMHO your best bet is to use ddrescue on disk3, use a disk with the same size for the new disk and if the recovery is fairly successful (like 90%+ recovered) we can than make that the existing disk3 to rebuild disk4, of course there would likely still be some data loss, mostly depending on the ddrescue results.

[bdcmrt07]

Thanks a lot for the reply, it's much appreciated. Do you mind explaining how this will work "we can than make that the existing disk3 to rebuild disk4" ? 

[JorgeB]

Like this:

https://forums.unraid.net/topic/128492-multiple-disabled-disks/?do=findComment&comment=1170517

 

[bdcmrt07]

Okay, I've read that post and trying to lay out the steps. Please excuse my ignorance as I don't okay around with unraid frequently.  I picked up a replacement disk for disk 3 and I've got ddrescue running now. Once that is done that essentially becomes the new disk3 which had the bad sectors. So I'd now shut down the system. Remove the old disk3, do a new configuration, assign this new replacement disk as disk 3 and rebuild the array? That essentially salvages whatever I had on disk 3. So because parity is still fine the thinking is that it would be able to rebuild the contents of the dead disk 4? Would I need to add another replacement disk for disk4 or will it rebuild from parity with the existing disks? 

[JorgeB]

22 minutes ago, bdcmrt07 said:

Would I need to add another replacement disk for disk4 or will it rebuild from parity with the existing disks? 

You need a replacement for disk4 since apparently it's dead, same size or larger than the old disk.

 

Once ddrescue finishes post the results and I can post the detailed instructions for your case.

[bdcmrt07]

I can't tell if this process is still going....i entered this command "ddrescue -f /dev/sdh /dev/sdc /boot/ddrescue.log" and left it to run but looks like it either completed or timed out when my pc went to sleep...i issued this command again in the terminal and it started..i went away and came back and after a cpl hours and the terminal is blank just back at root...it does look like it wrote something as i see writes to the disk...is there any way to tell if this finished? Should i issue the command again (I'll change my laptop to not sleep, i don't know why that would cause the terminal to end though) ? Thank you kindly for your time and assistance. 

tower-diagnostics-20220930-1634.zip

[JorgeB]

Run it again, if it didn't finish it will resume, if it did it will show the stats, also you should use the actual console or something like screen to not close the terminal window, it will stop if it does.

[trurl]

On 9/30/2022 at 1:12 PM, bdcmrt07 said:

read that post and trying to lay out the steps

The rest of your post didn't have all the steps so be sure to ask for more help before attempting anything.

[bdcmrt07]

I'll definitely need help with the step by step after the rescue is done. This is taking incredibly long... It went up 3% overnight. Normal? 

Edited October 2, 2022 by bdcmrt07

[JorgeB]

2 hours ago, bdcmrt07 said:

. It went up 3% overnight. Normal?

Depends on how bad the disk is, and that one doesn't look very good.

[bdcmrt07]

Still chugging along...58.96% rescued so far...guess it will take another couple of days.

[bdcmrt07]

Barely moved overnight and woke up to these warning notifications in attached...

[JorgeB]

Disk appears to be in very bad shape, ddrescue recovery below high 90s percentile will make recovering that disk and the other one very difficult.

[bdcmrt07]

Okay i've got a question here...This is at 71.33% now...but i was looking at processes in Unraid and i noticed there are 2 of these processes running.....I've only got 1 terminal open unless when i used screen the other one kept running in the background??? I can see different Start dates for both processes...did i mess up here and that's why this is taking so long? However with that said...it's been going faster the last couple of hours...up 10% in one hour and i haven't touched anything....

 

Edited October 5, 2022 by bdcmrt07

[JorgeB]

7 hours ago, bdcmrt07 said:

did i mess up here and that's why this is taking so long?

Maybe, you can cancel the current run and see if both processes stop, if yes just resume, if not kill the other one then resume.

 

 

[bdcmrt07]

Process has completed...please see attached. (Both instances are gone from processes now). Just to recap where we're at now....

-Single Parity

-Both Disk3 and Disk4 are listed as disabled, contents emulated. 

- Disk4 is "missing" because i physically removed it. This is the one that was clicking and i couldn't get UNRAID to recognize it.

-Bought a replacement disk (sdc) and ran ddrescue to rescue contents from disk3. DDrescue process now finished. 

 

Please assist me with step by step instructions on how to proceed further! Many thanks for the attention and assistance!

[JorgeB]

99.99% is good, do you also already have a replacement for disk4?

[bdcmrt07]

I can get one easily. You mentioned it could be larger than the original right? Disk4 was 6tB but i'd love to get a 10/12TB if the process will allow it. 

[JorgeB]

12 minutes ago, bdcmrt07 said:

You mentioned it could be larger than the original right?

Yes same size or larger, initially and to get the emulated disk you can even use a known bad disk in that slot, as long as it's detected, still needs to be same size or larger, but then you need a good one to rebuild.

[bdcmrt07]

Oh wait...but if my Parity drive is only 6TB i think that's the max size i can go right? I'm not allowed to have a data disk larger than parity?

[Kilrah]

Correct

[bdcmrt07]

Okay I've got the replacement drive for disk4. Please let me know how to proceed step by step. Cheers