Double Trouble? Two drives failed - advise and assistance required

[alinkognito]

Hello all.

Once again, I come to you for help and guidance.

 

During a drive (disk 2) rebuild Disk 9 is in a loop of what sounds (litarally) it trying to start up.

I stop the rebuild as it was saying it was going to take over 1000 days to complete and rebooted.

Now it is showing unmountable 2 x disk but which are available to format (I won't be doing this as in the past I formatted in error and lost all the data on that drive.)

I have 2 x new drives arriving tomorrow and want guidance on how to proceed with hopefully no data loss.

ie can I just add the 2 x new drives respectively and Unraid will re-build accordingly or is there more to it?

 

I look forward to your guidance.

 

Many thanks.

Al

 

PS: if you notice anything else in my diagnostic that I should look into, please let me know too.

kraken-diagnostics-20220811-1800.zip

Edited August 11, 2022 by alinkognito

[JorgeB]

Power down, check/replace cables on disk9, power back up and post new diags after array start.

[trurl]

29 minutes ago, alinkognito said:

can I just add the 2 x new drives respectively and Unraid will re-build accordingly or is there more to it?

Since you have single parity (you should consider dual parity with so many disks) there is no way to rebuild 2 disks at once. And rebuild won't fix unmountable anyway. And only one disk is disabled and needs rebuilding.

[alinkognito]

Done.

Took forever to mount the disks

kraken-diagnostics-20220811-1957.zip

[JorgeB]

Still some issues with disk9, but it mounted this time, check filesystem on the emulated disk2.

[alinkognito]

Thanks for your help.

I will do a check filessystem when I get home later today.

(Might need some confirmation on the correct command, but will reach out when in front of the server)

 

In the meanwhile, I can see the files, but disk9 is not sounding very healthy. Almost a ticking sound every 10 seconds or so.

This disk9 seems to be the culprit when disk2 was rebuilding and showing over 1000+ days to rebuild.

 

[JorgeB]

5 minutes ago, alinkognito said:

but disk9 is not sounding very healthy. Almost a ticking sound every 10 seconds or so.

Doesn't sound good, did you replace the power cable also?

[trurl]

2 hours ago, alinkognito said:

I will do a check filessystem when I get home later today.

(Might need some confirmation on the correct command, but will reach out when in front of the server)

Check filesystem, click the link.

 

Better if you don't try to do it at the command line. The webUI knows the correct command, just click on the disk to get to its page and use the Check button.

 

Be sure to capture the output so you can post it.

[alinkognito]

Cable replaced.

Rebooted

Disk9 not showing up in array, but in unassigned drives its come up as "35000c500ae408f73" and no longer as "ST16000NM002G_ZL20B2RR0000C943E5AA_35000c500ae408f73 - 16.0 TB"

 

 

Alas, was not able to do a check filesystem as unraid is now saying "Stopped. Invalid configuration."

 

PS: 2 x new 18TB drives available (was planning to use one as my 2nd parity and the other to replace disk2). Now holding tight (and slightly panicy) until further advise.

 

Hopefully something you can suggest to assist.

 

Thanks again,

 

[JorgeB]

New diags please.

[alinkognito]

kraken-diagnostics-20220812-1807.zip

[JorgeB]

Disk9 is failing:

 

=== START OF READ SMART DATA SECTION ===
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED

 

That leaves you in a bad spot, is the currently assigned disk2 a spare or the old disk2? It looks healthy, if it's the old one you could re-enable it, would likely only lose any data written to it after it got disabled, if there was any.

 

A for disk9 you can try ddrescue to recover as much as possible.

[alinkognito]

just rebooted and it's showed up

managed to get the array started in maintenance mode.

Shall I still run the check filesystem on disk9 (which is not sounding very well btw)?

[JorgeB]

P.S. for such a large array would definitely recommend dual parity, unless you have good backups.

[JorgeB]

Just now, alinkognito said:

Shall I still run the check filesystem

check filesystem was for disk2, or is disk9 not mounting again? But since we've now confirmed that disk9 is failing fixing the emulated disk might not be possible.

[alinkognito]

2 minutes ago, JorgeB said:

Disk9 is failing:

 

=== START OF READ SMART DATA SECTION ===
SMART Health Status: FAILURE PREDICTION THRESHOLD EXCEEDED

 

That leaves you in a bad spot, is the currently assigned disk2 a spare or the old disk2? It looks healthy, if it's the old one you could re-enable it, would likely only lose any data written to it after it got disabled, if there was any.

 

A for disk9 you can try ddrescue to recover as much as possible.

 

  

disk2 is the old disk which was being re-build. stopped at 89% when unraid started saying 1000+ days to complete

 

[JorgeB]

Just now, alinkognito said:

disk2 is the old disk which was being re-build. stopped at 89%

So it should still be mostly good.

[alinkognito]

How shall I proceed?

 

1) add one of the new disks on the server (let;s call it diskA)

2) ddrescue Disk9 so it recover/clone as much data to the new diskA

2) replace Disk9 with the cloned disk (diskA)

3) then replace Disk2 with another new disk (diskB) then re-build?

 

Apologies if I sound like a laman. Never had this problem before and ddrescue is also new to me.

 

Thanks

Al

 

 

[JorgeB]

18 minutes ago, alinkognito said:

3) then replace Disk2 with another new disk (diskB) then re-build?

For disk2 it's probably best just to use the existing disk without rebuilding, but first check its state, unassign disk2 from the array, start the array, stop the array, disk2 will now be in the unassigned devices section, see if it mounts with the UD and contents look correct, UD also has a check filesystem option if needed.

[alinkognito]

cool

• array running in maintenance mode
• check filesystem run on Disk2
• "Phase 1 - find and verify superblock... bad primary superblock - bad CRC in superblock !!! attempting to find secondary superblock... .found candidate secondary superblock... verified secondary superblock... would write modified primary superblock Primary superblock would have been modified. Cannot proceed further in no_modify mode. Exiting now."

 

[alinkognito]

and as per your guide:

 

• stopped array
• disk2 set to no device
• Disk2 now showing un UD and mounted
• Contents of drive can be seen, but no option for a check filesystem in UD
• Ran xfs_repair -n /dev/sds in terminal

Phase 1 - find and verify superblock...

bad primary superblock - bad magic number !!!

attempting to find secondary superblock...

 

lots of . . . . . . . . . . . . . . .  still going

[itimpi]

That command is incorrect for doing it from the command line as it such a case it needs the partition.   However doing so will invalidate parity.   The suggestion at this point was to see if it mounts under UD - not necessarily repair it.

[JorgeB]

6 minutes ago, alinkognito said:

Contents of drive can be seen, but no option for a check filesystem in UD

There is but if the disk is mounting it's not needed.

[JorgeB]

See if you can recover data form disk9 with ddrescue, then you'd do a new config with all the disks (including old disk2) and the clone from disk9, if successful.

[alinkognito]

damn sorry - anxiety kicked in, my bad

terminal closed.

 

so to back track... I can see the files in the mounted drive in UD