3 bad disks at once?!?! Please help me re-activate disabled data drive known good.

[WebAddict]

24 total disks, 22 data + 2 parity
I've dealt with this multiple times, not my first rodeo... but now THREE disabled at the same time! I'm treading carefully.
My question is primarily: My data disk9 was disabled after read errors, HOW DO I RE-ENABLE this DISABLED DATA DISK without WIPING it? I know this to be a good disk, and tested successfully with short smart and I believe is a cable-related issue (new cable arriving tonight).

Sequence of events

1. 16tb parity2 started having read errors (probably bad cable), I stopped array to replace with an already precleared 16tb disk. I made assignment to new disk and started the array (without rebooting).
2. As soon as array came online, multiple additional disks (parity1 & disk9) started having read errors and were disabled.
3. I stopped the automatic Data-Rebuild. And have been keeping the array stopped until I figure out how to proceed.
4. I downloaded diagnostics (noting after reboot), and have attached them

 

I'm not touching the former disk9, leaving in tact. If I can't re-enable it, I will plan on bringing online a new empty disk9 and rsync old disk9's data. Then will rebuild parity from scratch (as both parity are disabled anyways...)

 

Thanks in advance for any of your feedback

weapon-diagnostics-20231110-1412.zip

[JorgeB]

Nov 10 13:55:31 Weapon kernel: md: import disk9: (sdo) ST16000NM001G-2KK103_ZL20SN9Q size: 15625879500 erased
Nov 10 13:55:31 Weapon kernel: md: import_slot: 9 replaced

 

You didn't mention replacing disk9, is the old disk9 still intact?

[WebAddict]

Yes @JorgeB the old disk9 is still in tact. Unraid won't let me start the array with old disk9 assigned as disk9 due to "Too many wrong and/or missing disks!". 
 

I shutdown the server last night to move the spare disks off the Motherboard SATA controller to the LSI controller card by swapping with the disks that were disabled. So I'm attaching the latest diagnostics.

 

Is my best option going forward starting a "new config" but resetting the old disk9 to a valid status?

weapon-diagnostics-20231111-1028.zip

[JorgeB]

17 hours ago, WebAddict said:

Is my best option going forward starting a "new config" but resetting the old disk9 to a valid status?

If the old disk is in good health, that's what I would do, new config with old disk9 and re-sync parity, if there are any more errors during the sync post new diags.

[WebAddict]

Thank you @JorgeBfor helping me navigate that. I ran an Extended SMART Disk Check on all three disks that were affected, as well as my two spare drives. 18 hours later, the one parity2 disk that started the chain of events is actually failing, but each of the other four passed smart tests. So I put the original disk9 back in storage rack, and updated disk assignments and ran the UnRaid > Tools > New Config, keeping all drive and pool assignments. It worked like a charm, I now have all 22 data disks running and zero parity drives on there. I am getting a shipment of 4 new drives today, but will need to test and zero them before installing them. Was worth making slow and deliberate decisions and I kept all of the data. I just need to remember to pull diagnostics before rebooting.