[Solved] Dual drive failure?

[res2cou]

Good Afternoon

 

I done some work on my network the weekend and moved the server (very carefully), after getting everything fired back up everything was good and working.

Before bed i started a parity check (about 16 hours to complete) and woke up to a ton of emails reporting issues on 4 drives (8 in the array), these drives are all on the same controller (and the only drives on the controller).  These errors started with the parity drive (Time: 00:48), Disk 3 (Time: 00:50), Disk 2 (Time: 02:12) and Disk 1 (Time: 03:06).

 

My thoughts so far,

1 - cable issue

2 - drive failed causing the controller to "glitch"

3 - 4x failed drives

 

Post reboot i have one disk disabled by unraid (Disk 3) and another showing pre-failure smart issues (Parity) but the array is up.  I have attached the smart reports and diagnostics (pre-reboot).

I'm going to replace both drives, i have a spare which i'm going to replace drive 3 tonight and tomorrow i have a 10tb arriving to replace the parity.

 

Can anyone advise on the drives that are showing errors or as to what caused the issue?  Best course of action?

 

Thanks to anyone who can advise

ST8000DM004-2CX188_WCT00M8N-20200511-1157(Parity).txt ST4000DM000-2AE166_ZDH0JCWB-20200511-1327(Disk 3).txt tower-diagnostics-20200511-0822.zip

Edited May 16, 2020 by res2cou

[JorgeB]

Possibly a controller issue, though problem appeared to start with a disk first, in any case recommend updating LSI firmware to latest since it's on a very old one, and check all connections.

 

May 11 00:48:19 Tower kernel: mpt2sas_cm0: fault_state(0x4101)!
May 11 00:48:19 Tower kernel: mpt2sas_cm0: sending diag reset !!
May 11 00:48:20 Tower kernel: mpt2sas_cm0: diag reset: SUCCESS

 

[res2cou]

Thank you for your reply, I'll update it as soon as the array is rebuilt.

I have loaded in the cold spare 8tb drive and it's rebuilding now, about 24 hours at a guess by which time the new parity 10tb drive should have arrived and i can look at the firmware and preclear the new drive.

 

On a positive note the drive that has failed smart (Parity) looks to be within warranty.