HDD Failure, HBA Failure, or Both?

[natecook]

I've been rolling along for years with 2 cache, 2 parity, and 22 data disks. The data and parity disks run through two SASLP and one SAS2LP cards, and the cache directly connects to the mobo SATA ports. About 1/3 times, during a monthly parity check, one finicky SASLP (the one with data disks 1-8) would start spitting back errors for all 8 of its drives. I assumed it was just overheating, as when I added better ventilation, the issue became very sporadic.

 

On Monday, disk 5 on that SASLP was disabled. I/O errors, old drive, simple. I swapped it out, and felt great as started the rebuild. 5 hours later, the SASLP drops offline. As is tradition, I shut down and try again - still feeling fine. Another 4-5 hours and the SASLP is offline again. Now, when I reboot, disks 2 and 5 both show up as Unmountable. I try adding more fans, hoping to just get the data fixed before I start swapping hardware, and now the card fails after just 5 minutes. I ordered a SAS2LP to replace it, thinking all of my issues would be resolved without that bad SASLP.

 

Now, with the brand new SAS2LP installed, the parity sync runs at ~2 MB/s before it eventually fails. Looking at the logs, disk 8 is now throwing tons of I/O errors. FWIW, Disk 8 is from the same batch of Seagate 4TB hard drives as the failed disk 5, but disk 6 is too and it's fine.

 

What are my next steps here? I feel like I need to buy better HBA cards and trash some drives, but I don't know where to begin.

pangu-diagnostics-20200221-1859.zip

[Frank1940]

Start with this post and read the the rest of the thread:

 

    https://forums.unraid.net/topic/69018-sata-controller-replacement-question-and-advice/?tab=comments#comment-630097

 

[natecook]

Thank you @Frank1940! I have ordered three 9300-8i (three available PCI 3.0 x8 slots, so no problem there) and the corresponding SFF-8643 cables. That should all arrive tomorrow, and I'll report back after I get it all up and running.

[natecook]

All LSI cards installed. I noticed immediately that my parity sync speeds were much improved - 125-150 MB/s on the LSI controllers vs 80-100 MB/s with the SASLP controllers. Unfortunately, I am still experiencing these read errors issues on drives that are seemingly good.

 

The array's status is:

• disk 5 - disabled and unmountable (this disk was freshly installed and has not been rebuilt)
• disk 2 - unprotected and unmountable (this drive was fine until the rebuild issues began)
• all other disks - protected and mounted

However, when attempting a rebuild, I get random errors on a disk. Sometimes it's disk 4, sometimes 7, sometimes 8. I've tried swapping the cables and controllers around. The only consistent factor is that the read errors pop up on one of disks 1-8, which leads me to believe that the failing SASLP left some damage on those disks. What am I to do next?

pangu-diagnostics-20200223-1721.zip

[JorgeB]

These look like a connection/power issue, see if those disks share a power cable/splitter, or if possible try with a different PSU.

[natecook]

@johnnie.black that makes so much sense. I was so caught up in my AOC vs LSI controller mixup that I didn’t think about power. I tried a few repair options, but it appears that this whole issue was tied to a bad power cable from my PSU. With a new PSU SATA cable, it’s been running a sync for over 6 hours without issue. Absolutely wild! I would have never thought to check a power cable.

 

What did you see that made you think it was power?

[JorgeB]

Feb 23 17:14:00 Pangu kernel: sd 9:0:0:0: Power-on or device reset occurred
...
Feb 23 17:15:37 Pangu kernel: sd 9:0:3:0: Power-on or device reset occurred

These errors suggest a connection/power issue, and since you had already replaced/swapped the SATA cables power was the more likely culprit.
 

[natecook]

All done and stable.

@Frank1940, thank you for the LSI controller recommendation. My 24TB array with 10TB dual parity now takes ~24 hours to sync parity, and does so with no errors. This is down from ~34 hours and an expectation of 5-10 errors.

@johnnie.black, you saved my life by catching the power issue.

 

I am eternally grateful to you both!