Weird issues with disks showing invalid and unmountable

[DPKOZI]

System configuration:

Supermicro Motherboard MBD-X10DRL-I-B LGA2011 E5-2600v3 C612 DDR4 PCI-Express SATA ATX

 

Samsung M393A4K40BB0-CPB 32GB DDR4-2133 Memory MEM-DR432L-SL01-ER21.

(256GB Total)

 

Intel Processor Xeon LGA2011-3 2.60G 25M Proc E5-2660V3 10C DDR4 Up to 2133MHZ BX80644E52660V3

(Double CPU configuration)

 

Seagate Exos X18 18TB Enterprise HDD - CMR 3.5 Inch Hyperscale SATA 6Gb/s, 7200 RPM, 512e and 4Kn FastFormat, Low Latency with Enhanced Caching (ST18000NM000J)

(8 total drives all matching in one array)

 

MZHOU PCIe SATA Controller Card 10 Port with 10 SATA Cables and Low Profile Bracket - 6Gbps SATA 3.0 PCIe Card,Support 10 Port SATA 3.0 Devices

(All array drives connected to this card)

 

SAMSUNG 980 SSD 1TB PCle 3.0x4, NVMe M.2 2280, Internal Solid State Drive

(2 of these drives on pciE adapter card, only used for appdata and system cache)

 

SAMSUNG 870 EVO SATA III SSD 1TB 2.5” Internal Solid State Drive

 (Array drive caching only)

 

NVIDIA VCQP2000-PB Quadro P2000

(Plex encoding)

 

Thermaltake Toughpower PF1 850W 80+ Platinum Single Side SMD Compact Design

 

Ok with all that out of the way here is the issue we're having lately. The server had been running flawlessly for over a year serving our Plex users, and we ran out of storage with 6 drives. We upgraded the Sata controller and added 2 more additional drives to the system and immediately issues started happening slowly over time. We started noticing a few CRC errors on drives, replaced those and thought all was well. Today we noticed that 2 drives showed unmountable and bad filesystem, so we decided to change Sata cables to all drives, which did nothing. We followed the wiki to rebuild the 2 dropped drives after getting them to report they were being emulated, which is running right now. 

 

I'm going to attach our diagnostic files, we're just wondering if maybe it's something we are doing wrong, or maybe a hardware mismatch issue, or anything to explain why all the sudden we had the drives drop out after changing Sata controller and cables. All drives actually report healthy right but we just want to cover our bases and get ahead of any issues that may arise in the near future. 

 

We are also having issues with mover starting, cache drives filling up and having to manually run mover, as well as a starting services error on dashboard.

 

Any help would be greatly appreciated!!blackwidow-diagnostics-20230207-1548.zip

 

 

 

 

 

 

 

[trurl]

If you are doing the rebuild in Maintenance mode, no drives are mounted so can't tell anything about filesystems or shares.

 

Post new diagnostics after starting the array in Normal mode.

[DPKOZI]

33 minutes ago, trurl said:

If you are doing the rebuild in Maintenance mode, no drives are mounted so can't tell anything about filesystems or shares.

 

Post new diagnostics after starting the array in Normal mode.

blackwidow-diagnostics-20230207-1722.zip

 

Done with array started normally 

[JorgeB]

Check filesystem on disks 3 and 4.

[trurl]

Unrelated, but your system share has files on the array, probably you enabled Docker and/or VM Manager when you didn't have the specified pool so some of those files got created on the array.

[JonathanM]

18 hours ago, DPKOZI said:

MZHOU PCIe SATA Controller Card 10 Port

That card has a port multiplier chip, it's likely the issue. Probably can't handle the load.