I have 16 drives installed onto a LSI card onto the PCIEX 5 slot. I also have an intel ARC plugged into the other slot although it was not being used at the time of this incident.

 

I tried to replace my server motherboard/cpu/ram and it worked for about 24 hours without issue. As part of the upgrade, I added three nvme drives but didn't do anything with them.

 

24 hours after the upgrade, I created a pool from the drives and added the third to another pool.

 

Within minutes, 2 of my drives showed a corrupted filesystem. I was unable to do xfs repair so I rebooted to investigate and started a rebuild. One of my parity drives failed almost immediately during the rebuild. I tried swapping cables on parity 1 and I removed the nvme drives but parity 1 still continued to show read/write errors and wouldn't mount.

 

So I swapped out the hardware and put my old motherboard back. I told the array to accept my parity drive as valid and emulated the failed disk. I was at least able to successfully run xfs_repair on the emulated disks, which of course fixed the issue on parity.

 

I tried a rebuild and it made it 78% of the way before parity 1 failed so at this point i've written off the P1 drive + 2 data disks.

 

Is there any correlation between nvme drives and sas cards? The motherboard was being used by my desktop without issues for 2 years. This is the board: https://www.asus.com/us/motherboards-components/motherboards/tuf-gaming/tuf-gaming-z690-plus-wifi-d4/

From my googling, it appears that nvme might cause SATA ports to become disabled on the motherboard but it shouldn't cause any issues to my SAS card drives, other than perhaps reduced bandwidth on the interface? But that shouldn't corrupt filesystems should it?

Or did I just get really unlucky with a triple drive failure? The two failed data disks were on the same split HBA cable but the parity was on a different one.

 

I'm asking because im unsure of if I should try to put the new motherboard back in. My current hardware is really old and i'd like to be rid of it. But I don't want to risk losing more data drives.

The entire journey is chronicled here with diags below.  Unfortunately I don't have the diags from when the file system became corrupt, so im not i'll ever know what happened with those.  

 

Unfor

 

 

Wild guesses:

* Cables not fully seated or perhaps even damaged during the swap

* Power supply pushed over the edge, possibly now unstable

 

If nvme/pcie share lanes, devices simply won't show up.

 

I've never seen it myself but esd damage is technically a possibility also.

19 minutes ago, _cjd_ said:

Wild guesses:

* Cables not fully seated or perhaps even damaged during the swap

* Power supply pushed over the edge, possibly now unstable

 

If nvme/pcie share lanes, devices simply won't show up.

 

I've never seen it myself but esd damage is technically a possibility also.

 

i think it's one of the two mentioned.  im continuing to troubleshoot the issue and still having strange drive dropouts on the old hardware.  it's still the same 3 drives but im leaning towards power supply cable or sas card gone bad.  replacing both to see what happens.  

 

the part i just dont get is why it worked for a full day no problems and then a spike

Edited November 21, 20241 yr by oliver

If the power supply is the cause it would be surge related, and possibly hot surge vs cold (assuming all drives spin up at boot). It may get worse. How old is it, and what wattage? Did you change anything in how drive power is delivered? (Different power cable arrangement?)

Drives, I'd explore a couple at a time SATA to motherboard direct, if possible. Just see if they show up, maybe self test. And maybe on a temp os (temp unraid trial?).

Card failure seems less likely but not impossible.

I lost a couple HDD (fortunately retired 500gb, I was testing some stuff planning a backup unraid server) not long ago to a perfectly functional power supply - 3 SSDs were fine, but 3 spinners (even two) and only one would work - and the initial failure toasted two drives. It was an old Corsair PSU. Caps and solder joints and who knows what do age out and need attention - in this case just a new PSU.

1 hour ago, _cjd_ said:

If the power supply is the cause it would be surge related, and possibly hot surge vs cold (assuming all drives spin up at boot). It may get worse. How old is it, and what wattage? Did you change anything in how drive power is delivered? (Different power cable arrangement?)

Drives, I'd explore a couple at a time SATA to motherboard direct, if possible. Just see if they show up, maybe self test. And maybe on a temp os (temp unraid trial?).

Card failure seems less likely but not impossible.

I lost a couple HDD (fortunately retired 500gb, I was testing some stuff planning a backup unraid server) not long ago to a perfectly functional power supply - 3 SSDs were fine, but 3 spinners (even two) and only one would work - and the initial failure toasted two drives. It was an old Corsair PSU. Caps and solder joints and who knows what do age out and need attention - in this case just a new PSU.

 

unfortunately im not able to locate where i bought it, but it can't be more than 2 years old.  the wattage is 750w and it's corsair.  

 

i replaced the PS with a spare and the HBA card as well and am trying another rebuild.  the old SAS card is quite a bit slower than my new one, it's going to be a long 3 day wait to see what happens.