adammerkley

This is wild. Finally looked in my OPNsense logs and can see that Crowdsec was blocking all traffic from my server:



Removed that decision, and immediately everything on Unraid started working again. Crowdsec shows it was a decision based on firewallservices/pf-scan-multi_ports from 4 hours ago, which is how long I've been scratching my head over this.

In the reboot to install 7.1.3 I experienced the issue again, so I looked into it further and I believe it had to do with ZFS dataset vs a directory on ZFS. I ran mover to clear everything out of the Data share on my cache, then stopped all dockers and deleted the Data share from cache entirely. Then I rebooted again, and the Data share reappeared, with old files in it that had been moved weeks ago. After verifying that those files had in fact been moved to the array, I deleted the Data share from cache again, rebooted, and this time there was no Data share upon reboot. I then wrote a file to /mnt/user/data and confirmed that it was created on the cache drive.

It's been several days, and I've run mover and rebooted a few more times to test, and everything is working as expected. Oh, and I have SABnzbd writing to /mnt/user/data instead of /mnt/cache/data.

I replaced the sketchy 8087 to 8643 cables with genuine ones from Art of Server. 
 
I also realized that I was essentially running all 11 HDs off of a single PSU cable, since I had just one cable going from the PSU with each of its 3 headers plugged into the backplanes.  Now I have a dedicated cable coming from the PSU for each backplane.
 
Finally, I reconnected the intake fans and both of my HBAs have a small fan mounted to them.
 
It's been 17 days of uptime with no errors, so I'm marking this solved.

Replaced the failed Sabrent Rocket 4.0 nvme drive with a Samsung 980 Pro drive.  Thanks to zfs the procedure went without a hitch.

I replaced all of the SAS-SATA cables for drives connected to my HBA, and the SATA cables for drives connected to the motherboard.  I also removed the molex splitters I had deployed to deliver power to the backplane/drive cages.
 
Just completed a parity check with no errors.