supacupa

Agreed, it doesn't make sense. It's not the drives since this problem shows itself on other SAS drives. It's also not likely to be the controller as the problems showed itself on two separate controllers and I've not seen others report this problem on either of them. After a lot of testing and changing things out it's clear the problem is following the server and the UNRAID install. The server has not changed, but the UNRAID install has, so that is the most likely culprit. This same drive and controller setup works find on my EndeavourOS PC which is currently using 6.10.4-zen2-1-zen kernel, and my Debian server which is currently using the 6.9.7+bpo-amd64 kernel. If the 6.8.12-Unraid kernel is to blame there's not much I can do about that other than rollback and stick with slowly aging software with the hope some future update fixes it.
 
I do firmware development so I'm familiar with figuring this kind of problem on bare metal, but I don't really know where to start with Linux unless I can find information online on what component to debug, something I've had some trouble with doing on UNRAID. If something isn't doing what it should be doing it should be possible to trace it. Any ideas on where I can start? Any advice will be appreciated.
 
I need to have somewhat reliable speed which I did have before all this started. It was working perfectly fine on the previous UNRAID version and had been for many years. I was using these exact same drives with this exact same controller about three years ago. I switched to the older LSI card since I had an expander and could run more HDDs, but as I no longer have a need for hundreds of TBs of storage I simplified it to 4 primary 12 TB disks using the rest as cold storage. I primarily use UNRAID for VMs and Dockers. The UNRAID method of data storage and backup was nice back when I had an assortment of disks, but now that I have dozens of the same types of disks with many spares, it may not be the best method of data storage for me now, at least not when its this inconsistent.
 
My current options seem to be as follows:
1. Find and solve the problem so I can continue using UNRAID as I do. That may involve rolling back to an older UNRAID version, which may mean I no longer can update UNRAID which may or may not be a problem.
2. Perhaps attempt to pass the HBA through to a VM and use RAIDZ pools for data storage and let UNRAID focus on only VMs and Dockers. This is a somewhat silly option, but I could see it working. Probably not worth the headache, though.
3. Install another OS that works correctly and run Dockers and VMs the old fashioned way. When I started using UNRAID something like 4 or 5 years ago I didn't have the time to do that, but I've now have other servers I manage and have figured out better ways to do so, so this might be the best option. I've already spent way too much time on this problem and am worried more compatibility issues may crop up in future UNRAID builds.