You can also try updating the LSI firmware, you're using a release I've never seen before (16.00.08), latest stable was 16.00.01, I see now there's 16.00.10.
That's on the slow side, is that writing to cache or array? Your cache device is likely slower than you expect, you also have a slow disk1, still try with turbo write enable and write directly to the array, it should be considerably faster.
Iperf tests the actual LAN bandwidth, you're unlikely to get significantly more speed during a single transfer than that result, it's usually the hardware used limiting that, NIC, cable, switch, even board/CPU combo.
On the diags posted libvirt started correctly at first array start:
Mar 14 10:01:57 TOWER emhttpd: shcmd (463): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 2
Mar 14 10:01:57 TOWER kernel: BTRFS: device fsid 473e1a30-4df5-4093-9659-11deafd49e06 devid 1 transid 1987 /dev/loop3
Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): disk space caching is enabled
Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): has skinny extents
Mar 14 10:01:57 TOWER root: Resize '/etc/libvirt' of 'max'
Mar 14 10:01:57 TOWER kernel: BTRFS info (device loop3): new size for /dev/loop3 is 2147483648
Mar 14 10:01:57 TOWER emhttpd: shcmd (465): /etc/rc.d/rc.libvirt start
Mar 14 10:01:57 TOWER root: Starting virtlockd...
Mar 14 10:01:57 TOWER root: Starting virtlogd...
Mar 14 10:01:57 TOWER root: Starting libvirtd...
Then for some reason it tried to start again a few minutes later:
Mar 14 10:18:28 TOWER emhttpd: shcmd (552): /usr/local/sbin/mount_image '/mnt/user/system/libvirt/libvirt.img' /etc/libvirt 2
Mar 14 10:18:28 TOWER root: /mnt/user/system/libvirt/libvirt.img is in-use, cannot mount
Mar 14 10:18:28 TOWER emhttpd: shcmd (552): exit status: 1
Try rebooting in safe mode and post new diags.
There are some options here that might help with data recovery:
https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=543490
For the dockers you only need the appdata, the docker image can be easily recreated.
You could try that to at least get the array stable.
If you need VM hardware pass-trough and those errors keep happening, and like mentioned it's not that unusual with AMD boards, you might need a new board, bios update/newer kernel might also help, also note that I believe that on at least one case the user was still having issues even with IOMMU disable.
Problem with the onboard SATA controller:
Mar 14 02:52:18 Unraid kernel: ahci 0000:01:00.1: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x0000100000000000 flags=0x0010]
Mar 14 02:52:18 Unraid kernel: ahci 0000:01:00.1: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x0000100000000880 flags=0x0010]
Mar 14 02:52:18 Unraid kernel: ahci 0000:01:00.1: Event logged [IO_PAGE_FAULT domain=0x0000 address=0x0000100000000d80 flags=0x0010]
Seen it several times before on the forums with AMD boards, disabling IOMMU might help.
Yes, if the motherboard supports PCIe bifurcation, there's also a Supermicro model for two NVMe devices, but again it requires PCIe bifurcation for both devices to work.
The cache filesystem is corrupt and will need to be re-created, also make sure to check this:
https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173