I'm trying to a build a 3 in 1 virtualized desktop with 3 GPUs pass through to each VM, my hardware is as follows.
Ryzen TR 3960x
Gigabyte Aorus TRX40 wifi
128GB Ram
1660ti
Quadro m4000
Radeon RX5700 (flash with 5700 XT bios)
Intel x540-t1 NIC
Renasas USB addin card
3x NVME SSD
I got the 5700xt to post a couple of times before the board was fully populated but ran into the reset bug using the stock 6.8.3 kernel. I've since switched to a patched kernel with the AMD FLR fix for the Starship usb controller and the Navi reset fix. I'm not sure when exactly the 5700 stopped booting, but now no matter what I've tried I get the following error and a black screen when I attempt to the boot the VM with 5700 passed through.
[44087.701535] AMD-Vi: Completion-Wait loop timed out
[44087.838089] AMD-Vi: Completion-Wait loop timed out
[44088.000358] AMD-Vi: Completion-Wait loop timed out
[44088.134959] AMD-Vi: Completion-Wait loop timed out
[44088.282570] AMD-Vi: Completion-Wait loop timed out
[44088.405779] iommu ivhd1: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=4c:00.0 address=0x2006c88620]
[44089.281887] AMD-Vi: Completion-Wait loop timed out
[44089.407663] iommu ivhd1: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=4c:00.0 address=0x2006c88670]
[44089.716431] AMD-Vi: Completion-Wait loop timed out
[44090.409536] iommu ivhd1: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=4c:00.0 address=0x2006c886c0]
[44091.097331] AMD-Vi: Completion-Wait loop timed out
[44091.244344] AMD-Vi: Completion-Wait loop timed out
[44091.391406] AMD-Vi: Completion-Wait loop timed out
[44091.411416] iommu ivhd1: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=4c:00.0 address=0x2006c88750]
The card is currently in the bottom PCIe port on my motherboard, with some many add in cards it can only really fit in the top or bottom slot. I have absolutely no issues with the two Nvidia cards, the 1660ti which is my primary GPU boots up just fine in either EFI or CSM mode with the bios added to my VM config. The M4000 just worked out of the box, but 5700 refuses to boot. I noticed that when I booted into Windows on bare metal the Radeon didn't work at first because I had above 4g decoding turned off. With it on, Windows was able to work with all the add in cards without issue, however the problem in unraid persists.
Can anyone point me in the right direction on how to resolve this?
crushinator-diagnostics-20200601-0728.zip