Main GPU VM Crash, when starting 2nd GPU VM

[Skaterscare]

My Hardware + Settings

Mainboard B450 Aorus Pro-CF BIOS F60e, Ryzen 7 3800x, main GPU MSI rx 5700 xt Mech(passed through), second GPU MSI rx 480 Gaming X 8gb, 32 GiB DDR4. Unraid Version: 6.9.0-rc2, IOMMU=Auto (Enabled didn't change anything), HVM = Enabled, boot params: kernel /bzimage append pcie_acs_override=downstream,multifunction initrd=/bzroot pci=noats

 

What I did is updating my mainboard to F60e, but I forgot to enable HVM again, so as I started Unraid and tried to start my "main GPU VM" i got an error msg. Seeing that I remembered I had to change BIOS settings to run Unraid VMs. So I did and as I came back to my VM tab my VM was gone. So I had to reconfigure a new VM, same passthroughs same VBIOS file. It started up fine, but as soon as I started the second VM, my first one crashed. So I tried to startup the main VM(after dealing with AMD reset bug -.-) i got folowing Error msg.:

 

Initial Error: qemu unexpectedly closed the monitor.

 

Stoping my second Win 10 VM does let me startup my main VM normaly. The other way arround (starting second VM first), gives me the same error msg.

 

I had the same problem after changing one of my Drives. I thought that some files where corrupted or something so I made new VMs with new VioDrives, that helped until now?

 

My IOMMU groupings:

 

IOMMU group 0:[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 1:[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 2:[1022:1483] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 3:[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 4:[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 5:[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 6:[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 7:[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 8:[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 9:[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 10:[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 11:[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 12:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 13:[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0

[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1

[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2

[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3

[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4

[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5

[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6

[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7

IOMMU group 14:[15b7:5006] 01:00.0 Non-Volatile memory controller: Sandisk Corp WD Black 2019 / PC SN750 NVMe SSD

[N:0:8215:1] disk WDS100T3X0C-00SJG0__1 /dev/nvme0n1 1.00TB

IOMMU group 15:[1022:43d5] 02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01)

This controller is bound to vfio, connected USB devices are not visible.

IOMMU group 16:[1022:43c8] 02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)

[2:0:0:0] disk ATA WDC WD5000LPVX-6 1A01 /dev/sdb 500GB

[5:0:0:0] disk ATA ADATA SP600 2.9 /dev/sdc 128GB

[6:0:0:0] disk ATA ST1000DM003-1ER1 CC43 /dev/sdd 1.00TB

IOMMU group 17:[1022:43c6] 02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)

IOMMU group 18:[1022:43c7] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

IOMMU group 19:[1022:43c7] 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

IOMMU group 20:[1022:43c7] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

IOMMU group 21:[1022:43c7] 03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

IOMMU group 22:[1022:43c7] 03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

IOMMU group 23:[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

IOMMU group 24:[1002:67df] 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev c7)

IOMMU group 25:[1002:aaf0] 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]

IOMMU group 26:[1106:3483] 08:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)

This controller is bound to vfio, connected USB devices are not visible.

IOMMU group 27:[1002:1478] 09:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1)

IOMMU group 28:[1002:1479] 0a:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch

IOMMU group 29:[1002:731f] 0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c1)

IOMMU group 30:[1002:ab38] 0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio

IOMMU group 31:[1022:148a] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 32:[1022:1485] 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 33:[1022:1486] 0d:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP

IOMMU group 34:[1022:149c] 0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 005 Device 002: ID ffff:5678 USB Disk 2.0

Bus 005 Device 003: ID 046d:c539 Logitech, Inc. USB Receiver

Bus 005 Device 004: ID 1e7d:3098 ROCCAT ROCCAT Vulcan AIMO

Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

IOMMU group 35:[1022:1487] 0d:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

 

group 26 passed through to VM1

group 15 passed through to VM2

 

VM settings

 

1.

Machine: Q35-5.1 (i440fx GPU driver issues)

BIOS: OVMF

Hyper-V: Yes

USB Controller: 2.0 (EHCI)

OS Install ISO: /mnt/user/isos/Windows.iso

OS Install CDRom Bus: SATA

VirtIO Drivers ISO: /mnt/user/isos/virtio-win-0.1.190-1.iso

VirtIO Drivers CDRom Bus: SATA

Primary vDisk Location: Auto

/mnt/user/domains/*user*/vdisk1.img

Primary vDisk Bus: VirtIO

Graphics Card: AMD Radeon RX 5600 OEM/5600 XT / 5700/5700 XT (0b:00.0)

Graphics ROM BIOS:/mnt/user/isos/MSI.RX5700XT.8192.190805.dump(.rom same error)

Sound Card: AMD Navi 10 HDMI Audio (0b:00.1)

Network MAC: *my MAC*

Network Bridge: br0

Network Model: virtio-net

 

2.

Machine: Q35-4.2 (i440fx GPU driver issues Q35-5.1 performance issues)

BIOS: OVMF

Hyper-V: Yes

USB Controller: 2.0 (EHCI)

OS Install ISO: /mnt/user/isos/Windows.iso

OS Install CDRom Bus: SATA

VirtIO Drivers ISO: /mnt/user/isos/virtio-win-0.1.190-1.iso

VirtIO Drivers CDRom Bus: SATA

Primary vDisk Location: Auto

/mnt/user/domains/*user*/vdisk1.img

Primary vDisk Bus: VirtIO

Graphics Card: AMD Radeon RX 470/480/570/570X/580/580X/590 (06:00.0)

Graphics ROM BIOS:

Sound Card: AMD Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] (06:00.1)

Network MAC: *my MAC*

Network Bridge: br0

Network Model: virtio-net