I have recently migrated from old hardware, and in setting up my new server I'm faced with a VM issue. My gaming Windows VM won't boot, it goes into repair mode but as usual can't repair itself.

 

Through some troubleshooting I've found out it's my GTX 1070 doing some buggering, but I don't know what buggering is happening. It's bound through VFIO (with the VFIO-PCI CFG plugin) and is alone in the IOMMU group only with the soundcard on the graphics card (0000:09:00.0 and .1). My log is filled with endless lines of this:

Sep 12 17:04:44 <HOSTNAME> kernel: vfio-pci 0000:09:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]
Sep 12 17:04:44 <HOSTNAME> kernel: vfio-pci 0000:09:00.0: BAR 1: can't reserve [mem 0xd0000000-0xdfffffff 64bit pref]

Some things:

1) It worked on my old rig with VFIO binding in Syslinux configuration

2) No changes made to the vm .xml in migration other than re-picking the gpu with its new pcie address

3) The VM works with my secondary gpu (GT 710) which is bound in the same way, and in it's own IOMMU group

4) Tried with a backup vdisk image to no prevail.

 

I'm doing more troubleshooting tomorrow, but I'm at a loss from where to go from here. Going to try VFIO in Syslinux like my old system had, but I can't see how that should make a difference.

Edited September 15, 20205 yr by brin6thepayne

This might look like a mess, but it's from the vfio page in settings. Group 25 is the GTX 1070 which I think is the cause of these issues.

IOMMU	      Address	Bound	Reset	Vendor ID	Description
Group 0	00:01.0			1022:1482	Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
Group 1	00:01.2			1022:1483	PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
Group 2	00:02.0			1022:1482	Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
Group 3	00:03.0			1022:1482	Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
Group 4	00:03.1			1022:1483	PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
Group 5	00:04.0			1022:1482	Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
Group 6	00:05.0			1022:1482	Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
Group 7	00:07.0			1022:1482	Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
Group 8	00:07.1			1022:1484	PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Group 9	00:08.0			1022:1482	Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
Group 10	00:08.1			1022:1484	PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Group 11	00:08.2			1022:1484	PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Group 12	00:08.3			1022:1484	PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Group 13	00:14.0			1022:790b	SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
	00:14.3			1022:790e	ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
Group 14	00:18.0			1022:1440	Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
	00:18.1			1022:1441	Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
	00:18.2			1022:1442	Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
	00:18.3			1022:1443	Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
	00:18.4			1022:1444	Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
	00:18.5			1022:1445	Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
	00:18.6			1022:1446	Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
	00:18.7			1022:1447	Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
Group 15	01:00.0			1022:57ad	PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
Group 16	02:00.0			1022:57a3	PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
Group 17	02:03.0			1022:57a3	PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
Group 18	02:04.0			1022:57a3	PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
Group 19	02:08.0			1022:57a4	PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
	06:00.0			1022:1485	Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
	06:00.1			1022:149c	USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
					USB devices attached to this controller:
					Bus 002 Device 002: ID 2516:0051 Cooler Master Technology Inc. AMD SR4 lamplight Control
					Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
					Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
	06:00.3			1022:149c	USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
					USB devices attached to this controller:
					Bus 004 Device 003: ID 1b1c:1b50 Corsair CORSAIR K63 Wireless USB Receiver
					Bus 004 Device 002: ID 046d:c53f Logitech, Inc. USB Receiver
					Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
					Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Group 20	02:09.0			1022:57a4	PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
	07:00.0			1022:7901	SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
Group 21	02:0a.0			1022:57a4	PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
	08:00.0			1022:7901	SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
Group 22	03:00.0			10de:128b	VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
	03:00.1			10de:0e0f	Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
Group 23	04:00.0			8086:1539	Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
Group 24	05:00.0			10ec:816e	Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device 816e (rev 1a)
	05:00.1			10ec:8168	Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 1a)
	05:00.2			10ec:816a	Serial controller: Realtek Semiconductor Co., Ltd. Device 816a (rev 1a)
	05:00.4			10ec:816d	USB controller: Realtek Semiconductor Co., Ltd. Device 816d (rev 1a)
					USB devices attached to this controller:
					Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
	05:00.7			10ec:816c	IPMI Interface: Realtek Semiconductor Co., Ltd. Device 816c (rev 1a)
Group 25	09:00.0			10de:1b81	VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
	09:00.1			10de:10f0	Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
Group 26	0a:00.0			1022:148a	Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
Group 27	0b:00.0			1022:1485	Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
Group 28	0b:00.1			1022:1486	Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
Group 29	0b:00.3			1022:149c	USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
					USB devices attached to this controller:
					Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
					Bus 007 Device 002: ID 0781:5583 SanDisk Corp. Ultra Fit
					Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Group 30	0b:00.4			1022:1487	Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
Group 31	0c:00.0			1022:7901	SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
Group 32	0d:00.0			1022:7901	SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

 

Diagnostics.zip

EDIT: I got exactly zero replies on this, but figured out my issue was unraid grabbing my gpu for gui output since my new build doesnt have igpu. It was a common error, I found it by googling segments of the repeated log error one at a time.