Hi all,
I'm running the latest version of unraid 6.7.2.
I have 2 gpu's in my system, an old radeon 5770 and a gigabyte rtx 1660ti.
I have a Ryzen 7 2700 and a Gigabyte b450 aorus m motherboard.
I have tried everything I can think of or read to get the pass through working. I watched every spaceinvader one video on any related subject but it didn't help.
The nvidia card has all of it's devices in the same iommu group. I pinned them all to vfio-pci.
Whenever I try to start up a windows 10 vm , passing the card through, I get:
-device vfio-pci,host=06:00.0,id=hostdev0,bus=pci.3,addr=0x0,romfile=/mnt/user/system/vbios/Gigabyte.GTX1660Ti.6144.190113_1_no_header.rom \
-device vfio-pci,host=06:00.1,id=hostdev1,bus=pci.4,addr=0x0 \
-device vfio-pci,host=06:00.2,id=hostdev2,bus=pci.5,addr=0x0 \
-device vfio-pci,host=06:00.3,id=hostdev3,bus=pci.6,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2019-07-06 01:33:41.796+0000: Domain id=1 is tainted: high-privileges
2019-07-06 01:33:41.796+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2019-07-06T01:33:44.345969Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-07-06T01:33:44.349775Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-07-06T01:33:44.354756Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-07-06T01:33:44.358761Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
And then of course I have to kill the vm or it sits there doing nothing and not booting. If I try to start any other vm again with the card used for pass through, I'll get the error:
internal error: Unknown PCI header type '127' until I reboot.
I have tried running the vm with i440fx-3.1 , Q35-3.1, seabios, OVMF, with Hyper-V on/off , booted unraid from UEFI and without UEFI.
I set my bios to boot with the Radeon card as primary so I know I don't need the gpu bios to passthrough, but I did grab the bios and edited it to remove the headers as well, and tried that to see if that helped.
I have tried every combo of this for syslinux :
kernel /bzimage
append amd_iommu=on pcie_acs_override=downstream vfio-pci.ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed initrd=/bzroot
PCI Devices and IOMMU Groups
IOMMU group 0:[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 1:[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 2:[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 3:[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 4:[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge
IOMMU group 5:[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 6:[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 7:[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
IOMMU group 8:[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
IOMMU group 9:[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
IOMMU group 10:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 11:[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
IOMMU group 12:[1022:43d5] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01)
[1022:43c8] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
[1022:43c6] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
[1022:43c7] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[1022:43c7] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[1022:43c7] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
[10ec:8168] 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16)
[1002:68b8] 05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770]
[1002:aa58] 05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]
IOMMU group 13:[10de:2182] 06:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] (rev ff)
[10de:1aeb] 06:00.1 Audio device: NVIDIA Corporation Device 1aeb (rev ff)
[10de:1aec] 06:00.2 USB controller: NVIDIA Corporation Device 1aec (rev ff)
[10de:1aed] 06:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device 1aed (rev ff)
IOMMU group 14:[1022:145a] 07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
IOMMU group 15:[1022:1456] 07:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
IOMMU group 16:[1022:145f] 07:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller
IOMMU group 17:[1022:1455] 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function
IOMMU group 18:[1022:7901] 08:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 19:[1022:1457] 08:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller
I also get these errors in the system logs;
vfio-pci 0000:06:00.0: not ready 32767ms after FLR; waiting
vfio-pci 0000:06:00.1: Refused to change power state, currently in D3
vfio_bar_restore: 0000:06:00.0 reset recovery - restoring bars
vfio-pci 0000:06:00.0: timed out waiting for pending transaction; performing function level reset anyway
vfio-pci 0000:06:00.0: not ready 32767ms after FLR; waiting
vfio-pci 0000:06:00.0: not ready 65535ms after FLR; giving up
vfio_bar_restore: 0000:06:00.0 reset recovery - restoring bars
vfio_bar_restore: 0000:06:00.1 reset recovery - restoring bars
vfio_bar_restore: 0000:06:00.2 reset recovery - restoring bars
vfio_bar_restore: 0000:06:00.3 reset recovery - restoring bars
Jul 5 21:33:44 Redshift kernel: vfio_bar_restore: 0000:06:00.3 reset recovery - restoring bars Jul 5 21:33:44 Redshift kernel: vfio_bar_restore: 0000:06:00.2 reset recovery - restoring bars Jul 5 21:33:44 Redshift kernel: vfio_bar_restore: 0000:06:00.1 reset recovery - restoring bars Jul 5 21:33:44 Redshift kernel: vfio_bar_restore: 0000:06:00.0 reset recovery - restoring bars Jul 5 21:33:45 Redshift kernel: vfio-pci 0000:06:00.0: timed out waiting for pending transaction; performing function level reset anyway Jul 5 21:33:46 Redshift kernel: vfio-pci 0000:06:00.0: not ready 1023ms after FLR; waiting Jul 5 21:33:47 Redshift kernel: vfio-pci 0000:06:00.0: not ready 2047ms after FLR; waiting Jul 5 21:33:49 Redshift kernel: vfio-pci 0000:06:00.0: not ready 4095ms after FLR; waiting Jul 5 21:33:53 Redshift kernel: vfio-pci 0000:06:00.0: not ready 8191ms after FLR; waiting Jul 5 21:34:02 Redshift kernel: vfio-pci 0000:06:00.0: not ready 16383ms after FLR; waiting Jul 5 21:34:19 Redshift kernel: vfio-pci 0000:06:00.0: not ready 32767ms after FLR; waiting Jul 5 21:34:54 Redshift kernel: vfio-pci 0000:06:00.0: not ready 65535ms after FLR; giving up Jul 5 21:34:54 Redshift kernel: vfio_bar_restore: 0000:06:00.0 reset recovery - restoring bars Jul 5 21:34:54 Redshift kernel: vfio_bar_restore: 0000:06:00.1 reset recovery - restoring bars Jul 5 21:34:54 Redshift kernel: vfio_bar_restore: 0000:06:00.2 reset recovery - restoring bars Jul 5 21:34:54 Redshift kernel: vfio_bar_restore: 0000:06:00.3 reset recovery - restoring bars Jul 5 21:37:19 Redshift emhttpd: req (2): csrf_token=****************&title=Log+for%3AWindows+10+-+Bare+Metal&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=libvirt%2Fqemu%2FWindows+10+-+Bare+Metal.log Jul 5 21:37:19 Redshift emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log libvirt/qemu/Windows 10 - Bare Metal.log Jul 5 21:48:40 Redshift login[12997]: ROOT LOGIN on '/dev/pts/1' Jul 5 22:02:04 Redshift kernel: ata6.00: Enabling discard_zeroes_data Jul 5 22:02:04 Redshift kernel: sde: sde1 sde2 sde3 Jul 5 22:02:04 Redshift avahi-daemon[5027]: Interface vnet0.IPv6 no longer relevant for mDNS. Jul 5 22:02:04 Redshift avahi-daemon[5027]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe88:9fb7. Jul 5 22:02:04 Redshift kernel: br0: port 2(vnet0) entered disabled state Jul 5 22:02:04 Redshift kernel: device vnet0 left promiscuous mode Jul 5 22:02:04 Redshift kernel: br0: port 2(vnet0) entered disabled state Jul 5 22:02:04 Redshift avahi-daemon[5027]: Withdrawing address record for fe80::fc54:ff:fe88:9fb7 on vnet0. Jul 5 22:02:05 Redshift unassigned.devices: Disk with serial 'Samsung_SSD_840_Series_S14CNSAD200506M', mountpoint 'System_Reserved' is not set to auto mount and will not be mounted... Jul 5 22:02:05 Redshift unassigned.devices: Disk with serial 'Samsung_SSD_840_Series_S14CNSAD200506M', mountpoint 'Samsung_SSD_840_Series_S14CNSAD200506M-part3' is not set to auto mount and will not be mounted... Jul 5 22:02:05 Redshift unassigned.devices: Disk with serial 'Samsung_SSD_840_Series_S14CNSAD200506M', mountpoint 'Samsung_SSD_840_Series_S14CNSAD200506M-part2' is not set to auto mount and will not be mounted... Jul 5 22:02:05 Redshift kernel: vfio-pci 0000:06:00.0: timed out waiting for pending transaction; performing function level reset anyway Jul 5 22:02:07 Redshift kernel: vfio-pci 0000:06:00.0: not ready 1023ms after FLR; waiting Jul 5 22:02:08 Redshift kernel: vfio-pci 0000:06:00.0: not ready 2047ms after FLR; waiting Jul 5 22:02:10 Redshift kernel: vfio-pci 0000:06:00.0: not ready 4095ms after FLR; waiting Jul 5 22:02:14 Redshift kernel: vfio-pci 0000:06:00.0: not ready 8191ms after FLR; waiting Jul 5 22:02:23 Redshift kernel: vfio-pci 0000:06:00.0: not ready 16383ms after FLR; waiting Jul 5 22:02:40 Redshift kernel: vfio-pci 0000:06:00.0: not ready 32767ms after FLR; waiting Jul 5 22:02:56 Redshift ool www[14840]: /usr/local/emhttp/plugins/dynamix/scripts/bootmode '1' Jul 5 22:03:13 Redshift kernel: vfio-pci 0000:06:00.0: not ready 65535ms after FLR; giving up Jul 5 22:03:13 Redshift kernel: vfio-pci 0000:06:00.1: Refused to change power state, currently in D3 Jul 5 22:03:13 Redshift kernel: vfio-pci 0000:06:00.2: Refused to change power state, currently in D3 Jul 5 22:03:13 Redshift kernel: vfio-pci 0000:06:00.3: Refused to change power state, currently in D3
Please help! I don't know if it's my motherboard or video card that's the issue. I'll return either one if I can just fix this.