SOLVED - Can't pass through rtx 1660ti


Recommended Posts

Hi all,

I'm running the latest version of unraid 6.7.2.

I have 2 gpu's in my system, an old radeon 5770 and a gigabyte rtx 1660ti.

I have a Ryzen 7 2700 and a Gigabyte b450 aorus m motherboard.

 

I have tried everything I can think of or read to get the pass through working. I watched every spaceinvader one video on any related subject but it didn't help.

 

The nvidia card has all of it's devices in the same iommu group. I pinned them all to vfio-pci.

 

Whenever I try to start up a windows 10 vm , passing the card through, I get:

-device vfio-pci,host=06:00.0,id=hostdev0,bus=pci.3,addr=0x0,romfile=/mnt/user/system/vbios/Gigabyte.GTX1660Ti.6144.190113_1_no_header.rom \
-device vfio-pci,host=06:00.1,id=hostdev1,bus=pci.4,addr=0x0 \
-device vfio-pci,host=06:00.2,id=hostdev2,bus=pci.5,addr=0x0 \
-device vfio-pci,host=06:00.3,id=hostdev3,bus=pci.6,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2019-07-06 01:33:41.796+0000: Domain id=1 is tainted: high-privileges
2019-07-06 01:33:41.796+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2019-07-06T01:33:44.345969Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-07-06T01:33:44.349775Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-07-06T01:33:44.354756Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-07-06T01:33:44.358761Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

 

And then of course I have to kill the vm or it sits there doing nothing and not booting.  If I try to start any other vm again with the card used for pass through, I'll get the error:

internal error: Unknown PCI header type '127'   until I reboot.

 

I have tried running the vm with  i440fx-3.1 , Q35-3.1, seabios, OVMF, with Hyper-V on/off , booted unraid from UEFI and without UEFI.

 

I set my bios to boot with the Radeon card as primary so I know I don't need the gpu bios to passthrough, but I did grab the bios and edited it to remove the headers as well, and tried that to see if that helped.

 

I have tried every combo of this for syslinux :

kernel /bzimage
append amd_iommu=on pcie_acs_override=downstream vfio-pci.ids=10de:2182,10de:1aeb,10de:1aec,10de:1aed initrd=/bzroot

 

PCI Devices and IOMMU Groups

IOMMU group 0:[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 1:[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 2:[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 3:[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 4:[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge

IOMMU group 5:[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 6:[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 7:[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

IOMMU group 8:[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge

IOMMU group 9:[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B

IOMMU group 10:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 11:[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0

[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1

[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2

[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3

[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4

[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5

[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6

[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7

IOMMU group 12:[1022:43d5] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01)

[1022:43c8] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)

[1022:43c6] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)

[1022:43c7] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[1022:43c7] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)

[10ec:8168] 03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 16)

[1002:68b8] 05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770]

[1002:aa58] 05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]

IOMMU group 13:[10de:2182] 06:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 Ti] (rev ff)

[10de:1aeb] 06:00.1 Audio device: NVIDIA Corporation Device 1aeb (rev ff)

[10de:1aec] 06:00.2 USB controller: NVIDIA Corporation Device 1aec (rev ff)

[10de:1aed] 06:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device 1aed (rev ff)

IOMMU group 14:[1022:145a] 07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function

IOMMU group 15:[1022:1456] 07:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor

IOMMU group 16:[1022:145f] 07:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller

IOMMU group 17:[1022:1455] 08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function

IOMMU group 18:[1022:7901] 08:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 19:[1022:1457] 08:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller

 

 

 

I also get these errors in the system logs;

vfio-pci 0000:06:00.0: not ready 32767ms after FLR; waiting

vfio-pci 0000:06:00.1: Refused to change power state, currently in D3

vfio_bar_restore: 0000:06:00.0 reset recovery - restoring bars

vfio-pci 0000:06:00.0: timed out waiting for pending transaction; performing function level reset anyway

vfio-pci 0000:06:00.0: not ready 32767ms after FLR; waiting
vfio-pci 0000:06:00.0: not ready 65535ms after FLR; giving up

vfio_bar_restore: 0000:06:00.0 reset recovery - restoring bars

vfio_bar_restore: 0000:06:00.1 reset recovery - restoring bars
vfio_bar_restore: 0000:06:00.2 reset recovery - restoring bars
vfio_bar_restore: 0000:06:00.3 reset recovery - restoring bars

 

Jul 5 21:33:44 Redshift kernel: vfio_bar_restore: 0000:06:00.3 reset recovery - restoring bars Jul 5 21:33:44 Redshift kernel: vfio_bar_restore: 0000:06:00.2 reset recovery - restoring bars Jul 5 21:33:44 Redshift kernel: vfio_bar_restore: 0000:06:00.1 reset recovery - restoring bars Jul 5 21:33:44 Redshift kernel: vfio_bar_restore: 0000:06:00.0 reset recovery - restoring bars Jul 5 21:33:45 Redshift kernel: vfio-pci 0000:06:00.0: timed out waiting for pending transaction; performing function level reset anyway Jul 5 21:33:46 Redshift kernel: vfio-pci 0000:06:00.0: not ready 1023ms after FLR; waiting Jul 5 21:33:47 Redshift kernel: vfio-pci 0000:06:00.0: not ready 2047ms after FLR; waiting Jul 5 21:33:49 Redshift kernel: vfio-pci 0000:06:00.0: not ready 4095ms after FLR; waiting Jul 5 21:33:53 Redshift kernel: vfio-pci 0000:06:00.0: not ready 8191ms after FLR; waiting Jul 5 21:34:02 Redshift kernel: vfio-pci 0000:06:00.0: not ready 16383ms after FLR; waiting Jul 5 21:34:19 Redshift kernel: vfio-pci 0000:06:00.0: not ready 32767ms after FLR; waiting Jul 5 21:34:54 Redshift kernel: vfio-pci 0000:06:00.0: not ready 65535ms after FLR; giving up Jul 5 21:34:54 Redshift kernel: vfio_bar_restore: 0000:06:00.0 reset recovery - restoring bars Jul 5 21:34:54 Redshift kernel: vfio_bar_restore: 0000:06:00.1 reset recovery - restoring bars Jul 5 21:34:54 Redshift kernel: vfio_bar_restore: 0000:06:00.2 reset recovery - restoring bars Jul 5 21:34:54 Redshift kernel: vfio_bar_restore: 0000:06:00.3 reset recovery - restoring bars Jul 5 21:37:19 Redshift emhttpd: req (2): csrf_token=****************&title=Log+for%3AWindows+10+-+Bare+Metal&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=libvirt%2Fqemu%2FWindows+10+-+Bare+Metal.log Jul 5 21:37:19 Redshift emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log libvirt/qemu/Windows 10 - Bare Metal.log Jul 5 21:48:40 Redshift login[12997]: ROOT LOGIN on '/dev/pts/1' Jul 5 22:02:04 Redshift kernel: ata6.00: Enabling discard_zeroes_data Jul 5 22:02:04 Redshift kernel: sde: sde1 sde2 sde3 Jul 5 22:02:04 Redshift avahi-daemon[5027]: Interface vnet0.IPv6 no longer relevant for mDNS. Jul 5 22:02:04 Redshift avahi-daemon[5027]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe88:9fb7. Jul 5 22:02:04 Redshift kernel: br0: port 2(vnet0) entered disabled state Jul 5 22:02:04 Redshift kernel: device vnet0 left promiscuous mode Jul 5 22:02:04 Redshift kernel: br0: port 2(vnet0) entered disabled state Jul 5 22:02:04 Redshift avahi-daemon[5027]: Withdrawing address record for fe80::fc54:ff:fe88:9fb7 on vnet0. Jul 5 22:02:05 Redshift unassigned.devices: Disk with serial 'Samsung_SSD_840_Series_S14CNSAD200506M', mountpoint 'System_Reserved' is not set to auto mount and will not be mounted... Jul 5 22:02:05 Redshift unassigned.devices: Disk with serial 'Samsung_SSD_840_Series_S14CNSAD200506M', mountpoint 'Samsung_SSD_840_Series_S14CNSAD200506M-part3' is not set to auto mount and will not be mounted... Jul 5 22:02:05 Redshift unassigned.devices: Disk with serial 'Samsung_SSD_840_Series_S14CNSAD200506M', mountpoint 'Samsung_SSD_840_Series_S14CNSAD200506M-part2' is not set to auto mount and will not be mounted... Jul 5 22:02:05 Redshift kernel: vfio-pci 0000:06:00.0: timed out waiting for pending transaction; performing function level reset anyway Jul 5 22:02:07 Redshift kernel: vfio-pci 0000:06:00.0: not ready 1023ms after FLR; waiting Jul 5 22:02:08 Redshift kernel: vfio-pci 0000:06:00.0: not ready 2047ms after FLR; waiting Jul 5 22:02:10 Redshift kernel: vfio-pci 0000:06:00.0: not ready 4095ms after FLR; waiting Jul 5 22:02:14 Redshift kernel: vfio-pci 0000:06:00.0: not ready 8191ms after FLR; waiting Jul 5 22:02:23 Redshift kernel: vfio-pci 0000:06:00.0: not ready 16383ms after FLR; waiting Jul 5 22:02:40 Redshift kernel: vfio-pci 0000:06:00.0: not ready 32767ms after FLR; waiting Jul 5 22:02:56 Redshift ool www[14840]: /usr/local/emhttp/plugins/dynamix/scripts/bootmode '1' Jul 5 22:03:13 Redshift kernel: vfio-pci 0000:06:00.0: not ready 65535ms after FLR; giving up Jul 5 22:03:13 Redshift kernel: vfio-pci 0000:06:00.1: Refused to change power state, currently in D3 Jul 5 22:03:13 Redshift kernel: vfio-pci 0000:06:00.2: Refused to change power state, currently in D3 Jul 5 22:03:13 Redshift kernel: vfio-pci 0000:06:00.3: Refused to change power state, currently in D3

 

Please help!  I don't know if it's my motherboard or video card that's the issue. I'll return either one if I can just fix this.

Link to comment

Well it looks like this was due to the bios on my motherboard. I had the latest revision F40. I tried downgrading to F32 but had the same problem. I then downgraded to F5 and I am now able to pass-through successfully. Ugh. I will be reluctant to update my bios now.

Link to comment
  • Caduceus changed the title to SOLVED - Can't pass through rtx 1660ti

Are you sure, you don't miss some settings in new BIOS? CSM? IOMMU?

Aso, did you try this patch: 

lI know its related to Threadripper, but maybe will also help you for this reset problem?

 

There is also a parameter for vfio-pci module - disable_idle_d3:Disable using the PCI D3 low power state for idle, unused devices (bool)

Edited by belliash
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.