Unraid 6.8 GTX 1650 Super Passthrough Issues

[Lucict]

Hello, 

 

I'm having trouble passing through my GTX 1650 Super to VMs. When I try to start the VM, it hangs and won't start. This in turn hangs the VM manager and then the whole system. I can't even restart, I have to hard power the system off and back on. I do not receive any error message.

 

I am trying to passthrough the GPU to a Windows 10 VM. It works with just VNC. I'm not sure what is happening. Any help would really be appreciated.

 

This is a new build for me. System specs are as follows:

Unraid 6.8 with Linuxserver Nvidia plugin

Ryzen 5 3600

Gigabyte B450 Aorus Pro Wifi (BIOS version F50)

G.Skill DDR4 Ram - 2x16g sticks

SeaSonic M12II-620 PSU 

Nvidia GTX 1650 Super

Nvidia Quadro P400

 

So far, I've done the following:

--Turned on SVM and IOMMU in BIOS

--Updated BIOS to version F50 (released 11/19)

--Extracted vBIOS from the GTX 1650 Super with GPU-Z and edited out the header

--Edited the template to make sure the device is multifunction and on the same slot in the VM

--Turned on ACS override (both downstream and multifunction) to separate IOMMU groups

 

None of these things have changed anything. It hangs each time I try to start the VM.

[Lucict]

For anyone who finds this helpful, I was able to solve my problem by using VFIO-pci.ids.

[Henk Geels]

 

On 1/12/2020 at 7:05 PM, Lucict said:

For anyone who finds this helpful, I was able to solve my problem by using VFIO-pci.ids.

Hi, could you please give me some direction on how you fixed it with VFIO-pci.ids

 

 

[Lucict]

Sure! I’d be happy to help.

 

So, VFIO-pci.id is a way to limit the host machine (i.e., Unraid) from interacting with a piece of hardware.  Instead of the host system attaching a driver to a piece of hardware, the kernel attaches a virtual driver, leaving it available for the VM. This method makes that hardware unavailable to the host machine. So, you need a second graphics card if you want to boot into the unraid GUI separately.

 

To use the VFIO-PCs.ids method, go to the Unraid web UI and click on Main. Then click on your flash drive (the blue title that says “Flash” is a link). Once inside, scroll down to the Syslinux configuration section. In the Unraid OS section (or the UnRaid GUI mode if you boot to that), type in vfio-pci.ids=<device id1>,<device id2>. You can find the device id by going to Tools->System Devices. The device is in brackets. So here’s how mine looks: vfio-pci.ids=10de:1aeb,10de:2187,10de:1aec,10de:1aed

 

 

Here’s a post the might help:

https://forums.unraid.net/topic/80036-new-method-for-passthrough-devices-in-unraid-67-vfiobind/
 

 

 

[bugsysiegals]

I've the same card and cannot get Windows VM to load.   I've tried Windows VM with i440fx-4.2 and Q35-4.2 without success. 

 

FWIW - when I reboot unRAID, I can see the beginning of the boot code up until PCI 0000.06:00.0 and then it stops showing any additional code.  When I start the Windows VM, the TV switches to no signal.

 

I also tried following the link you posted ... I removed the ids from vfio-pci.ids and added BIND=06:00.0 to /config/vfio-pci.cfg but then I can then see the entire unRAID boot code display when it boots and I suspect that shouldn't happen.

 

I'm booting in UEFI mode if that matters.

 

unraid OS

kernel /bzimage
append vfio-pci.ids=8086:150e,10de:2187,10de:1aeb,10de:1aec,10de:1aed pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interupts=1 initrd=/bzroot 

 

System Devices

IOMMU group 14:	
[10de:2187] 06:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1)
[10de:1aeb] 06:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)
[10de:1aec] 06:00.2 USB controller: NVIDIA Corporation Device 1aec (rev a1)
[10de:1aed] 06:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1)

After enabling pcs override it changed to this ... is this even needed?

IOMMU group 22:	[10de:2187] 06:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev ff)
IOMMU group 23:	[10de:1aeb] 06:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev ff)
IOMMU group 24:	[10de:1aec] 06:00.2 USB controller: NVIDIA Corporation Device 1aec (rev ff)
IOMMU group 25:	[10de:1aed] 06:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev ff)

 

 

Edited April 25, 2020 by bugsysiegals

[Lucict]

What happens when you start the VM?

[bugsysiegals]

It would give me a D3 error in the log and VNC would be Black screen.  I updated my BIOS which included 1.0.0.4 B Patch below and now I can boot but am having code 43 in Device Manager.  Perhaps I need to not boot in UEFI mode ... I'll be playing with various settings today to get it sorted out.

 

 

[N¿¿B]

Hello guys! I'm having an issue passing through my 1650 Super. @Lucict I'd appreciate any help you could provide.

 

Here's my log:

 

-uuid 32b5f37e-0939-1ff7-fbca-58e86196be84 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x3 \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 10/vdisk1.img","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-3-format,id=virtio-disk2,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/windows/Win10_2004_English_x64.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=ide.0,unit=0,drive=libvirt-2-format,id=ide0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/windows/virtio-win-0.1.173-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.0,unit=1,drive=libvirt-1-format,id=ide0-0-1 \
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:63:63:75,bus=pci.0,addr=0x2 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=37,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device vfio-pci,host=0000:83:00.0,id=hostdev0,bus=pci.0,multifunction=on,addr=0x5,romfile=/mnt/user/domains/VBIOS/gigabyte_gtx1650_super.rom \
-device vfio-pci,host=0000:83:00.1,id=hostdev1,bus=pci.0,addr=0x5.0x1 \
-device vfio-pci,host=0000:83:00.2,id=hostdev2,bus=pci.0,addr=0x5.0x2 \
-device vfio-pci,host=0000:83:00.3,id=hostdev3,bus=pci.0,addr=0x5.0x3 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-08-08 03:50:31.038+0000: Domain id=11 is tainted: high-privileges
2020-08-08 03:50:31.038+0000: Domain id=11 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)

Here's the parts of my XML that deal with the GPU:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x83' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/domains/VBIOS/gigabyte_gtx1650_super.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x83' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</hostdev>
 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x83' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x83' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x3'/>
</hostdev>

I'll start the VM and then try to boot in using Splashtop or TigerVNC, I just keep getting a timed out issue. It won't let me connect to the VM. I have no idea what I'm doing wrong. Any help would be appreciated.

Edited August 8, 2020 by N¿¿B

[Lucict]

I’m confused—do you want to use VNC or are you passing through your graphics card to connect to a monitor?

[meep]

12 hours ago, N¿¿B said:

:ll start the VM and then try to boot in using Splashtop or TigerVNC, I just keep getting a timed out issue. It won't let me connect to the VM. I have no idea what I'm doing wrong. Any help would be appreciated.

While the GPU portions of the XMl look OK, there could be any number of other issues.

For example, the system could be stuck at the windows login prompt and I beleive Spalshtop needs to access a desktop session. (Did you use netplwiz to auto-login? Did splashtop work BEFORE you added the GPU?).

 

Or yiour system could be stuck in the boot process somewhere, like in recovery mode, and you cannot see it?

 

Check out the steps in the link in my sig to help you get to a point where at least you can vide the boot process to see whats going on (virt-manager docker is your friend)

 

 

[bigbangus]

Just got a 1650 super today and managed to pass it through as primary gpu on first go with a modified vbios (no header) and by adding "video=efifb:off" to the Syslinux config. Just for fun, I did the multifunction='on' edit in the xml but didn't seem to affect anything. BTW I'm on 6.9beta30 nvidia.

 

Question: I do get this in my VM log though:

2020-10-17T14:28:58.117847Z qemu-system-x86_64: vfio: Cannot reset device 0000:0b:00.4, depends on group 31 which is not owned.
2020-10-17T14:28:58.125684Z qemu-system-x86_64: vfio: Cannot reset device 0000:09:00.1, depends on group 28 which is not owned.
2020-10-17T14:28:58.153772Z qemu-system-x86_64: vfio: Cannot reset device 0000:0b:00.4, depends on group 31 which is not owned.

09:00.1 is the 1650 super. (0b:00.4 is a USB controller). Passing through a 1050 Ti prior, I never got the gpu warning before. Just curious if it's an issue I should be worried about.

 

See VM xml attached.

 

 

Win10VM.txt

Edited October 17, 2020 by bigbangus

[bigbangus]

Nevermind! I didn't pass the GPU usb controllers! The "Cannot reset device 0000:09:00.1" has gone away. Didn't expect USB on there!

 

See updated Win10VM_v2 xml in case anyone else has issue with passing a 1650S.

Win10VM_v2.xml