VMs, Passthrough, 127 Error and Failing to Boot with Nvidia

[xer01ne]

Okay, so I am not sure what I did.  About 2 months ago, everything was working fine and now everything seems to have gone to hell.  I have an Nvidia GTX 980 that I was successfully passing through to a Win10 VM.  No issues.  I could reboot the VM multiple times and no problems. 

 

Now, I reboot the VM once, I get an error 127.  Furthermore, the VM doesn't start when the Nvidia card is passed through.  If I switch to VNC, the VM starts fine.  AND... every time I reboot Unraid, the address of the Nvidia card changes.  REALLY annoying.

 

VT-d is enabled in the bios.  I have tried using the rom file (The one I dumped and the downloaded one, both with the same results).

 

I have tried turning the card into a multi-function card in XML.

 

Here is the logs when I start with passthrough:

-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=31,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \
-device pcie-pci-bridge,id=pci.6,bus=pci.1,addr=0x0 \
-device pcie-root-port,port=0xd,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x5 \
-device ich9-usb-ehci1,id=usb,bus=pcie.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pcie.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pcie.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pcie.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/cache/domains/Virtual Gaming OS/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-1-format,id=virtio-disk2,bootindex=1,write-cache=on \
-netdev tap,fd=33,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:cf:cc:15,bus=pci.3,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=34,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device vfio-pci,host=0000:02:00.0,id=hostdev0,bus=pci.5,addr=0x0,romfile=/mnt/user/storage/storage/files/dump.rom \
-device vfio-pci,host=0000:02:00.1,id=hostdev1,bus=pci.7,addr=0x0 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-06-19 21:47:51.396+0000: Domain id=2 is tainted: high-privileges
2021-06-19 21:47:51.396+0000: Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2021-06-19T21:47:54.476697Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2021-06-19T21:47:54.487652Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

 

I don't know where to go from here or what to do.  I have wiped the Unraid load and rebuilt the VMs, all with crap results that leave me with a broken system where passthrough doesn't work anymore.

 

Any suggestions or help would be greatly appreciated.  Thanks.

[ghost82]

16 hours ago, xer01ne said:

I have tried turning the card into a multi-function card in XML

 

16 hours ago, xer01ne said:

-device vfio-pci,host=0000:02:00.0,id=hostdev0,bus=pci.5,addr=0x0,romfile=/mnt/user/storage/storage/files/dump.rom \

-device vfio-pci,host=0000:02:00.1,id=hostdev1,bus=pci.7,addr=0x0 \

in the target, video is attached to 05:00:00 (bus:slot:function), audio is attached to 07:00:00 (bus:slot:function); attach audio to 05:00:01 (bus:slot:function)

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/storage/storage/files/dump.rom'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>

Retry, I don't know if this will solve your issue, as it seems that the gpu is in use somewhere else, but the correct topology for audio/video is that I described, so fix one thing at a time.

 

16 hours ago, xer01ne said:

every time I reboot Unraid, the address of the Nvidia card changes

This is weird and should never happen, unless you modify the hardware of the host..do you remember if you updated your bios in the time between working and not working passthrough?

Error 127 sounds like gpu not resetting properly: since this is nvidia, it's not the amd gpu reset bug, check if there are updates for your motherboard bios and make sure to have a proper vbios (dumped from your gpu).

Edited June 20, 2021 by ghost82

[Gee1]

something new here? u guys get it to work ?