VM won't boot with GPU passthrough "Unknown PCI header type '127'"

[Dubz]

Hello guys,

 

I am having problems with GPU passthrough to my VM. It won't boot and I'm getting the "Unknown PCI header type '127'" error.

After a restart it "boots" but I get "qemu-system-x86_64: vfio: Unable to power on device, stuck in D3". But after some time it goes back to error 127

I have tried finding help through threads here but can't seem to find any solutions to my problem.

 

What I have tried:

Changing OS from win10 to Linux

changing i440fx-4.2 to Q35

Turning off Hyper-V

Following Spaceinvaders Videos

 

Hardware:

CPU: AMD Ryzen 1700

GPU: MSI GTX970 

Motherboard: B450 GAMING PLUS MAX

 

Here is my VM log:

Quote

ErrorWarningSystemArrayLogin

-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x7.0x2 \
-device ahci,id=sata0,bus=pci.0,addr=0x3 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Windows 10.iso","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \
-device ide-cd,bus=sata0.0,drive=libvirt-3-format,id=sata0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.173-2.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=sata0.1,drive=libvirt-2-format,id=sata0-0-1 \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 10 Game/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-hd,bus=sata0.2,drive=libvirt-1-format,id=sata0-0-2,bootindex=1,write-cache=on \
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:38:f5:48,bus=pci.0,addr=0x2 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=38,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device vfio-pci,host=0000:26:00.0,id=hostdev0,bus=pci.0,addr=0x5 \
-device vfio-pci,host=0000:26:00.1,id=hostdev1,bus=pci.0,addr=0x6 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-03-19 18:19:43.744+0000: Domain id=1 is tainted: high-privileges
2020-03-19 18:19:43.744+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
2020-03-19T18:19:46.462957Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2020-03-19T18:19:46.463215Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2020-03-19T18:20:05.050233Z qemu-system-x86_64: terminating on signal 15 from pid 7989 (/usr/sbin/libvirtd)
2020-03-19 18:20:07.251+0000: shutting down, reason=destroyed
2020-03-19 18:38:29.935+0000: shutting down, reason=failed
2020-03-19 18:38:35.546+0000: shutting down, reason=failed

 

and is my VM xml:

      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.173-2.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 Game/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:38:f5:48'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x26' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x26' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

Hopefully someone smarter than me can help me resolve the issue.

Thanks

[Dubz]

Here is also the syslog

syslog

[toastman]

Do you have the latest BIOS for the motherboard?  There are issues with passthrough that was only fixed recently

 

Edited March 20, 2020 by toastman

[Dubz]

8 hours ago, toastman said:

Do you have the latest BIOS for the motherboard?  There are issues with passthrough that was only fixed recently

 

Thanks! Just updated my motherboard and it boots but without a screen output. So I got my vbios and tried it out and... IT WORKS i am getting an display output. Trying to install Windows now and see how it does.

[Dubz]

So I installed Windows and everything worked like a charm until I needed to reboot the VM to add mouse and keyboard passthrough. Now I'm back to the 127 error and the "stuck in D3" error. I remove keyboard and mouse passthrough but that did not fix the problem.

I tried adding a new VM but got the same error.

[toastman]

That's "normal".  unRAID can't reset the card after the VM stops using it.  But it can reset it after a full server reboot.  So you're stuck in the situation where anytime you power down the VM, you need to reboot the server before powering on a VM that uses the card.

 

If your BIOS is up to date, do a search for 127 errors.  There are a LOT of posts on here about that

[Dubz]

Good to know. I tried rebooting and it did not work. I rebooted again with autoboot on and it worked. Seems a little bit unstable but it gets the job done.

Thanks for all the help toastman