Skylake, NVIDIA GPU passthrough (Error 43)


Recommended Posts

I'm having trouble passing my nvidia 960 to my Windows 10 VM since I've upgraded my motherboard/cpu. This was working on my old motherboard/cpu (P8Z68-V PRO) with on-board graphics, but the new board (ASUS x299-Prime) doesn't have on-board video. I have a good 960 rom, but I'm left with an error 43 on the nvidia driver in Windows. The virtio drivers are installed as well as the QEMU Guest Agent. I have used VNC in my troubleshooting and have heard that might be a problem?

 

Now there's 2 things worth noting.

1. I've been running unraid since early v6 and have upgraded through to current. I'm wondering if I have some old settings getting in the way?

2. Error 43 went away after a BIOS update and the GPU worked properly for a week! At about the same time the GPU passthrough stopped working again, Windows 10 had updated to build 1709 & I had opened up my case and moved my m2 drive from slot 2 to slot 1. The next time I started the VM, I got error 43. I've since moved the drive back. Re-flashed the BIOS, cleared all settings, & have reviewed all the BIOS settings multiple times. 

 

I've attached my system diagnostics, KVM template, and a recent VM log. 

 

I've gone through the forums, watched all of the videos on the subject from Spaceinvader One, and googled until I couldn't see straight any more. I just can't figure this one out. I might even try out, https://github.com/sk1080/nvidia-kvm-patcher.

 

VM Log

Quote

2018-02-21 04:55:23.204+0000: starting up libvirt version: 3.8.0, qemu version: 2.10.2, hostname: Sol
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/local/sbin/qemu -name 'guest=Windows 10 (Primary),debug-threads=on' -S -object 'secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-Windows 10 (Primary)/master-key.aes' -machine pc-i440fx-2.10,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on -drive file=/etc/libvirt/qemu/nvram/d46cb528-03ca-3c1e-7058-8309b7b855fa_VARS-pure-efi.fd,if=pflash,format=raw,unit=1 -m 8192 -realtime mlock=off -smp 4,sockets=1,cores=2,threads=2 -uuid d46cb528-03ca-3c1e-7058-8309b7b855fa -display none -no-user-config -nodefaults -chardev 'socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-1-Windows 10 (Primary)/monitor.sock,server,nowait' -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -boot stri)/org.qemu.guest_agent.0,server,nowait' -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device vfio-pci,host=65:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/user/domains/vBIOS/Asus.GTX960.4096.150618.dump -device vfio-pci,host=65:00.1,id=hostdev1,bus=pci.0,addr=0x6 -device usb-host,hostbus=1,hostaddr=2,id=hostdev2,bus=usb.0,port=1 -device usb-host,hostbus=1,hostaddr=8,id=hostdev3,bus=usb.0,port=2 -device usb-host,hostbus=1,hostaddr=6,id=hostdev4,bus=usb.0,port=3 -device usb-host,hostbus=1,hostaddr=7,id=hostdev5,bus=usb.0,port=4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on
2018-02-21 04:55:23.204+0000: Domain id=1 is tainted: high-privileges
2018-02-21 04:55:23.204+0000: Domain id=1 is tainted: host-cpu
2018-02-21T04:55:23.253395Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/0 (label charserial0)
2018-02-21T04:55:23.857106Z qemu-system-x86_64: -device vfio-pci,host=65:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/cache/domains/vBIOS/Asus.GTX960.4096.150618.dump: Failed to mmap 0000:65:00.0 BAR 3. Performance may be slow
2018-02-21T04:55:37.818227Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x1088, 0xfffe11,8) failed: Device or resource busy
2018-02-21T04:55:37.818258Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11000, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818266Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11008, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818272Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11010, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818279Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11018, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818285Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11020, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818291Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11028, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818297Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11030, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818303Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11038, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818309Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11040, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818316Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11048, 0x0,8) failed: Device or resource busy
2018-02-21T04:55:37.818322Z qemu-system-x86_64: vfio_region_write(0000:65:00.0:region3+0x11050, 0x0,8) failed: Device or resource busy

...

Goes on forever

 

 

 

 

 

 

Windows 10 VM Template

Troubleshooting: Disabled HyperV, removed all USB devices, tried different amounts of RAM. I have tried different append options in syslinux.cfg, but those didn't help so I went back to default. 

Quote

<domain type='kvm'>
  <name>Windows 10 (Primary)</name>
  <uuid>d46cb528-03ca-3c1e-7058-8309b7b855fa</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='14'/>
    <vcpupin vcpu='2' cpuset='7'/>
    <vcpupin vcpu='3' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.10'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/d46cb528-03ca-3c1e-7058-8309b7b855fa_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disks/Samsung_SSD_960_PRO_512GB_S3EWNX0JA09070E/Windows 10 (vDisk)/windows10_primary.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_1709_English_x64.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.141-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5c:d3:4d'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x65' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/domains/vBIOS/Asus.GTX960.4096.150618.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x65' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x02e6'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0b05'/>
        <product id='0x1867'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0b05'/>
        <product id='0x1868'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x004a'/>
      </source>
      <address type='usb' bus='0' port='4'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

sol-diagnostics-20180220-2024.zip

Edited by BusterBrawls
updated details of old system
Link to comment
  • 2 weeks later...
  • 2 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.