GTX970 Pass Through Issues on AMD 990FX Chipset


Recommended Posts

Hi Everyone,

 

I've spent the last two weekends fighting with this and am at my wits end.

 

First off here is my system info:

 

unRaid: 6.0-rc6a

Motherboard: 990FXA-UD5 (rev 3.1 with current bios, IOMMU enabled)

CPU: AMD FX-8320 FX-Series 8-Core Black Edition (AMD-Vi with IOMMU)

GPU: MSI GTX 970 Gaming (with current bios & driver, gpu-z shows IOMMU support)

Guest OS: Win 8.1 Enterprise x64

 

I've noticed that most users are in the Intel camp but I'm hoping some of the AMD users will chime in with some assistance.

 

I've gone through the forums and tried all the suggestions I could find and gotten two final results. Under OVMF I end up with a Code 12 error for the GPU (both i44fx-2.3 & Q35-2.3). Under SeaBIOS I end up with a Code 43 error for the GPU (both i44fx-2.3 & Q35-2.3). I'm not really sure where to go from here.

 

Things I have tried so far:

 

append amd_iommu=on iommu=pt iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot added to Syslinux.cfg

kvm hidden state patch

manual gpu rom selection

 

Nothing seems to resolve the error codes.

 

Below are my XML files, and attached is a screenshot of my system devices in unRaid.

 

Any help you can offer is much appreciated.

 

Thanks!

 

James

 

OVMF:

 

<domain type='kvm'>

  <name>Win81EntOVMF</name>

  <uuid>c306284e-43eb-e9b8-4a47-d5a9efa551d2</uuid>

  <metadata>

    <vmtemplate name="Custom" icon="windows.png" os="windows"/>

  </metadata>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>2097152</currentMemory>

  <memoryBacking>

    <nosharepages/>

    <locked/>

  </memoryBacking>

  <vcpu placement='static'>4</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='0'/>

    <vcpupin vcpu='1' cpuset='1'/>

    <vcpupin vcpu='2' cpuset='2'/>

    <vcpupin vcpu='3' cpuset='3'/>

  </cputune>

  <os>

    <type arch='x86_64' machine='pc-q35-2.3'>hvm</type>

    <loader type='pflash'>/usr/share/qemu/ovmf-x64/OVMF-pure-efi.fd</loader>

  </os>

  <features>

    <acpi/>

    <apic/>

    <kvm>

      <hidden state='on'/>

    </kvm>

  </features>

  <cpu mode='host-passthrough'>

    <topology sockets='1' cores='4' threads='1'/>

  </cpu>

  <clock offset='localtime'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='raw' cache='writeback'/>

      <source file='/mnt/cache/VM/Win81EntOVMF/vdisk1.img'/>

      <target dev='hdc' bus='virtio'/>

      <boot order='1'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>

    </controller>

    <controller type='sata' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pcie-root'/>

    <controller type='pci' index='1' model='dmi-to-pci-bridge'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>

    </controller>

    <controller type='pci' index='2' model='pci-bridge'>

      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:a6:a8:0e'/>

      <source bridge='br0'/>

      <model type='virtio'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <channel type='unix'>

      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Win81EntOVMF.org.qemu.guest_agent.0'/>

      <target type='virtio' name='org.qemu.guest_agent.0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>

      </source>

      <rom file='/mnt/user/VM/GTX970.rom'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>

      </source>

      <rom file='/mnt/user/VM/GTX970.rom'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>

    </hostdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>

    </memballoon>

  </devices>

</domain>

 

 

SeoBIOS:

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

  <name>Win81EntSeaBios</name>

  <uuid>616531db-ef1d-9315-fe46-5af0818e8e57</uuid>

  <metadata>

    <vmtemplate name="Custom" icon="windows.png" os="windows"/>

  </metadata>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>2097152</currentMemory>

  <memoryBacking>

    <nosharepages/>

    <locked/>

  </memoryBacking>

  <vcpu placement='static'>4</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='0'/>

    <vcpupin vcpu='1' cpuset='1'/>

    <vcpupin vcpu='2' cpuset='2'/>

    <vcpupin vcpu='3' cpuset='3'/>

  </cputune>

  <os>

    <type arch='x86_64' machine='pc-q35-2.3'>hvm</type>

  </os>

  <features>

    <acpi/>

    <apic/>

    <kvm>

      <hidden state='on'/>

    </kvm>

  </features>

  <cpu mode='host-passthrough'>

    <topology sockets='1' cores='4' threads='1'/>

  </cpu>

  <clock offset='localtime'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='raw' cache='writeback'/>

      <source file='/mnt/cache/VM/Win81EntSeaBios/vdisk1.img'/>

      <target dev='hdc' bus='virtio'/>

      <boot order='1'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <source file='/mnt/user/ISO/en_windows_8.1_enterprise_with_update_x64_dvd_6054382.iso'/>

      <target dev='hda' bus='sata'/>

      <readonly/>

      <boot order='2'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <source file='/mnt/user/ISO/virtio-win-0.1.96.iso'/>

      <target dev='hdb' bus='sata'/>

      <readonly/>

      <address type='drive' controller='0' bus='0' target='0' unit='1'/>

    </disk>

    <controller type='usb' index='0' model='ich9-ehci1'>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>

    </controller>

    <controller type='usb' index='0' model='ich9-uhci1'>

      <master startport='0'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>

    </controller>

    <controller type='sata' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pcie-root'/>

    <controller type='pci' index='1' model='dmi-to-pci-bridge'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>

    </controller>

    <controller type='pci' index='2' model='pci-bridge'>

      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:31:6d:60'/>

      <source bridge='br0'/>

      <model type='virtio'/>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <channel type='unix'>

      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Win81EntSeaBios.org.qemu.guest_agent.0'/>

      <target type='virtio' name='org.qemu.guest_agent.0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>

    </memballoon>

  </devices>

  <qemu:commandline>

    <qemu:arg value='-device'/>

    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=01:00.0,bus=pcie.0,multifunction=on,x-vga=on,romfile=/mnt/user/VM/GTX970.rom'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>

  </qemu:commandline>

</domain>

system-devices.jpg.89a019010de0dfca7adcae1203fb50cc.jpg

Link to comment

I can't help you with VMs and GPU issues, and what I'll comment on is probably completely unrelated, but I noticed 2 Marvell disk controllers listed, one being based on the 88SE9172 chipset.  I've just finished writing a Defect Report concerning that, and I'm curious if you are having problems with the drives attached to it.  Please see the Defect Report.  Would you be able to attach a full diagnostics zip?

Link to comment

I have almost exactly the same setup as you and everything works and is rock solid. It took a while to get there but at this point it works great so it should be able to work with your hardware. Please see my settings. I do see some differences. Hopefully something will make a difference.

 

line in Go script

 

/usr/local/sbin/vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:07:00.0

 

line in syslinix.cfg

append pcie_acs_override=downstream amd_iommu=on iommu=pt iommu=1 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

I found that the audio was not great until I switched windows over to using MSI interrupts for the audio and video controller on the GPU.

 

my xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>windows81nvidiaMSIRAW</name>
  <uuid>cc411d70-4463-4db7-bf36-d364c0cdaa3c</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <iothreads>6</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='7'/>
    <iothreadpin iothread='1' cpuset='2'/>
    <iothreadpin iothread='2' cpuset='3'/>
    <iothreadpin iothread='3' cpuset='4'/>
    <iothreadpin iothread='4' cpuset='5'/>
    <iothreadpin iothread='5' cpuset='6'/>
    <iothreadpin iothread='6' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hap/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='6' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup' track='guest'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='directsync' io='native'/>
      <source file='/mnt/disk/vmdisk/Image Media/Win8.1ProN.raw'/>
      <target dev='hda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/vmdisk/Images/en_windows_8_1_n_x64_dvd_2707896.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/vmdisk/Images/virtio-win-0.1-100.iso'/>
      <target dev='hdd' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:46:29:be'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.x-data-plane=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=07:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0'/>
  </qemu:commandline>
</domain>

 

Link to comment

Hopefully Jude's comments will help you, but a few things I noticed:

 

line in Go script

 

/usr/local/sbin/vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:07:00.0

 

Totally unnecessary.  We vfio-bind devices automatically now through the webGui.  This will not do anything to help you with pass through.  In fact, Jude can remove this from his go file as well.

 

my xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>windows81nvidiaMSIRAW</name>
  <uuid>cc411d70-4463-4db7-bf36-d364c0cdaa3c</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <iothreads>6</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='7'/>
    <iothreadpin iothread='1' cpuset='2'/>
    <iothreadpin iothread='2' cpuset='3'/>
    <iothreadpin iothread='3' cpuset='4'/>
    <iothreadpin iothread='4' cpuset='5'/>
    <iothreadpin iothread='5' cpuset='6'/>
    <iothreadpin iothread='6' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hap/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='6' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup' track='guest'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='directsync' io='native'/>
      <source file='/mnt/disk/vmdisk/Image Media/Win8.1ProN.raw'/>
      <target dev='hda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/vmdisk/Images/en_windows_8_1_n_x64_dvd_2707896.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/vmdisk/Images/virtio-win-0.1-100.iso'/>
      <target dev='hdd' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:46:29:be'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.x-data-plane=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=07:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0'/>
  </qemu:commandline>
</domain>

 

A few things from Jude's XML:

 

  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>

 

2.3 is the current version of QEMU we use, so you can adjust that from 2.1 in the above section.

 

    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.x-data-plane=on'/>

 

Above isn't needed anymore.  Didn't actually have the impact we wanted.

Link to comment

Hopefully Jude's comments will help you, but a few things I noticed:

 

line in Go script

 

/usr/local/sbin/vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:07:00.0

 

Totally unnecessary.  We vfio-bind devices automatically now through the webGui.  This will not do anything to help you with pass through.  In fact, Jude can remove this from his go file as well.

 

my xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>windows81nvidiaMSIRAW</name>
  <uuid>cc411d70-4463-4db7-bf36-d364c0cdaa3c</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <iothreads>6</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='7'/>
    <iothreadpin iothread='1' cpuset='2'/>
    <iothreadpin iothread='2' cpuset='3'/>
    <iothreadpin iothread='3' cpuset='4'/>
    <iothreadpin iothread='4' cpuset='5'/>
    <iothreadpin iothread='5' cpuset='6'/>
    <iothreadpin iothread='6' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hap/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='6' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup' track='guest'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='directsync' io='native'/>
      <source file='/mnt/disk/vmdisk/Image Media/Win8.1ProN.raw'/>
      <target dev='hda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/vmdisk/Images/en_windows_8_1_n_x64_dvd_2707896.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disk/vmdisk/Images/virtio-win-0.1-100.iso'/>
      <target dev='hdd' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:46:29:be'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.x-data-plane=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=07:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0'/>
  </qemu:commandline>
</domain>

 

A few things from Jude's XML:

 

  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>

 

2.3 is the current version of QEMU we use, so you can adjust that from 2.1 in the above section.

 

    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.x-data-plane=on'/>

 

Above isn't needed anymore.  Didn't actually have the impact we wanted.

 

Thanks for the input on my current settings. Everything has been working so well that I hadn't even thought to look for tweaks or updates. I will make those changes.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.