Need help with PCIe pass though of an Intel quad port pro/1000 nic


Aa617

Recommended Posts

Not sure what I am doing wrong but when I try to start the vm after editing the XML i get this error "Execution error Requested operation is not valid: PCI device 0000:04:00.0 is not assignable" I have PCIe ACS Override set to yes as without it half of my components are all in the same iommu group. With it set to yes everything is in a separate group, in fact the nic is in 2 groups (2 ports in each group). Hardware is an Asus z87 gryphon MB, xeon 1231v3, 16gb ram, gtx 960 in first x16 slot, intel card in second x16 slot, and gs8400 in third x16 (this was the only way I could get all three cards in while still being able to set unraid to use the 8400 in the uefi. XML for the pfsense vm i am trying to pass the card to is below... Please help before I pull all my hair out.

 

<domain type='kvm'>

  <name>Router</name>

  <uuid>3380bded-2ac3-9696-8712-8993f32d2a87</uuid>

  <metadata>

    <vmtemplate name="Custom" icon="freebsd.png" os="freebsd"/>

  </metadata>

  <memory unit='KiB'>1048576</memory>

  <currentMemory unit='KiB'>1048576</currentMemory>

  <memoryBacking>

    <nosharepages/>

    <locked/>

  </memoryBacking>

  <vcpu placement='static'>1</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='6'/>

  </cputune>

  <os>

    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <cpu mode='host-passthrough'>

    <topology sockets='1' cores='1' threads='1'/>

  </cpu>

  <clock offset='utc'>

    <timer name='rtc' tickpolicy='catchup'/>

    <timer name='pit' tickpolicy='delay'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='raw' cache='writeback'/>

      <source file='/mnt/user/VDisks/Router/vdisk1.img'/>

      <target dev='hdb' bus='virtio'/>

      <boot order='1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

    </disk>

    <disk type='file' device='cdrom'>

      <driver name='qemu' type='raw'/>

      <source file='/mnt/user/ISOs/pfSense-LiveCD-2.2.5-RELEASE-amd64-20151104-1549.iso'/>

      <target dev='hda' bus='ide'/>

      <readonly/>

      <boot order='2'/>

      <address type='drive' controller='0' bus='0' target='0' unit='0'/>

    </disk>

    <controller type='usb' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'/>

    <controller type='ide' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:13:00:b6'/>

      <source bridge='br0'/>

      <model type='virtio'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </interface>

    <serial type='pty'>

      <target port='0'/>

    </serial>

    <console type='pty'>

      <target type='serial' port='0'/>

    </console>

    <channel type='unix'>

      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Router.org.qemu.guest_agent.0'/>

      <target type='virtio' name='org.qemu.guest_agent.0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <input type='tablet' bus='usb'/>

    <input type='mouse' bus='ps2'/>

    <input type='keyboard' bus='ps2'/>

    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>

      <listen type='address' address='0.0.0.0'/>

    </graphics>

    <video>

      <model type='vmvga' vram='16384' heads='1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </video>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>

      </source>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>

    </hostdev>

    <memballoon model='virtio'>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </memballoon>

  </devices>

</domain>

 

Link to comment

Jon, again I can't say enough how much I appreciate your time and looking into this. I wanted to post an update. In hoping we would figure something out that would allow me to finish the pfSense vm, I went ahead and created my Windows vm. I was able to pass through my gtx 960, the on-board audio and a usb controller without any issues. While I am happy I was able to get this done it has left me more confused as to why I can not get the nic to pass through. 

Link to comment

Ok so another update, I have had some success. I have been able to pass through a single port on the nic with the following added to the xml

 

  <qemu:commandline>

    <qemu:arg value='-device'/>

    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=04:00.0,bus=root.1,addr=00.0'/>

  </qemu:commandline>

 

However when I try to add any more than one port I get this error "Execution error internal error: process exited while connecting to monitor: 2015-11-21T21:19:12.969032Z qemu-system-x86_64: -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1: PCI: single function device can't be populated in function 0.1  2015-11-21T21:19:12.969055Z qemu-system-x86_64: -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1: Device 'vfio-pci' could not be initialized"  below is the change in the xml when I receive this error

 

  <qemu:commandline>

    <qemu:arg value='-device'/>

    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=04:00.0,bus=root.1,addr=00.0'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=04:00.1,bus=root.1,addr=00.1'/>

  </qemu:commandline>

 

Any thought?

Link to comment

Final update (I hope)... I was finally able to pass through all four ports with the following

 

  <qemu:commandline>

    <qemu:arg value='-device'/>

    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=04:00.0,bus=root.1,addr=00.0'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=04:00.1,bus=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=05:00.0,bus=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=05:00.1,bus=root.1'/>

  </qemu:commandline>

 

Apparently even though they are separate devices since they are all on the same PCIe card only the first should have an "addr" assigned.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.