1650 super passthrough error on dl380p g8

BinglesWorth · June 10, 2021

Hey guys, i am a total noob to unraid and can't make heads or tails of this. please try to specific, as im not sure how to export logs and such, when asking for more info; as i am sure we will need it.

When passing through my graphics card to unraid i get the following error:

Execution error

internal error: qemu unexpectedly closed the monitor: 2021-06-10T12:00:26.146517Z qemu-system-x86_64: -device vfio-pci,host=0000:24:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio 0000:24:00.0: failed to setup container for group 78: Failed to set iommu for container: Operation not permitted

i have checked the group 78, in the system devices to bind at vfio, as i was getting an error for that initially. i am passing through audio and video as well as selecting the check boxes at the bottom for the usb options.

BinglesWorth · June 26, 2021

Is there a way I can put a bounty in this? Don't know what's fair but I've got 50$-75$ to whoever can fix it.

ghost82 · June 26, 2021

Please attach diagnostics or vm xml and iommu groups.

Diagnostics:

Tools tab, click on the Diagnostics icon, then click on the Download button. After the diagnostic data collection is complete, it will save a diagnostics zip file to your computer, to the download location you specify or is configured in your browser

NB: not interested in the bounty, just interested to fix issues

Edited June 26, 2021 by ghost82

BinglesWorth · June 26, 2021

Here you are, and bounty or no, ill find a way to pay you back if we can get it working. I dont want to have to try freenas since i already bought unraid lol.

image.png.3342c079a25975ef3655eca86f9d4f0d.png

unraid-diagnostics-20210626-1156.zip vm xml.txt vfio-pci log.txt iommu groups.txt

Edited June 26, 2021 by BinglesWorth
added photo of error message

ghost82 · June 27, 2021

15 hours ago, BinglesWorth said:

Here you are

Some edits to your xml, put the components of the gpu in a single multifunction device, copy and replace:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>10b4697d-edcf-d7e0-6700-705388f94d3b</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>20971520</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='16'/>
    <vcpupin vcpu='1' cpuset='36'/>
    <vcpupin vcpu='2' cpuset='17'/>
    <vcpupin vcpu='3' cpuset='37'/>
    <vcpupin vcpu='4' cpuset='18'/>
    <vcpupin vcpu='5' cpuset='38'/>
    <vcpupin vcpu='6' cpuset='19'/>
    <vcpupin vcpu='7' cpuset='39'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/10b4697d-edcf-d7e0-6700-705388f94d3b_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disk1/Windows/Windows 10/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows10.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.185.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:74:21:ee'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='2'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

Make sure you enabled virtualization in the bios:

https://support.hpe.com/hpesc/public/docDisplay?docId=mmr_kc-0121549

Make sure that in unraid system information you have both IOMMU and HVM set to "enabled".

Check that all the 4 components of the gpu have a check for vfio (vga, audio and the 2 controllers).

If it still doesn't work, try with acs override:

Settings -> vm manager -> switch to "advanced view" (top right) -> pcie acs override --> set it to both

Edited June 27, 2021 by ghost82

ghost82 · June 27, 2021

16 hours ago, BinglesWorth said:

Here you are

In your syslog you have also a couple of these entries:

Jun 26 09:55:59 UnRaid kernel: vfio-pci 0000:21:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

21:00.1 is the audio part of the gpu.

Here:

@1812 solved the same(?) issue by not passing through that audio part. NOTE: you need to use acs override to split your iommu group 78 to have the audio component in its iommu group.

And you may need vfio_iommu_type1.allow_unsafe_interrupts=1 added to your syslinux config in unraid.

If you want to give it a try, copy and replace:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>10b4697d-edcf-d7e0-6700-705388f94d3b</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>20971520</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='16'/>
    <vcpupin vcpu='1' cpuset='36'/>
    <vcpupin vcpu='2' cpuset='17'/>
    <vcpupin vcpu='3' cpuset='37'/>
    <vcpupin vcpu='4' cpuset='18'/>
    <vcpupin vcpu='5' cpuset='38'/>
    <vcpupin vcpu='6' cpuset='19'/>
    <vcpupin vcpu='7' cpuset='39'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/10b4697d-edcf-d7e0-6700-705388f94d3b_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disk1/Windows/Windows 10/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows10.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.185.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:74:21:ee'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='2'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

Edited June 27, 2021 by ghost82

BinglesWorth · June 27, 2021

With the first config given the error goes away but the vm dosnt start. no errors though so i added the latest log. theres no change when i do the second config, i figure we can wait to fix the audio problem until we can get it to boot

unraid-diagnostics-20210627-0827.zip

ghost82 · June 27, 2021

14 minutes ago, BinglesWorth said:

With the first config given the error goes away but the vm dosnt start. no errors though so i added the latest log. theres no change when i do the second config, i figure we can wait to fix the audio problem until we can get it to boot

unraid-diagnostics-20210627-0827.zip 94.75 kB · 0 downloads

Can you please attach the iommu groups txt file as you did before?

From the diagnostic I cannot find a way to look at devices/groups.

Does the audio part 21:00.1 splits and it's alone in a iommu group?

Don't bind to vfio the audio part (I can still see it binded in syslinux config), because you have the same error in syslog as before:
Jun 27 06:22:55 UnRaid kernel: vfio-pci 0000:21:00.1: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement. Contact your platform vendor.

And obviously, don't pass the audio part to the vm.

It's not "an audio problem": the problem is with your mainboard which doesn't allow to passthrough the audio part of the gpu, so the not booting is related to this. Are you running the latest bios?

Edited June 27, 2021 by ghost82

BinglesWorth · June 27, 2021

Will do I misunderstood, I'll get that done in the next few hours and update you.

BinglesWorth · June 27, 2021

I finally got it to boot, drivers installed and I haven't tested anything yet but there are no issues. With Nvidia finally supporting virtualization on the 10 series and newer, we should be fine. Seems like I won't have audio but that's nothing a USB adapter can't fix.

Unless I'm mistaking somthing again, seems like my last step is go get an SSD cause my disk usage is a bit to high right now so I need somthing snappier.

I'm running sata drives on 6gb sas ports which is slowing them to 3gbs Sata and they are 8 year old drives lol.

2.6k · June 27, 2021

@BinglesWorth/ @ghost82

Correct, the workaround was to not use audio. But there is a better way now that patches out the RMRR issue:

I don't currently have any proliants, but this worked well and reliably when last I used it.

ghost82 · June 27, 2021

1 hour ago, 1812 said:

But there is a better way now that patches out the RMRR issue

uuu yes, I remember I read that and also pointed another user to that thread, thank you very much.

So, @BinglesWorth
you may want to follow this last thread with a better solution to this specific issue.

Anyway you already know that this workaround works, not the best solution but it works.

1650 super passthrough error on dl380p g8

Recommended Posts

BinglesWorth

Link to comment

BinglesWorth

Link to comment

ghost82

Link to comment

BinglesWorth

Link to comment

ghost82

Link to comment

ghost82

Link to comment

BinglesWorth

Link to comment

ghost82

Link to comment

BinglesWorth

Link to comment

BinglesWorth

Link to comment

1812

Link to comment

ghost82

Link to comment

Join the conversation