Jump to content

Cannot Run GPU and PCIE Sound Card Simultaneously

2SERFlex

By 2SERFlex
in VM Engine (KVM)

Recommended Posts

2SERFlex Noob

2SERFlex

Posted
Posted

We're currently trying to run 3 Virtual Machines for an audio playback system. We've been running into issues where one of the three virtual machines cannot boot with both PCIE Cards it needs.

 

Our configuration is on Unraid 6.11.1. We have three virtual machines. The first (VM1 from here) runs Windows 10, headless to operate our audio playback server. The second (VM2) runs Windows 10 as a user-operable workstation with an AMD 6500XT for video output and a Presonus USB Audio Interface. The third (VM3), our problem child, runs Windows 10, an Audio Science PCIE Sound Card, and should also be running with our second AMD 6500XT.

 

Whenever we go to boot VM3 with our desired configuration, it returns the following error: 

internal error: qemu unexpectedly closed the monitor: YYYY-MM-DDTHH:mm:SS.####### qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:0b:00.0","id":"hostdev3","bus":"pci.8","addr":"0x0"}: vfio 0000:0b:00.0: Failed to set up TRIGGER eventfd signaling for interrupt INTX-0: VFIO_DEVICE_SET_IRQS failure: Device or resource busy

 

Device 0b:00.0 is our Audio Science sound card. If we remove the AMD 6500XT or the Audio Science card from the VM3 Config, it boots perfectly fine and has been consistently stable, but we have not been able to get them both to run at the same time.

 

This currently means users are having to VNC into VM3 from VM2 to use our audio playback system.

 

The VM Config is as follows; 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>##############################</name>
  <uuid>##############################</uuid>
  <description>Wide Orbit Trobuleshooting</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='13'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/53256559-f8ed-4338-0e56-993bf476107f_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/cachea/vdisk1.qcow2'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/mnt/user/domains/vdisk2.img'/>
      <target dev='hdd' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.221-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='##:##:##:##:##:##'/>
      <source bridge='virbr0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x01' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <address bus='5' device='3'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

I've also attached the system diagnostics. My PCIE ACS Override Mode is Multifunction, and I'm binding the GPUs and Soundcard to VFIO at boot. Both GPUs are showing up with their video and audio components in separate IOMMU Groups. The sound card has two devices (0a:00.0 and 0b:00.0) both in IOMMU Group 25, and these don't get separated in any ACS Override Mode.

 

Is there something in this VM config I can adjust to get both PCIE devices to run? Any other suggestions on how to get both PCIE devices running at the same time?

2serdiagnostics-20230411-1215.zip

Link to comment
ghost82 Proficient

ghost82

Posted
Posted (edited)

Your issue is with irq conflicts:
  

Apr 10 19:13:28 2SERMQ0138 kernel: genirq: Flags mismatch irq 129. 00000000 (vfio-intx(0000:0b:00.0)) vs. 00000080 (vfio-intx(0000:09:00.1))

 

where 0b:00.0 (audioscience) conflicts with 09:00.1 (audio part of gpu).

 

1. update the bios (if any is available) and see if the irq conflict disappears

2. change the phisical slot of the gpu or of the audioscience and see if the irq conflict disappears

3. check in your bios if you can manually assign irqs to devices (I don't think you have this option..)

4. if you don't need digital audio from the gpu, you can try to not pass the gpu audio to the vm

 

If you aren't able to solve with these suggestions I think there's nothing more you can do..

 

Edited by ghost82
Link to comment
2SERFlex Noob

2SERFlex

Posted
  • Author
Posted (edited)
On 4/11/2023 at 7:08 PM, ghost82 said:

4. if you don't need digital audio from the gpu, you can try to not pass the gpu audio to the vm

 

If you aren't able to solve with these suggestions I think there's nothing more you can do..

 

 

This has allowed me to boot with video from the GPU and sound from the ASI sound card, which was the desired result, but I then cannot restart VM3 without rebooting the entire server. This is because I'm not passing through the whole GPU so it isn't resetting properly, correct? Is there a workaround for this?

 

I should add - non-admin users interact with VM3 without supervision, so user scripts to reset the GPU such as this video wouldn't be viable, I believe.

Edited by 2SERFlex
further information
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...