I am having an issue when it comes to passing through both my gpu and usb controller into my windows vm.  i had been running both together until i recently went back to the XML and fixed my gpu.  i had been getting error 43 on the gpu driver because i reverted the original XML to make changes to the vm on the form view side of things. i forgot to go back and redo the xml side but once i fixed that, this new issue popped up smh.

 

when i try to run the vm passing through both the gpu and usb controller, my cpu 2 gets pinned to 100% and doesnt allow the vm to start.  I have to terminate the process for the core to go back to idle.  if i remove the usb controller from the vm, it starts perfectly fine. so something about the two together is causing the problem.  any ideas of what i can do here?

 

 

 

bus='0x04' is the USB controller i am wanting to pass through.

 

my XML for the vm:

Quote

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>417b850e-6264-70ec-8f38-5f2d51e2cba3</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>17301504</memory>
  <currentMemory unit='KiB'>17301504</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>12</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='10'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='12'/>
    <vcpupin vcpu='6' cpuset='5'/>
    <vcpupin vcpu='7' cpuset='13'/>
    <vcpupin vcpu='8' cpuset='6'/>
    <vcpupin vcpu='9' cpuset='14'/>
    <vcpupin vcpu='10' cpuset='7'/>
    <vcpupin vcpu='11' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/417b850e-6264-70ec-8f38-5f2d51e2cba3_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='6' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <serial>vdisk1</serial>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='52:54:00:c8:91:40'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/MSI.RTX3070.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x0800'/>
      </source>
      <boot order='2'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>
 

 

my log for the vm starting:

Quote

text  error  warn  system  array  login  

2024-04-04 09:21:08.004+0000: starting up libvirt version: 8.7.0, qemu version: 7.2.0, kernel: 6.1.79-Unraid, hostname: PCserver
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME='/var/lib/libvirt/qemu/domain-1-Windows 10' \
XDG_DATA_HOME='/var/lib/libvirt/qemu/domain-1-Windows 10/.local/share' \
XDG_CACHE_HOME='/var/lib/libvirt/qemu/domain-1-Windows 10/.cache' \
XDG_CONFIG_HOME='/var/lib/libvirt/qemu/domain-1-Windows 10/.config' \
/usr/local/sbin/qemu \
-name 'guest=Windows 10,debug-threads=on' \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-Windows 10/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/417b850e-6264-70ec-8f38-5f2d51e2cba3_VARS-pure-efi.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-i440fx-7.2,usb=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu host,migratable=on,host-cache-info=on,l3-cache=off \
-m 16896 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":17716740096}' \
-overcommit mem-lock=off \
-smp 12,sockets=1,dies=1,cores=6,threads=2 \
-uuid 417b850e-6264-70ec-8f38-5f2d51e2cba3 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=35,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pci.0","addr":"0x7"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x3"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Windows 10/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-1-format","id":"virtio-disk2","bootindex":1,"write-cache":"on","serial":"vdisk1"}' \
-netdev tap,fd=36,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:c8:91:40","bus":"pci.0","addr":"0x2"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=34,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.0","multifunction":true,"addr":"0x6","romfile":"/mnt/user/isos/MSI.RTX3070.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:01:00.1","id":"hostdev1","bus":"pci.0","addr":"0x6.0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:04:00.0","id":"hostdev2","bus":"pci.0","addr":"0x8"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/007","id":"hostdev3","bootindex":2,"bus":"usb.0","port":"2"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
2024-04-04T09:23:22.703177Z qemu-system-x86_64: terminating on signal 15 from pid 10033 (/usr/sbin/libvirtd)
2024-04-04 09:23:23.703+0000: shutting down, reason=destroyed
 

 

 

System Information

Model:Custom

M/B:Micro-Star International Co., Ltd. Z590 PRO WIFI (MS-7D09) Version 1.0 s/n 07D0910_L31E243506

BIOS:American Megatrends International, LLC. Version 1.90 Dated 06/06/2023

CPU:11th Gen Intel® Core™ i9-11900KF @ 3.50GHz

HVM:Enabled

IOMMU:Enabled

Cache:L1 Cache: 384 KiB, L1 Cache: 256 KiB, L2 Cache: 4 MiB, L3 Cache: 16 MiB

Memory:32 GiB DDR4 (max. installable capacity 64 GiB)

Network:eth0: 1000 Mbps, full duplex, mtu 1500

Kernel:Linux 6.1.79-Unraid x86_64

OpenSSL:1.1.1v

pcserver-diagnostics-20240404-0237.zip

i should have also mentioned that i recently also switched from UEFI mode to Legacy for boot as per spaceinvaders advice.  That helped with my GPU passthrough but maybe thats the cause of the problem here?

I don't think you have stubbed the 3070 and usb controller. You are missing check mark boxes.

 

 

is iomu enabled.?  

  

On 4/2/2024 at 12:26 AM, bmartino1 said:

you will not be able to use the unraid nvdia driver if you have it bound.

the card will need to be bound via iommu to use in a vm properly.

you will need to edit the xml config to fix thr graphic and audio of the g card.

see space invaders video

*this is the secret sauce fix... ^ Thank you Space Invader 

 

You will need to edit the xml near the bottom and fix your audio to be the next function and to stay on the correct bus.

Optional but helpful edits:
#################################

 

Main > flash drive > to add to your grub option:
video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off

 

We want to make sure the 3070 is fully set to vfio in use driver and that nothing touch it's FB to passit for full use in the VM. We also don't want unraid to control the power setting to it, the vm will handle that. This is why the advance eidt options...

example grub config that i run for my unraid same gpu:

Current Grub boot command:

kernel /bzimage
append initrd=/bzroot nvme_core.default_ps_max_latency_us=5500 default_hugepagesz=1G hugepagesz=1G pcie_acs_override=downstream,multifunction transparent_hugepage=always video=vesafb:off,efifb:off,simplefb:off,astdrmfb initcall_blacklist=sysfb_init pci=noaer pcie_aspm=off

 

I also recommend adding this to your vfio modprobe to assist with the GPU passing.
tools > system drivers > inuse drop down set to all

scroll to vfio.conf hit the pencil at the right and add this for your system:

*don't forget to save it at the right...
 

we want to edit this config and rebuild the modules and rebuild:

/etc/modprobe.d/vfio.conf

 

softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidiafb pre: vfio-pci
softdep nvidia_drm pre: vfio-pci
softdep drm pre: vfio-pci

options vfio-pci ids=(your vifds here...) disable_idle_d3=1 enable_sriov disable_denylist disable_vga=1

 

reboot
#########################

lspci -v will tell you what in use drivers there are running in you want to see vfio under the 3070 and audio device.
Fix your xml edits to add mutifunciton to the video portion of the Gcard and fix the bus to match the gcard audio function.

 

On 4/4/2024 at 11:16 AM, bmartino1 said:

Verifying IOMMU parameters

 

dmesg | grep -e DMAR -e IOMMU

Intel will show something similar:

 

There should be a line that looks like "DMAR: IOMMU enabled".

If there is no output, something is wrong.

 

Verify IOMMU interrupt remapping is enabled

 

It is not possible to use PCI passthrough without interrupt remapping. Device assignment will fail with 'Failed to assign device "[device name]": Operation not permitted' or 'Interrupt Remapping hardware not found, passing devices to unprivileged domains is insecure.'.

All systems using an Intel processor and chipset that have support for Intel Virtualization Technology for Directed I/O (VT-d), but do not have support for interrupt remapping will see such an error. Interrupt remapping support is provided in newer processors and chipsets (both AMD and Intel).

To identify if your system has support for interrupt remapping:

 

dmesg | grep 'remapping'

 

If you see one of the following lines:

AMD-Vi: Interrupt remapping enabled

DMAR-IR: Enabled IRQ remapping in x2apic mode ('x2apic' can be different on old CPUs, but should still work)

then remapping is supported.

If your system doesn't support interrupt remapping, you can allow unsafe interrupts
^enabled under unraid VM settings...

your xml looks good. I never had to use a g-card rom file. it helps but this can also break things if using worng bios file. From your pictures, You appear to have issues with your vifo configurations ad device stubbing.

please post your vfio log form system devices.

so this seems to have worked for me: removed the binding from the vfio and USB controller completely so it can be read by the unraid server system device. Remove the ticked box in the VM edit form view to make sure the usb is no longer connected. In the XML view is where I added the USB controller in the hostdev section. I noticed at this point i could get into the vm but that if things were plugged into the usb controller while starting it, there would be some pinning now and then.  i then switched back to UEFI mode just to test and everything worked straight away.

 

by making sure i had the full multifunction section of the XML in place for the GPU i no longer got errors within the VM.  by using UEFI mode again I no longer had pinning issues with the USB controller. I hope this helps others