jwmxyz

Going to mark this as solved. I have managed to get it working. Whilst I'm not 100% sure on the how, the following two things (along with the above) seem to have solved the problem: Using VNC as the display driver, boot the VM and install the latest GPU Drivers. Once installed stop the VM. Before rebooting the VM remove the Graphics ROM BIOS. Start the VM and success.

Im trying to run two VMs on both of my GPUS. Both GPUS are identical. Specs: --- Current setup (from googling/searching forums): - Binded both IOMMU GPU groups - Tried removing video rending when starting up UnRaid --- - Using a VBios for GPU with header removed (N.B - Confirmed this works when using secondary GPU - same .rom) --- Current VM XML: N.B - When changing and pointing to the second GPU it works fine! <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='6'> <name>Windows 10</name> <uuid>049628b9-ad3e-4be4-d25f-53f1cf93c543</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='0'/> <vcpupin vcpu='1' cpuset='16'/> <vcpupin vcpu='2' cpuset='1'/> <vcpupin vcpu='3' cpuset='17'/> <vcpupin vcpu='4' cpuset='2'/> <vcpupin vcpu='5' cpuset='18'/> <vcpupin vcpu='6' cpuset='3'/> <vcpupin vcpu='7' cpuset='19'/> <vcpupin vcpu='8' cpuset='4'/> <vcpupin vcpu='9' cpuset='20'/> <vcpupin vcpu='10' cpuset='5'/> <vcpupin vcpu='11' cpuset='21'/> <vcpupin vcpu='12' cpuset='6'/> <vcpupin vcpu='13' cpuset='22'/> <vcpupin vcpu='14' cpuset='7'/> <vcpupin vcpu='15' cpuset='23'/> </cputune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/049628b9-ad3e-4be4-d25f-53f1cf93c543_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='8' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/nvme0n1' index='2'/> <backingStore/> <target dev='hdc' bus='sata'/> <boot order='1'/> <alias name='sata0-0-2'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/sdc' index='1'/> <backingStore/> <target dev='hdd' bus='sata'/> <alias name='sata0-0-3'/> <address type='drive' controller='0' bus='0' target='0' unit='3'/> </disk> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='sata' index='0'> <alias name='sata0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <alias name='usb'/> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <alias name='usb'/> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <alias name='usb'/> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:d7:01:27'/> <source bridge='virbr0'/> <target dev='vnet5'/> <model type='virtio-net'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty'> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-6-Windows 10/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='5'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <audio id='1' type='none'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/disk1/isos/x570dump.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x046d'/> <product id='0xc534'/> <address bus='5' device='2'/> </source> <alias name='hostdev2'/> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0810'/> <product id='0x0003'/> <address bus='3' device='2'/> </source> <alias name='hostdev3'/> <address type='usb' bus='0' port='2'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0b05'/> <product id='0x18f3'/> <address bus='3' device='4'/> </source> <alias name='hostdev4'/> <address type='usb' bus='0' port='3'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0e8d'/> <product id='0x0608'/> <address bus='1' device='3'/> </source> <alias name='hostdev5'/> <address type='usb' bus='0' port='4'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain> Diagnostics: tower-diagnostics-20230105-1225.zip My current thought process is UnRaid is using the GPU from boot - but wont free it up. - With a HDMI plugged into the primary GPU on boot I see unraid loading along with all the startup console dumps. -- When trying to start a VM on the primary GPU the screen goes black saying there is no input source UPDATED: - Here are the logs for a successful boot: boot_OK - Here are the logs for a failed boot: boot_FAIL Seems to get stuck after: Jan 5 14:24:23 Tower kernel: virbr0: port 1(vnet4) entered forwarding state Jan 5 14:24:23 Tower kernel: virbr0: topology change detected, propagating Any help appreciated.