Wes1

Yeah, this might be the crux of the issue. Which sucks, because after the 40 series nvidia and potentially higher Vram on all cards, we'll be running into this issue with any board that doesn't have Rebar support when trying to passthrough the gpu.

6.11.5 With that patch kernel to allow rebar.

Yes, that is part of the xml for the VM I have running, still didn't resolve it. Looks like it executes it fine in the logs, didn't see any errors.

Wasn't sure what the specific item being referenced was. I assumed it was the block below, is that not the case? <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline>

Currently having issues getting an Nvidia P40 (24GB vram) to work on windows 10 pro vm, device manager is reporting Code 12 (This device cannot find enough free resources that it can use). I'm looking to use the card for OobaBooga and Stable diffusion. Vram being king right now and the P40 having 24GB, it is ideal since it's cost is 1/8 of a 3090 or 4090 that also have 24GB Vram. Though i'm sure i'd have issues with either card right now given the Vram size is larger than the average right now. I've supplied the steps I've tried and resource information. If you have any ideas let me know. Host Specs. Motherboard: Supermicro X10DRG-Q Version 3.2, BIOS dated: Fri 22 Nov 2019 12:00:00 AM PST (Latest bios) Processor: 2x Intel® Xeon® CPU E5-2699 v4 @ 2.20GHz, 22 c, 44 t Memory: 512 GB DDR4 PC-2133 ecc Based on research I've done in other posts that suggested the following: Host BIOS Enable ReBAR support Uncertain if my Bios allows this option. Pg 95 of the manual : https://www.supermicro.com/manuals/motherboard/C612/MNL-1677.pdf Only specifies Above 4G Decoding, but no specifics for ReBar like i've seen in other Bios. Host BIOS Enable 4G Decoding Enabled Enable & Boot Custom Kernel syslinux configuration (near beginning of this thread) Done, applied the patch found in post, by copying the new bzimage to the boot usb and rebooted unraid Boot Unraid in UEFI Mode Done VM must use UEFI BIOS Done VM must have the top line of XML from <domain type='kvm'> to: <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> VM must have added the following (after the </device> line, before the </domain> line): <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Settings > Vm Manager > PCIe ACS override : Both Unbound and Rebound P40 to "VFIO at boot." tools > devices > PCI Devices and IOMMU groups. Modifying resource 0 doesn't seem to work, I get permission errors even as root and the card being unbound. cd /sys/bus/pci/devices/0000:84:00.0/ echo 13 > resource0_resize XML of VM: LSPCI for P40 once bound to "VFIO at boot" Boot Log, doesn't show any errors with the resize: A few posts I've looked through: