I think I'm close, but possibly not modifying something correctly.   The first problem is when I boot unraid it still tries to use the onboard GPU and I get output while booting.  So I'm unsure how to disable that.  Currently my bios is set to CSM mode(I read that helps), but that only changed it from outputting in native 4k to lower res.

Second, I think I'm not setting the multifunction thing and IOMMU group correctly.  Here is what my IOMMU config looks like for my 3090(I have the 2 devices selected and clicked the "Bind selected to VFIO at boot" and restarted):

 

 

So with that I go in to my VM settings and update the following(before modification):

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk1/isos/GA102.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>

 

to:
 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk1/isos/GA102.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>

 

 

But after that change it still doesn't work. Note: I only change the slot from 0x06 to 0x05 to match the GPU.  

 

 

Before all of this I installed the OS via VNC mode, and then at one point on a reboot it actually booted in to windows since I installed the OS directly on one of my NVME's, so I took that opportunity to install GPUZ and dump my vbios.  I also tried using a downloaded vbios and removing the needed section....but I can't tell if I did it correctly/that didn't work either.

 

 

 

So to recap: 

• How to I edit my unraid config to disable it from using the GPU on boot up if that is an issue?
• Am I editing my xml incorrectly above?
• Did I hurt something by letting the the VM boot directly from the PC outside of KVM?

13 hours ago, 00100100 said:

How to I edit my unraid config to disable it from using the GPU on boot up if that is an issue?

It's normal that you have some video output when unraid boots, vfio attaches after.

 

13 hours ago, 00100100 said:

Am I editing my xml incorrectly above?

Yes, multifunction is applied correctly

 

13 hours ago, 00100100 said:

Did I hurt something by letting the the VM boot directly from the PC outside of KVM?

You didn't

 

Attach diagnostics and the vbios file you are using. Note that if you dump the vbios using gpuz you still needs to remove the header.