VM subsystem (libvirt) freezes Unraid when vbios is loaded.

[Gryman]

Hello, I have a newer install of Unraid (6.9.2) as I'm moving away from an older Ubuntu server.

 

ASRock X399 Phantom Gaming 6

AMD Threadripper 2950X

64GB RAM

Sapphire 5700XT Nitro+ 8GB

 

Everything is working great, except for when I try to do GPU passthrough on a VM. Currently, if I set a vbios for my GPU, it won't allow the VM to start or stop properly, and eventually I can no longer access any functionality related to the VM system. I have to do a hard reset of the server as the system will not gracefully shutdown with the VM still running as it cannot unmount media, etc, etc.

 

I've also noticed that now when I build a new VM, the VNC server doesn't seem to work anymore, even if the QXL adapter is the only video device assigned.

I've attached the diagnostic file and here is a video of the actually occurrence which is repeatable. I've tried two different vbios files both from https://www.techpowerup.com

 

 

zeus-diagnostics-20220226-0858.zip

[ghost82]

Quote

vfio-pci 0000:43:00.1: can't change power state from D0 to D3hot (config space inaccessible)
iommu ivhd1: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=43:00.0 address=0x1000e7d50]
[drm:amdgpu_atombios_init [amdgpu]] *ERROR* Failed to create device file for VBIOS version
amdgpu 0000:43:00.0: amdgpu: Fatal error during GPU init

1. All of these may suggest a crappy vbios, dump your own and use it (if needed).

 

2. You didn't isolate gpu video and gpu audio, attach them to vfio at boot.

 

Quote

Feb 26 08:32:26 Zeus kernel: pci 0000:43:00.0: BAR 0: assigned to efifb

3. You gpu video may be assigned to efifb even if isolated, add video=efifb:off to your syslinux config

 

Quote

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user0/Greg/Sapphire.RX5700XT.8192.190820.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

4. Multifunction is not set:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user0/Greg/vbiosToBeReplacedWithYourOwn.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>