6.11 GPU Passthrough working with issues

[sterling90]

I’m having an issue that’s stumping unraid support and myself. 

I can only do a GPU passthrough when there is no VFIO bind and no amdgpu blacklist. But when I shutdown the VM I will receive these errors:

 

Sep 28 17:06:29 Tower kernel: amdgpu

0000:03:00.0: amogpu: failed to read ip discovery binary from file

 

Sep 28 17:06:29 Tower kernel:

[drm:amdgpu_discovery_set_ip_blocks [amdgpu]]

*ERROR* amdgpu_discovery_init failed

 

Sep 28 17:06:29 Tower kernel: amdgpu

0000:03:00.0: amdgpu: Fatal error during GPU init

 

Sep 28 17:06:29 Tower kernel: amdgpu: probe of

0000:03:00.0 failed with error -22

 

if I did bind the VFIO or do the blacklist, I will get just a black screen. When I shutdown the VM no errors. 
 

I‘ll also get the header 127 reset bug.
 

Troubleshooting:

- VFIO GPU Bind

- PCI ACS Override - all the modes 

- Vbios in the vm settings. I was unable to do the Vbios Dump script. 
- blacklist AmdGPU

Passthrough only works with no VFIO bind and no blacklists 

 

Setup:

i5 12600k

Gigabyte z690 Elite AX DDR5

AMD 6600 XT

 

 

Diagnostic:

tower-diagnostics-20221002-1522-beforeVmStart 2.zip

tower-diagnostics-20221002-1524-VMShutdown 2.zip

 

System Devices:

Edited October 3, 2022 by sterling90

[ghost82]

Can you check the link? Cannot find anything.

[sterling90]

25 minutes ago, ghost82 said:

Can you check the link? Cannot find anything.

Which link?

 

this one? https://forums.unraid.net/support/10473/

[ghost82]

yes, it could be a private ticket which only you and the official support can see.

[sterling90]

10 minutes ago, ghost82 said:

yes, it could be a private ticket which only you and the official support can see.

Ah thanks for pointing that out. I’ve updated the entire post. 

[sterling90]

Bump for help 

[ghost82]

16 hours ago, sterling90 said:

I can only do a GPU passthrough when there is no VFIO bind and no amdgpu blacklist

Do it the right way:

1. bind audio and video to vfio

2. set gpu as multifunction and pass a rom file, your settings are wrong, change from this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
    </hostdev>

 

to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom file='/path/to/Powercolor.RX6600XT.8192.210701.rom'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>

 

Replace /path/to/Powercolor.RX6600XT.8192.210701.rom with the correct path. Check that your rx6600xt is the red devil version.

3. Reboot and try

----

If you still get errors like:

Refused to change power state from D0 to D3hot

Unable to change power state from D3cold to D0

 

you have something wrong in your bios configuration, or the bios is simply bugged: in this case I suggest to check the vfio group on reddit to see if anyone else has this issue for that motherboard and to contact Gigabyte directly, since your motherboard is still supported and you run the latest F7 available bios.

Powercolor.RX6600XT.8192.210701.rom

Edited October 4, 2022 by ghost82

[sterling90]

I will try this step. 
 

I have the fighter. But I’ll try the red devil. 
 

Also, I receive sound without multifunction. Adding multifunction is interesting. I will either have sound or not have sound. 
 

Edit:

 

I found the group and there was one post with my motherboard and they happened to have the close to the same setup. 
 

 

Looks like it’s the 12th gen intel cpu and the Amd gpus not playing together 

 

Edited October 4, 2022 by sterling90

[ghost82]

Use the attached vbios for the fighter version.

It seems another issue on that reddit post, yours seems like the reset bug, but it shouldn't be, never found a 6000 series gpu with that bug, and my concern is about the mb bios...

Powercolor.RX6600XT.8192.210701_1.rom

Edited October 4, 2022 by ghost82

[sterling90]

If is from TechPower Up. I’ve been using this rom. It works with it and without it. 
 

Also, same for the multifunction xml change. Actually works better without it. 
 

so the first time I got the motherboard I tried the Q-Flash with the F7 Bios.  
 

I’ve also tried it with the the normal way of using a USB and going through the bios motion. 
 

What I’ve been told is that the reset bug is not present within the 6000 series. 
 

But a person(s) has seen an issue with the 6000 Series and the intel 12 motherboard. 
 

if it’s a bios issue. Are you thinking it could be a bios issue or a motherboard issue?

Edited October 4, 2022 by sterling90

[sterling90]

Just tried a 6800XT and it’s not working either. I don’t know what to do.

 

Feels like I’m forced to use an Nivida card and I don’t know if that will work. 

[ghost82]

9 hours ago, sterling90 said:

Are you thinking it could be a bios issue or a motherboard issue?

Bios config or bugged bios.

[sterling90]

10 hours ago, ghost82 said:

Bios config or bugged bios.

I feel like I can make the assumption that my motherboard is too new for virtualization and gigabyte has not made an update to

handle it. 
 

For the configurations 

 

VT-d - enabled 

internal graphics - enabled 

Graphics - I set it to the iGPU

Resizable Bar - disabled 

Above 4k Decoding - enabled 

CSM - disabled 

C state - disabled 

 

 

The system time is finally right. I I accidentally figured out how to lock the time and date. 
 

did I miss anythin?

 

Edited October 5, 2022 by sterling90