Passing through AMD RX6700XT to Win 10 VM

[ouabain]

Hi all,

 

Hoping someone will be able to assist.

 

I have a unraid server with three graphics cards

- 1st slot has a nvidia GT710

- 2nd slot has a AMD RX6700XT

- 3rd slot has a nvidia 3080

 

I have successfully passed through the 3080 to a win10 VM with no issues.

 

I am struggling to pass through the 6700XT to a separate win10 VM. I normally connect to my VMs via RDP, and it works ok when the 6700XT is not passed through, but times out when the 6700XT is passed through. Interestingly, I have installed nomachine on the Win10 vm, and it shows up as 'available' on the network - but times out when I attempt to connect.

 

I have attached my diagnostics - the first is with the 6700XT not passed through, and the second has the 6700XT passed through.

 

Thanks.

odin-diagnostics-20220103-0816.zip odin-diagnostics-20220103-0842.zip

[ouabain]

Anyone?

[ghost82]

Apart suggesting setting the gpu as multifunction:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>

 

I would suggest to check your disks health because you have errors on ata.3 / sdb.

[ouabain]

4 hours ago, ghost82 said:

Apart suggesting setting the gpu as multifunction:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>

 

I would suggest to check your disks health because you have errors on ata.3 / sdb.

Thank you for your suggestion - it is much appreciated.

 

What does the multifunction option offer? - I tried it, but still no luck. Any suggestion on where to from here?

 

I have checked my disks - now passing all tests.

[ghost82]

3 hours ago, ouabain said:

What does the multifunction option offer?

A multifunction device is a device incorporating different functions, same bus, same slot, different functions.

GPU, ethernet cards, usb controllers, can be multifunction.

If a device has different portions of the same device in the same bus/slot and different functions it's a multifunction device.

It is good practice to have the topology of the vm reflecting that of the hardware on the host you want to passthrough.

 

But no idea why it isn't working.

Edited January 5, 2022 by ghost82

[ouabain]

So I have been doing more reading, and this led to the bright idea that I should try dumping my ROM to help with the passthrough.

 

I tried spaceinvader ones script etc - but it just simply doesn't work. I wonder if my card is a lemon.

 

Pity I don't have a spare box to do some more testing. That said - I will try and swap a different GPU in and see if I get any joy that way.

 

Thanks for your help.

On 1/5/2022 at 5:57 PM, ghost82 said:

Apart suggesting setting the gpu as multifunction:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>

 

I would suggest to check your disks health because you have errors on ata.3 / sdb.

[ouabain]

Argh.

 

Swapped a different GPU in - same problem, including the inability to dump bios.

 

I have a feeling I am missing something relatively minor here....

[ghost82]

did you try q35+ovmf instead of i440fx?

 

[ouabain]

1 hour ago, ghost82 said:

did you try q35+ovmf instead of i440fx?

 

Interesting question.

 

So - I swapped in a 5700 instead, and had all the previous issues.

 

I then watched @SpaceInvaderOne's video on how he passed through his 5700XT and made the following changes

- I tried q35+ovmf

- I passed through a bios that I downloaded from techpowerup.......

 

And it worked!

 

Managed to install drivers etc, and then rebooted.

 

Blank screen again!

 

I figured it was the AMD reset bug - so I tried @SpaceInvaderOne's script to reset the card without rebooting the server, and it got as far as putting the server to sleep.

 

I could not wake the server up, so had to reset which ended up causing errors on my parity drive (now rebuilding) and and caused a few issues with some of my other VMs, so am trying to recover those before trying again.

 

Any advice for the next attempt?

[ghost82]

3 hours ago, ouabain said:

I tried q35+ovmf

I think this was the solution, i440fx is quite old and it was optimized for pci passthrough, q35 is more recent and optimized for pcie passthrough. A newer gpu may have issues with i440fx.

 

3 hours ago, ouabain said:

I figured it was the AMD reset bug

I would update to unraid 6.10rc and use the amd reset bug plugin.