Radeon RX 7900 XT passthrough

[Ancalagon]

On 3/31/2023 at 4:06 PM, celborn said:

When you say that you are "binding all 4 devices to vfio" are you referring to your equivalent of?

@celborn sorry for the late reply. The bottom 2 of those 4 devices are correct. The top 2 are not though. You need the other 2 below that are on 04:00.2 and 04.00.3. This is mine:

[Stixbnr32]

I have been trying for a few hours now all the combos - my situation is i need the iGPU for transcodes and if i turn CSM on then i cant have the igpu in multi monitor mode bois wont even post.  It takes away the "set primary GPU" option and then wont post.

 

Why is it so difficult to passthrough an 7900xt?

 

I just did a google looks like you need linux kernal 6.+ - looks like i need to upgrade from 6.11 to 6.12 rc4

 

 

To get my 7900xt working I had to do a few things.

 

1) Disable SAM (asrock z790)

2) Had to add vbios from Techpowerup

3) Update to 6.12rc4 for new Linux kernel (has amd support for 7000 series)

4) add to vm xml

 

  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>

 

Go to settings for your Windows VM, and enable XML view in the upper right corner. We will need to edit 2 blocks:

add the following to <hyperv mode='custom'> block

<vpindex state='on'/> <synic state='on'/> <stimer state='on'/>

add the following to <clock offset='localtime'> block

<timer name='hypervclock' present='yes'/>

 

Few things I set while testing but have not turned on/off yet

1) Not legacy boot as I need igpu to run in unraid as primary for transcodes

2) have turned on PCIE override to both

3) Allow unsafe interrupts 

 

I now get similar performance as my baremetal 5800x3d did.  pretty happy with that.

Edited May 3, 2023 by Stixbnr32

[MrChoke]

I'm struggling myself with a 7900xtx. I managed to get everything working with a 6900xt manually resized BAR and all but moving to the 7900xtx I can only get it to boot with CSM which, as far as I can tell, disables BAR resizing somewhat defeating the point of the card. I've even upgraded to 6.12-rc6 hoping that that might help but with no luck. Wondering if anyone has any suggestions on things to try.

[shpitz461]

Did you add the following params to your xml?

Quote

  <qemu:commandline>
    <qemu:arg value='-fw_cfg'/>
    <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/>
  </qemu:commandline>

At the end before the closing </domain> tag.

The VM will not boot if ReBAR is enabled and these params are not added.

Edited June 1, 2023 by shpitz461

[MrChoke]

On 6/1/2023 at 6:47 AM, shpitz461 said:

Did you add the following params to your xml?

At the end before the closing </domain> tag.

The VM will not boot if ReBAR is enabled and these params are not added.

No dice with rebar on or off after adding those params.

[shpitz461]

Ok, provide your full XML while VM is off.

 

Also, provide the output from this command:

 

Quote

lspci -vvvs 85:00.0

 

Adjust 85:00.0 to your card (the addresses are listed with an 'lspci' command with no parameters).

Edited June 3, 2023 by shpitz461

[MrChoke]

2 hours ago, shpitz461 said:

Ok, provide your full XML while VM is off.

 

Also, provide the output from this command:

 

 

Adjust 85:00.0 to your card (the addresses are listed with an 'lspci' command with no parameters).

Managed to get it working, not sure exactly what it was but I'm here now. Getting some weird freezing but I'll work on that on my own or start another thread. Thanks for the help.

[shpitz461]

Would be great if you can post your full XML for future reference for other AMD GPU's.

As for the freezing, maybe play with the cpu-features section.

Also try with and without vbios in the XML.

Also try to set the card to not multifuction and each piece of it on another pci slot in the xml. Or vice-versa: set the .0 piece of the card to multifunction='yes' and then set the .1 piece of the card on the pci slot.

[ghost82]

12 hours ago, shpitz461 said:

Also try with and without vbios in the XML.

Also try to set the card to not multifuction and each piece of it on another pci slot in the xml. Or vice-versa

Please don't take it badly, I'm only trying to give proper info on this.

You use vbios when it's needed, i.e. when the gpu that has to be passed is flagged as boot vga by the host; in all the other cases there's no need to pass a vbios; to check if it's flagged as "boot vga" one has to simply check for that string in the system log and check the corresponding address.

This is because the video rom saved by the host it's not a 1:1 copy of the video rom of the gpu if it's flagged as boot vga; if it's not the boot vga the video rom is a 1:1 copy.

As far as multifunction, one has to always set the gpu as a multifunction device: in bare metal hardware the gpu is a multifunction device so we emulate it as a multifunction device. Although the gpu can work if the gpu is not set as multifunction in the guest, in some cases, if it is not set as multifunction drivers can play badly.

Moreover it's always strongly suggested to pass all the subdevices to the vm (video, audio, usb controller, etc.), for the same reason described above.

 

+1 for checking cpu features.

Edited June 4, 2023 by ghost82

[shpitz461]

11 hours ago, ghost82 said:

As far as multifunction, one has to always set the gpu as a multifunction device: in bare metal hardware the gpu is a multifunction device so we emulate it as a multifunction device. Although the gpu can work if the gpu is not set as multifunction in the guest, in some cases, if it is not set as multifunction drivers can play badly.

Moreover it's always strongly suggested to pass all the subdevices to the vm (video, audio, usb controller, etc.), for the same reason described above.

Odd, I've been running my VM with a 3090 for over a year now and I did not set it to multifunction:

Quote

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x85' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/isos/vbios/HP.RTX.3090.v94.02.42.80.FE.dump.rom'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x85' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

I remember experimenting a long time ago and multifunction gave me issues, maybe I misconfigured it.

I'll look at it later and see if it makes any difference, right now my Win11 VM is running very smooth.

And criticism is always welcome, that's how we learn...

[Donach]

Hey all wanted to chime in with my experience...
I had several issues over the years with GPU passthrough, eventually I lately settled on having working 3060Ti in 2nd slot (x8) PCI-E, which works flawlessly.

However, my main goal was to allow two machines running at once, yet the first GPU, whatever I used, was having either reset bug, or Error 43...

Now I've tested just today, and my VM resets and shutsdown-powers up flawlessly - using RX 6800.

Here are all the settings used, hope I remember all:

HW:

R9-5900X

AsRock Phantom Gaming 4S (X570 mobo)

64GB RAM (4 sticks)

 

 

UEFI:
Enabled 4G, Resize-Bar

Booting from UEFI unRaid on USB stick

Latest AsRock bios (as of 09/2023 or so).

IOMMU, HVM etc on ofc

unRaid: 
Version: 6.12.3
Enabled ACS Override = "both"
VFIO allow unsafe interrupts: = "yes"
Syslinux config:
kernel /bzimage
append video=efifb:off isolcpus=3-11,15-23 pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

Ofc make sure to bind EVERYTHING from the GPU to VFIO - RX 6800 has 4 devices on it: GPU, Sound, USB controller + USB serial bus
(IDs 0d:00:0 -> 0d:00:3 for me...)

VM config:
Win10 - I actually didn't install "from scratch", I had an old image lying around with Win10 working from testing the RTX 3080 in 1st PCI-E slot - so I cannot say if this helped in any way...
Q35v7.1
Bios: OVMF
Hyper-V: YES
USB: 3.0 (qemu XHCI)

Make sure ALL FOUR devices are passed through to VM

I did NOT do any manual edits, I wanna avoid them as much as possible so...

 

To install the driver, I also used the VNC console, which I've now removed after installing Barrier to the VM.

 

With this, I'm able to restart and/or shutdown and bring the VM back up, no issues.
Haven't done performance testing yet, I'll compare the two VMs to see if there is any considerable difference (the 2nd VM has the 3060ti in 2nd PCIE slot).

Edited November 26, 2023 by Donach
Minor corrections and styling update