Ancalagon Posted April 26, 2023 Author Share Posted April 26, 2023 On 3/31/2023 at 4:06 PM, celborn said: When you say that you are "binding all 4 devices to vfio" are you referring to your equivalent of? @celborn sorry for the late reply. The bottom 2 of those 4 devices are correct. The top 2 are not though. You need the other 2 below that are on 04:00.2 and 04.00.3. This is mine: Quote Link to comment
Stixbnr32 Posted May 1, 2023 Share Posted May 1, 2023 (edited) I have been trying for a few hours now all the combos - my situation is i need the iGPU for transcodes and if i turn CSM on then i cant have the igpu in multi monitor mode bois wont even post. It takes away the "set primary GPU" option and then wont post. Why is it so difficult to passthrough an 7900xt? I just did a google looks like you need linux kernal 6.+ - looks like i need to upgrade from 6.11 to 6.12 rc4 To get my 7900xt working I had to do a few things. 1) Disable SAM (asrock z790) 2) Had to add vbios from Techpowerup 3) Update to 6.12rc4 for new Linux kernel (has amd support for 7000 series) 4) add to vm xml <features> <acpi/> <apic/> <hyperv mode='custom'> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='1234567890ab'/> </hyperv> <kvm> <hidden state='on'/> </kvm> Go to settings for your Windows VM, and enable XML view in the upper right corner. We will need to edit 2 blocks: add the following to <hyperv mode='custom'> block <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> add the following to <clock offset='localtime'> block <timer name='hypervclock' present='yes'/> Few things I set while testing but have not turned on/off yet 1) Not legacy boot as I need igpu to run in unraid as primary for transcodes 2) have turned on PCIE override to both 3) Allow unsafe interrupts I now get similar performance as my baremetal 5800x3d did. pretty happy with that. Edited May 3, 2023 by Stixbnr32 Quote Link to comment
MrChoke Posted June 1, 2023 Share Posted June 1, 2023 I'm struggling myself with a 7900xtx. I managed to get everything working with a 6900xt manually resized BAR and all but moving to the 7900xtx I can only get it to boot with CSM which, as far as I can tell, disables BAR resizing somewhat defeating the point of the card. I've even upgraded to 6.12-rc6 hoping that that might help but with no luck. Wondering if anyone has any suggestions on things to try. Quote Link to comment
shpitz461 Posted June 1, 2023 Share Posted June 1, 2023 (edited) Did you add the following params to your xml? Quote <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> At the end before the closing </domain> tag. The VM will not boot if ReBAR is enabled and these params are not added. Edited June 1, 2023 by shpitz461 Quote Link to comment
MrChoke Posted June 2, 2023 Share Posted June 2, 2023 On 6/1/2023 at 6:47 AM, shpitz461 said: Did you add the following params to your xml? At the end before the closing </domain> tag. The VM will not boot if ReBAR is enabled and these params are not added. No dice with rebar on or off after adding those params. Quote Link to comment
shpitz461 Posted June 3, 2023 Share Posted June 3, 2023 (edited) Ok, provide your full XML while VM is off. Also, provide the output from this command: Quote lspci -vvvs 85:00.0 Adjust 85:00.0 to your card (the addresses are listed with an 'lspci' command with no parameters). Edited June 3, 2023 by shpitz461 Quote Link to comment
MrChoke Posted June 3, 2023 Share Posted June 3, 2023 2 hours ago, shpitz461 said: Ok, provide your full XML while VM is off. Also, provide the output from this command: Adjust 85:00.0 to your card (the addresses are listed with an 'lspci' command with no parameters). Managed to get it working, not sure exactly what it was but I'm here now. Getting some weird freezing but I'll work on that on my own or start another thread. Thanks for the help. 1 Quote Link to comment
shpitz461 Posted June 3, 2023 Share Posted June 3, 2023 Would be great if you can post your full XML for future reference for other AMD GPU's. As for the freezing, maybe play with the cpu-features section. Also try with and without vbios in the XML. Also try to set the card to not multifuction and each piece of it on another pci slot in the xml. Or vice-versa: set the .0 piece of the card to multifunction='yes' and then set the .1 piece of the card on the pci slot. 1 Quote Link to comment
ghost82 Posted June 4, 2023 Share Posted June 4, 2023 (edited) 12 hours ago, shpitz461 said: Also try with and without vbios in the XML. Also try to set the card to not multifuction and each piece of it on another pci slot in the xml. Or vice-versa Please don't take it badly, I'm only trying to give proper info on this. You use vbios when it's needed, i.e. when the gpu that has to be passed is flagged as boot vga by the host; in all the other cases there's no need to pass a vbios; to check if it's flagged as "boot vga" one has to simply check for that string in the system log and check the corresponding address. This is because the video rom saved by the host it's not a 1:1 copy of the video rom of the gpu if it's flagged as boot vga; if it's not the boot vga the video rom is a 1:1 copy. As far as multifunction, one has to always set the gpu as a multifunction device: in bare metal hardware the gpu is a multifunction device so we emulate it as a multifunction device. Although the gpu can work if the gpu is not set as multifunction in the guest, in some cases, if it is not set as multifunction drivers can play badly. Moreover it's always strongly suggested to pass all the subdevices to the vm (video, audio, usb controller, etc.), for the same reason described above. +1 for checking cpu features. Edited June 4, 2023 by ghost82 Quote Link to comment
shpitz461 Posted June 4, 2023 Share Posted June 4, 2023 11 hours ago, ghost82 said: As far as multifunction, one has to always set the gpu as a multifunction device: in bare metal hardware the gpu is a multifunction device so we emulate it as a multifunction device. Although the gpu can work if the gpu is not set as multifunction in the guest, in some cases, if it is not set as multifunction drivers can play badly. Moreover it's always strongly suggested to pass all the subdevices to the vm (video, audio, usb controller, etc.), for the same reason described above. Odd, I've been running my VM with a 3090 for over a year now and I did not set it to multifunction: Quote <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x85' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/isos/vbios/HP.RTX.3090.v94.02.42.80.FE.dump.rom'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x85' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> I remember experimenting a long time ago and multifunction gave me issues, maybe I misconfigured it. I'll look at it later and see if it makes any difference, right now my Win11 VM is running very smooth. And criticism is always welcome, that's how we learn... 1 Quote Link to comment
Donach Posted November 26, 2023 Share Posted November 26, 2023 (edited) Hey all wanted to chime in with my experience... I had several issues over the years with GPU passthrough, eventually I lately settled on having working 3060Ti in 2nd slot (x8) PCI-E, which works flawlessly. However, my main goal was to allow two machines running at once, yet the first GPU, whatever I used, was having either reset bug, or Error 43... Now I've tested just today, and my VM resets and shutsdown-powers up flawlessly - using RX 6800. Here are all the settings used, hope I remember all: HW: R9-5900X AsRock Phantom Gaming 4S (X570 mobo) 64GB RAM (4 sticks) UEFI: Enabled 4G, Resize-Bar Booting from UEFI unRaid on USB stick Latest AsRock bios (as of 09/2023 or so). IOMMU, HVM etc on ofc unRaid: Version: 6.12.3 Enabled ACS Override = "both" VFIO allow unsafe interrupts: = "yes" Syslinux config: kernel /bzimage append video=efifb:off isolcpus=3-11,15-23 pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot Ofc make sure to bind EVERYTHING from the GPU to VFIO - RX 6800 has 4 devices on it: GPU, Sound, USB controller + USB serial bus (IDs 0d:00:0 -> 0d:00:3 for me...) VM config: Win10 - I actually didn't install "from scratch", I had an old image lying around with Win10 working from testing the RTX 3080 in 1st PCI-E slot - so I cannot say if this helped in any way... Q35v7.1 Bios: OVMF Hyper-V: YES USB: 3.0 (qemu XHCI) Make sure ALL FOUR devices are passed through to VM I did NOT do any manual edits, I wanna avoid them as much as possible so... To install the driver, I also used the VNC console, which I've now removed after installing Barrier to the VM. With this, I'm able to restart and/or shutdown and bring the VM back up, no issues. Haven't done performance testing yet, I'll compare the two VMs to see if there is any considerable difference (the 2nd VM has the 3060ti in 2nd PCIE slot). Edited November 26, 2023 by Donach Minor corrections and styling update Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.