6500XT Passthrough on X99

[Evedoescomputerstuff]

Alright,

I've been trying for 2 days now to get my 6500XT passed through to VMs successfully (and reliably).

I've tried various fixes like multifunction, hiding the relevant devices from VFIO, etc. Tried 6.9 and now 6.10 . I've been able to get video out of the GPU to boot specific OSes (Debian 11 worked for example, but no hardware acceleration) and occasionally I've been able to get it to show up in LSPCI in said VM. I've dumped the bios on an actual windows machine as well, the card does indeed work.

I figured okay, Ubuntu LTS has AMD driver support so I'll try that. VM boots, and gives me ubuntu's boot options (through the GPU's video out), however hitting "try / install" causes a VFIO error which leaves the screen on "Tianocore" (the last frame is always left hanging on the GPU's output). Doing safemode, I obviously get further but that isn't going to really help my situation as the notification I get as safemode launches is that the AMD drivers won't be loaded.

 

 

qemu-system-x86_64: vfio_err_notifier_handler(0000:05:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest

qemu-system-x86_64: vfio_err_notifier_handler(0000:05:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest

 

 

What else can I try? I can post VM configs but I've tried a bunch of different ones by this point.

 

Also, curiously, there's 4 relevant devices for my GPU in VFIO, but only 2 have little check boxes. I did attempt to add the remaining 2 manually (following the style of the 2 which I was able to enable via the GUI) to my vfio config file but it didn't make a difference. First 2 here, the "Upstream" and "Downstream" have no checkbox.

 

IOMMU group 31:[1002:1478] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1)

IOMMU group 32:[1002:1479] 04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch

IOMMU group 33: [1002:743f] 05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 24 [Radeon RX 6400 / 6500 XT] (rev c1)

IOMMU group 34: [1002:ab28] 05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller

 

 

Is it possible I need to write the xml config for the VM to include the first 2 devices? How would that look?

X99 Jingyue board if that helps. 12c V3 Xeon. This isn't my primary GPU (I have an old AMD card in there and it seems to only output console to that, not the 6500xt so I believe Unraid's OS is correctly preferring the crappy card)

[ghost82]

I would try to enable allow unsafe interrupts in unraid settings (reboot after change).

If this doesn't work I would try kernel parameter pci=noaer (reboot after change).

 

Do not make any manual changes, you should pass only 05:00.0 and .1, do not passthrough pci bridges, set 05:00.0 and .1 in a multifunction device.

Edited September 16, 2022 by ghost82

[Evedoescomputerstuff]

8 hours ago, ghost82 said:

I would try to enable allow unsafe interrupts in unraid settings (reboot after change).

If this doesn't work I would try kernel parameter pci=noaer (reboot after change).

 

Do not make any manual changes, you should pass only 05:00.0 and .1, do not passthrough pci bridges, set 05:00.0 and .1 in a multifunction device.

I should have mentioned unsafe interrupts, already have those on. I haven't tried pci=noaer though. 

[Evedoescomputerstuff]

10 hours ago, ghost82 said:

I would try to enable allow unsafe interrupts in unraid settings (reboot after change).

If this doesn't work I would try kernel parameter pci=noaer (reboot after change).

 

Do not make any manual changes, you should pass only 05:00.0 and .1, do not passthrough pci bridges, set 05:00.0 and .1 in a multifunction device.

pci=noaer works *more*, the 6500xt has a problem on Ubuntu but I'm getting closer. 

https://gitlab.freedesktop.org/drm/amd/-/issues/1933

Displays the exact error as this. Working on it now to see what I can do on a newer kernel / mesa version 

 

I assume this is beyond the Unraid community's help but do let me know if there's another option.

 

[Evedoescomputerstuff]

Kernel 5.19.3 + Mesa drivers from ppa:oibaf/graphics-drivers seems to work with the newest ubuntu LTS at least provisionally. Haven't tested anything heavy but my screen works and apparently I have hw accel. 

 

If you're trying to get this to work, try this. Install Ubuntu in GPU safe mode with just the VNC passed. Get it installed and do the kernel upgrade + mesa drivers. Then pass your GPU + HDMI audio, manually adding in multifunction. As a bonus, edit grub to allow the menu display by default, this will give you an easy road back in to 'recovery' / cli mode if a change breaks your GPU. 

 

Can't get both mouse and keyboard passed (either by doing the GUI config, and then re-editing the xml) and ALSO have display show up no idea why? I get conflicts over ports and stuff. Mouse and Keyboard sucessfully passed to other VMs however. It's not a *massive* deal since I don't plan on having physical HID devices connected in the long run. 

 

RealVNC server is my choice for control with a passed through GPU. Install it, then "libcanberra-gtk-module" (for some reason it's missing for me after these steps) , then configure it, then disable Wayland. Wayland presents major issues for RealVNC and on the KB+M control front (like synergy / barrier) there isn't a decent option yet.

 

EDIT: Correct Kernel is 5.19.3 sorry for the confusion. https://github.com/pimlie/ubuntu-mainline-kernel.sh is what I used. You're probably better to do it yourself but this worked for me.

 

Uniengine benchmarking works. GPU still shows up as a non specific AMD device in lshw but presents correctly to software I guess? 

 

Driver: amdgpu OpenGL 4.6 

OS: Ubuntu 22.04.1 LTS 

Kernel: 5.19.3-051903-generic 

 

 

 

 

Edited September 16, 2022 by Evedoescomputerstuff
corrected kernel version #