AMD APU Ryzen 5700G iGPU Passthrough on 6.9.2


Recommended Posts

  • 2 weeks later...
On 9/1/2022 at 3:58 PM, Name_0901 said:

 

I was successful on PVE 7.2. The iGPU can be passthrough in both Windows 10 and Ubuntu 20.04 VM, and works well.

Hi, have you done anything special to passthrough the igpu  under ubuntu? 

For me, I can only boot it with modeset parameter otherwise the screen will turn off and in dmesg I will see amdgpu giving errors. Due to modeset there is no video acceleration, so it is not usable.

 

Thanks

Link to comment
16 hours ago, thomas said:

Hi, have you done anything special to passthrough the igpu  under ubuntu? 

For me, I can only boot it with modeset parameter otherwise the screen will turn off and in dmesg I will see amdgpu giving errors. Due to modeset there is no video acceleration, so it is not usable.

 

Thanks

Try reboot your PVE host before do the iGPU passthrough. This is because the AMD GPU reset bug, you can only use the iGPU in passthrough once per host startup. If you want to use it next time, or your VM rebooted, you need to restart your host machine to re-active the iGPU.

 

wx-Rmt posted a solution, you can take a look here at page 7. But since I'm a C noob, there's nothing I can do to rewrite the patch.

 

You also can googling "AMD GPU/iGPU reset bug" for more details.

Edited by Name_0901
Link to comment
On 9/16/2022 at 2:34 PM, Name_0901 said:

Try reboot your PVE host before do the iGPU passthrough. This is because the AMD GPU reset bug, you can only use the iGPU in passthrough once per host startup. If you want to use it next time, or your VM rebooted, you need to restart your host machine to re-active the iGPU.

 

Thanks. Unfortunately for me Ubuntu or any other linux is not working. Windows is always fine with the RadeonReset service from https://github.com/inga-lovinde/RadeonResetBugFix

For context, I'm using an Asrock deskmini X300W with proxmox.

Link to comment

Nothing new I guess?

Read through the whole Thread and tried everything.
VBIOS extraction worked for my AsRock A520M-HVS.

No matter what I do, it always ends with 

 

qemu-system-x86_64: -device vfio-pci,host=0000:30:00.0,id=hostdev0,bus=pci.4,multifunction=on,addr=0x0,romfile=/mnt/cache/test12345/vbios_1638.rom: Failed to mmap 0000:30:00.0 BAR 0. Performance may be slow


In the VM-Log and it stays on the Unraid Login screen...

Attached my VBIOS if it may help anyone.

vbios_1638.rom

 

 

If I use VNC and HD Audio I get sound over HDMI, that works.

Edited by ven0m
Link to comment
  • 2 weeks later...

I thought I would provide a quick update.  I ditched my AMD rig; 5700G + 5500XT, and replaced it with Intel i5 12400 + 1050ti.  Plex natively supports intel apu and it worked without any drivers or effort at all, just pass device and turn it on.  It seems plenty powerful.  I also was able to pass my 1050ti into a VM with zero issues, no need to extract BIOS or anythign fancy.  I even have it working with UNRAID 6.11 and Windows 11 with TPM.  It was crazy how easy this just worked, with all the pain I had with AMD rig.  The AMD stuff was more power efficient, but honestly worth the trade off.  I know this isn't overly helpful here, but know that this AMD + APU stuff is probably just not even worth the effort.  The performance was not good, and overall nothing just works on this platform.  Happy unraider now unsubscribing.

Link to comment
  • 2 months later...

Just wanted to thank Tanne for the guide. By following it I managed to get my 5600G iGPU passed through successfully.  Additionally to all the points listed here, I had to change a few things (which took the best part of a day to figure out, so here they are for everyone else!):

  • In your BIOS
    • Make sure that the iGPU is enabled as the default (my ASRock mobo had external GPU as default.  But I have no external GPU!).
    • I disabled CSM as well - no idea if it was needed or not, but didn't seem to hurt.
  • I think the notation used in step 6.3 might be obsolete.  I was getting error messages about "hostdev0" not being a valid device.  I had to change that whole section to:
      <qemu:override>
        <qemu:device alias='hostdev0'>
          <qemu:frontend>
            <qemu:property name='x-vga' type='bool' value='true'/>
          </qemu:frontend>
        </qemu:device>
      </qemu:override>
  • I was seeing the same warning message as @ven0m above.  Unraid was using the GPU for its terminal output, I had to modify the Syslinux configuration in Unraid and add  video=efifb:off to the kernel argument list.  After a reboot, the warning was gone (and indeed Unraid stopped writing output to my monitor during the boot sequence!)
  • In general, I found myself "fighting" against whatever xml validation mechanism is in place for the VM configuration in Unraid.  Often I'd make changes, save them, but re-opening the config would reveal that my configuration had been removed.  I guess you guys already know this, but I found it quite confusing at first...

On my Mobo I had to set PCIe ACS override=both and VFIO allow unsafe interrupts=yes, as my VGA and HDMI audio controllers shared a group with a tonne of other devices.

 

I tested the system for an hour or so yesterday, and everything seemed OK, apart from some slight audio stutters every now and then.  I'll be keeping it monitored.

 

Hopefully someone will find these additions helpful.

 

Luca

Edited by optiluca
  • Thanks 1
Link to comment
On 12/12/2022 at 3:10 AM, optiluca said:

Just wanted to thank Tanne for the guide. By following it I managed to get my 5600G iGPU passed through successfully.  

Optiluca, after seeing you were successful I decided to try it myself. For flavor text, I've been snubbed this issue since 2018 with a raven ridge 2400 G, and I've long since given up on this ever actually working in a windows VM. Trying all of the steps you followed including your modifications, I'll admit that this is closer to "actually working" than it's ever been before, but still with quite a number of issues sadly.

Something about the initial iGPU handoff is going buggy for me - I can start the VM cleanly, but as soon as I remote desktop into it, unraid loses connection to it's flash drive (I lose a good chunk of the webUI functionality, but the array doesn't crash).
Within the VM, the vega graphics device is correctly showing but the display driver has a yellow alert next to it - Windows reports that the device has issues and the driver isn't properly installed (code 43). As a result of all this, I'm not able to get any HDMI out from the server itself. Really not sure what to try next, I can tell it's sort-of working because the remote desktop is full resolution and whatnot, but I have no idea how to iron out what's going on behind the scenes. I believe the driver issue is just a symptom of a bigger underlying problem.

 

One thing I'm really curious about is the <qemu:override> section, I am also seeing the XML file seemingly revert that section when I reopen it, which makes me think there is something invalid there which is causing the document to not save properly. I also got the "hostdev0 not valid" issue when I used the code snippet in Tanne's guide, but using that code the XML at least saved properly...

I'm still holding onto a flicker of hope that this can work, because there's no room in my server for a dedicated GPU and I would hate to have to give up/start over with a different processor after all this. But man, I really can't say I recommend using AMD APUs with unraid, considering all this hassle.

Link to comment
6 hours ago, Manthony said:

I am also seeing the XML file seemingly revert that section when I reopen it

Make sure you have this at the top of your xml:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

otherwise the qemu:override section is considered not valid.

 

6 hours ago, Manthony said:

I also got the "hostdev0 not valid" issue

hostdev0 is an alias for the device; if not specified in the xml it is assigned automatically by libvirt/qemu, so yours can be different than hostdev0 (look at the libvirt/qemu logs in your diagnostics and you will find what is your alias); you can assign manually your alias with what you want in the hostdev section of your xml.

Quote

The <qemu:device> sub-element groups overrides for a device identified via the alias attribute. The alias corresponds to the <alias name=''> property of a device. It's strongly recommended to use user-specified aliases for devices with overridden properties.

 

Note also that you need at least libvirt 8.2.0 to use qemu:frontend

Edited by ghost82
Link to comment
14 hours ago, ghost82 said:

Make sure you have this at the top of your xml:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

otherwise the qemu:override section is considered not valid.

 

Thanks for the tip, not having any issues saving the XLM document after adding that in, so at least that's working better.

As for actually getting a VM working, it's somehow going much worse today than yesterday. I did see a brief blip of HDMI out, which was a new first and probably a good sign, but every time I start the VM unraid loses the flash drive after a few moments, the VM becomes unreachable, and I need to reboot everything.

If I had to guess, either it's attempting to pass through all of the USB ports and losing the flash drive, or maybe it's just more bad igpu passthrough issues dropping the ball. Really baffling because it worked at least a couple of times yesterday, but today turning on the VM instantly breaks it, and I can't remote desktop at all. I've tried making a brand new VM following the instructions in this thread to a T, XML config is good, IOMMU groups are good, iGPU and audio are in their own groups without any USB controllers, so I don't know why the flash drive would be getting bumped off while the rest of the array stays 'on'

 

 

EDIT, took a peek at the array logs when I kick off the VM. what stands out to me is that a bunch of things are timing out, but really I don't have a clue what this is about or how to fix it. Snippet ends when bunch of USBs disconnect which leads to all of the errors associated with the flash drive being gone.

 

Edit 2: flash drive doesn't disconnect after I moved it to the front panel - not ideal for aesthetic reasons but maybe not a permanent change. I can now attempt to spin up the VM as much as I want without it impacting the array. Still consistently not taking off properly when using igpu passthrough though, logs continue to show many timeouts and remoting in is a no-go. Adding the server logs

 

 

logs.txt VMlogs.txt

Edited by Manthony
updated logs
Link to comment
11 hours ago, ghost82 said:

I'm not confident it will work, but you could try to add to your syslinux config:

iommu=soft

and/or

pci=noats

 

Thanks, I included both of those and it seems like pci=noats fixes the timeout issue without causing any other problems (iommu=soft didn't work on it's own and didn't seem to make a difference when set). With that out of the way, the next issue is that the kernel gets stuck at

kernel: vfio-pci 0000:09:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

afterwards nothing else happens and the VM doesn't start.
 

spent some time on google today looking into this one, a fair number of amd VM users, apparently. I couldn't find any definitive solution though, at least not any that seemed applicable.

Link to comment
  • 4 weeks later...

unraid 6.92版本有问题 ,我的配置是 5700g 32g 3200频率内存条 主板微星b450m 迫击炮 ,6.92时候怎么搞都不行显卡直通后来 直接用 6.11.5 版本直接搞定了 在win10 虚拟机 显示ok  ,方法从官网下载微星主板对应的固件 然后用UBU_v1_79_17 提取vblos 固件 就是 vbios_1638.dat 这个文件 然后其他就是照常配置虚拟机就行了 然后虚拟机里面安装驱动可以正确识别 apu 显卡。我的建议是如果是新的硬件尽量用新的unraid 版本!

E7}CTU)H)YP]NPQYR~6)T@V.png

)KHUXZW98GDJZ]PL`FQKV_G.png

OFF@DJD9{(1~HJ`JY(_T0}G.png

  • Like 1
Link to comment
  • 2 weeks later...

Hi everyone, i have a 5700g, on a msi x470 gaming plus max, with the lates bios.
i used the "UBU" tool to extract the rom, renamed it with "name.rom" and follow several times the step previusly posted.

The unraid version of my server is 6.11.5, my flash setup is:

kernel /bzimage
append pcie_acs_override=downstream,multifunction initrd=/bzroot vfio_iommu_type1.allow_unsafe_interrupts=1 video=efifb:off

and i created the amdgpu.config in /boot/config/modprobe.d

i have other 2 gpu in my server, for other 2 VM (1 gpu each VM), and it works great.

when i try to run my 3rd vm, with the 5700g GPU and few cores, looks like everything is ok, (i follow all step to get this point like:
install windows with no gpu
enable rdp

disable windows update and autoinstall driver

but nothing let me use my onboard graphics (even unplug the HDMI before start and plug after the VM is "ready" works).

The only problem i found is this:

i cannot use the code:

  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.hostdev0.x-vga=on'/>
  </qemu:commandline>

because i get this error:

image.png.f60ad4e8ff79c92984a0dd2296e80a34.png

so, and i use

<qemu:override>
  <qemu:device alias='hostdev0'>
    <qemu:frontend>
      <qemu:property name='x-vga' type='bool' value='true'/>
    </qemu:frontend>
  </qemu:device>
</qemu:override>

and the machine is running, the gpu is in the device list, has no error, latest driver installed but i wont show any video.

 

i almost forgot to say: if remove the "virtual GPU" an select the "Radeon" one, the system wont start, and i need to make a new XML.

 

Can someone tell me what im doing wrong? there is any chance to fix my issiue for 6.11.5?

Thanks everyone for your work and support.

 

this is the entire schema of my windows 10 machine

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows 10</name>
  <uuid>9f862754-778a-12b8-9566-ec1e1988d38c</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='9'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='3D6L09A17K3O'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Win10iGPU/vdisk.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:b6:00:04'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='it'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <audio id='1' type='none'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x30' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/gpubios/radeonIGPU2.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x30' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x30' slot='0x00' function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <qemu:override>
    <qemu:device alias='hostdev0'>
      <qemu:frontend>
        <qemu:property name='x-vga' type='bool' value='true'/>
      </qemu:frontend>
    </qemu:device>
  </qemu:override>
</domain>

 

Edited by Goldino
forgot something
Link to comment

a little update: if i try to boot the machine without the disk and the Windows iso, and the iGPU RUN! But.... it wont boot if i set the disk or the iso.... the Firts core is stuck at 100% and i get only a black screen.

image.png.ab3a4ce60e9eb19fe8d629c085b18e0a.png

the only think is running on my server is the VM with the iGPU.

Can someone tell me what is going on?

Thanks

Link to comment
On 1/10/2023 at 9:12 AM, ludaye said:

unraid 6.92版本有问题 ,我的配置是 5700g 32g 3200频率内存条 主板微星b450m 迫击炮 ,6.92时候怎么搞都不行显卡直通后来 直接用 6.11.5 版本直接搞定了 在win10 虚拟机 显示ok  ,方法从官网下载微星主板对应的固件 然后用UBU_v1_79_17 提取vblos 固件 就是 vbios_1638.dat 这个文件 然后其他就是照常配置虚拟机就行了 然后虚拟机里面安装驱动可以正确识别 apu 显卡。我的建议是如果是新的硬件尽量用新的unraid 版本!

E7}CTU)H)YP]NPQYR~6)T@V.png

)KHUXZW98GDJZ]PL`FQKV_G.png

OFF@DJD9{(1~HJ`JY(_T0}G.png

兄弟,你这个能直接HDMI接口显示虚拟win画面吗

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.