I spent an ungodly amount of time trying to get this VM to work, so here's what I've done to hopefully streamline the process for others using 6000 series Radeon GPUs.

 

1. Disable SAM/Resizable Bar in BIOS (this was the crucial step for me), I also disabled above 4G encoding, but I'm unsure of the effects so far

2. I have had no issues installing the latest AMD GPU drivers, so I assume that these are safe to install on the guest system

3. Only use Q35 in the BIOS because it has better support for pcie passthrough devices. (which AMD GPUs require)

4. Dump the bios from your card and set it as the vbios in the VM creation page if your AMD GPU is your vga-out card

5. Set your primary sound card to the GPU sound card

 

That should leave you all set! If anyone else has anything else they feel should be added, please reply and I'll update this topic!

 

Thanks @ghost82

Edited October 27, 20223 yr by PixOnePro

Nice that you found the solution however:

7 hours ago, PixOnePro said:

2. Despite what spaceinvaderone's video on gpu passthrough says, you do not need to create a multifunction device in VM creation xml, you can leave it as the form has made it and it will work fine.

It could work, but bare metal hardware is multifunction, so it is strongly recommended to make the gpu a multifunction device in the vm too.

If it's not multifunction, you could:

1. be unable to install drivers in the guest, because of crappy drivers, they could check for a multifunction device, and could fail the installation if they don't find it; have unexpected drivers behaviour too.

2. in some oses digital audio will not work correctly if the device is not set as multifunction, mac os for example.

 

7 hours ago, PixOnePro said:

4. Only use Q35 in the BIOS, i440 will not work to my knowledge as AMD only supports UEFI bios

Q35 should be the default these days, for every os, because it has better support for pcie passthrough devices. UEFI is not linked to q35, you could have a vm booting in uefi both with q35 and i440fx; q35 and i440fx are only emulated chipsets, q35 being more recent. What is important for uefi is booting with a uefi capable bios (ovmf).

 

7 hours ago, PixOnePro said:

5. Do dump the bios from your card and set it as the vbios in the VM creation page (again, unsure if this makes a difference - regardless, it's good practice)

This is not strictly necessary; it is necessary when the gpu you want to passthrough is flagged by the host os as "boot vga"; if it's flagged that way the host os will make a shadow copy of the vbios, but it's not a 1:1 copy of the gpu vbios, so when the vm will try to use it, it will fail; passing through a vbios file will fix it. This is true for linux hosts only, windows doesn't have this issue, in fact you are able to dump a 1:1 copy of the vbios on a bare metal windows machine even if the gpu is the boot gpu.

Edited October 23, 20223 yr by ghost82

On 10/23/2022 at 2:21 PM, ghost82 said:

Nice that you found the solution however:

It could work, but bare metal hardware is multifunction, so it is strongly recommended to make the gpu a multifunction device in the vm too.

If it's not multifunction, you could:

1. be unable to install drivers in the guest, because of crappy drivers, they could check for a multifunction device, and could fail the installation if they don't find it; have unexpected drivers behaviour too.

2. in some oses digital audio will not work correctly if the device is not set as multifunction, mac os for example.

 

Q35 should be the default these days, for every os, because it has better support for pcie passthrough devices. UEFI is not linked to q35, you could have a vm booting in uefi both with q35 and i440fx; q35 and i440fx are only emulated chipsets, q35 being more recent. What is important for uefi is booting with a uefi capable bios (ovmf).

 

This is not strictly necessary; it is necessary when the gpu you want to passthrough is flagged by the host os as "boot vga"; if it's flagged that way the host os will make a shadow copy of the vbios, but it's not a 1:1 copy of the gpu vbios, so when the vm will try to use it, it will fail; passing through a vbios file will fix it. This is true for linux hosts only, windows doesn't have this issue, in fact you are able to dump a 1:1 copy of the vbios on a bare metal windows machine even if the gpu is the boot gpu.

 

Hello ghost82, I have some difficulties to run a gaming VM on unRAID.

I created one with W11, my 10600k a rx 6600 and my B460m plus mb.
I had set the rx 6600 as primary GPU and the IGPU as secondary on the BIOS.
Then i disable "PCIe ACS override" and Bind the Graphic and audio of the rx 6600.

On the BIOS VMX is enable, Resizable Bar is disable, CSM is set on UEFI.

The VM started and the rx 6600 is perfectly working, but when i stop and restart the VM, it's crashing all unRAID. The connection il no longer working for unRAID but the server still run. So i have to manually reboot it..

Here are the unRAID logs :
```Feb 19 10:50:44 Nwiconeflix  ool www[14674]: /usr/local/emhttp/plugins/dynamix/scripts/rsyslog_config
Feb 19 11:04:43 Nwiconeflix  rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="30840" x-info="https://www.rsyslog.com"] start
Feb 19 11:04:47 Nwiconeflix kernel: br0: port 2(vnet1) entered blocking state
Feb 19 11:04:47 Nwiconeflix kernel: br0: port 2(vnet1) entered disabled state
Feb 19 11:04:47 Nwiconeflix kernel: device vnet1 entered promiscuous mode
Feb 19 11:04:47 Nwiconeflix kernel: br0: port 2(vnet1) entered blocking state
Feb 19 11:04:47 Nwiconeflix kernel: br0: port 2(vnet1) entered forwarding state
Feb 19 11:04:47 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
Feb 19 11:04:47 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
Feb 19 11:04:47 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x26@0x410
Feb 19 11:04:47 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x27@0x440
Feb 19 11:04:49 Nwiconeflix kernel: vfio-pci 0000:03:00.0: No more image in the PCI ROM
Feb 19 11:04:49 Nwiconeflix kernel: vfio-pci 0000:03:00.0: No more image in the PCI ROM```

Do you what is happening ?

Here is my config :

Edited February 19, 20233 yr by nwicone

On 10/23/2022 at 6:50 AM, PixOnePro said:

I spent an ungodly amount of time trying to get this VM to work, so here's what I've done to hopefully streamline the process for others using 6000 series Radeon GPUs.

 

1. Disable SAM/Resizable Bar in BIOS (this was the crucial step for me), I also disabled above 4G encoding, but I'm unsure of the effects so far

2. I have had no issues installing the latest AMD GPU drivers, so I assume that these are safe to install on the guest system

3. Only use Q35 in the BIOS because it has better support for pcie passthrough devices. (which AMD GPUs require)

4. Dump the bios from your card and set it as the vbios in the VM creation page if your AMD GPU is your vga-out card

5. Set your primary sound card to the GPU sound card

 

That should leave you all set! If anyone else has anything else they feel should be added, please reply and I'll update this topic!

 

Thanks @ghost82

what are the sittings you use ?
Do you enable IOMMU ?
Do you disable "PCIe ACS override" ?
if you have a 6000 series, i could take the same settings you have
Thanks

1 hour ago, nwicone said:

I had set the rx 6600 as primary GPU and the IGPU as secondary on the BIOS.

I think the host is taking back the gpu and it's not able to release it resulting in a crash.

I would set the igpu as primary in bios and not the 6600; a setup with the 6600 as primary is possible but you need the vbios to be passed through for the vm.

Bind the gpu audio and video to vfio.

Check the syslog in unraid and see if efifb is attaching to the 6600 (check the address of the gpu), if it attaches blacklist it in unraid sysconfig.

2 minutes ago, ghost82 said:

I think the host is taking back the gpu and it's not able to release it resulting in a crash.

I would set the igpu as primary in bios and not the 6600; a setup with the 6600 as primary is possible but you need the vbios to be passed through for the vm.

Bind the gpu audio and video to vfio.

Check the syslog in unraid and see if efifb is attaching to the 6600 (check the address of the gpu), if it attaches blacklist it in unraid sysconfig.

Thank you for your answer.
I'm actually not really confident with all this, i'm a beginner with VM. So i will summarize what i have to do just to check that I understood correctly :

- unBIND AMD GPU
- set the IGPU as primary and the AMD GPU as secondary
- Bind AMD GPU to VFIO
- Check the logs to see if the AMD GPU is blacklisted

Correct ?
it is especially the reading of the logs (check blacklist) that I have never done

I can also understand that :
I keep "PCIe ACS override" disable 
I keep enable IOMMU
Last thing, here is my syslinux config for unraid os :

`kernel /bzimage
append video=efifb:off initrd=/bzroot`

Many thanks for your help !

yes correct

4 hours ago, ghost82 said:

yes correct

i got this :

Feb 19 19:32:35 Nwiconeflix  rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="21527" x-info="https://www.rsyslog.com"] start
Feb 19 19:33:42 Nwiconeflix kernel: br0: port 2(vnet0) entered blocking state
Feb 19 19:33:42 Nwiconeflix kernel: br0: port 2(vnet0) entered disabled state
Feb 19 19:33:42 Nwiconeflix kernel: device vnet0 entered promiscuous mode
Feb 19 19:33:42 Nwiconeflix kernel: br0: port 2(vnet0) entered blocking state
Feb 19 19:33:42 Nwiconeflix kernel: br0: port 2(vnet0) entered forwarding state
Feb 19 19:33:42 Nwiconeflix kernel: vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
Feb 19 19:33:42 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
Feb 19 19:33:42 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
Feb 19 19:33:42 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x26@0x410
Feb 19 19:33:42 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x27@0x440
Feb 19 19:33:42 Nwiconeflix kernel: vfio-pci 0000:03:00.1: enabling device (0000 -> 0002)
Feb 19 19:37:04 Nwiconeflix  ntpd[1260]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

there's nothing wrong in here, describe your issue and post full diagnostics

1 hour ago, ghost82 said:

there's nothing wrong in here, describe your issue and post full diagnostics

it crashed again at startup
when i first start it it's working, the rx 6600 is perfectly working.
But when i shut it down and then i start it, it's crashing..

It crashed at 21h49, but here are the logs :
 

Feb 19 21:42:37 Nwiconeflix  avahi-daemon[6569]: Joining mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe4a:9f6f.
Feb 19 21:42:37 Nwiconeflix  avahi-daemon[6569]: New relevant interface vnet0.IPv6 for mDNS.
Feb 19 21:42:37 Nwiconeflix  avahi-daemon[6569]: Registering new address record for fe80::fc54:ff:fe4a:9f6f on vnet0.*.
Feb 19 21:44:00 Nwiconeflix root: Fix Common Problems Version 2023.02.17
Feb 19 21:44:00 Nwiconeflix root: Fix Common Problems: Warning: Docker Application sonarr has an update available for it
Feb 19 21:44:00 Nwiconeflix root: Fix Common Problems: Warning: Docker Application sonarr_4k has an update available for it
Feb 19 21:44:06 Nwiconeflix root: Fix Common Problems: Warning: Syslog mirrored to flash
Feb 19 21:47:35 Nwiconeflix  avahi-daemon[6569]: Interface vnet0.IPv6 no longer relevant for mDNS.
Feb 19 21:47:35 Nwiconeflix  avahi-daemon[6569]: Leaving mDNS multicast group on interface vnet0.IPv6 with address fe80::fc54:ff:fe4a:9f6f.
Feb 19 21:47:35 Nwiconeflix kernel: br0: port 2(vnet0) entered disabled state
Feb 19 21:47:35 Nwiconeflix kernel: device vnet0 left promiscuous mode
Feb 19 21:47:35 Nwiconeflix kernel: br0: port 2(vnet0) entered disabled state
Feb 19 21:47:35 Nwiconeflix  avahi-daemon[6569]: Withdrawing address record for fe80::fc54:ff:fe4a:9f6f on vnet0.
Feb 19 22:04:03 Nwiconeflix kernel: microcode: microcode updated early to revision 0xf4, date = 2022-07-31

Here is the diagnostic file : nwiconeflix-diagnostics-20230219-2208.zip

Edited February 19, 20233 yr by nwicone

Try the followings:
1. enable acs override --> both (meaning downstream,multifunction); restart the server, check if you are able to split the current iommu group 1, try to split the amd pci bridges from vga and audio; do not attach to vfio pci bridges at 01:00.0 and 02:00.0, only video and audio of the gpu

2. change in your vm xml the gpu part from this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

 

to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>

 

Edited February 20, 20233 yr by ghost82

22 minutes ago, ghost82 said:

Try the followings:
1. enable acs override --> both (meaning downstream,multifunction); restart the server, check if you are able to split the current iommu group 1, try to split the amd pci bridges from vga and audio; do not attach to vfio pci bridges at 01:00.0 and 02:00.0, only video and audio of the gpu

2. change in your vm xml the gpu part from this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

 

to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>

 

 

Okay thanks, what do you mean by split ?

Correct me if this list of action is not correct :
-unbind AMD GPU

- enable acs override --> both

- restart

- Bind Video and audio of the AMD GPU
- Change the xml file like you show me

Correct ?

yes.

By split I mean having the two amd pci bridges not in the same iommu group of the vga and audio.

Now 2 amd pci bridges together with vga and audio are in iommu group 1.

Check if with acs override you can have vga and audio in one or 2 different iommu groups, other than the iommu group of the pci bridges.

Then attach to vfio only vga and audio and not the bridges.

Edited February 20, 20233 yr by ghost82

19 minutes ago, ghost82 said:

yes.

By split I mean having the two amd pci bridges not in the same iommu group of the vga and audio.

Now 2 amd pci bridges together with vga and audio are in iommu group 1.

Check if with acs override you can have vga and audio in one or 2 different iommu groups, other than the iommu group of the pci bridges.

Then attach to vfio only vga and audio and not the bridges.

Okay, so i need to different IOMMU group for for the AMD audio and AMD video ? maybe you have an exemple to show me to make sure i understood ?
This is what i have with acs override --> both
If i'm not wrong, now i just have to select them and Bind them

On the bios, i have to keep IOMMU enable ?

Thanks 

Edited February 20, 20233 yr by nwicone

Yes now select both 13 and 14.

6 minutes ago, nwicone said:

On the bios, i have to keep IOMMU enable ?

ok course otherwise you wont be able to passthrough anything.

1 minute ago, ghost82 said:

Yes now select both 13 and 14.

ok course otherwise you wont be able to passthrough anything.

Okay perfect, here i am :

 

config before :

Config now :


All correct ?

It crashed again, again it was perfectly working the first time, then i shuted dow the VM and then a started it. As each time, it crashed at the second start.

Here are the unRAID logs, crash happened between 26 and 30, at 31 all unraid was starting :

 

Feb 20 11:24:32 Nwiconeflix webGUI: Successful login user root from 10.253.0.2
Feb 20 11:26:46 Nwiconeflix kernel: br0: port 2(vnet0) entered disabled state
Feb 20 11:26:46 Nwiconeflix kernel: device vnet0 left promiscuous mode
Feb 20 11:26:46 Nwiconeflix kernel: br0: port 2(vnet0) entered disabled state
Feb 20 11:27:15 Nwiconeflix kernel: br0: port 2(vnet1) entered blocking state
Feb 20 11:27:15 Nwiconeflix kernel: br0: port 2(vnet1) entered disabled state
Feb 20 11:27:15 Nwiconeflix kernel: device vnet1 entered promiscuous mode
Feb 20 11:27:15 Nwiconeflix kernel: br0: port 2(vnet1) entered blocking state
Feb 20 11:27:15 Nwiconeflix kernel: br0: port 2(vnet1) entered forwarding state
Feb 20 11:27:16 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x19@0x270
Feb 20 11:27:16 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x1b@0x2d0
Feb 20 11:27:16 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x26@0x410
Feb 20 11:27:16 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_ecap_init: hiding ecap 0x27@0x440
Feb 20 11:27:32 Nwiconeflix kernel: vfio-pci 0000:03:00.0: vfio_bar_restore: reset recovery - restoring BARs
Feb 20 11:31:46 Nwiconeflix kernel: microcode: microcode updated early to revision 0xf4, date = 2022-07-31

 

and here is the diagnostic :nwiconeflix-diagnostics-20230220-1135.zip

Edited February 20, 20233 yr by nwicone

I heard of some possible issues with the 6000 series and some specific brands.

When you say it crashes, does it crash the whole hotst or you are not able to run only the vm?

 

Check in bios:

1. resizable bar: disabled

2. decode above 4g: disabled

 

if this doesn't work I have no more ideas.

8 minutes ago, ghost82 said:

I heard of some possible issues with the 6000 series and some specific brands.

When you say it crashes, does it crash the whole hotst or you are not able to run only the vm?

 

Check in bios:

1. resizable bar: disabled

2. decode above 4g: disabled

 

if this doesn't work I have no more ideas.

When i say it crash, it's the entire unRAID that is not available. the dashboard of unraid is no more available. I think it crash the internet, because physically i can see that it's running, but 192.168.1.50 is not responding.. So i have to manually reboot unRAID.

 

1. resizable bar: disabled  --> this is already done

2. decode above 4g: disabled. --> i will try that

I also think the mb can be the problem.

Thanks

Edited February 20, 20233 yr by nwicone

I will reset all MB settings to check and set

 

Enable :
- VMX     - VT-d

 

Disable :

- decode above 4g decoding

- resizable bar        - CSM

Okay the problem was from my MB, VT-d was "unsupported"..

I reset the MB and i've set again everything.

Now VT-d is enable.

How should I start now ?

13 hours ago, nwicone said:

VT-d was "unsupported"

I doubt you had unsupported vt-d: vt-d is needed for vfio, so it had to be available..

anyway just proceed the same way as described in the above posts.

5 minutes ago, ghost82 said:

I doubt you had unsupported vt-d: vt-d is needed for vfio, so it had to be available..

anyway just proceed the same way as described in the above posts.

I already tried with 4g disable yesterday night.

Not working too.

If I can't find a solution I will pass on an Nvidia which should work better 

All I can say is that from what I saw your configuration is correct and unfortunately the errors recall the so called 'amd reset bug', which the 6000 series shouldn't have..but some reports show that some of them could have it, it's not clear if it depends on the gpu or the bios or whatever, so trying a nvidia is the right way to go, just to see if it depends on the gpu itself.

1 minute ago, ghost82 said:

All I can say is that from what I saw your configuration is correct and unfortunately the errors recall the so called 'amd reset bug', which the 6000 series shouldn't have..but some reports show that some of them could have it, it's not clear if it depends on the gpu or the bios or whatever, so trying a nvidia is the right way to go, just to see if it depends on the gpu itself.

Okay I see.

I already have a gt 610, if it's working on the gt 610 it will work on others Nvidia or I have to try on a newer version ?