[almost solved] Windows 10 top - macOS flop... Apple will not work with GPU passthrough..

[derpuma]

First I was struggeling but later today I was able to install a stable and reliable Windows 10 VM on a virtual Disk with full GPU passthrough.

I tested it with some rounds of high FPS 3d Gaming and it was rock solid and stable!
So my Hardware (Ryzen 3900X and Gigabyte Aorus Pro Z570) is able to deal with UNRAID.

I splitted my IOMMU Groups via PCIe ACS override with the setting "Both", did not bound any IOMMU via VFIO-PCI Config

and it worked great.
To get this far, I used a new kernel from this thread...
Unfortunatly my Installation of macOS was a complete fail.
Installation was easy and booting via VNC was no problem at all. Touble started when I tried to GPU passthrough like I did with Windows.
I was not able to get it together, Attatched you will find my IoMMU Groups and my XML setting of both, my working Windows 10 and my

messy Catalina install.
I tried different combinations with and without USB passthrough.  Maybe someone could have a look at my files an give me some tips and help.

In general I guess it should be possible, to get a working macOS too, if Windows 10 ist flawless?

My passthrough GPU ist a Radon 5700 XT. My Unraid Host GPU an older RX480 set to first PCIe device after boot, to be alble to have the 5700 XT for the passthrough.

If you need anything other background information, please let me know.

 

 

IOMMU group 0:[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 1:[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 2:[1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 3:[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 4:[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 5:[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 6:[1022:1483] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 7:[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 8:[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 9:[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 10:[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 11:[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 12:[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 13:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 14:[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0

[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1

[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2

[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3

[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4

[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5

[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6

[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7

IOMMU group 15:[1987:5012] 01:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01)

IOMMU group 16:[1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream

IOMMU group 17:[1022:57a3] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 18:[1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 19:[1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 20:[1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 21:[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

IOMMU group 22:[1022:1485] 05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 23:[1022:149c] 05:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 24:[1022:149c] 05:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 25:[1022:7901] 06:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 26:[1022:7901] 07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 27:[1002:1478] 08:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1)

IOMMU group 28:[1002:1479] 09:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch

IOMMU group 29:[1002:731f] 0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c1)

IOMMU group 30:[1002:ab38] 0a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio

IOMMU group 31:[1002:67df] 0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev c7)

IOMMU group 32:[1002:aaf0] 0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]

IOMMU group 33:[1022:148a] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 34:[1022:1485] 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 35:[1022:1486] 0d:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP

IOMMU group 36:[1022:149c] 0d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 37:[1022:1487] 0d:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

 

CPU Thread Pairings

Pair 1:cpu 0 / cpu 12

Pair 2:cpu 1 / cpu 13

Pair 3:cpu 2 / cpu 14

Pair 4:cpu 3 / cpu 15

Pair 5:cpu 4 / cpu 16

Pair 6:cpu 5 / cpu 17

Pair 7:cpu 6 / cpu 18

Pair 8:cpu 7 / cpu 19

Pair 9:cpu 8 / cpu 20

Pair 10:cpu 9 / cpu 21

Pair 11:cpu 10 / cpu 22

Pair 12:cpu 11 / cpu 23

 

USB Devices

Bus 001 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 002:ID 048d:8297 Integrated Technology Express, Inc. ITE Device(8595)

Bus 001 Device 003:ID 1e7d:38a4 ROCCAT ROCCAT Kave Headset

Bus 001 Device 004:ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 002 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 003 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 003 Device 002:ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 003 Device 003:ID 03f0:2b4a Hewlett-Packard

Bus 003 Device 004:ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive

Bus 004 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 005 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 005 Device 002:ID 0b05:18e1 ASUSTek Computer, Inc. ROG STRIX IMPACT II

Bus 005 Device 003:ID 03eb:3301 Atmel Corp. at43301 4-Port Hub

Bus 005 Device 004:ID 1038:0100 SteelSeries ApS Ideazon Zboard

Bus 006 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

 

SCSI Devices

[0:0:0:0]disk /dev/sda 3.99GB

[N:0:1:1]disk PNY CS3030 2TB SSD__1 /dev/nvme0n1 2.00TB


Log of the failing macOS GPU Passthrough:

-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device ide-hd,bus=ide.2,drive=libvirt-3-format,id=sata0-0-2,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/MacinaboxCatalina/Catalina-install.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-hd,bus=ide.3,drive=libvirt-2-format,id=sata0-0-3,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/MacinaboxCatalina/macos_disk.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-hd,bus=ide.4,drive=libvirt-1-format,id=sata0-0-4,write-cache=on \
-netdev tap,fd=35,id=hostnet0,vhost=on,vhostfd=36 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:04:b9:bd,bus=pci.3,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=38,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-device vfio-pci,host=0000:0a:00.0,id=hostdev0,bus=pci.4,addr=0x0,romfile=/mnt/disk1/isos/Powercolor_5700XT.rom \
-device vfio-pci,host=0000:0a:00.1,id=hostdev1,bus=pci.5,addr=0x0 \
-usb \
-device usb-kbd,bus=usb-bus.0 \
-device '************************' \
-smbios type=2 \
-cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2020-06-19 16:56:10.708+0000: Domain id=1 is tainted: high-privileges
2020-06-19 16:56:10.708+0000: Domain id=1 is tainted: custom-argv
2020-06-19 16:56:10.708+0000: Domain id=1 is tainted: host-cpu
char device redirected to /dev/pts/3 (label charserial0)
2020-06-19T16:56:10.775914Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2020-06-19T16:56:10.777224Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2020-06-19T16:56:10.778138Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2020-06-19T16:56:10.778998Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2020-06-19T16:56:10.779874Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2020-06-19T16:56:10.780785Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2020-06-19T16:56:10.781505Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2020-06-19T16:56:10.782190Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.01H:ECX.pcid [bit 17]
2020-06-19T16:56:17.038207Z qemu-system-x86_64: vfio: Cannot reset device 0000:0a:00.1, no available reset mechanism.
2020-06-19T16:56:17.042396Z qemu-system-x86_64: vfio: Cannot reset device 0000:0a:00.1, no available reset mechanism.
2020-06-19T16:56:31.836311Z qemu-system-x86_64: terminating on signal 15 from pid 8553 (/usr/sbin/libvirtd)
2020-06-19 16:56:32.236+0000: shutting down, reason=destroyed

 

Win10_Working.xml MacinaboxCatalina.xml

Edited June 20, 2020 by derpuma
problem solved

[derpuma]

Just want to give a quick update. After hassling around the RX 5700 XT macOS is booting with the GPU right now.

The pita was not the GPU or the Reset Bug, it was Clover and I had to set some boot args and deselect some preselected options while booting.

 

1) The GPU needs the boot-arg agdpmod=pikera

2) by default clover does ATI Inject, I deselected this

3) I selected load VBIOS in clover

4) I changed SMBIOS to MacPro1,1 (MacPro7,1 will also work)

 

These are the main reasons, why I was not able to passthrough my GPU to macOS.

 

If a reset bug happens, I use the script from spacinvador via User Scripts plugin:

#!/bin/bash
# 
#replace xx\:xx.x with the number of your gpu and sound counterpart
#
#
echo "disconnecting amd graphics"
echo "1" | tee -a /sys/bus/pci/devices/0000\:0a\:00.0/remove
echo "disconnecting amd sound counterpart"
echo "1" | tee -a /sys/bus/pci/devices/0000\:0a\:00.1/remove
echo "entered suspended state press power button to continue"
echo -n mem > /sys/power/state
echo "reconnecting amd gpu and sound counterpart"
echo "1" | tee -a /sys/bus/pci/rescan
echo "AMD graphics card succesfully reset"

 

I also updated to the latest Bios for my Gigabyte Z570 Aorus Pro (F20s).
Iommu is like a charm and almost every device has his own IOMMU Group.

 

Single USB passthrough also works.

 

Next step is to passthrough the different USB Controllers to get USB hotplugable.

Another thing I will try is to passthrough HDD and SSD devices and boot from their OS.

 

So happy days!

[Coshy93]

Would you be able to explain to a dummy like me how I make these changes go my unraid server / Mac os vm to pass through my 5700xt?

 

Im unsure where to add those boot args and don't know how to perform the other steps with clover etc. But eager to learn! 

[podlipny]

yeah some quick guide would be great. I am currently trying to migrate from bare metal Hackintosh to unRaid setup. But I m running i9 9900K