wbsmolen Posted March 23, 2020 Share Posted March 23, 2020 (edited) I'm trying to passthrough 2 GPUs on an x570 Taichi mobo: a 5700 XT in the primary slot and a GTX 1080 in the secondary. I have no problems passing through the secondary gpu. I'm having mild success passing through the primary gpu but hitting the same wall each time. I am trying to pass the gpu through to a macOS vm (which I have booted up/configed in VNC). OVMF and Clover share identical resolutions. I get halfway through boot and hang at "Unable to power on device, stuck in D3". I'm under the impression that this kernel patch is included in unraid stable. Looking for any/all assistance. Please let me know what logs I can share. Syslinux: kernel /bzimage append vfio-pci.ids=1002:731f,1002:ab38,1002:1478,1002:1479 isolcpus=1-4,6,8,10,13-16,18,20,22 pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 video=efifb:off initrd=/bzroot IOMMU Groups: IOMMU group 0: [1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 1: [1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 2: [1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 3: [1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 4: [1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 5: [1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 6: [1022:1483] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 7: [1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 8: [1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 9: [1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 10: [1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 11: [1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 12: [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 13: [1022:1484] 00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 14: [1022:1484] 00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 15: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 16: [1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 IOMMU group 17: [1987:5016] 01:00.0 Non-Volatile memory controller: Phison Electronics Corporation E16 PCIe4 NVMe Controller (rev 01) IOMMU group 18: [1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream IOMMU group 19: [1022:57a3] 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 20: [1022:57a3] 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 21: [1022:57a3] 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 22: [1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 23: [1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 24: [1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 25: [144d:a802] 04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM951/PM951 (rev 01) IOMMU group 26: [1987:5016] 05:00.0 Non-Volatile memory controller: Phison Electronics Corporation E16 PCIe4 NVMe Controller (rev 01) IOMMU group 27: [1b21:1184] 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port IOMMU group 28: [1b21:1184] 07:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port IOMMU group 29: [1b21:1184] 07:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port IOMMU group 30: [1b21:1184] 07:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port IOMMU group 31: [1b21:1184] 07:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e PCIe Switch Port IOMMU group 32: [8086:2723] 08:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a) IOMMU group 33: [8086:1539] 0a:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 34: [1022:1485] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 35: [1022:149c] 0c:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 36: [1022:149c] 0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 37: [1022:7901] 0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 38: [1022:7901] 0e:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 39: [1002:1478] 0f:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1) IOMMU group 40: [1002:1479] 10:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch IOMMU group 41: [1002:731f] 11:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c1) IOMMU group 42: [1002:ab38] 11:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio IOMMU group 43: [10de:1b80] 12:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1) IOMMU group 44: [10de:10f0] 12:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1) IOMMU group 45: [1022:148a] 13:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 46: [1022:1485] 14:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 47: [1022:1486] 14:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP IOMMU group 48: [1022:149c] 14:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 49: [1022:1487] 14:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller IOMMU group 50: [1022:7901] 15:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 51: [1022:7901] 16:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) diagnostics.zip Edited March 23, 2020 by wbsmolen Quote Link to comment
cat2devnull Posted March 23, 2020 Share Posted March 23, 2020 Not sure if this will help but I have a similar setup. x570 Gaming X (basically the Taichi with a 2.5GB Realtek ethernet port) and a 5700XT in the first PCIe x16 and a RX550 in the second. I haven't found it necessary to stub the card (with vfio-pci.ids). I pass the 5700XT to a Win10 VM and the RX550 to a Catalina VM. Both VMs work well with a couple of annoying little foibles. - If I power off the Win10 VM there is about a 20% chance the machine will hard-lock. If I sleep the VM or kill it, then no problem. - The Mac VM runs 24/7 as a Plex server and so I have the display power down after 10min. It never wakes cleanly. If I try to wake it, then it just displays a black screen. I have to leave it for 10sec until it power saves again, then it wakes fine the second time. Because the 5700XT is the primary GPU and used for Win10, I have to give it a ROM file in the VM config. Other than that it's all pretty straight forward. So some things to try; - Maybe see how you go without stubbing the primary GPU at all. - or Stub the GPU but leave the Navi 10 XL Upstream/Downstream Ports. - If that doesn't work, try using BIND (in /boot/config/vfio-pci.cfg) instead. The VFIO-PCI Config plugin makes it easy if you don't like hitting the command line. I'm not actually sure if using BIND is actually just stubbing the GPU just via a different config file or if it is actually a different method all together. As I said, I haven't needed to do it for the GPU but I use it to lock out the onboard USB and a Broadcom PCIe bluetooth/wifi card for the Mac VM. Quote Link to comment
wbsmolen Posted March 23, 2020 Author Share Posted March 23, 2020 10 hours ago, cat2devnull said: Not sure if this will help but I have a similar setup. x570 Gaming X (basically the Taichi with a 2.5GB Realtek ethernet port) and a 5700XT in the first PCIe x16 and a RX550 in the second. I haven't found it necessary to stub the card (with vfio-pci.ids). I pass the 5700XT to a Win10 VM and the RX550 to a Catalina VM. Both VMs work well with a couple of annoying little foibles. - If I power off the Win10 VM there is about a 20% chance the machine will hard-lock. If I sleep the VM or kill it, then no problem. - The Mac VM runs 24/7 as a Plex server and so I have the display power down after 10min. It never wakes cleanly. If I try to wake it, then it just displays a black screen. I have to leave it for 10sec until it power saves again, then it wakes fine the second time. Because the 5700XT is the primary GPU and used for Win10, I have to give it a ROM file in the VM config. Other than that it's all pretty straight forward. So some things to try; - Maybe see how you go without stubbing the primary GPU at all. - or Stub the GPU but leave the Navi 10 XL Upstream/Downstream Ports. - If that doesn't work, try using BIND (in /boot/config/vfio-pci.cfg) instead. The VFIO-PCI Config plugin makes it easy if you don't like hitting the command line. I'm not actually sure if using BIND is actually just stubbing the GPU just via a different config file or if it is actually a different method all together. As I said, I haven't needed to do it for the GPU but I use it to lock out the onboard USB and a Broadcom PCIe bluetooth/wifi card for the Mac VM. Really appreciate the help. I have tried the things you mention above and am already using the VFIO-PCI Config plugin and binding per Unraid's guidance (cfg is in the diagnostics in OP). My latest attempt including stubbing but it had no impact on the result. I am using a ROM file from tech powerup and that seems to be working since I can get through OVMF and Clover pre-boot no problem. Booting into macOS hangs predictably each time. I'm using the macinabox method and whatever default clover config is included in that. I updated the kexts (lilu/weg/etc) and configured SMBIOS but maybe I need to do more in clover. Do you mind sharing your EFI? I'd think both our AMD cards are supported by macOS OOB. Quote Link to comment
cat2devnull Posted March 23, 2020 Share Posted March 23, 2020 Sure thing. I originally used Macinabox but had issues getting it to boot. I think @SpaceInvaderOne has done amazing work with Macinabox (and if he ever makes it to Oz, there is a cold beer waiting) but his installs usually rely on booting from an image file that he makes available and is a bit of a blackbox. I like being able to build everything from first principles so I just made a .img file on my spare notebook, installed the latest clover and a bunch of kexts then moved that to my Unraid server. I started a new VM pointing at that EFI image, the install media and a blank NVME drive passed through and away I went. I did end up needed to use his OVMF files and I would love @SpaceInvaderOne to let us know what he did to create them and how they are different. I've attached my EFI folder and my VM template. Just make sure you edit the config.plist file as I have replaced my serial numbers etc with "xxx" and in the VM file I've removed the Apple OSK key. EFI.zip Catalina VM.xml Quote Link to comment
wbsmolen Posted March 24, 2020 Author Share Posted March 24, 2020 2 hours ago, cat2devnull said: Sure thing. I originally used Macinabox but had issues getting it to boot. I think @SpaceInvaderOne has done amazing work with Macinabox (and if he ever makes it to Oz, there is a cold beer waiting) but his installs usually rely on booting from an image file that he makes available and is a bit of a blackbox. I like being able to build everything from first principles so I just made a .img file on my spare notebook, installed the latest clover and a bunch of kexts then moved that to my Unraid server. I started a new VM pointing at that EFI image, the install media and a blank NVME drive passed through and away I went. I did end up needed to use his OVMF files and I would love @SpaceInvaderOne to let us know what he did to create them and how they are different. I've attached my EFI folder and my VM template. Just make sure you edit the config.plist file as I have replaced my serial numbers etc with "xxx" and in the VM file I've removed the Apple OSK key. EFI.zip 3.61 MB · 0 downloads Catalina VM.xml 9.57 kB · 0 downloads You are a saint. I’m going to boot into VNC, copy/paste your EFI into mine, then look for gaps in our XML and take action. I’ll report back in a bit. Quote Link to comment
wbsmolen Posted March 26, 2020 Author Share Posted March 26, 2020 On 3/23/2020 at 4:57 PM, cat2devnull said: Sure thing. I originally used Macinabox but had issues getting it to boot. I think @SpaceInvaderOne has done amazing work with Macinabox (and if he ever makes it to Oz, there is a cold beer waiting) but his installs usually rely on booting from an image file that he makes available and is a bit of a blackbox. I like being able to build everything from first principles so I just made a .img file on my spare notebook, installed the latest clover and a bunch of kexts then moved that to my Unraid server. I started a new VM pointing at that EFI image, the install media and a blank NVME drive passed through and away I went. I did end up needed to use his OVMF files and I would love @SpaceInvaderOne to let us know what he did to create them and how they are different. I've attached my EFI folder and my VM template. Just make sure you edit the config.plist file as I have replaced my serial numbers etc with "xxx" and in the VM file I've removed the Apple OSK key. EFI.zip 3.61 MB · 4 downloads Catalina VM.xml 9.57 kB · 1 download No luck, thanks anyways. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.