bastl Posted October 22, 2020 Share Posted October 22, 2020 16 hours ago, Vaggeto said: That makes sense on the audio. Do you mind sharing the code to prevent unRAID to use the audio controller that you are checking as a USB audio device? (or is it the xen-pciback.hide= code?) The PCI IDs in my syslinux.config is for the audio controller itself, not for the USB audio device. 1022:1487 = onboard audio controller 1b21:2142 = onboard USB 3.1 controller 16 hours ago, Vaggeto said: For whatever reason when I try to pass my controllers, the VM won't load. They are in their own IOMMU groups and everything. I posted here about it. I do have one additional controller on my motherboard, but it is the only controller with USB2 ports so I don't want to pass it through even if I could since I'd prefer for my unRAID USB flash drive to be on a USB2 port. I have kinda similar issue with one of the other USB controllers and an extra PCI card like you. The FLR issue should be fixed in newer kernels. Unraid 6.9 should come with a fix for this. Quote Link to comment
SteelCityColt Posted October 22, 2020 Share Posted October 22, 2020 Apologies for jumping in on this topic. I'm slowly losing my mind with my TRX40 board (AS Rock TRX40 Creator) and 3960x combo. I want to be able to pass through a GPU (The only one in the system), a USB Controller, NVME, and the onboard sound. Every time I tried to start the VM it killed unRaid to the point of a hard reboot. It's also killed the flash drive twice, requiring a rebuild. Things I've worked out so far by trial and error and lots of VM config tests: 1) The original GPU (Vega 56) just wasn't having it. Swapping to a RX580 works. 2) Passing through the NVME controller using the VFIO plugin works fine 3) As soon as I try pass through either the sound or the USB controller it kills everything. With the USB I've tired isolating the group with the USB and also the non-essential instrumentation and then passing through (with no sound). Kills unRaid. 4) I tried passing through just the sound, both under soundcard without using the plugin, and also using the plugin. Kills unRaid. Where am I going wrong?! Iommu Groups below and diagnostics attached. IOMMU group 0: [1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 1: [1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 2: [1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 3: [1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 4: [1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 5: [1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 6: [1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 7: [1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 8: [1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 9: [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 10: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 11: [1022:1490] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1491] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1492] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1493] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1494] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1495] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1496] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1497] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 IOMMU group 12: [1002:67df] 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev e7) [1002:aaf0] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] IOMMU group 13: [1022:148a] 02:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 14: [1022:1485] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 15: [1022:148c] 03:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller IOMMU group 16: [1022:1482] 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 17: [1022:1482] 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 18: [1022:1482] 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 19: [1022:1483] 20:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 20: [1022:1482] 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 21: [1022:1482] 20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 22: [1022:1482] 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 23: [1022:1484] 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 24: [1022:1482] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 25: [1022:1484] 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 26: [1000:0072] 21:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) IOMMU group 27: [1022:148a] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 28: [1022:1485] 23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 29: [1022:1486] 23:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP IOMMU group 30: [1022:148c] 23:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller IOMMU group 31: [1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller IOMMU group 32: [1022:1482] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 33: [1022:1483] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 34: [1022:1483] 40:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 35: [1022:1483] 40:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 36: [1022:1482] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 37: [1022:1482] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 38: [1022:1482] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 39: [1022:1482] 40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 40: [1022:1482] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 41: [1022:1484] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 42: [1022:1482] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 43: [1022:1484] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 44: [1022:57ad] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream IOMMU group 45: [1022:57a3] 42:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 46: [1022:57a3] 42:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 47: [1022:57a3] 42:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 48: [1022:57a3] 42:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 49: [1022:57a3] 42:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 50: [1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:1485] 48:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:149c] 48:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 48:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 51: [1022:57a4] 42:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:7901] 49:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 52: [1022:57a4] 42:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:7901] 4a:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 53: [1987:5012] 43:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) IOMMU group 54: [1b21:3242] 44:00.0 USB controller: ASMedia Technology Inc. Device 3242 IOMMU group 55: [1d6a:07b1] 45:00.0 Ethernet controller: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02) IOMMU group 56: [8086:2723] 46:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a) IOMMU group 57: [10ec:8125] 47:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 01) IOMMU group 58: [1987:5012] 4b:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) IOMMU group 59: [1987:5012] 4c:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) IOMMU group 60: [1022:148a] 4d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 61: [1022:1485] 4e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 62: [1022:1482] 60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 63: [1022:1482] 60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 64: [1022:1482] 60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 65: [1022:1482] 60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 66: [1022:1482] 60:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 67: [1022:1482] 60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 68: [1022:1484] 60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 69: [1022:1482] 60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 70: [1022:1484] 60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 71: [1022:148a] 61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 72: [1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP diagnostics-20201022-1404.zip Quote Link to comment
bastl Posted October 22, 2020 Share Posted October 22, 2020 5 hours ago, SteelCityColt said: Where am I going wrong?! As I said earlier, I also have issues similiar to yours, trying to passthrough the onboard audio controller. Unraid will kinda freeze and I have to reset the server. That was the first issue i stumbled across when I got my TRX40. Issues with the sound card are fixed for me since I noticed the sound card itself is split into 2 USB audio devices which I'am able to passthrough to different VMs and can run them both at the same time. GPU passthrough never was an issue for me. 2 GPUs for 2 VMs, rock stable from day one. Onboard USB Controller passthrough showed similiar issues with Unraid becoming unresponsible. But not all controllers on my board. I think one of the two Starship ones caused the issues for me. The 3.1 controller passes through fine and is the only one I use and need. The FLR warning/error produced by the other controller will be fixed with newer linux kernels. Maybe check the Unraid 6.9 beta and test it or use a FLR patched Kernel. I think there is a custom kernel in one of the unraid forums which include the FLR patch and the vega patch. Quote Link to comment
Vaggeto Posted October 22, 2020 Share Posted October 22, 2020 (edited) 9 hours ago, SteelCityColt said: Apologies for jumping in on this topic. I'm slowly losing my mind with my TRX40 board (AS Rock TRX40 Creator) and 3960x combo. I want to be able to pass through a GPU (The only one in the system), a USB Controller, NVME, and the onboard sound. Every time I tried to start the VM it killed unRaid to the point of a hard reboot. It's also killed the flash drive twice, requiring a rebuild. Things I've worked out so far by trial and error and lots of VM config tests: 1) The original GPU (Vega 56) just wasn't having it. Swapping to a RX580 works. 2) Passing through the NVME controller using the VFIO plugin works fine 3) As soon as I try pass through either the sound or the USB controller it kills everything. With the USB I've tired isolating the group with the USB and also the non-essential instrumentation and then passing through (with no sound). Kills unRaid. 4) I tried passing through just the sound, both under soundcard without using the plugin, and also using the plugin. Kills unRaid. Where am I going wrong?! Iommu Groups below and diagnostics attached. diagnostics-20201022-1404.zip 101.22 kB · 0 downloads I had the same issue from a USB and Audio device perspective. Have you tried just using the VM checkboxes for USB devices vs trying to pass through an entire USB controller? I would start there and if that works then try the audio and once you launch the VM, look at the live log file and see what error you're getting. Edited October 22, 2020 by Vaggeto Quote Link to comment
Vaggeto Posted October 22, 2020 Share Posted October 22, 2020 (edited) 12 hours ago, bastl said: The PCI IDs in my syslinux.config is for the audio controller itself, not for the USB audio device. 1022:1487 = onboard audio controller 1b21:2142 = onboard USB 3.1 controller I have kinda similar issue with one of the other USB controllers and an extra PCI card like you. The FLR issue should be fixed in newer kernels. Unraid 6.9 should come with a fix for this. Thanks! This is very interesting. I'll do the same "blocking out". Sadly I'm kinda stuck with these 2 Starship USB controllers that won't pass. I will go ahead and try the RC version of unRAID though as that sounds interesting for the FLR issues. I realize a USB card might work, but I really want to save my PCI-E slots for graphics. Well that was easy. I went to the Beta unRAID 6.9 and the USB controller passed without any issues so easily. Wow! and thanks again! Now time to add another graphic card and for VM #2 Edited October 22, 2020 by Vaggeto 1 Quote Link to comment
Cadal Posted October 23, 2020 Share Posted October 23, 2020 On 10/21/2020 at 2:53 AM, jwoolen said: I did my testing with the Asus Zenith II Alpha. I used the two x16 slots. No overrides or unsafe interrupts. I dumped the bios files in a bare metal windows installation and removed the header from the ROM files with HxD. Okay got passthrough working on the second GPU... i found faulty PCIe power cables..... Got everything working now - thanks for your help. I am running beta though. 1 Quote Link to comment
SteelCityColt Posted October 23, 2020 Share Posted October 23, 2020 @bastl Sorry I didn't twig straight away you're on a different mobo. I don't have the equivalent controller to pass through sadly. So experimentation has show that 6.9 beta solves all issues, but I'm a bit twitchy about running a beta on my main server. I've got a PCIE USB card landing today that I'll try and maybe use as a temporary solution until 6.9 stable is released. Quote Link to comment
methanoid Posted December 10, 2020 Share Posted December 10, 2020 Great info thread (esp since I dont have a TRX40 board yet!). Re the two Matisse USB controllers with one only able to reset. Presumably that means unRAID USB on the OTHER Matisse one but all other controllers CAN be passed through? Also (sorry not run UnRAID for a year or more) is this FLR thing done yet? Thanks guys.. Wondering whether to splurge for a 24C TR for 3 VMs with GPUs 🙂 Quote Link to comment
turnipisum Posted December 10, 2020 Share Posted December 10, 2020 56 minutes ago, methanoid said: Great info thread (esp since I dont have a TRX40 board yet!). Re the two Matisse USB controllers with one only able to reset. Presumably that means unRAID USB on the OTHER Matisse one but all other controllers CAN be passed through? Also (sorry not run UnRAID for a year or more) is this FLR thing done yet? Thanks guys.. Wondering whether to splurge for a 24C TR for 3 VMs with GPUs 🙂 Forgot all about this thread! haha. I just swapped around the usb's that i pass through the other day without any issues on my system, so now thinking about it i don't think it's a issue anymore. Yeah do it get one! beast chip's so many threads haha! although you never have enough just keep using them. 😂 Quote Link to comment
jwoolen Posted December 11, 2020 Share Posted December 11, 2020 (edited) I pass both Matisse Controllers to my VM. I'm using multifunction to pass the whole IOMMU group. Edited December 11, 2020 by jwoolen Quote Link to comment
Vaggeto Posted December 11, 2020 Share Posted December 11, 2020 (edited) 2 hours ago, methanoid said: Great info thread (esp since I dont have a TRX40 board yet!). Re the two Matisse USB controllers with one only able to reset. Presumably that means unRAID USB on the OTHER Matisse one but all other controllers CAN be passed through? Also (sorry not run UnRAID for a year or more) is this FLR thing done yet? Thanks guys.. Wondering whether to splurge for a 24C TR for 3 VMs with GPUs 🙂 I just recently setup a system on the 3970x with 3 VMs (2 with pass-through graphics) and to be honest although I spent a couple days troubleshooting and figuring out how to pass things through, it's actually pretty easy to do and would only take me a couple hours if I had to start from scratch. I knew very little about VMs prior which also caused research to take me some time. unRAID 6.9 fixed one specific issue I was having and I wouldn't recommend trying without that although in many setups it likely would work but just with more manual tweaking. The biggest issue you'll have with 3 VMs is PCIe slots and their spacing with the motherboards but PCIe extender cables would easily solve that and I can't think of any other issues you might have. I used the Gigabyte TRX40 Aorus Master and would recommend it. Easy to pass at least 2 different sets of USBs to VMS, easy to pass both separate on-board audio outputs. Easy to pass graphics. Easy to pass NVME ports. Edited December 11, 2020 by Vaggeto Quote Link to comment
methanoid Posted December 11, 2020 Share Posted December 11, 2020 Looks to me very much like ALL TRx40 boards have 2 Starship and 2 Matisse (same group but can be separated) controllers, plus some ALSO have Asmedia controllers. So min 4, poss 5 passable controllers. So one for unRAID, and 3 or 4 for VMs. I can work out how to get 3 diff GPUs Might splurge after Xmas 🙂 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.