MikeW Posted September 12, 2015 Share Posted September 12, 2015 I purchased a High Point RocketU 1144C USB 3.0 PCIe adapter because I read in different forums that each of the individual USB 3 controllers on the card can be assigned to different VMs. There is a PLX PCIe switch on the card which is used to connect to the four controllers. But when I installed the card and looked at the devices in UnRAID all four controllers are in the same IOMMU group. I tried different PCIe slots, but I get the same results. I have the ACS override setting enabled so I don't know why they devices aren't being put into their own IOMMU groups. Can anyone help me figure out why this is happening and if there is a way to solve this problem? Thanks. Quote Link to comment
MikeW Posted September 12, 2015 Author Share Posted September 12, 2015 Here is some more information from my system: PCI Devices: 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06) 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) 00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06) 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0) 00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0) 00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0) 00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0) 00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0) 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller 01:00.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) 01:00.1 System peripheral: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) 02:01.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) 02:05.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) 02:07.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) 02:09.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba) 03:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller 04:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller 05:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller 06:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller 07:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 730] (rev a1) 07:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1) 09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11) 0a:00.0 PCI bridge: ASMedia Technology Inc. Device 1184 0b:01.0 PCI bridge: ASMedia Technology Inc. Device 1184 0b:03.0 PCI bridge: ASMedia Technology Inc. Device 1184 0b:05.0 PCI bridge: ASMedia Technology Inc. Device 1184 0b:07.0 PCI bridge: ASMedia Technology Inc. Device 1184 0d:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 0f:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02) 10:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM] 10:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] 11:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller IOMMU Groups: /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/2/devices/0000:00:01.1 /sys/kernel/iommu_groups/3/devices/0000:00:02.0 /sys/kernel/iommu_groups/4/devices/0000:00:14.0 /sys/kernel/iommu_groups/5/devices/0000:00:16.0 /sys/kernel/iommu_groups/6/devices/0000:00:19.0 /sys/kernel/iommu_groups/7/devices/0000:00:1a.0 /sys/kernel/iommu_groups/8/devices/0000:00:1c.0 /sys/kernel/iommu_groups/9/devices/0000:00:1c.2 /sys/kernel/iommu_groups/10/devices/0000:00:1c.3 /sys/kernel/iommu_groups/11/devices/0000:00:1c.4 /sys/kernel/iommu_groups/12/devices/0000:00:1c.6 /sys/kernel/iommu_groups/13/devices/0000:00:1d.0 /sys/kernel/iommu_groups/14/devices/0000:00:1f.0 /sys/kernel/iommu_groups/14/devices/0000:00:1f.2 /sys/kernel/iommu_groups/14/devices/0000:00:1f.3 /sys/kernel/iommu_groups/15/devices/0000:01:00.0 /sys/kernel/iommu_groups/15/devices/0000:01:00.1 /sys/kernel/iommu_groups/15/devices/0000:02:01.0 /sys/kernel/iommu_groups/15/devices/0000:02:05.0 /sys/kernel/iommu_groups/15/devices/0000:02:07.0 /sys/kernel/iommu_groups/15/devices/0000:02:09.0 /sys/kernel/iommu_groups/15/devices/0000:03:00.0 /sys/kernel/iommu_groups/15/devices/0000:04:00.0 /sys/kernel/iommu_groups/15/devices/0000:05:00.0 /sys/kernel/iommu_groups/15/devices/0000:06:00.0 /sys/kernel/iommu_groups/16/devices/0000:07:00.0 /sys/kernel/iommu_groups/16/devices/0000:07:00.1 /sys/kernel/iommu_groups/17/devices/0000:09:00.0 /sys/kernel/iommu_groups/18/devices/0000:0a:00.0 /sys/kernel/iommu_groups/19/devices/0000:0b:01.0 /sys/kernel/iommu_groups/20/devices/0000:0b:03.0 /sys/kernel/iommu_groups/21/devices/0000:0b:05.0 /sys/kernel/iommu_groups/22/devices/0000:0b:07.0 /sys/kernel/iommu_groups/23/devices/0000:0d:00.0 /sys/kernel/iommu_groups/24/devices/0000:0f:00.0 /sys/kernel/iommu_groups/25/devices/0000:10:00.0 /sys/kernel/iommu_groups/25/devices/0000:10:00.1 /sys/kernel/iommu_groups/26/devices/0000:11:00.0 I believe IOMMU Group 15 contains all the USB controllers on the RocketU 1144C. Quote Link to comment
MikeW Posted September 19, 2015 Author Share Posted September 19, 2015 Could it be possible that something specific to the motherboard is preventing these independent controllers from being in their own IOMMU group? Others have used this board in KVM/ESXi/Xen because of how easy it is to assign the controllers to different VMs. Quote Link to comment
jonp Posted September 19, 2015 Share Posted September 19, 2015 One thing you can try is to avoid using VFIO and instead, use the KVM PCI device assignment mechanism. This may or may not work for you, but here's how you do it. First remove any XML edits you've made to pass through the controllers. Now add this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> </hostdev> This would pass through controller 03:00.0. Change the bus for each additional controller respectively. Let me know if this works for you or not. Quote Link to comment
johmei Posted January 28, 2016 Share Posted January 28, 2016 One thing you can try is to avoid using VFIO and instead, use the KVM PCI device assignment mechanism. This may or may not work for you, but here's how you do it. First remove any XML edits you've made to pass through the controllers. Now add this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> </hostdev> This would pass through controller 03:00.0. Change the bus for each additional controller respectively. Let me know if this works for you or not. Well I don't know if this helped the OP or not, but I wanted to let you know that this worked for me with my PCI sound card! I figured this would be some reassurance to someone that this will potentially solve the problem, and I also wanted to thank you! Quote Link to comment
jonp Posted January 28, 2016 Share Posted January 28, 2016 One thing you can try is to avoid using VFIO and instead, use the KVM PCI device assignment mechanism. This may or may not work for you, but here's how you do it. First remove any XML edits you've made to pass through the controllers. Now add this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> </hostdev> This would pass through controller 03:00.0. Change the bus for each additional controller respectively. Let me know if this works for you or not. Well I don't know if this helped the OP or not, but I wanted to let you know that this worked for me with my PCI sound card! I figured this would be some reassurance to someone that this will potentially solve the problem, and I also wanted to thank you! Really...that's very interesting. Thank you for the heads up. Quote Link to comment
johmei Posted January 28, 2016 Share Posted January 28, 2016 One thing you can try is to avoid using VFIO and instead, use the KVM PCI device assignment mechanism. This may or may not work for you, but here's how you do it. First remove any XML edits you've made to pass through the controllers. Now add this: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> </hostdev> This would pass through controller 03:00.0. Change the bus for each additional controller respectively. Let me know if this works for you or not. Well I don't know if this helped the OP or not, but I wanted to let you know that this worked for me with my PCI sound card! I figured this would be some reassurance to someone that this will potentially solve the problem, and I also wanted to thank you! Really...that's very interesting. Thank you for the heads up. Well I seem to have spoken too soon. It does boot up, and passes it through and it shows up in my Device manager, however, when I go to shut down the VM, the network goes splat and I can no longer access the webgui. I'm assuming this has something to do with the fact that unRAID's network controller is in the same group as the sound card. Sigh...well it works somewhat at least. Quote Link to comment
MikeW Posted March 29, 2016 Author Share Posted March 29, 2016 So it's only been 6+ months later and I'm finally getting around to trying jonp's suggestion for passing through the 1144C USB controller, but I'm still not having any luck even after editing the XML file as suggested. I get the following error when trying to start the VM: internal error: process exited while connecting to monitor: 2016-03-29T04:41:53.279928Z qemu-system-x86_64: -device pci-assign,configfd=24,host=10:00.0,id=hostdev0,bus=pci.0,addr=0x7: Failed to assign device "hostdev0": Invalid argument The HighPoint RocketU 1144C is a single PCIe card with four independent USB3 controllers. PCI Devices: 0b:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0b:00.1 System peripheral [0880]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0c:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0c:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0c:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0c:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 0e:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 0f:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 10:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] IOMMU Groups: /sys/kernel/iommu_groups/27/devices/0000:10:00.0 /sys/kernel/iommu_groups/27/devices/0000:0b:00.0 /sys/kernel/iommu_groups/27/devices/0000:0b:00.1 /sys/kernel/iommu_groups/27/devices/0000:0c:01.0 /sys/kernel/iommu_groups/27/devices/0000:0c:05.0 /sys/kernel/iommu_groups/27/devices/0000:0c:07.0 /sys/kernel/iommu_groups/27/devices/0000:0c:09.0 /sys/kernel/iommu_groups/27/devices/0000:0d:00.0 /sys/kernel/iommu_groups/27/devices/0000:0e:00.0 /sys/kernel/iommu_groups/27/devices/0000:0f:00.0 This is the code I added to the XML file: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='kvm'/> <source> <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </hostdev> I'm completely out of ideas. Any ideas why this isn't working? Thanks. Quote Link to comment
MikeW Posted March 30, 2016 Author Share Posted March 30, 2016 SUCCESS!!! These are all the devices associated with the High Point RocketU 1144C 4-port USB3 PCIe card and they were all in the same IOMMU group: 0b:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0b:00.1 System peripheral [0880]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0c:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0c:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0c:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0c:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba) 0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 0e:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 0f:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] 10:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] I applied the pcie_acs_override option, but all 4 USB controllers were always in the same IOMMU group. After googling a ton and reading many comments by Alex Williamson I decided to add the following to the pcie_acs_override option in the syslinux.cfg file: append pcie_acs_override=downstream,id:10b5:8609 initrd=/bzroot I can't seem to highlight the part I added, but I added the "id:10b5:8609". I'm not sure if this PCIe switch is on the motherboard or if it's on the 1144C card. By specifically adding this device to the pcie_acs_override option it split all of the USB controllers on the card into their own IOMMU group. I now have a single PCIe USB3 controller card which passes through individual USB3 ports to 4 different VMs. 1 Quote Link to comment
WildAngel Posted April 17, 2017 Share Posted April 17, 2017 Hello, I have the High Point RocketU 1144C 4-port USB3 PCIe card and I am trying to do exactly the same as you. I have in the Syslinux configuration this: "append pcie_acs_override=downstream,id:10b5:8609 initrd=/bzroot" Now it seems that each port belongs to it's own IOMMU group like: IOMMU group 23 [1b21:1142] 05:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller IOMMU group 24 [1b21:1142] 06:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller IOMMU group 25 [1b21:1142] 07:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller IOMMU group 26 [1b21:1142] 08:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller What is the next step to pass through each one of them into a VM ? Thanks, Alex Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.