Can't Split RocketU 1144C USB Controllers Into Separate IOMMU Groups


MikeW

Recommended Posts

I purchased a High Point RocketU 1144C USB 3.0 PCIe adapter because I read in different forums that each of the individual USB 3 controllers on the card can be assigned to different VMs. There is a PLX PCIe switch on the card which is used to connect to the four controllers. But when I installed the card and looked at the devices in UnRAID all four controllers are in the same IOMMU group. I tried different PCIe slots, but I get the same results. I have the ACS override setting enabled so I don't know why they devices aren't being put into their own IOMMU groups.

 

Can anyone help me figure out why this is happening and if there is a way to solve this problem? Thanks.

Link to comment

Here is some more information from my system:

 

PCI Devices:

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0)
00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0)
00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0)
00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
01:00.1 System peripheral: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
02:01.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
02:05.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
02:07.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
02:09.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
03:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
04:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
05:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
06:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
07:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 730] (rev a1)
07:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)
0a:00.0 PCI bridge: ASMedia Technology Inc. Device 1184
0b:01.0 PCI bridge: ASMedia Technology Inc. Device 1184
0b:03.0 PCI bridge: ASMedia Technology Inc. Device 1184
0b:05.0 PCI bridge: ASMedia Technology Inc. Device 1184
0b:07.0 PCI bridge: ASMedia Technology Inc. Device 1184
0d:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
0f:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
10:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
10:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]
11:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

 

IOMMU Groups:

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:19.0
/sys/kernel/iommu_groups/7/devices/0000:00:1a.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.2
/sys/kernel/iommu_groups/10/devices/0000:00:1c.3
/sys/kernel/iommu_groups/11/devices/0000:00:1c.4
/sys/kernel/iommu_groups/12/devices/0000:00:1c.6
/sys/kernel/iommu_groups/13/devices/0000:00:1d.0
/sys/kernel/iommu_groups/14/devices/0000:00:1f.0
/sys/kernel/iommu_groups/14/devices/0000:00:1f.2
/sys/kernel/iommu_groups/14/devices/0000:00:1f.3
/sys/kernel/iommu_groups/15/devices/0000:01:00.0
/sys/kernel/iommu_groups/15/devices/0000:01:00.1
/sys/kernel/iommu_groups/15/devices/0000:02:01.0
/sys/kernel/iommu_groups/15/devices/0000:02:05.0
/sys/kernel/iommu_groups/15/devices/0000:02:07.0
/sys/kernel/iommu_groups/15/devices/0000:02:09.0
/sys/kernel/iommu_groups/15/devices/0000:03:00.0
/sys/kernel/iommu_groups/15/devices/0000:04:00.0
/sys/kernel/iommu_groups/15/devices/0000:05:00.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/16/devices/0000:07:00.0
/sys/kernel/iommu_groups/16/devices/0000:07:00.1
/sys/kernel/iommu_groups/17/devices/0000:09:00.0
/sys/kernel/iommu_groups/18/devices/0000:0a:00.0
/sys/kernel/iommu_groups/19/devices/0000:0b:01.0
/sys/kernel/iommu_groups/20/devices/0000:0b:03.0
/sys/kernel/iommu_groups/21/devices/0000:0b:05.0
/sys/kernel/iommu_groups/22/devices/0000:0b:07.0
/sys/kernel/iommu_groups/23/devices/0000:0d:00.0
/sys/kernel/iommu_groups/24/devices/0000:0f:00.0
/sys/kernel/iommu_groups/25/devices/0000:10:00.0
/sys/kernel/iommu_groups/25/devices/0000:10:00.1
/sys/kernel/iommu_groups/26/devices/0000:11:00.0

 

I believe IOMMU Group 15 contains all the USB controllers on the RocketU 1144C.

Link to comment

One thing you can try is to avoid using VFIO and instead, use the KVM PCI device assignment mechanism.  This may or may not work for you, but here's how you do it.  First remove any XML edits you've made to pass through the controllers.  Now add this:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='kvm'/>

      <source>

        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>

      </source>

    </hostdev>

 

This would pass through controller 03:00.0.  Change the bus for each additional controller respectively.

 

Let me know if this works for you or not.

Link to comment
  • 4 months later...

One thing you can try is to avoid using VFIO and instead, use the KVM PCI device assignment mechanism.  This may or may not work for you, but here's how you do it.  First remove any XML edits you've made to pass through the controllers.  Now add this:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='kvm'/>

      <source>

        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>

      </source>

    </hostdev>

 

This would pass through controller 03:00.0.  Change the bus for each additional controller respectively.

 

Let me know if this works for you or not.

 

Well I don't know if this helped the OP or not, but I wanted to let you know that this worked for me with my PCI sound card!  I figured this would be some reassurance to someone that this will potentially solve the problem, and I also wanted to thank you!

Link to comment

One thing you can try is to avoid using VFIO and instead, use the KVM PCI device assignment mechanism.  This may or may not work for you, but here's how you do it.  First remove any XML edits you've made to pass through the controllers.  Now add this:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='kvm'/>

      <source>

        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>

      </source>

    </hostdev>

 

This would pass through controller 03:00.0.  Change the bus for each additional controller respectively.

 

Let me know if this works for you or not.

 

Well I don't know if this helped the OP or not, but I wanted to let you know that this worked for me with my PCI sound card!  I figured this would be some reassurance to someone that this will potentially solve the problem, and I also wanted to thank you!

Really...that's very interesting. Thank you for the heads up.

Link to comment

One thing you can try is to avoid using VFIO and instead, use the KVM PCI device assignment mechanism.  This may or may not work for you, but here's how you do it.  First remove any XML edits you've made to pass through the controllers.  Now add this:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='kvm'/>

      <source>

        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>

      </source>

    </hostdev>

 

This would pass through controller 03:00.0.  Change the bus for each additional controller respectively.

 

Let me know if this works for you or not.

 

Well I don't know if this helped the OP or not, but I wanted to let you know that this worked for me with my PCI sound card!  I figured this would be some reassurance to someone that this will potentially solve the problem, and I also wanted to thank you!

Really...that's very interesting. Thank you for the heads up.

 

 

Well I seem to have spoken too soon.  It does boot up, and passes it through and it shows up in my Device manager, however, when I go to shut down the VM, the network goes splat and I can no longer access the webgui.  I'm assuming this has something to do with the fact that unRAID's network controller is in the same group as the sound card.  Sigh...well it works somewhat at least. :P

Link to comment
  • 2 months later...

So it's only been 6+ months later and I'm finally getting around to trying jonp's suggestion for passing through the 1144C USB controller, but I'm still not having any luck even after editing the XML file as suggested. I get the following error when trying to start the VM:

 

internal error: process exited while connecting to monitor: 2016-03-29T04:41:53.279928Z qemu-system-x86_64: -device pci-assign,configfd=24,host=10:00.0,id=hostdev0,bus=pci.0,addr=0x7: Failed to assign device "hostdev0": Invalid argument

 

The HighPoint RocketU 1144C is a single PCIe card with four independent USB3 controllers.

 

PCI Devices:

0b:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0b:00.1 System peripheral [0880]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0c:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0c:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0c:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0c:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
0e:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
0f:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
10:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]

 

IOMMU Groups:

/sys/kernel/iommu_groups/27/devices/0000:10:00.0
/sys/kernel/iommu_groups/27/devices/0000:0b:00.0
/sys/kernel/iommu_groups/27/devices/0000:0b:00.1
/sys/kernel/iommu_groups/27/devices/0000:0c:01.0
/sys/kernel/iommu_groups/27/devices/0000:0c:05.0
/sys/kernel/iommu_groups/27/devices/0000:0c:07.0
/sys/kernel/iommu_groups/27/devices/0000:0c:09.0
/sys/kernel/iommu_groups/27/devices/0000:0d:00.0
/sys/kernel/iommu_groups/27/devices/0000:0e:00.0
/sys/kernel/iommu_groups/27/devices/0000:0f:00.0

 

This is the code I added to the XML file:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='kvm'/>
      <source>
        <address domain='0x0000' bus='0x10' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>

 

I'm completely out of ideas. Any ideas why this isn't working? Thanks.

Link to comment

SUCCESS!!!

 

These are all the devices associated with the High Point RocketU 1144C 4-port USB3 PCIe card and they were all in the same IOMMU group:

0b:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0b:00.1 System peripheral [0880]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0c:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0c:05.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0c:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0c:09.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA [10b5:8609] (rev ba)
0d:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
0e:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
0f:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
10:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]

 

I applied the pcie_acs_override option, but all 4 USB controllers were always in the same IOMMU group. After googling a ton and reading many comments by Alex Williamson I decided to add the following to the pcie_acs_override option in the syslinux.cfg file:

append pcie_acs_override=downstream,id:10b5:8609 initrd=/bzroot

 

I can't seem to highlight the part I added, but I added the "id:10b5:8609".

 

I'm not sure if this PCIe switch is on the motherboard or if it's on the 1144C card. By specifically adding this device to the pcie_acs_override option it split all of the USB controllers on the card into their own IOMMU group.

 

I now have a single PCIe USB3 controller card which passes through individual USB3 ports to 4 different VMs.  ;D

  • Upvote 1
Link to comment
  • 1 year later...

Hello,

 

I have the High Point RocketU 1144C 4-port USB3 PCIe card and I am trying to do exactly the same as you.

 

I have in the Syslinux configuration this:   "append pcie_acs_override=downstream,id:10b5:8609 initrd=/bzroot"

 

Now it seems that each port belongs to it's own IOMMU group like:

 

IOMMU group 23
	[1b21:1142] 05:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
IOMMU group 24
	[1b21:1142] 06:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
IOMMU group 25
	[1b21:1142] 07:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
IOMMU group 26
	[1b21:1142] 08:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

 

What is the next step to pass through each one of them into a VM ?

 

Thanks,

Alex

 

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.