Functional Multi-Controller USB PCI-e Adapter


Prograde

Recommended Posts

Hello,

I have had great success with both ESXI and unRAID passing through a USB controller to VMs using the Buffalo IFC-PCIE2U3, which is a simple dual port card powered by a single Renesas µPD720202 surface mount.

Now you can imagine physical PCI-e ports can disappear quickly if you want to pass-through such native USB controllers to multiple VMs. What I have been looking for is a quad-port four-controller USB (3.0 or 2.0, just want some I/O) PCI-e card that can be split into four different pass-through devices to use with four different VMs similar to how something like an Intel I350-T4 can be split up into four separate NICs each with a single port.

I have found and tested a few devices that use a PLX chip to feed four separate USB controllers, and have had no success finding a reliable solution to work with unRAID.

Here are the two models I have tried so-far:

 

HighPoint RocketU 1144C with four Asmedia 1042A Controllers - Latest Firmware dated to 2014

 

IOMMU Splits up OK with ACS Override and VFIO blacklisting.
Passes through to VMs OK, being seen in the OS.

 

Windows: HighPoint drivers do work OK, and all is well on first boot. USB seems to work fine. Shutting down the VM and starting back up logs an error:

DMAR: [DMA Read] Request device [0a:00.0] fault addr ed000 [fault reason 06] PTE Read access is not set.

The port will not come back to functionality until the entire host is rebooted.

 

Ubuntu: Works perfectly with native drivers on first boot. Shutting down the VM and starting back up logs an error:

DMAR: [DMA Read] Request device [0a:00.0] fault addr ed000 [fault reason 06] PTE Read access is not set.

The port will not come back to functionality until the entire host is rebooted.

 

ESXI/Windows: Same issue after clean shut down and cold boot with not handing controller back to the host/VM properly.

 

StarTech PEXUSB3S44V with four Renesas µPD720202 Controllers - Latest Firmware dated to 2012


IOMMU Splits up OK with ACS Override and VFIO blacklisting.
Passes through to VMs OK, being seen in the OS.

 

Windows: Throws a Code 10 error, device cannot start directly on hub. Startech drivers and Microsoft xHCI drivers both do not work.

 

Ubuntu: Does not work at all, presumably due to same issue the windows VM was seeing.

 

ESXI/Windows: Seems to work identically to the IFC-PCIE2U3, in that everything functions both in and outside the VMs and between shut-downs and reboots. This is where I think there should be hope for the StarTech card with unRAID/KVM with some discrete setting, or as seen in the last link below, a kernel supporting USB Attached SCSI.


Are there any 4 or more port USB PCI-e controllers other users have found to work properly with unRAID? Or even two port cards? This would add a whole lot of value to the platform, having a reliable way to pass USB controllers en masse to VMs.

I have found a few references looking about the web for both cards, and some recent information regarding the Startech PEXUSB3S44V and a possible custom kernel:

 

HighPoint 1144C References:

 

https://www.redhat.com/archives/vfio-users/2016-June/msg00102.html

 

 

Startech PEXUSB3S44V References:

 

The last link is the most promising, and something I am looking to try out. I am willing to try out just about anything, having two spare test unRaid servers currently going and hardware in hand. Any insight or experience is welcomed.

 

Thank you for your help!

Edited by Prograde
Link to comment
  • 2 months later...

grr.  ok.  so people are saying that we need to have:

 

A quick fix consists in replacing the boot kernel parameter ‘intel_iommu=on’ with ‘intel_iommu=pt’ “to set up pass through (PT) mode in context mapping entry”. DMAR is then disabled in the GNU/Linux kernel but KVM still benefits IOMMU and interrupt remapping.

 

and ALSO:

intel_iommu=igfx_off

 

to not get the dma read error.

 

havent gotten a chance to try it since im doing a rebuild on my array.

 

Sources:

https://centoros.wordpress.com/2015/07/22/iommu-dmar-fault/

https://bbs.archlinux.org/viewtopic.php?id=176398

Link to comment

testing now, because people suck. aaaaand finished testing.  adding those lines into the config do not fix the problem with ASMedia usb controllers.  

 

maybe im adding the lines to the wrong part of the config?

 

label unRAID OS
  menu default
  intel_iommu=on
  intel_iommu=pt
  intel_iommu=igfx_off
  kernel /bzimage

 

but regardless this sucks for people with asmedia crap usb controllers.  

Link to comment

rob from another post states:

"

So I guess the best advice is to keep looking for updates to firmware, BIOS, and Linux kernel (the kernel core itself as well as the virtualization support, especially IOMMU, and the included drivers).

 

Some workarounds that are generally suggested, with possibly undesirable side-effects (choose only one!) -

- iommu=pt

- iommu=pt r8169.use_dac=1

- intel_iommu=pt

- intel_iommu=igfx_off

- iommu=off

- intel_iommu=off

- replace problem hardware

- disable virtualization support in BIOS
"

 

 

i tried the iommu=pt just by itself, and my intel usb controller started working.  but i wonder if the - iommu=pt r8169.use_dac=1  will do a more thorough job.

Link to comment
  • 3 weeks later...
On 10/29/2017 at 1:21 PM, TheFullTimer said:

I'm very curious about this because I've been having issues with a 7 port USB card ORICO Hub USB 3.0 PCI-E Expansion Card Adapter.

 

When I pass through a USB audio device the other devices connected stop working until the audio ceases.



That was one of the cards i was curious about.  are you able to pass through the slave and master controllers separately?  or does it just only show as one usb controller?

Link to comment

Here is a dump of my devices and you can see that it only passes through the VL805.  If I didn't stub it I wouldn't be able to pass it to anything.  If I cannot figure out how to make it work with USB audio without freezing the other devices on the controller then I'll swap it for something else.

 

Bus 1 --> 0000:00:1a.0 (IOMMU group 3)
Bus 001 Device 003: ID 0781:5575 SanDisk Corp. 
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 2 --> 0000:00:1d.0 (IOMMU group 7)
Bus 002 Device 004: ID 051d:0002 American Power Conversion Uninterruptible Power Supply
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

IOMMU group 7
[RESET] 00:1d.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 [8086:1c26] (rev 05)
IOMMU group 5
[RESET] 00:1c.4 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 [8086:1c18] (rev b5)
IOMMU group 3
[RESET] 00:1a.0 USB controller [0c03]: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 [8086:1c2d] (rev 05)
IOMMU group 11
[RESET] 06:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU group 1
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0101] (rev 09)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:0105] (rev 09)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb9] (rev a1)
[RESET] 02:00.0 USB controller [0c03]: VIA Technologies, Inc. VL805 USB 3.0 Host Controller [1106:3483] (rev 01)
IOMMU group 8
00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev a5)
[RESET] 07:05.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 10)
IOMMU group 6
[RESET] 00:1c.5 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 [8086:1c1a] (rev b5)
IOMMU group 4
[RESET] 00:1c.0 PCI bridge [0604]: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 [8086:1c10] (rev b5)
IOMMU group 2
00:06.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port [8086:010d] (rev 09)
[RESET] 03:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
IOMMU group 10
[RESET] 05:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU group 0
00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller [8086:0108] (rev 09)
IOMMU group 9
00:1f.0 ISA bridge [0601]: Intel Corporation C204 Chipset Family LPC Controller [8086:1c54] (rev 05)
[RESET] 00:1f.2 SATA controller [0106]: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller [8086:1c02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller [8086:1c22] (rev 05)

Link to comment
  • 2 months later...

I've been looking into this as well and wondered if anyone had any success with any multi port card so far...

Looked at the  High Point RocketU 1144C USB 3.0 PCIe as well though from this read it doesn't look like anyone is having any success with it so far... :(

Looks like some success has been had with this card:  

http://www.sonnettech.com/product/allegroprousb3pcie.html

Intel built a 4 VM machine for VR with each machine getting one of the USB 3.0 ports.  You can read all about it and watch the video here:

https://www.pcworld.com/article/3222652/gaming/how-we-hosted-a-star-trek-vr-party.html

They talk about the Allegro card at the 1:15 mark or so.

I'm looking to give 4 VMs each a port to run a keyboard, mouse and USB audio to each though I've heard some bad stuff about VMs and USB audio so far...

Sigh...

Link to comment
  • 4 weeks later...

Has anyone progressed the topic of multi-controller USB cards further?  I had read on another thread about the High Point RocketU 1144C working so ordered one and have had some success with it but am chasing what I think are some power related issues due to long USB runs (using active USB extenders).  I have the 1144C working and seems to survive VM re-boots/power downs when individual ports are passed to the VM's.

 

Unfortunately I didn't see this thread until after I ordered the Startech card.  I was thinking since it has a perhaps more robust power source via molex connector it may work better in my case.  I've run into the same as the others with it looking like it should work but the errors in Windows and non-functionality are present.   For fun I tried in another PC and works fine in bare metal Windows.  As well I tired a very recent Openelec via a USB boot and the Startech card appears to work fine (sees a USB disk attached into Openelec bare metal).

 

So it looks like the Startech card can work in Linux too... Unraid just doesn't see it nor can it be passed to a VM.   I'm just not sure what to check or change in Unraid and would be open to any suggestions to try before I send the card back.  

 

For the High point card.  On my server I could not get it to split into individual IOMMU's until I added the following to kernel boot parameters:

 

vfio-pci.ids=12d8:2608,1912:0015,8086:1d26,8086:1d2d vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=id:12d8:2608 irqpoll

12d8:2608 is the High point card.  Basically I have both onboard USB ports and the 1144C card set so UnRaid won't use/see the USB ports and it's the pcie_acs override part that finally broke the IOMMU into individual groups to allow VM passthrough:

 

Looks like this after that is added:

 

IOMMU group 22
	[10b5:8609] 04:00.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
IOMMU group 23
	[10b5:8609] 04:00.1 System peripheral: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
IOMMU group 24
	[10b5:8609] 05:01.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
IOMMU group 25
	[10b5:8609] 05:05.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
IOMMU group 26
	[10b5:8609] 05:07.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
IOMMU group 27
	[10b5:8609] 05:09.0 PCI bridge: PLX Technology, Inc. PEX 8609 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch with DMA (rev ba)
IOMMU group 28
	[1b21:1142] 06:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
IOMMU group 29
	[1b21:1142] 07:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
IOMMU group 30
	[1b21:1142] 08:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
IOMMU group 31
	[1b21:1142] 09:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

 Oddly, this is not at all needed for the Startech card... it split out fine with no overrides.

 

Just posting my results so that others may benefit and would also love to get the StarTech going, but am at a loss so giving up unless someone has some suggestions.  Happy to post more info if needed.  I'm running an ASRock EP2C602-4L/D16 & Dual Xeon E5 2680 setup.

Edited by mikeg_321
Link to comment
  • 1 month later...
On 3/3/2018 at 4:06 PM, Ph9214 said:

Linus tech tips found this one in this video, but its 100$, yikes

I can 100% confirm that this card does not support AMD's x399 chipset due to lack of thunderbolt support. No intel systems to test it on. I just returned it and picked up the PEXUSB3S44V to test and so far haven't even been able to get unraid to boot when devices are plugged into it. Very interested in this thread.

Link to comment
16 hours ago, Starlord said:

I can 100% confirm that this card does not support AMD's x399 chipset due to lack of thunderbolt support. No intel systems to test it on. I just returned it and picked up the PEXUSB3S44V to test and so far haven't even been able to get unraid to boot when devices are plugged into it. Very interested in this thread.

Do you have any powered USB hubs or powered USB devices plugged in when trying to boot?  I found that many hubs back-feed 5V into your USB system which can cause the PC to not boot.  I had to cut the 5v line back into the USB port on my hub because of this and just rely on an external power brick for the devices on that hub. My server wouldn't boot and/or would cause instability issues otherwise.... something to check anyhow. These cards seem very picky.

Link to comment
On 3/8/2018 at 11:29 AM, mikeg_321 said:

Do you have any powered USB hubs or powered USB devices plugged in when trying to boot?  I found that many hubs back-feed 5V into your USB system which can cause the PC to not boot.  I had to cut the 5v line back into the USB port on my hub because of this and just rely on an external power brick for the devices on that hub. My server wouldn't boot and/or would cause instability issues otherwise.... something to check anyhow. These cards seem very picky.

Even with everything unplugged I can't get it to post. 

Link to comment
  • 1 month later...

Posting back more results.  I just couldn't get the RocketU card to be stable.  Kept running into some type of problem where one of the ports would stop working after a while.  I suspect there's a power overload protection of some sort going on that I was bumping up against.  Card works fine with single devices per USB port , but once a hub (powered or un-powered) is in play it is problematic for me.  Now one of the Four ports now won't work at all.

 

I decided to try the Sonnet Allegro Pro USB 3.0 PCIe card (USB3-PRO-4PM-E).  It has been flawless so far (2 weeks) with hubs plugged into 3 of the 4 ports and no issues with many USB devices passed to various VM's.  IOMMU split out without any extra settings or overrides on my EP2C606-4L server.  

Edited by mikeg_321
clarified Allegro Pro as USB 3.0 & model number (the one I have functioning) - there is anew 3.1 card as well... I have not tried it
Link to comment
  • 2 weeks later...
On 5/5/2018 at 3:31 PM, mikeg_321 said:

I decided to try the Allegro Pro USB 3 card.  It has been flawless so far (2 weeks) with hubs plugged into 3 of the 4 ports and no issues with many USB devices passed to various VM's.  IOMMU split out without any extra settings or overrides on my EP2C606-4L server.  

 

Can't believe I'm going to spend $100 on a USB card, but sometimes 'just works' is worth the premium. Thanks for the update

Link to comment
On 5/5/2018 at 3:31 PM, mikeg_321 said:

I decided to try the Allegro Pro USB 3 card.  It has been flawless so far (2 weeks) with hubs plugged into 3 of the 4 ports and no issues with many USB devices passed to various VM's.  IOMMU split out without any extra settings or overrides on my EP2C606-4L server.  

 

Did you have to cut wires or do anything non-standard to use USB hubs with the Allegro Pro card? I haven't been able to get hubs to work with any of the cheaper USB cards I have.

Link to comment

Yeah, $100 for USB is nuts.  Think of it this way, it's actually 4 USB cards. which you can basically turn into 4 different computers via VM's  ?

 

I just looked and I have 2 non powered USB hubs that are working fine which didn't work on the other cards I had. They are out of the box with no mods. The other powered USB hub does appear to have a wire cut.  Suspect I cut +5v as it's powered and I was having back feed issues in the past.  My system is working so well I hate to play around and see if it would tolerate the back feeds or other hubs better or not though.  

 

Another thing to note is that my bios has an ACS on/of option which I have enabled. That may have helped in splitting the addresses out to individual IOMMU's.  Point being, there's so much variability between systems I'd just say make sure you order from somewhere you can return the board in case you don't have the same luck I have.  Amazon was great when I had to return the Startech.   

Link to comment
  • 4 weeks later...
On 3/7/2018 at 8:52 PM, Starlord said:

I can 100% confirm that this card does not support AMD's x399 chipset due to lack of thunderbolt support. No intel systems to test it on. I just returned it and picked up the PEXUSB3S44V to test and so far haven't even been able to get unraid to boot when devices are plugged into it. Very interested in this thread.

 

First post here on the forums, but with my long USB battle I wanted to post up my results.

I can 100% confirm that the Sonnet Allegro Pro card is the only one I have gotten to work on my X399/1920X build. I fumbled with the PEXUS Startech units and could pass them through to my VMs, but the drivers would either crash or the controller would fail to start correctly. They also had issues powering multiple HIDs when a hub was used.

I now have a dedicated controller passed to each of my three LAN gaming VMs with the Sonnet card. Each USB port has a hub plugged into the end to allow players to have their HIDs and their headsets connected. Audio is perfect and hot swap sure is nice to have now. Especially since a lot of the HIDs were of the same make/model and webGUI USB passthrough was very buggy for me.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.