USB Controller Card Separation

[littleman11186]

I have added a 4 port USB card to my unraid setup to allow VM assignment but I'm having difficulty breaking the new device out of the primary motherboard IOMMU group. Here is what I see when I plug in the device:

 

IOMMU group 13:[1022:43ba] 01:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset USB 3.1 xHCI Controller (rev 02)

[1022:43b6] 01:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset SATA Controller (rev 02)

[1022:43b1] 01:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X399 Series Chipset PCIe Bridge (rev 02)

[1022:43b4] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 02:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 02:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 02:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 02:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[168c:003e] 03:00.0 Network controller: Qualcomm Atheros QCA6174 802.11ac Wireless Network Adapter (rev 32)

[1ae9:0310] 04:00.0 Network controller: Wilocity Ltd. Wil6200 802.11ad Wireless Network Adapter (rev 02)

[8086:1539] 05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

[1912:0014] 06:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)

[1b21:2142] 08:00.0 USB controller: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller

 

1912:0014 is the id for the device and I've attempted to isolate it on the Syslinux configuration:

 

kernel /bzimage
append amd_iommu=on iommu=pt iommu=1 pcie_acs_override=downstream vfio-pci.ids=1912:0014 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,/bzroot-gui

 

I am unable to assign the device to the VM because the rest of the devices do not apply. What can I do?

 

 

 

Edited December 5, 2019 by littleman11186

[littleman11186]

I tried with overrides set to multifunction and downstream which did separate out the IOMMU group, but the VM will not start up correctly when I attach the device. The VM just shows the splash screen of the open source framework. Once I unassign the USB controller it is able to start into windows no problem. I don't see any errors in the VM logs. Any suggestions on how to proceed?

[testdasi]

Tools -> Diagnostics -> attach zip file.

 

Also, have you tried passing through the chipset USB 3.0 controller instead? Saving yourself the trouble with additional USB cards.

[littleman11186]

Here's the diagnosti cs, I did try to pass through the chipset USB controller but had connectivity issues with one of them. Here's what I've done

 

1. Setup 2 VM's with Windows 10

2. Attach a USB controller to each

     a. 2 of the 5 controllers (that are not in group 13) are assignable

     b. 1 of those 2 work perfectly for what I need. 

     c. the other has connectivity issues

          i. attached USB controller and started VM

          ii. connected Occulus Rift S to the assigned slots

          iii. Occulus device drops intermittently during setup, rendering it unusable

 

3. purchase USB card

4. have difficulty assigning USB card

     a. cannot isolate from IOMMU group 13 with acs overrides

     b. cannot utilize "downstream,multifunction" overrides for some reason because VM won't start after USB card assignment

tower-diagnostics-20191205-1839.zip

Edited December 6, 2019 by littleman11186

[testdasi]

A few points:

• Any particular reason why you are still on 6.6.6? You should update to the latest Unraid unless there's a compelling reason not to.
  • If 6.7.2 or 6.8.0 (when it gets released as stable) doesn't help, you might want to wait for 6.9.0-rc1 (which LT said should come out rather quickly after 6.8.0 stable) since it has 5.x kernel which, in my case, helps stabilise ACS Override.
• Your syslinux is a mash up of various things. You should:
  • Remove iommu=pt
  • Remove iommu=1
  • Remove pcie_acs_override=id:1912:0014
  • Add pcie_acs_override=downstream,multifunction
  • Add vfio-pci.ids=1912:0014
• For your chipset USB controller, you might want to check BIOS for any funky settings e.g. stuff for external sound card etc. That might be the cause of the instability of the other port.

Now I know you said multifunction doesn't work but having the card in the same IOMMU as other critical devices definitely won't work.

We can try tweaking the xml instead.

 

Can you attach the xml of the troublesome machine. If copy-pasting, please use the code functionality (the </> button next to the smiley button).

Edited December 6, 2019 by testdasi

[littleman11186]

No reason for 6.6.0 other than not clicking update. I have made the suggested syslinux config changes and rebooted. I see the IOMMU groups much more spread out and I am able to assign the device to the VM. However, when the VM starts the attached screenshot and nothing else. I will attempt the upgrade now and see if that helps.

 

vmNotUtilizingUSBXml.xml VMlogs.txt

[testdasi]

Let's try these:

• Update to latest (6.8.0-rc9)
• Change your Q35 machine type to Q35 4.0.1 (6.8.0 has qemu 4.x which has better PCIe support for Q35 machine type).
• Add <ioapic driver='kvm'/> to before </features> in your xml
• Tools -> Diagnostics -> attach new zip file (every time you make major changes like ACS Override etc, it's always worthwhile to dump a new set of diagnostics since your config would change rather drastically).

[littleman11186]

Same behavior after update to 6.7.2.

 

Edit: Just saw your post, pushing up to 6.8.0-RC9 now, and making other recommended changes.

Edited December 8, 2019 by littleman11186

[littleman11186]

Posting updated tower diagnostics. I have attempted the following with no luck:

 

- Q35- 4.0.1

- Q35- 4.1

- USB Controller - 3.0 qemu

- USB Controller - 3.0 nec

tower-diagnostics-20191208-1557.zip

[littleman11186]

VM starts no problem when card is deselected.

[littleman11186]

Re-tested USB Occulus headset playability and got through a whole song in Beat Saber before hearing a high pitched beep then the game froze and went black. The whole time it had a very static-y fuzz to all the audio as well.

Edited December 8, 2019 by littleman11186

[littleman11186]

After doing some snooping I found that there were 2 more USB 3 ports available on my motherboard under the ethernet slot that I was able to plug into the 1st vm. I can now play occulus on both VM's at once. Time to setup echo VR.