Split IOMMU Group to separate USB devices of one USB controller for VM pass-through

[DonPadro]

I believe that I bought the wrong hardware and that I can't do what I want.

 

My goal is to build one PC to handle several tasks. This should result in a low power consumption :

1. High speed network between PC and Unraid via direct connection 2.5MBit/s, ZFS incl. data compression on 14T harddisk => Works perfectly

2. Wlan acces point via OpenWRT VM using the internal wlan card => That's I'm working on.

3. TV/media via Windows VM (VPN to Zattoo), Youtube web browsing,...)

 

My hardware is

   Mobile CPU powered Mini PC PN41 from ASUStec with 6W max power and 2.5MBit/s network.

   I exchanged the wlan card from INTEL to MediaTek to have OpenWRT driver support

   I added external 12V power supply for 3,5" hard disc (PC works only with 5V)

 

Now to my issue.

I found out very late that the USB controller is handling too much devices, that I need to separate for the VMs.

Number one issue is the USB stick of Unraid (Bus 002 Device 003 Flash Drive).

If I give the complete group to one VM, than Unraid has no access to the boot device. This can't be healthy:

   IOMMU group 3:[8086:4ded] 00:14.0 USB controller: Intel Corporation Device 4ded (rev 01)

      Bus 001 Device 001 Port 1-0 ID 1d6b:0002 Linux Foundation 2.0 root hub

      Bus 001 Device 002 Port 1-8 ID 0e8d:0608 MediaTek Inc. Wireless_Device

      Bus 002 Device 001 Port 2-0 ID 1d6b:0003 Linux Foundation 3.0 root hub

      Bus 002 Device 002 Port 2-2 ID 090c:1000 Silicon Motion, Inc. - Taiwan (formerly Feiya Technology Corp.) Flash Drive

      Bus 002 Device 003 Port 2-5 ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter

   IOMMU group 14:[10ec:8125] 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)

 

I have another USB wlan stick and USB ethernet 2.5GB/s stick which are in the same IOMMU group 3.

The USB ethernet 2.5GB/s worked already and I could find it in the network setting.

But after the last actual update of Unraid, now I can't find anymore.

It would be interesting to know why but it's not my main issue.

The USB wlan stick have never been seen in the network setup.

 

To all what I read already about (and seen on spaceinvader One video), I can't splitt the devices from USB controller.

I tried "Enable PCIe ACS Override=Downstream, Multi-Function, Both" but nothing changed.

The only thing that I didn't tryed is, as I believe its the same as  Enable PCIe ACS Override=both:

   append pcie_acs_override=downstream,multifunction initrd=/bzroot

 

What do you thing?

Do you see a chance to get the MediaTek card out of the IOMMU group 3 to use it in a VM?

 

Best regards,

Edited September 20, 2022 by DonPadro
To read better the text.

[JonathanM]

2 hours ago, DonPadro said:

If I give the complete group to one VM, than Unraid has no access to the boot device.

Yeah, that's not gonna fly.

 

Have you tried the USB manager plugin? It may give you the flexibility you need with devices and VM's.

 

2 hours ago, DonPadro said:

The USB ethernet 2.5GB/s worked already and I could find it in the network setting.

But after the last actual update of Unraid, now I can't find anymore.

That may be the result of a driver that got missed in the latest versions, I believe it should be added back in the next RC

 

2 hours ago, DonPadro said:

The USB wlan stick have never been seen in the network setup.

Unraid doesn't support wireless. Server + wireless is a bad combo to try to support.

[ghost82]

9 hours ago, DonPadro said:

The USB wlan stick have never been seen in the network setup

If you want to use wifi for the server and you have a spare ethernet port, you can use an ethernet --> wifi bridge; for my servers I use:

TP-link TL-WR802N (amazon euro 23,99)

Vonets VAP11G-500 (amazon euro 36,99)

 

Or any other ethernet --> wifi bridge, setup as client.

I prefer the tp-link bridge over the vonets and it's running stable on my servers from a long time.

[DonPadro]

I wasn't aware of the USB manager.

I'll give it a try and let you know about my results.

But first I need to read the 10 pages...

Thanks for you help so far.

[DonPadro]

I'd like to give me my feedback using USB manager:

I selected the following devices to the VM:

   ASIX Electronics Corp. AX88772B (0b95:772b)

   RealtekCorp. USB 10/100/1G/2.5G LAN (0bda:8156)

   MediaTek Inc. Wireless_Device (0e8d:0608)

 

In VM I installed Win10 and had access to:

   ASIX Electronics Corp. AX88772B (0b95:772b)

   RealtekCorp. USB 10/100/1G/2.5G LAN (0bda:8156)

 

About ediaTek Inc. Wireless_Device (0e8d:0608):

   Windows could see only the Bluethooth part and had problems with this.

   Maybe the device is broken anyhow?!?!

 

I did a speed test with RealtekCorp. USB 10/100/1G/2.5G LAN (0bda:8156) connected to unraid via a second 2.5GB/sec Realtek, but performance was very low (maybe 5% speed). I checked with the VM Network card and this was very fast.

 

 I installed Ubuntu VM and OpenWRT VM, but MediaTek Inc. Wireless_Device (0e8d:0608) was not detected.
 

I tried with kernel boot option:

append pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,

 

Than I was trying to us the PC as media player connecting a monitor. But I wasn't able that the VM uses the monitor.

I tryed with and without sound, as sound is in a different IOMMU group, and sound isn't allown:

IOMMU group 13:

[8086:4d87] 00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)

[8086:4dc8] 00:1f.3 Audio device: Intel Corporation Jasper Lake HD Audio (rev 01)

[8086:4da3] 00:1f.4 SMBus: Intel Corporation Jasper Lake SMBus (rev 01)

[8086:4da4] 00:1f.5 Serial bus controller: Intel Corporation Jasper Lake SPI Controller (rev 01)

 

Thus up to know, my hardware don't make me happy.

I was expecting, that the monitor for VMs don't make me trouble as I was thinking that this works without extra work, but I failed.

 

Maybe I just continue to use the x86 as Unraid server and pfsense firewall.

 

If you have some more ideas to test, please let me know.

 

[SimonF]

27 minutes ago, DonPadro said:

I'd like to give me my feedback using USB manager:

I selected the following devices to the VM:

   ASIX Electronics Corp. AX88772B (0b95:772b)

   RealtekCorp. USB 10/100/1G/2.5G LAN (0bda:8156)

   MediaTek Inc. Wireless_Device (0e8d:0608)

 

In VM I installed Win10 and had access to:

   ASIX Electronics Corp. AX88772B (0b95:772b)

   RealtekCorp. USB 10/100/1G/2.5G LAN (0bda:8156)

 

About ediaTek Inc. Wireless_Device (0e8d:0608):

   Windows could see only the Bluethooth part and had problems with this.

   Maybe the device is broken anyhow?!?!

 

I did a speed test with RealtekCorp. USB 10/100/1G/2.5G LAN (0bda:8156) connected to unraid via a second 2.5GB/sec Realtek, but performance was very low (maybe 5% speed). I checked with the VM Network card and this was very fast.

 

 I installed Ubuntu VM and OpenWRT VM, but MediaTek Inc. Wireless_Device (0e8d:0608) was not detected.
 

I tried with kernel boot option:

append pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,

 

Than I was trying to us the PC as media player connecting a monitor. But I wasn't able that the VM uses the monitor.

I tryed with and without sound, as sound is in a different IOMMU group, and sound isn't allown:

IOMMU group 13:

[8086:4d87] 00:1f.0 ISA bridge: Intel Corporation Device 4d87 (rev 01)

[8086:4dc8] 00:1f.3 Audio device: Intel Corporation Jasper Lake HD Audio (rev 01)

[8086:4da3] 00:1f.4 SMBus: Intel Corporation Jasper Lake SMBus (rev 01)

[8086:4da4] 00:1f.5 Serial bus controller: Intel Corporation Jasper Lake SPI Controller (rev 01)

 

Thus up to know, my hardware don't make me happy.

I was expecting, that the monitor for VMs don't make me trouble as I was thinking that this works without extra work, but I failed.

 

Maybe I just continue to use the x86 as Unraid server and pfsense firewall.

 

If you have some more ideas to test, please let me know.

 

Which USB emulation are you using on the VM Template? 2.0 or 3.0?

 

[DonPadro]

I tried the 2 versions of 3.0.

2.0 shows error message starting the VM.

 

With Redhad 10G internal interface, I get 300-130MB having 100% CPU load (Yes it's a low performance CPU).

With Realtek 2.5G in and out I get 17-25MB with 80 CPU load.

I copy from/to the same SSD.

 

If I copy between Laptop and Unraid via 2,5G interface, I get full network performance