Automatic USB hotplugging for specific USB ports


midgard00

Recommended Posts

Hello,

I want to demonstrate a method of enabling USB hotplugging on specific USB ports to specific VMs, that is only active while the VM is running. The behaviour is the similar to USB controller passthrough, but is applied to specific USB ports rather than all ports on the controller. It does not require passthrough or VFIO binding of the controller to which the port belongs. It can thus be used for USB ports connected to the controller that hosts the Unraid USB Drive. Calling this method "USB port passthrough" would be a fitting description of the behaviour, but this is not what is actually happening.

The way this works is to use a script that monitors which VMs are currently running. This script enables (or disables) udev rules when a VM is started (or stopped). These udev rules manage the hotplugging and specify which USB port shall be "passed through" to which VM. They are automatically triggered when a device inserted into the USB port and then call another script that attaches the newly connected device to the specified VM.
That means, while the VM is running, all USB devices plugged into the specified ports are automatically attached to the VM. While the VM is shut down, the USB devices are accessible to Unraid. It is also possible to have one USB port assigned to multiple VMs, if no more than one of these VMs is running simultaneously.


When to use this method?
I wanted to have a way to automatically pass any USB device connected to specific ports through to specific VMs. Passing an entire USB controller to a VM is the easiest way to get this to work. However, if we want to use more VMs than we have USB controllers, or we can't pass a controller through for some other reason, we need another option.
I would recommend USB controller passthrough over this method, if possible, as it is much easier to set up or modify. This method takes some time and technical understanding to set up, but has now worked reliably for several months for me.
There is only one thing that is currently not working: If a USB device is already plugged into a port designated for "passthrough" before the corresponding VM is started, it is not automatically attached to the VM, but needs to be unplugged and plugged in again. I have an idea on how to address this, but have not gotten around to testing it.

Unfortunately, since it took me quite some time to develop this method I am not able to give full credit to all the sources that helped me. Among them were definitely these two posts:
http://kicherer.org/joomla/index.php/en/blog/48-automatic-hotplugging-of-usb-devices-for-libvirt-managed-vms
https://www.labsrc.com/unraid-automatic-usb-hotplugging-libvirt/
as well as this repository, which also contains the script I am using for attaching the USB devices to the VMs:
https://github.com/olavmrk/usb-libvirt-hotplug


How to set this up?
The first and most complicated step is to identify the addresses of the USB ports that we want to "pass through". After that we can write the udev rules for these ports. The rules specify to which VM the USB device should be passed. After testing these rules, we will setup the script to enable or disable the rules when the corresponding VMs are started or stopped.

 

Identifying USB controllers
First, we need to get to know our motherboards USB layout. I am using an MSI Creator X299 and will use it as an example here to go through the process. First we can take a look at Unraids System Devices page to learn what USB controllers we have and what their addresses are:


Unraid System Devices

 

On this motherboard there are 3 USB controllers (marked red) with the addresses 00:14.0, 03:00.0 and 04:00.0. The latter two ( ASM3242 and ASM2142) are chips that bridge PCIe to USB. Thus, they can be passed through in the traditional way. From the motherboards manual I know that the ASM3242 provides the USB-C port on the rear I/O and the ASM2142 provides the internal Type-C header.


Motherboard Spec

 

The remaining controller is the one that is integrated into the X299 Chipset. Of the devices attached to it I can identify the Unraid USB drive (green), some other known USB devices (yellow) and two other ASM chips (blue). These chips are USB hubs (ASM1074) with one upstream and four downstream ports. The manual also tells me that the USB 3 ports on the rear I/O are connected to these ASM1074s, while the USB 2 ports and the internal headers are connected directly to the chipset. Thus the USB layout looks like this:


USB Layout

 

To identify which port belongs to which controller you can also simply plug a USB device into the port in question, refresh the Unraid System Devices page and note under which controller the device is listed.

You might have noticed that the two ASM1074 hubs were listed twice in the Unraid System Devices page. Once in bus 001 and once in bus 002. Bus 001 is the USB 2 bus, which i know because the two water pumps are connected to USB 2 headers. Bus 002 is the USB 3 bus, where also the Unraid flash drive is connected through one of the internal USB 3 headers. This will be important later on.

 

Identifying USB ports
Now we know which USB ports on the motherboard belong to which controller. Now we need to find out the device addresses of the USB ports. In an unraid terminal, navigate to /sys/devices.


sys_devices.thumb.png.a169e96f164837cd8ea688c426e00cf1.png

 

Among other things, there should be a folder called pci0000:XX, where XX are the first to digits in the address of the USB controller in the Unraid System Devices page. I will focus on the chipset controller here, so it is 00. Enter that folder.


sys_devices_2.thumb.png.99841ee9d481dfefed5a925922c48380.png


Inside that folder there should be more folders with names that match the addresses of several chipset devices from the Unraid System Devices page. We already know that the chipset USB controller has the address 00:14.0, so we enter the corresponding folder.

 

sys_devices_3.thumb.png.aa7433b77313a505268183e960bba9a5.png


In here vendor and device should contain the numbers shown in brackets before the address on the Unraid System Devices page. The usb1 and usb2 folders correspond to bus 001 and bus 002 from the Unraid System Devices page. Lets start with usb1. As mentioned above this is the USB 2 bus, so inside this folder we should be able to find all USB 2 ports connected to this controller (which in this example are all USB 2 ports on the motherboard).


sys_devices_4.thumb.png.de93e81ad6465dcac769981c36c0fa96.png


There are 6 folders inside (marked red), each corresponding to one of the 6 attached USB 2 devices. Note that the two ASM1074 hubs are included in this count, and any external USB hubs plugged into one of these ports would also be, because this is a tree-like structure. A folder corresponding to a hub will itself contain a folder for each port on the hub. Note also that we exclude the root hub and the folder 1-0:1.0 (meked yellow) because these are not of interest here.
When I insert a USB stick into one of the USB 2 ports on the rear I/O, it now looks like this:


sys_devices_5.thumb.png.e34d5d21651dd57aa3c4abdf7ae931a3.png


The folder 1-11 has appeared. Any time I insert any device into that specific port, the folder 1-11 will appear here. I will refer to 1-11 as the address of that specific port, I don't know if that is technically the correct term, but the important thing is that it does not change. Even after a reboot, that number is specific to that port, unlike e.g. the "Device" number shown by lsusb, which changes each time the USB stick is plugged out and in again.
We can also get some information about the connected devices:


sys_devices_6.thumb.png.f87ba9fd258fc6966010ac05a78f6ed5.png


Each of the 1-XX folders will also contain another folder called 1-XX:1.0. With that, we have the full address of the USB port and the device that is connected to it.


sys_devices_7.thumb.png.0e8e6137f8a2e932a58e0c59e5d32014.png


The USB drive I just inserted has the full address

/sys/devices/pci0000:00/0000:00:14.0/usb1/1-11/1-11:1.0

In fact, any USB device connected to the same port will have that same address.


address.thumb.png.ed737d5ec8ab2533f24088a6c834037e.png


Things get a little more complicated when the USB device is not as simple as a USB stick, a mouse or a keyboard:


sys_devices_8.thumb.png.cd29b3850d20e0fff585fd4b0e5c0458.png


Some devices consist of several seperate devices, as is the case here with the Intel Bluetooth adapter on port 1-13 marked in green. For hubs like the ASM1074 on port 1-6, the folder 1-6:1.0 describes the hub itself, while the other folders 1-6.1 through 1-6.4 correspond to the ports provided by the hub and only appear, if a device is plugged into that port. I have here inserted devices into the ports 3 and 4 of the hub on port 1-6. The former has the complete address

/sys/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.3/1-6.3:1.0

If I were to insert a USB hub into that same port and the device into the first port of that hub, the address would look like this:

/sys/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.3/1-6.3.1/1-6.3.1:1.0

 

If we now take a look at the usb2 folder, the structure is the same:


sys_devices_9.thumb.png.e3f7d3f0bcd7dc058509a201020b6320.png


It is important to note that one physical USB 3 port has two separate addresses, one for operation in USB 3 mode and one for operation in USB 2 mode. My Unraid flash drive has the address

/sys/devices/pci0000:00/0000:00:14.0/usb2/2-3/2-3:1.0

If I were plug a USB 2 stick into the same port, it would have the address

/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0

I recommend pluggin one USB device into one port after another, checking which folders pop up in the usb1 and usb2 folders, and thus noting which USB port has which address.

 

Writing udev rules
Now that we know the addresses of each physical port on out motherboard, it is time to write the udev rules that define which ports should be "passed" to which VM. We will use the udev rule to automatically execute a script whenever a device is plugged in or out of the port specified in the rule. The script will attache (or detach) the device to the VM. I am using the "usb-libvirt-hotplug" script from here for this purpose. Download it and save it somewhere on the Unraid machine.

 

The udev rule will look something like this:

SUBSYSTEM=="usb",DEVPATH=="path_to_usb_port",RUN+="path_to_script VM_name"

As previously determined, a USB device plugged into one of the USB 2 ports on my rear I/O has the address

/sys/devices/pci0000:00/0000:00:14.0/usb1/1-11/1-11:1.0

If I want to automatically pass devices connected to that port through to a VM called "Ubuntu", the udev rule for that will look like this:

SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-11",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"

Note that in the "path_to_usb_port" the leading /sys is excluded. Also the last section /1-11:1.0 is excluded, meaning we only use the address to the USB port and not to the device attached.

If that port were a USB 3 port and I also wanted to attach USB 3 devices to the VM, a second rule would be required:

SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-11",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"

Write this rule (or these two rules, each in one line) into a file called 90-testrule.rules and save it into the /etc/udev/rules.d/ directory. Then execute

udevadm control --reload-rules

This tells Unraid to reload the udev rules, activating this newly added one. Do this while the target VM is running.
USB devices plugged into this port should now automatcally be attached to the VM.

 

What about hubs?
As mentioned above, my motherboard has two ASM1074 chips, so essentially two USB hubs directly on board. These have the addresses 2-5 (1-5 when operating at USB 2 speed) and 2-6 (1-6 when operating at USB 2 speed). The latter one hosts four USB 3 ports on my rear I/O. If I want to atutomatically attach all USB device inserted into these four ports to a VM called "Ubuntu", the .rules file for that would look like this:

SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.2",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"

The first four lines handle USB 2 devices, the other handle USB 3 devices. Now say I know that I will additionally attach a USB 2 hub to port 2, which in turn hosts four USB 2 ports. The .rules file would look like this:

SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2/1-6.2.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2/1-6.2.2",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2/1-6.2.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.2/1-6.2.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb1/1-6/1-6.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.1",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.3",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"
SUBSYSTEM=="usb",DEVPATH=="/devices/pci0000:00/0000:00:14.0/usb2/2-6/2-6.4",RUN+="/mnt/user/appdata/scripts/usb-libvirt-hotplug.sh Ubuntu"

The line for 2-6.2 is missing, because there wont be any USB 3 devices attached to that port. Also the four rules for the additional hub are only present for the USB 2 bus, because the hub only supports USB 2. (As far as I know adding more rules that will never trigger does not harm anything.)

So for hubs this method is not very flexible, but as long as the hub stays in a fixed position it is fine.

 

Enabling and disabling rules
As we have seen before, activating a rule is simply done by copying the rules file into /etc/udev/rules.d/ and running udevadm control --reload-rules. Disabling the rule is equally simple: Remove the .rules file from /etc/udev/rules.d/ and run udevadm control --reload-rules again.

If a rule is active but the corresponding VM is shut down and a USB device is inserted into a port managed by the rule, the script will attempt to attach the device to the VM and fail, because it is shut down. This will generate some error messages in the syslog but is otherwise not a problem. Still, it would be nice to have rules automatically enabled when their corresponding VM ist started and disabled when the VM is stopped. This would also make it possible to use the same ports for two VMs that only run mutually exclusive: VM1 starts -> rules for VM1 are loaded -> VM1 is shut down -> rules for VM1 are disabled -> VM2 starts -> rules for VM2 are loaded -> ...


For this purpose I have written a script that does exactly that: You can specify the names of all VMs for which you want rules to be loaded and provide a .rules file for each of these VMs. The script monitors the state of these VMs and enables (or disables) the rules when the VMs are started (or stopped) by copying the corresponding .rules files into /etc/udev/rules.d/ and running udevadm control --reload-rules.

 

The code:

#!/usr/bin/python3
import subprocess
import time
import syslog

#####################################
### User configuation starts here ###
#####################################

# This is the list of VMs managed by this script. 
# Add the names of the VMs for which you want to enable hotplugging here.
# For each VM listed here a .rules file is required, see documentation for details.
managed_vms = ['VM-name1','VM-name2','VM-name3']

# The path to the directory where the .rules files are stored.
path_to_rules = '/mnt/user/appdata/udevrules/'

# The .rules files should be named 'someprefix-vmname.rules'. The 'vmname' part must
# be the same as in the 'managed_vms' list above and the prefix is defined here.
rules_prefix = '90-usbhotplug-'

#################################################
### No changes should be necessary below here ###
#################################################



# Uses the 'virsh list' command to get the state and names of all VMs on the host
# and returns it as a formated list.
def get_vms():
    vms = []
    # call 'virsh list --all' to get all stopped and running VMs
    p = subprocess.run(['virsh', 'list', '--all'], stdout=subprocess.PIPE, universal_newlines=True)
    # split the returned string into lines and exclude the header
    p = p.stdout.strip().split('\n')[2:]
    # for each line of the output
    for line in p:
        # add the name and the state to the return list
        vms.append(line.split()[1:3])
    return vms



# Initialize states of all managed VMs. The state is a tuple of the VM name and the
# state string, which is either 'shut' when stopped or 'running' when running.
vm_state = []
for vm in managed_vms:
    # initially all vms are stopped
    vm_state.append([vm,'shut'])



syslog.syslog("VM USB hotplug monitor started.")
while(True):
    # for each VM defined on the host
    for vm in get_vms():
    
        # get previous state of the vm
        old_state = []
        for v in vm_state:
            if vm[0] == v[0]: # match VM names
                old_state = v
        
        # if len(old_state) == 0, the VM is not in the managed_vms list
        if len(old_state) > 0:
            
            # if VM was previously running and is now shut down
            if old_state[1] == 'running' and vm[1] == 'shut':
                # remove udev rule for this VM
                subprocess.run(['udevadm', 'control', '--reload-rules'])

                syslog.syslog("Stopped " + vm[0] + " hotplugging.")
                # update vm_state
                vm_state[vm_state.index([vm[0], 'running'])][1] = 'shut'

            # if VM was previously shut down and is now running
            if old_state[1] == 'shut' and vm[1] == 'running':
                # copy the corresponding .rules file to '/etc/udev/rules.d' and activate it
                s = path_to_rules + rules_prefix + vm[0] + '.rules'
                subprocess.run(['cp', s, '/etc/udev/rules.d'])
                subprocess.run(['udevadm', 'control', '--reload-rules'])

                syslog.syslog("Started " + vm[0] + " hotplugging.")
                # update vm_state
                vm_state[vm_state.index([vm[0], 'shut'])][1] = 'running'
                
            # if no VM state change, do nothing
    
    # wait for 3 seconds before next check
    time.sleep(3)

 

Setting up the script
The script is meant for usage with CA User Scripts. You will also need python, which can be installed through NerdPack. Add a User Script and paste the code inside. You should only need to modify the three variables managed_vms, path_to_rules, and rules_prefix.


managed_vms: This is the list of VMs for which you want to load udev rules. The names must be written in single quotes and comma separated. The name must be the exact name of the VM, i.e. what is shown on the Dashboard and in the "Name" field in the VM configuration. For each VM listed here a .rules file, that is to be loaded when the VM is started and disabled when the VM is stopped, is required. These files should contain the udev rules as shown above.
path_to_rules must be the path to the directory where these rules are stored. (This must not be /etc/udev/rules.d/ !)
The .rules files must be named "someprefix-vmname.rules", where "vmname" must be the name of the corresponding VM and "someprefix-" is defined by rules_prefix.

So if I want to manage 3 VMs called "Win1", "Win2" and "Ubuntu", I have:

managed_vms = ['Win1','Win2','Ubuntu']

I store my .rules files in /mnt/user/appdata/udevrules, so:

path_to_rules = '/mnt/user/udevrules'

I set the prefix to

rules_prefix = '90-usbhotplug-'

thus I need 3 files called 90-usbhotplug-Win1.rules, 90-usbhotplug-Win2.rules, and 90-usbhotplug-Ubuntu.rules in /mnt/user/udevrules.

In User Scrips I set the script to run at startup of the array. In the last line of the script you can also modify the time between VM state checks if you desire.

 

 

I have not tested this on other motherboards yet, but the procedure should be the same. I developed this to use it myself, but I figured there might be some other people interested in this method.

 

  • Like 1
Link to comment
38 minutes ago, jonathanm said:

 

Interesting. Last time I searched the forum I did not find this. It does however appear to work on a per device basis, as in you can define what actions shoud be taken if a certain device is detected, no matter the physical port. This method targets ports instead. So you could do something like attach the USB device to VM1 if inserted into port 1 and attach it to VM2 if inserted into port 2. That does not currently seem possible with that plugin.

Link to comment

Nice write up. I am already adding an option to the plugin to set a port to a VM. Specific devices will take precedence if selected over a port.

 

The way I work out device to a port mapping is as follows in the php.

 

            #udevadm info -a   --name=/dev/bus/usb/003/002 | grep KERNEL==

            $udevcmd = "udevadm info -a   --name=/dev/bus/usb/".$arrMatch['bus']."/".$arrMatch['dev']." | grep KERNEL==" ;

            exec( $udevcmd , $udev);

        

            $physical_busid = trim(substr($udev[0], 13) , '"') ;

 

 

I have already added icon for the port mapping setting, I just to finish the processing part. Please give my plugin a spin, any feedback is appreciated. 

 

The plugin will also add a device into the VM when starts also, which I am planning to do for the port option also.

 

 

image.thumb.png.5f8f2331c2a37f7f2ccd828ad965c95f.png

Link to comment
21 hours ago, midgard00 said:

I have no experience with php and Unraid plugin development, but I imagine the difficult thing would be to automatically detect ports where currently no device is connected, since there are no folders for those in the directories.

I have added support for ports to my plugin.  if you connect a device to a port is will should in the GUI then you can add a mapping. Look for unraid usbip gui in Community Apps if you want to try it out.

Link to comment
  • 1 month later...

Hello,  first off this plugin is amazing.  thank you.  I was finally able to passthru my USB devices that came with the htc vive vr headset to my win10 vm however I seem to be missing 2 USB device in the list to map.  Unraid sees 8 devices (057 - 064) and plugin only sees 6.  probably because of the 2 duplicate devices (061 and 062 have duplicate ID as well as 060 and 064).  it's also why steam vr still cannot see the headset.  unfortunately I cannot isolate the usb controller into a separate IOMMU group even when adding a 2nd usb pci controller.   any suggestion is welcome.  thanks again. 

 

1619580960_ScreenShot2021-04-17at12_34_13AM.thumb.png.bab0bc630d854baf28dc8193942950a9.png

 

2134025980_ScreenShot2021-04-17at12_33_12AM.thumb.png.542eebe7d25cf9792f84db5a8b5a95d9.png

Link to comment
  • 3 months later...
  • 2 months later...

Does anyone know how things change for USB Type C ports? I see the device under usb1, but after I pass that through, the VM doesn't recognize the device on that port. (An Oculus Quest.) I'm ordering a Type C -> USB A cable in the meantime to try a USB3 port.

Edited by coppit
Link to comment
  • 1 month later...
On 10/24/2021 at 7:57 PM, coppit said:

Does anyone know how things change for USB Type C ports? I see the device under usb1, but after I pass that through, the VM doesn't recognize the device on that port. (An Oculus Quest.) I'm ordering a Type C -> USB A cable in the meantime to try a USB3 port.

Did you get usb c working?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.