**VIDEO GUIDE** How to easily and safely pass through a USB controller **


Recommended Posts

Hi, Guys. I have made a tutorial showing how to easily and safely pass through a USB controller on your unRAID server.

You can see how to quickly identify all USB controllers in the server and see which devices and buses are connected to each controller and

also which iommu group each controller is in. So you can easily avoid accidentally passing through the controller used for the unRAID flash drive.

You will also see how to easily identify which USB controllers support resetting. I hope you find it useful :)

 

How to easily  pass though a USB controller

 

Link to comment

Hi Gridrunner,

 

Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6

 

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1)

Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

Link to comment

Hi Gridrunner,

 

Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6

 

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1)

Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

 

please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset

 

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

 

 

edit . if you are using  in syslinux maybe try the older pci-stub.ids= instead

so

vfio-pci.ids=1b73:1100

would be

pci-stub.ids=1b73:1100

 

may not solve your prob but worth a try as i dont think stub also it to go into d3

Link to comment

Can I pass through 2 USB controler to 2VMs? Where can I put an second Pci stub

 

Yes just be careful not to pass through unRAID's usb flash controller.

 

To stub 2 devices (well interesting point or maybe not lol ... its not stub anymore from linux 4.1 its vfio-pci.ids which is a newer version of 'stub' that allows devices to go into D3 state(low power state) when they are not in use)  :)

 

just put a comma between the ids heres my syslinux line below with 2 devices

 

append vfio-pci.ids=1b73:1100,8086:15a1 initrd=/bzroot 

  • Like 1
Link to comment

Hi Gridrunner,

 

Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6

 

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1)

Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

 

please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset

 

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

 

It should be able to reset itself:

 

IOMMU group 13

[RESET] 03:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev ff)

Link to comment

Hi Gridrunner,

 

Running the same USB pci card as you have. However, I keep running into problems when I restart my VM's. Either UNRAID freezes up or I am not able to start up the VM as the USB resource is busy. Are you experiencing the same? Is there anything I can do to check why my usb card is not resetting properly? Only a cold restart seems to resolve the problem. Running 6.3 rc6

 

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: vfio_ecap_init: 0000:03:00.0 hiding ecap 0xffff@0xffc

Dec 17 13:26:18 unRAID kernel: genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:00.0)) vs. 00000080 (ehci_hcd:usb1)

Dec 17 13:26:18 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 13:26:19 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 13:26:20 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:20 unRAID kernel: device vnet0 left promiscuous mode

Dec 17 13:26:20 unRAID kernel: br0: port 2(vnet0) entered disabled state

Dec 17 13:26:21 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

 

please ssh into your server and copy the result of this. Should say reset next to your usb card/controller if it supports reset

 

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d);do echo "IOMMU group $(basename "$iommu_group")"; for device in $(\ls -1 "$iommu_group"/devices/); do if [[ -e "$iommu_group"/devices/"$device"/reset ]]; then echo -n "[RESET]"; fi; echo -n $'\t';lspci -nns "$device"; done; done

 

It should be able to reset itself:

 

IOMMU group 13

[RESET] 03:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev ff)

 

yeah thats strange.

not sure if you saw the edit i made on the last post to you but if using vfio-pci.ids= then try pci-stub.ids=  and vice versa

Link to comment

Problem persists using pci-stub, although stub is claiming the pci.

 

A cold boot into VM, everything works as it should. A VM reset, I can still login to the VM, but it is extremely slow. Third reboot and I cannot even start the VM and receive a pop up with the following message:

 

Execution error

internal error: qemu unexpectedly closed the monitor: 2016-12-17T15:42:06.737278Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy

2016-12-17T15:42:08.620251Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: Device initialization failed

 

Log VM:

2016-12-17T15:42:06.737278Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: vfio: Error: Failed to setup INTx fd: Device or resource busy

2016-12-17T15:42:08.620251Z qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev2,bus=pci.0,addr=0x9: Device initialization failed

2016-12-17 15:42:08.664+0000: shutting down, reason=failed

 

Log unraid:

Dec 17 16:38:46 unRAID kernel: vfio-pci 0000:03:00.0: timed out waiting for pending transaction; performing function level reset anyway

Dec 17 16:38:47 unRAID kernel: vfio-pci 0000:03:00.0: Failed to return from FLR

Dec 17 16:38:48 unRAID kernel: vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=none

Dec 17 16:38:48 unRAID kernel: vfio-pci 0000:03:00.0: Refused to change power state, currently in D3

Dec 17 16:38:48 unRAID kernel: pci-stub 0000:03:00.0: claimed by stub

Link to comment

I dont know what else to suggest. I have not had any of the same problems with my fresco logic usb. Mine is  Inateck PCI-E to USB 3.0 5-Port PCI Express Card (KTU3FR-5O2I) you can see it here http://amzn.to/2f2OS5U?

I am using an x99 motherboard and dont have any acs overide enabled or anything else.

this is my syslinux file here.

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
  append vfio-pci.ids=1b73:1100,8086:15a1 initrd=/bzroot  
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest

 

Perhaps the card is just faulty??

Link to comment
  • 4 months later...

Thank you for the great videos.  I have been trying to pass through the exact same USB card to my windows VM.  However, it never shows up in "Other PCI Devices"

 

Below are the steps I have tried:

Confirmed that it is in its own IOMMU group (21) and shows RESET.  

Used the following string: vfio-pci.ids=1b73:1100

Added it to the flash drive syntax:  append vfio-pci.ids=1b73:1100 pcie_acs_override=downstream initrd=/bzroot

 

Note that unlike the tutorial mine has "pcie_acs_override=downstream" in it as well.  Was not sure how to handle that.

 

Assistance appricated!

 

 

 

 

Link to comment
  • 2 weeks later...
  • 2 years later...

Thanks so much for your videos, Ed! As a newbie, I can't thank you enough for all your help!

 

I followed this one to the T, but sadly, my Unraid won't boot unless it's booted in GUI Mode. I'm assuming it's because of the syslinux config

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label Unraid OS
  menu default
  kernel /bzimage
  append vfio-pci.ids=1022:149c initrd=/bzroot
label Unraid OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label Unraid OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Unraid OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

My USB Hub is in its own IOMMU group. However, it is occupying two devices on the Bus 005 and Bus 006 each. It's a TP-Link 7-port Hub with 2 charging ports.

Bus 1 --> 0000:07:00.1 (IOMMU group 21)
Bus 001 Device 002: ID 0781:5571 SanDisk Corp. Cruzer Fit
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 2 --> 0000:07:00.1 (IOMMU group 21)
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 3 --> 0000:07:00.3 (IOMMU group 21)
Bus 003 Device 003: ID 0b05:18f3 ASUSTek Computer, Inc. AURA LED Controller
Bus 003 Device 002: ID 046d:c31c Logitech, Inc. Keyboard K120
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 4 --> 0000:07:00.3 (IOMMU group 21)
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 5 --> 0000:0c:00.3 (IOMMU group 31)
Bus 005 Device 003: ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 005 Device 002: ID 0bda:5411 Realtek Semiconductor Corp. 4-Port USB 2.0 Hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 6 --> 0000:0c:00.3 (IOMMU group 31)
Bus 006 Device 003: ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 006 Device 002: ID 0bda:0411 Realtek Semiconductor Corp. 4-Port USB 3.0 Hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

I think I'm passing the right ID too

 

IOMMU group 31
[RESET] 0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

Any help would be greatly appreciated!

  • Like 1
Link to comment
  • 4 weeks later...

Followed the guide, see it in my pass through:

 

image.png.b5315d00a03a2453873691fce14399c1.png

 

Check the box, save the settings.  Try to start the VM and get the following:

 

image.png.424fa2f8c9083d2e1212901b7fd1c1fd.png

 

edit: I've also tried all 3 different settings for ACS override and the error persists.

Edited by JasonK
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.